← All insightsAutomation

Automating Apache Kafka from scratch

CodeWizard Academy · 1 March 2026 · 3 min read · Intermediate

Anything you do to Apache Kafka twice should be automated. This walks the path from a manual install to a repeatable, pipeline-driven setup.

1. Install it reproducibly

bash
docker run -p 9092:9092 apache/kafka:latest
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3

2. Describe the desired state

Produce and consume with the Python client.

python
from kafka import KafkaProducer, KafkaConsumer
p = KafkaProducer(bootstrap_servers='localhost:9092')
p.send('orders', b'{"id": 42}'); p.flush()

c = KafkaConsumer('orders', bootstrap_servers='localhost:9092', group_id='billing')
for msg in c:
    print(msg.value)

3. Run it from CI

Spin up Kafka as a CI service for integration tests.

yaml
services:
  kafka:
    image: apache/kafka:latest
    ports: ['9092:9092']

Want senior help putting Apache Kafka to work - or a team bootcamp on it? CodeWizard runs Israel's largest DevOps community and embeds named senior engineers with a transparent hours log. Book a free consult or explore the Academy.

Like where this is going?

Get hands-on help from the engineers who write these posts.

Book a free consult