← All insightsIntegration

How to use Apache Kafka with Kubernetes

Nir Geier, CodeWizard · 25 April 2025 · 3 min read · Intermediate

Apache Kafka and Kubernetes are a common pairing in a modern DevOps stack. Here is how they fit together and a practical way to wire them up.

Where Apache Kafka fits

Kafka is a distributed log for streaming events: producers append to partitioned topics, consumers read at their own pace, and data is durably replicated - the backbone of event-driven architectures.

python
from kafka import KafkaProducer, KafkaConsumer
p = KafkaProducer(bootstrap_servers='localhost:9092')
p.send('orders', b'{"id": 42}'); p.flush()

c = KafkaConsumer('orders', bootstrap_servers='localhost:9092', group_id='billing')
for msg in c:
    print(msg.value)

Where Kubernetes fits

Kubernetes is a container orchestrator: you declare the desired state of your workloads in YAML and controllers continuously reconcile the cluster toward it, handling scheduling, scaling, healing and networking.

yaml
apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
  replicas: 3
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - name: web
          image: ghcr.io/acme/web:1.0
          ports: [{ containerPort: 3000 }]
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits:   { cpu: 500m, memory: 256Mi }
          readinessProbe:
            httpGet: { path: /healthz, port: 3000 }
---
apiVersion: v1
kind: Service
metadata: { name: web }
spec:
  selector: { app: web }
  ports: [{ port: 80, targetPort: 3000 }]

Making them work together

  • Size partitions for target consumer parallelism up front
  • Monitor consumer lag and set alerts on it
  • Set requests/limits and use the HorizontalPodAutoscaler for elastic workloads
  • Separate config (ConfigMap) from secrets and mount them, don't bake them in

Want this pairing set up properly, with docs and handover? That is exactly the kind of work CodeWizard's embedded senior engineers do - and both Apache Kafka and Kubernetes have active community groups you can join for free.

Like where this is going?

Get hands-on help from the engineers who write these posts.

Book a free consult