Apache Kafka and Kubernetes are a common pairing in a modern DevOps stack. Here is how they fit together and a practical way to wire them up.
Where Apache Kafka fits
Kafka is a distributed log for streaming events: producers append to partitioned topics, consumers read at their own pace, and data is durably replicated - the backbone of event-driven architectures.
from kafka import KafkaProducer, KafkaConsumer
p = KafkaProducer(bootstrap_servers='localhost:9092')
p.send('orders', b'{"id": 42}'); p.flush()
c = KafkaConsumer('orders', bootstrap_servers='localhost:9092', group_id='billing')
for msg in c:
print(msg.value)Where Kubernetes fits
Kubernetes is a container orchestrator: you declare the desired state of your workloads in YAML and controllers continuously reconcile the cluster toward it, handling scheduling, scaling, healing and networking.
apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
replicas: 3
selector: { matchLabels: { app: web } }
template:
metadata: { labels: { app: web } }
spec:
containers:
- name: web
image: ghcr.io/acme/web:1.0
ports: [{ containerPort: 3000 }]
resources:
requests: { cpu: 100m, memory: 128Mi }
limits: { cpu: 500m, memory: 256Mi }
readinessProbe:
httpGet: { path: /healthz, port: 3000 }
---
apiVersion: v1
kind: Service
metadata: { name: web }
spec:
selector: { app: web }
ports: [{ port: 80, targetPort: 3000 }]Making them work together
- Size partitions for target consumer parallelism up front
- Monitor consumer lag and set alerts on it
- Set requests/limits and use the HorizontalPodAutoscaler for elastic workloads
- Separate config (ConfigMap) from secrets and mount them, don't bake them in
Want this pairing set up properly, with docs and handover? That is exactly the kind of work CodeWizard's embedded senior engineers do - and both Apache Kafka and Kubernetes have active community groups you can join for free.