You do not actually know your system is resilient until something fails. Chaos engineering makes that failure happen on your terms - a controlled experiment with a hypothesis, a small blast radius and an abort button - instead of at 3am.
Frame it as an experiment
- Steady state: a metric that means 'healthy' (e.g. checkout success rate)
- Hypothesis: 'killing one API pod does not move that metric'
- Blast radius: one pod, in staging first, with a stop condition
- Observe, then either confirm resilience or file the bug you found
A minimal pod-kill experiment
# kill one random pod and watch the steady-state metric
kubectl delete pod -l app=api \
$(kubectl get pod -l app=api -o name | shuf -n1 | cut -d/ -f2)# or declaratively with a chaos tool (LitmusChaos)
kind: ChaosEngine
spec:
experiments:
- name: pod-delete
spec: { components: { env: [{ name: TOTAL_CHAOS_DURATION, value: '30' }] } }The goal of chaos engineering is not to break production - it is to find out what is already broken before your users do.
Start in staging, automate one small experiment, and only graduate to production once you trust your guardrails. Each experiment either builds confidence or hands you a bug worth fixing.