← All insightsSRE

Chaos engineering: breaking things on purpose, safely

CodeWizard Engineering · 23 June 2025 · 3 min read · Intermediate

You do not actually know your system is resilient until something fails. Chaos engineering makes that failure happen on your terms - a controlled experiment with a hypothesis, a small blast radius and an abort button - instead of at 3am.

Frame it as an experiment

  • Steady state: a metric that means 'healthy' (e.g. checkout success rate)
  • Hypothesis: 'killing one API pod does not move that metric'
  • Blast radius: one pod, in staging first, with a stop condition
  • Observe, then either confirm resilience or file the bug you found

A minimal pod-kill experiment

bash
# kill one random pod and watch the steady-state metric
kubectl delete pod -l app=api \
  $(kubectl get pod -l app=api -o name | shuf -n1 | cut -d/ -f2)
yaml
# or declaratively with a chaos tool (LitmusChaos)
kind: ChaosEngine
spec:
  experiments:
    - name: pod-delete
      spec: { components: { env: [{ name: TOTAL_CHAOS_DURATION, value: '30' }] } }
The goal of chaos engineering is not to break production - it is to find out what is already broken before your users do.

Start in staging, automate one small experiment, and only graduate to production once you trust your guardrails. Each experiment either builds confidence or hands you a bug worth fixing.

Like where this is going?

Get hands-on help from the engineers who write these posts.

Book a free consult