Reliability is not the absence of incidents - it is how fast you detect, diagnose and recover. Mean time to recovery (MTTR) is the number that captures it, and most of MTTR is human process, not tooling.
A lightweight incident flow
- Declare early - a named incident commander, a single channel
- Mitigate before you diagnose - roll back, scale out, shed load
- Communicate status on a cadence, even 'still investigating'
- Write the timeline as you go, while it is fresh
A runbook is a checklist, not an essay
## Runbook: high 5xx on web
1. Check dashboard: is it one pod or all? `kubectl get pods -l app=web`
2. Recent deploy? Roll back: `kubectl rollout undo deploy/web`
3. DB saturated? Check connections, scale read replica
4. Still bad? Page the on-call DBA (see escalation)Blameless postmortems
The goal is a fixed system, not a blamed person. Focus on the contributing causes and the action items, and track those items to done.
Postmortems without follow-through are just paperwork. The action items are the whole point.
Fast rollback, current runbooks and honest postmortems shrink MTTR more than any amount of extra monitoring.