← All insightsSRE

Cutting MTTR: incident response, runbooks and blameless postmortems

CodeWizard Academy · 28 June 2025 · 3 min read · Intermediate

Reliability is not the absence of incidents - it is how fast you detect, diagnose and recover. Mean time to recovery (MTTR) is the number that captures it, and most of MTTR is human process, not tooling.

A lightweight incident flow

  • Declare early - a named incident commander, a single channel
  • Mitigate before you diagnose - roll back, scale out, shed load
  • Communicate status on a cadence, even 'still investigating'
  • Write the timeline as you go, while it is fresh

A runbook is a checklist, not an essay

markdown
## Runbook: high 5xx on web
1. Check dashboard: is it one pod or all? `kubectl get pods -l app=web`
2. Recent deploy? Roll back: `kubectl rollout undo deploy/web`
3. DB saturated? Check connections, scale read replica
4. Still bad? Page the on-call DBA (see escalation)

Blameless postmortems

The goal is a fixed system, not a blamed person. Focus on the contributing causes and the action items, and track those items to done.

Postmortems without follow-through are just paperwork. The action items are the whole point.

Fast rollback, current runbooks and honest postmortems shrink MTTR more than any amount of extra monitoring.

Like where this is going?

Get hands-on help from the engineers who write these posts.

Book a free consult