At 3am, nobody wants a wiki novel, a “check Grafana” suggestion, or a Slack scavenger hunt. Runbooks that actually get used are short, specific, and safe: they tell you what to check first, what “normal” looks like, how to mitigate without making things worse, and when to escalate.
Quickstart
If you only have one hour this week to improve operations, do these steps in order. They create an immediately usable runbook (even if you refine it later).
Fast win #1: Write the “first 5 minutes”
- Copy/paste the alert name into the runbook title (match search terms)
- List 3–5 read-only checks (dashboards, recent deploys, error logs)
- Define “normal” ranges and “bad” ranges (even approximate)
- Timebox: after 5 minutes, either mitigate or escalate
Fast win #2: Add one safe mitigation
- Choose the lowest-risk action: rollback, scale-out, feature flag off, rate limit
- Write a verification step: what metric/log confirms improvement?
- Write a stop condition: when not to do this (risk factors)
- Include a fallback: what to do if it doesn’t work in 10 minutes
Fast win #3: Make it reachable
- Link the runbook from the alert itself (Pager/Slack/Email)
- Add the owner/on-call link and escalation path
- Store it near the code (repo) or in a catalog that is searchable
- Keep it one screen for the critical path (details can be collapsed or linked)
Fast win #4: Add the “context panel”
- Service overview: purpose, tier, dependencies
- Last deploy link and changelog link
- Dashboards and logs links (exact ones, not “go to Grafana”)
- Permissions checklist (what access is needed to run steps)
People don’t “read” runbooks during incidents—they scan them. Use short bullets, strong labels (READ-ONLY vs WRITE), and single-purpose links. If a step can’t be executed quickly, move it to a follow-up section.
Any step that mutates state (deleting pods, dropping caches, scaling to zero, disabling retries) should have preconditions, blast-radius notes, and a verification + rollback plan.
Overview
A runbook is an operational guide for a specific situation: “High error rate on checkout-api,” “Database connections exhausted,” “Queue lag rising,” “Latency p95 spikes,” “Pod crash loop.” The best runbooks are designed around two truths: incidents are time-pressured, and the responder may not be the original author.
This post shows how to write runbooks that actually get used: a structure that works under stress, what to include (and what to exclude), and how to keep runbooks accurate over time without turning documentation into a full-time job.
| Doc type | Best for | What it contains |
|---|---|---|
| Runbook | Responding to a known failure mode | Entry conditions, first checks, mitigations, verification, escalation |
| Playbook | Managing the incident process | Roles, comms, severity levels, timelines, stakeholder updates |
| Postmortem | Learning after the incident | Root causes, contributing factors, action items, follow-ups |
Why runbooks get ignored
- They’re too long or too generic (“check the logs”)
- They don’t match alert names, so nobody can find them
- They’re outdated (links rot, commands changed, dashboards moved)
- They lack safe mitigations or verification steps
- They assume access/permissions the responder doesn’t have
What “actually used” looks like
- Critical path fits on one screen
- First 5 minutes are obvious and low-risk
- Mitigations are timeboxed and include stop conditions
- Links go to exact dashboards/log queries, not home pages
- It’s tested and improved after real incidents
The hidden cost in incident response is cognitive load: deciding what to do, where to look, what’s safe, and who to ask. A good runbook doesn’t replace expertise—it makes expertise reusable.
Core concepts
A runbook is not a brain dump. It’s a decision aid for a specific failure mode. When you write one, assume: the responder is tired, the system is changing, and every action has risk. Your goal is to provide clarity and safety.
The “critical path” and the “reference path”
Critical path (must fit on one screen)
- Entry conditions: when to use this runbook
- First checks (read-only) with “normal” and “bad” signals
- One or two safe mitigations
- Verification steps (how you know it worked)
- Escalation: who, when, and what to include
Reference path (details, optional)
- Deep diagnostics, rare edge cases, advanced queries
- Dependency mapping and failure modes
- Architecture context for new responders
- Known pitfalls and “don’t do this” actions
- Follow-up checklist after mitigation
Safety labels: READ-ONLY vs WRITE
Most incidents start with investigation. Your first steps should be read-only by default. When you include mutating actions, label them clearly and include guardrails.
| Label | Examples | What to include |
|---|---|---|
| READ-ONLY | Dashboards, log queries, describing resources, viewing configs | Normal vs abnormal, time ranges, links to exact queries |
| WRITE | Rollback, scaling, toggling flags, changing limits, draining traffic | Preconditions, blast radius, stop conditions, verification, rollback plan |
Entry conditions and exit criteria
A runbook needs a clear “when to use this” section. Otherwise responders will either misuse it or ignore it. Equally important: define when you can stop (exit criteria) and when you should escalate.
- Entry condition: “Alert X firing for 5 minutes” or “p95 latency > Y for Z minutes”
- Exit criteria: “Errors < threshold for 10 minutes and backlog decreasing”
- Escalate when: “Customer impact confirmed” or “Mitigation doesn’t help within 15 minutes”
Runbooks must be attached to alerts
Search is unreliable during incidents. The best place for a runbook link is the alert payload itself (Pager/Slack/Email), with the same keywords as the alert title. If the runbook isn’t linked, it won’t be used consistently.
If a runbook requires “knowing the system,” it’s incomplete. A good runbook contains just enough context to move forward: what the service does, which dashboards to open, which dependencies to suspect, and which mitigations are safe.
Step-by-step
This process produces a runbook that’s useful immediately and improves over time. Start with one incident pattern (one alert), ship the critical path, then iterate after each real incident.
Step 1 — Pick a single scenario (one alert, one runbook)
Don’t write “The Service Runbook.” Write “High error rate on Service” or “Database connections exhausted.” Specificity makes it usable.
- Choose the top 3 alerts by frequency or impact
- Write a runbook per alert (or per tightly related group)
- Ensure the runbook title matches the alert title (same keywords)
Step 2 — Use a standard template (so scanning works)
A consistent structure reduces “where do I look?” time. Here’s a compact Markdown template you can copy into a repo
as RUNBOOK.md or runbooks/ALERT_NAME.md. Keep the critical path at the top.
Example: minimal runbook template (Markdown)
# Runbook: High error rate — checkout-api
**Owner:** team-payments
**On-call:** https://internal/oncall/team-payments
**Dashboards:** https://internal/obs/d/checkout
**Logs:** https://internal/logs?q=service:checkout-api
**Last deploy:** https://internal/deployments/checkout-api
## Entry conditions
- Alert: `checkout-api / http_5xx_rate_high` firing for 5 minutes
- Customer impact: elevated failures at /checkout
## Critical path (first 5 minutes)
**READ-ONLY**
1) Confirm impact (scope + time window)
- Look for: 5xx rising, p95 latency rising, queue/backlog rising
2) Check recent deploys (last 60 minutes)
- If deploy occurred: compare error rates before/after
3) Check dependency health (payments gateway, orders-db)
- Look for timeouts, connection errors, rate limits
## Mitigations (timebox each to 10 minutes)
**WRITE (guardrails apply)**
A) Roll back last deploy (preferred if correlation is strong)
- Preconditions: deploy occurred within last 60 minutes and rollback is safe
- Verify: 5xx rate decreases within 5–10 minutes
B) Reduce load / fail open
- Actions: enable rate limiting, disable non-critical features, shed traffic
- Verify: error budget burn slows and latencies stabilize
## Verification (exit criteria)
- 5xx rate below threshold for 10 minutes
- p95 latency trending back to baseline
- Backlog decreasing
## Escalation
- Escalate immediately if: data loss risk, widespread outage, mitigation unsafe
- Include: incident timeline, dashboards screenshots/links, actions attempted
## Follow-up
- Create a ticket: root cause + prevention
- Update this runbook with new learnings (what worked, what didn’t)Step 3 — Attach the runbook to the alert payload
Runbooks are most effective when they’re one click away. Add a Runbook URL field to alert annotations (or message templates) so responders don’t hunt for docs. Also ensure the alert includes the minimum diagnostic context.
Alert context that helps
- Service name, environment, region/cluster
- Metric that fired and threshold (with time window)
- Links: dashboard, logs query, runbook
- Recent deploy info (or a link to deploy history)
Alert context that hurts
- “Something is wrong” without a metric or threshold
- Links to home pages instead of specific views
- No environment (prod/stage) or service identifier
- Firehose of unrelated graphs in a single message
Step 4 — Write read-only diagnostics as copy/paste commands
Commands are useful only if they’re safe and specific. Prefer read-only commands that give quick signal: is this a deploy issue, a dependency issue, capacity, or a config change?
Example: safe triage commands for a Kubernetes service
Keep commands grouped and labeled. If you include mutating commands, separate them and add preconditions. This snippet stays read-only.
# READ-ONLY triage (Kubernetes)
# Set these for your environment
NS="payments"
APP="checkout-api"
# 1) Are pods healthy and stable?
kubectl -n "$NS" get pods -l app="$APP" -o wide
kubectl -n "$NS" get deploy "$APP" -o wide
kubectl -n "$NS" describe deploy "$APP" | sed -n '1,120p'
# 2) Are there recent restarts or crash loops?
kubectl -n "$NS" get pods -l app="$APP" --sort-by=.status.containerStatuses[0].restartCount
# 3) What do recent logs say? (limit scope)
POD="$(kubectl -n "$NS" get pods -l app="$APP" -o jsonpath='{.items[0].metadata.name}')"
kubectl -n "$NS" logs "$POD" --since=10m | tail -n 200
# 4) Is the service responding internally?
kubectl -n "$NS" get svc "$APP" -o wide
kubectl -n "$NS" port-forward svc/"$APP" 18080:8080
# In another terminal:
# curl -sS http://localhost:18080/health
# curl -sS http://localhost:18080/ready
# 5) Quick dependency signal (example: DNS + connectivity from a pod)
kubectl -n "$NS" exec -it "$POD" -- sh -lc 'getent hosts orders-db; getent hosts payments-gateway'Step 5 — Add one mitigation, then add guardrails
Runbooks become valuable when they include safe actions. Start with a mitigation that’s reversible and well understood: rollback, scale out, disable a feature flag, or reduce load. For every mitigation, add guardrails: when to do it, when not to do it, and how to verify.
- Preconditions: what must be true before you act?
- Blast radius: what might break or get worse?
- Stop conditions: when do you stop trying this?
- Verification: which metric/log confirms improvement?
- Rollback: how do you undo it?
Step 6 — Keep runbooks up to date (without heroics)
The biggest reason runbooks fail is rot. You can fight rot with small automation: validate required sections exist, ensure links aren’t obviously malformed, and keep ownership metadata current. A simple “runbook lint” check in CI catches many issues early.
Example: tiny runbook linter you can run in CI
This script checks that required headings exist and that the document includes a critical path and escalation section. It won’t prove correctness, but it prevents “empty runbook” failures.
#!/usr/bin/env python3
"""
runbook_lint.py - minimal checks to prevent runbook rot.
Usage:
python runbook_lint.py path/to/RUNBOOK.md
Exit codes:
0 = ok
2 = lint failures
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
REQUIRED_HEADINGS = [
r"^##\s+Entry conditions\b",
r"^##\s+Critical path\b",
r"^##\s+Mitigations\b",
r"^##\s+Verification\b",
r"^##\s+Escalation\b",
]
def main() -> int:
if len(sys.argv) != 2:
print("Usage: python runbook_lint.py path/to/RUNBOOK.md", file=sys.stderr)
return 2
path = Path(sys.argv[1])
if not path.exists():
print(f"File not found: {path}", file=sys.stderr)
return 2
text = path.read_text(encoding="utf-8", errors="replace")
failures = []
for pattern in REQUIRED_HEADINGS:
if not re.search(pattern, text, flags=re.MULTILINE):
failures.append(f"Missing heading: {pattern}")
# Encourage safety labeling
if "READ-ONLY" not in text:
failures.append("Missing safety label: include 'READ-ONLY' steps in the critical path.")
if "WRITE" not in text:
failures.append("Missing safety label: include 'WRITE' mitigations with guardrails.")
if failures:
print("Runbook lint failed:")
for f in failures:
print(f"- {f}")
return 2
print("Runbook lint OK")
return 0
if __name__ == "__main__":
raise SystemExit(main())Step 7 — Test runbooks like you test code
A runbook is “correct” only when it works during an incident. Schedule a short review cycle: once per quarter (or after every major incident), validate links, rerun key commands, and update mitigations.
The best time to update a runbook is within 24 hours of using it. Add what you learned: what was confusing, which signal mattered, which mitigation worked, and which links were missing.
Common mistakes
Most runbooks fail because they’re written like documentation, not like an operational tool. Here are the pitfalls that make runbooks unusable under pressure—and fixes that keep them practical.
Mistake 1 — “Check the logs” (no specifics)
- Problem: responders lose time figuring out which logs, which filters, which time window.
- Fix: link to an exact query and show what “bad” looks like (error types, timeouts, status codes).
Mistake 2 — No critical path
- Problem: important actions are buried under architecture explanations.
- Fix: put entry conditions + first 5 minutes + safe mitigations at the top.
Mistake 3 — Unsafe actions with no guardrails
- Problem: runbooks cause secondary incidents (state changes, traffic shifts, data risk).
- Fix: label WRITE actions, include preconditions, blast radius, stop conditions, verification, rollback.
Mistake 4 — Outdated or missing links
- Problem: dashboards moved, repos renamed, log systems changed.
- Fix: keep runbooks near code, add ownership, run periodic link/structure checks, update after incidents.
Mistake 5 — No permissions plan
The runbook assumes the responder can access prod logs, deployments, or kubectl. At 3am, permission requests are slow.
- Fix: list required roles/tools, and provide a fallback path if access is missing.
- Fix: pre-provision on-call roles with least privilege and audit trails.
Mistake 6 — No verification / exit criteria
A mitigation “feels” like it worked, then the incident returns. Without verification, you don’t know when to stop.
- Fix: define exit criteria (metric below threshold for a window, backlog decreasing).
- Fix: include “what to do next if it didn’t improve within 10 minutes.”
A runbook should explicitly say when to stop experimenting and call the owner/SRE/security. Escalation isn’t failure—it’s how you control risk and reduce time to recovery.
FAQ
What should every runbook include?
At minimum: entry conditions, a “first 5 minutes” critical path (read-only checks), one or two mitigations with guardrails, verification/exit criteria, and escalation contacts. If any of those are missing, the runbook won’t be reliably usable during incidents.
How long should a runbook be?
The critical path should fit on one screen. You can add a reference section below for deeper diagnostics, but responders should be able to take meaningful action within 60–120 seconds of opening the page.
Where should we store runbooks?
Store them where they are easiest to find and keep up to date: commonly in the service repo (versioned with code) or in a service catalog that indexes ownership and links. Either way, the runbook should be linked directly from the alert payload.
Runbook vs playbook: what’s the difference?
A runbook is for fixing a specific technical failure mode. A playbook is for managing the incident process (roles, comms, severity, timelines). Most teams need both: playbooks keep coordination sane; runbooks reduce MTTR.
Do we need a runbook for every alert?
Not for every alert on day one. Start with alerts that are frequent, high-impact, or complex to debug. Over time, a healthy practice is: if an alert pages someone, it should have a linked runbook—even if the first version is short.
How do we keep runbooks from going stale?
Tie updates to reality: update within 24 hours after an incident, run a quarterly link/command check, and add lightweight automation (linting for required sections, ownership metadata). “Living docs” works when it’s part of the incident loop, not a separate initiative.
What makes “Runbooks That Actually Get Used” different from regular documentation?
They’re written for stressed responders: short, labeled for safety, full of specific links and copy/paste commands, and they include verification + escalation guidance. Regular documentation explains systems; runbooks move incidents forward.
Cheatsheet
Print this mentally. If your runbook meets these checks, it will be used more often and reduce time-to-recovery.
Runbook critical path checklist
- Title matches alert title (searchable keywords)
- Entry conditions are explicit (when to use)
- First 5 minutes are READ-ONLY and concrete
- Includes “normal vs bad” signals
- Has at least one safe mitigation (WRITE with guardrails)
- Verification + exit criteria are defined
- Escalation path is clear (who + what to include)
Guardrails for WRITE actions
- Preconditions (what must be true)
- Blast radius (what could get worse)
- Stop conditions (when to stop trying)
- Verification (what proves improvement)
- Rollback plan (how to undo)
- Timebox (usually 10 minutes per mitigation)
| Runbook fitness check | Pass criteria | Quick fix if it fails |
|---|---|---|
| Findability | Linked in alert + searchable by alert keywords | Add Runbook URL to alert annotations/templates |
| Scan speed | Critical path in one screen | Move deep info to “Reference” section |
| Specificity | Exact dashboards/log queries included | Replace generic links with pre-filtered views |
| Safety | READ-ONLY/WRITE labels + guardrails | Add labels and stop conditions |
| Freshness | Updated after incidents + quarterly review | Automate reminders and add a tiny lint check |
Include the alert name, thresholds, and a screenshot-worthy dashboard link near the top. When the alert fires, the runbook should already “look like the incident.”
Wrap-up
Good runbooks aren’t “nice documentation.” They’re operational tooling: optimized for speed, safety, and clarity under stress. If you want runbooks that actually get used, focus on the critical path, label actions for safety, include verification, and attach the runbook to the alert.
Next actions (do this in the next sprint)
- Pick your top 3 paging alerts and write one runbook per alert
- Add runbook links directly into alert notifications
- Ensure each runbook has one safe mitigation + verification
- Schedule a quarterly runbook review (and update after incidents)
- Adopt a lightweight quality bar (template + lint checks)
“Runbook rot” is normal unless you connect updates to real incidents. Treat every incident as a chance to improve: add missing links, clarify confusing steps, and document what worked.
If you’re building stronger operations, pair runbooks with solid observability, SLOs, and incident hygiene. The goal isn’t to eliminate incidents—it’s to recover quickly, safely, and consistently.
Quiz
Quick self-check. This quiz covers runbook structure, safety, and what makes runbooks usable during incidents.