Runbooks
A runbook is written for the person who did not build the system, at 3 a.m., under pressure. If it assumes context that person does not have, it is not a runbook.
The standard. A runbook for every routine operation and known failure mode, written for the responder, automated where deterministic, and exercised before it is needed.
What a runbook covers
| Category | Examples |
|---|---|
| Routine operations | Deployment, rollback, scaling, certificate renewal, key rotation |
| Failure modes | Unhealthy instance, high error rate, database failover, queue backlog |
| Data operations | Snapshot, restore, failover, schema migration |
| Security events | Credential exposure, WAF block, GuardDuty finding |
| Recovery | Regional failover, restore from backup, rebuild from image |
Anatomy of a runbook
Every runbook has the same structure:
Title
The operation, in plain language — "Roll back a failed production deployment".
Trigger
The alarm, symptom or schedule that means this runbook applies.
Impact
What is affected while this runs, and for how long.
Prerequisites
Access, tools, credentials and approvals needed before starting.
Procedure
Numbered, copy-pasteable steps with the expected output after each.
Verification
How to confirm the operation succeeded — a specific check with a specific result.
Rollback
What to do if the procedure fails part-way.
Escalation
Who to contact, and at what point.
Owner
The team accountable for keeping the runbook current.
Example: application rollback
## Roll back a failed production deployment
**Trigger** — Production error rate above 2% within 30 minutes of a deployment.
**Impact** — The service runs the previous version. In-flight requests may fail.
**Prerequisites** — Access to the deployment pipeline; the previous version tag.
**Procedure**
1. Confirm the timing correlates with the deployment:
`aws cloudwatch get-metric-statistics ...`
Expected: error rate rises at or after the deployment timestamp.
2. Identify the previous version:
`aws ecs describe-task-definition --task-definition example-service`
Expected: note the previous revision number.
3. Roll back:
`aws ecs update-service --cluster example-prod --service example-service \
--task-definition example-service:<previous> --force-new-deployment`
4. Watch the deployment:
`aws ecs wait services-stable --cluster example-prod --services example-service`
**Verification** — Error rate returns below 0.5% within 10 minutes; the
health endpoint returns 200 from all targets.
**Rollback of this procedure** — Re-deploy the failed version only with the
approval of the incident lead.
**Escalation** — If error rate does not fall within 15 minutes, escalate to the
platform on-call and the application owner.
Runbooks as automation
Where a procedure is deterministic, it is implemented as AWS Systems Manager Automation and the runbook becomes a thin wrapper around it.
schemaVersion: '0.3'
description: 'Restart application processes on an unhealthy instance'
parameters:
InstanceId:
type: String
description: Target EC2 instance
mainSteps:
- name: restartService
action: aws:runShellScript
inputs:
runCommand:
- 'systemctl restart app'
- 'sleep 10'
- 'systemctl is-active app'
- name: verifyHealth
action: aws:runShellScript
inputs:
runCommand:
- 'curl -sf http://localhost:8080/health'
Automation is preferred when:
- The action is safe to run without judgement.
- The action is time-sensitive (failover, scaling).
- Human error in the manual procedure is plausible.
Automation is not preferred when the action requires interpretation — for example, deciding whether to fail over a database with replication lag. Those runbooks stay human, with an automated diagnostic step.
Writing rules
- Assume no context. Name the account, the cluster, the service and the exact command. The reader has not seen the architecture.
- Every command has expected output. "Run this" without "you should see this" is an invitation to proceed blindly.
- Verify, do not assume. The last step is always a check.
- Include the rollback. A runbook with no failure path is a wish.
- Date and version it. A runbook that describes last year's environment is worse than none.
- Keep it short. If it takes more than ten minutes to read, split it.
- Link from the alarm. The person receiving the alert should land on the runbook, not search for it.
Exercising runbooks
A runbook that has never been used is a hypothesis.
| Exercise | Frequency | What it proves |
|---|---|---|
| Tabletop walkthrough | On handover | The steps are comprehensible |
| Game day — induced failure | Quarterly | The steps work against the real system |
| Restore drill | Per RTO commitment | Backups are usable and the procedure is accurate |
| Failover drill | Per RTO commitment | The recovery path works |
| Post-incident review | After every incident | The runbook covered the failure — or revealed a gap |
During a game day, the person following the runbook is not its author. That is the only way to find the assumptions.
Maintenance
- Reviewed quarterly, and after every incident.
- Updated in the same pull request as the change that invalidates them.
- Retired when the component is retired — a stale runbook is a trap.
- Indexed by alarm, so every alarm resolves to exactly one runbook.
Evidence
- Runbook index mapped to alarms.
- A complete runbook for each critical failure mode.
- Automation documents for the automatable procedures.
- Game day or drill records, including issues found and fixed.