How to Write an Incident Runbook a Tired Responder Can Use

In one sentence

A runbook is a short, tested document that gives whoever was paged the next safe decision, in the order they need it, without requiring them to understand the whole system first.

What a runbook is for

At 03:12 the person holding the pager is not at their best. They have partial context, one screen, no colleague within reach, and a decision to make. A runbook is written for that moment and no other.

Its job is not to explain the service. Its job is to shorten the distance between "I have been paged" and "I have done something safe and useful". Everything that does not serve that belongs somewhere else.

This reframing settles most editorial arguments. If a paragraph would not change what the responder does in the next ten minutes, move it to the architecture document and link to it.

Three documents, three jobs

Mixing them produces a document nobody can use. A runbook that opens with a design tour gets scrolled past. A postmortem action that never becomes a runbook step, an alert change or a code change is a note, not an improvement.

The anatomy of a usable runbook

Eight sections, in this order, because that is the order a responder needs them.

1. Trigger. The exact alert names that lead here, plus one paragraph on what the alert means in user-visible terms — and what it does not mean. Link the runbook from the alert payload so it arrives with the page.
2. First five minutes. Three to five numbered actions: acknowledge, confirm whether users are affected, check for recent change, open the incident channel. No branching yet.
3. Safety boundaries. What must not be done without approval. See below.
4. Diagnosis. A small set of checks, each with the expected healthy value and what a deviation implies. "If queue depth is rising and consumer count is zero, go to Mitigation 2" beats a list of dashboard links.
5. Mitigation options. For each: what it does, its blast radius, how to undo it, and how long to wait before judging whether it worked. An option without a rollback is not an option a tired person should take.
6. Escalation and stop conditions. The point at which the responder stops trying and calls someone: a time limit, a severity threshold, or a step they are not authorised to take.
7. Communication. Who needs to know, where updates go, roughly what to say, and at what interval.
8. Resolution and follow-up. How to confirm recovery rather than assume it, what to record, and what to file so the next person is better off.

Safety boundaries: the section most runbooks are missing

Most runbooks list what to do. What they leave out is what a responder must not do alone, and that omission is where the irreversible damage lives. Restarting a stateless service is recoverable. Purging a queue is not.

Write one line per boundary, and always give the route around it:

Three rules make this section work.

A boundary without a named approver is a blocker. "Check with someone before failing over" leaves a responder at 3am with no idea who someone is. Name the rotation and say how to reach it.

State the exception before it is needed. If a severity-1 incident allows the incident commander to authorise a boundary action when the approver has not answered within ten minutes, write that down. Otherwise the responder makes that call alone, under pressure, and is second-guessed afterwards.

Say why. A boundary with a reason survives contact with a clever engineer who can see a faster path. A bare prohibition does not.

Writing for someone who does not own the service

Assume the reader has no context, possibly no access, and is reading on a laptop tethered to a phone.

• Spell out service names in full the first time. Internal shorthand that everyone on the owning team knows is exactly what the responder does not know.
• Make every command complete and copy-pasteable, and say what healthy output looks like.
• Link straight to the specific dashboard and time range, not to the monitoring tool's front page.
• Put required access at the top — roles, network access, credentials. If the reader lacks it, the runbook should say who to page instead. Checking access is step zero, not a surprise at step four.
• Give baselines. A number without a normal range is not a signal.
• Prefer "if you see X, do Y" over description. Prose forces the reader to infer; at 3am they will infer wrong.

Keeping runbooks true

Three mechanics, all cheap:

• One owner. A team that maps to a real rotation, named in the header, so a fix request reaches someone who is on call for the service.
• Two dates. Last reviewed, and last tested. A runbook untested for a year should be treated as unverified, and the header should say so plainly.
• Deletion. When an alert is retired, delete its runbook the same day. A missing runbook makes a responder careful. A wrong one makes them confident.

Link every paging alert to its runbook. That single habit exercises the library continuously: if an alert has no runbook, either write one or ask whether it should be waking anyone.

How to test a runbook

1. Cold read. Give it to an engineer who does not own the service and ask them to narrate what they would do. Watch where they stop, hesitate or ask a question. This takes fifteen minutes and finds more problems than the author expects.
2. Access dry-run. Have that same person confirm they can open every link and run every command in read-only form, with their own credentials.
3. Simulated firing. Trigger the condition in a non-production environment, or walk it through in a scheduled exercise. Measure the time to first safe action.
4. Real-incident capture. At the end of every incident, ask what the runbook failed to tell you, and fix it within a week while the answer is still specific.

Record the date after each of these. A tested date is the only honest measure of whether a runbook still works.

A worked example

The following is an invented example, not a real service.

Trigger: alert checkout-api-latency-p99, which fires when p99 latency stays above 2 seconds for 10 minutes. It means checkout requests are slow and customers may be waiting at payment. It does not mean checkout is down — check the error rate for that.

First five minutes

1. Acknowledge the page.
2. Open the checkout service dashboard, last 60 minutes. Read latency, error rate and request rate together.
3. Check the deploy log for checkout-api and payment-gateway-client in the last two hours.
4. If a deploy landed inside that window, go to Mitigation 1.
5. Post in the incident channel: what is slow, since when, whether a deploy is suspected.

Safety boundaries: do not restart the payments worker fleet without payments on-call approval — in-flight authorisations may be retried and charge twice. Do not raise the database connection pool limit without database on-call.

Mitigation 1 — roll back the most recent deploy. Blast radius: all checkout traffic. Rollback: redeploy the previous version with the same command. Wait 10 minutes of live traffic before judging.

Escalation: if latency is unchanged 20 minutes after the first mitigation, or the error rate rises above 5 percent, page the service owner and raise severity to 2.

That is the whole runbook. Its brevity is the feature.

What to do next

Pick the alert that paged someone most recently and work through this list today.

• Write the trigger paragraph: what the alert means in user terms, and what it does not mean.
• Write the first five minutes as numbered actions with no branching.
• List three safety boundaries, each with a reason and a named approver.
• Give every mitigation a rollback and a wait time.
• Write the stop condition: the moment the responder escalates rather than continues.
• Add the header line: owner, last reviewed, last tested.
• Link the runbook from the alert itself.
• Book a fifteen-minute cold read with someone who does not own the service.