How to Write Runbooks That Actually Get Used During Incidents

A runbook is a real-time system with a human in the loop. The human is the most unreliable component. Design for that fact or accept that your runbook is decorative.

I learned this at 2:47 AM on a Wednesday in Lagos, during a database failover that should have taken 90 seconds and took 47 minutes. The runbook was written by a consultant who had never operated anything in West Africa. Step 4 said: “Wait 30 seconds and verify replication status on the replica.” The replica was reachable only over a shared 4G APN with 800ms RTT under no load—and during the failover window, because failover windows are when everything on the network decides to talk at once, the RTT spiked to roughly 3 seconds with packet loss around 4%. The “30-second” verification step took 11 minutes. The primary was already gone. We were blind.

The consultant was not incompetent. The runbook was clean, well-structured, and had obviously been reviewed by someone who cared. But it assumed an operator who could reach the replica over a low-latency private link, type commands without typos induced by sleep deprivation, and trust that the network would cooperate during the most stressful 60 seconds of the quarter. Every one of those assumptions was wrong. None of them was written down.

What follows is a field guide to writing runbooks that survive contact with reality: sleep-deprived engineers, 3G tethers, terminals with no GUI, brownouts that might kill the machine in 30 seconds. The core thesis is that a runbook is not documentation—it is an embedded system with a human peripheral. You should engineer it the way you would engineer any safety-critical control loop under degraded conditions.

Why Most Runbook Advice Assumes a World That Does Not Exist

The canonical reference for site reliability engineering practice, the Google SRE book, devotes entire chapters to being on-call, effective troubleshooting, emergency response, and managing critical state. These chapters are excellent. Read them. But they describe an environment with private fiber between datacenters, dedicated SRE rotations, and automated remediation for most failure modes. When the book talks about a human executing a procedure, that human is typically at a workstation with a stable link, a second monitor showing a dashboard, and a teammate on chat.

That is not the world I operate in. I suspect it is not yours either. The last runbook I followed during a real incident was executed from a parking lot in Ikeja, on a laptop tethered to a phone whose battery was at 14%, while a UPS in the server room beeped its low-battery warning every 30 seconds. There was no dashboard. There was a terminal, an SSH session that dropped twice, and a colleague on a voice call reading me steps from a wiki page that kept timing out on load.

The Google SRE framework is still the right starting point. The discipline of treating incident response as a practiced activity with documented procedures—not ad-hoc heroics—is the correct mental model. The problem is that most teams stop at the framework and never adapt the procedures to their actual operating environment. A runbook that assumes a calm, well-rested operator with stable connectivity is not wrong. It is incomplete in a way that becomes dangerous at the exact moment it is needed.

The Lagos Failover: What the Runbook Said vs. What Happened

The original runbook for our PostgreSQL failover had seven steps. I will not reproduce it verbatim, but the structure was familiar:

1. Confirm the primary is unreachable via pg_isready -h primary -p 5432.
2. Check the replica’s pg_stat_replication to confirm it is receiving WAL.
3. Promote the replica using pg_ctl promote -D /var/lib/postgresql/data.
4. Wait 30 seconds and verify the replica accepts writes.
5. Update the connection string in the application config.
6. Restart the application pool.
7. Verify the application can write.

Clean. Linear. Wrong for our environment in at least four ways.

First, step 1 assumed pg_isready would return quickly. Over the 4G link, a TCP connect to the primary took 3–8 seconds when the host was up, and 75 seconds (the default tcp_syn_retries timeout) when it was down. The runbook did not specify a timeout, so the operator waited the full 75 seconds, concluded the primary was down, and moved on. Correct in this case, but only by accident. If the primary had been up but slow, the operator would have promoted a replica against a live primary.

Second, step 2 assumed the operator could query the replica. The replica was in a secondary datacenter reachable over the 4G APN. The pg_stat_replication query—trivial on a LAN—took 11 minutes over the degraded link because the SSH session had to be re-established twice and the psql client’s default connect timeout was 60 seconds, which was sometimes not enough.

Third, step 4 said “wait 30 seconds and verify.” Verify how? The runbook did not say. The operator ran SELECT 1; and it returned. But SELECT 1 does not confirm the node is a primary—it confirms the node is alive. The operator needed SELECT pg_is_in_recovery(); returning false. The runbook omitted this.

Fourth, step 5 said “update the connection string in the application config.” Which file? On which hosts? Via what mechanism? The runbook assumed a config management tool that could push changes. Our config management ran from the same datacenter as the primary, which was down.

Every one of these gaps was survivable in a review meeting. Every one was fatal at 2:47 AM.

A Framework: Runbooks as Embedded Firmware Documentation

After that incident, I rewrote every failover runbook using a mental model borrowed from embedded systems. Treat the runbook as firmware documentation for a control loop where the actuator is a sleep-deprived human and the sensor is a degraded network. In embedded systems, you do not write “set the register and check the flag.” You write: “Write 0x01 to register CTRLA. Within 10ms, read STATUS. If bit 3 is set, the device is ready. If STATUS reads 0xFF, the bus is hung—power-cycle the device and retry from step 1. Do not proceed past this step if STATUS is not 0x01 or 0x02 within 50ms.”

Every step has a precondition, a timeout, an expected result, a failure mode, and a “stop here if” branch. The human never has to decide what to do next under stress. The decision tree is already in the document.

The rewritten runbook has the following structure for each step:

• Precondition: What must be true before you start this step. If it is not true, do not proceed—go to the specified fallback.
• Command: The exact command, with all flags, including timeouts. No psql without -c. No ssh without -o ConnectTimeout=10.
• Expected: The exact output you should see, or a pattern. Not “it should work”—the literal string or a regex.
• If unexpected: What to do if the output does not match. Usually: stop, try once more with a longer timeout, then go to a named fallback step.
• Time budget: How long this step should take under normal conditions and under degraded conditions. If it exceeds the degraded budget, something is wrong. Do not keep waiting.
• Stop here if: The conditions under which you must halt the entire procedure and escalate. This is the circuit breaker for the human.

Here is what step 1 of the failover runbook looks like now:

Step 1: Confirm primary is unreachable.

Precondition: You have SSH access to the replica host. If you do not, go to Fallback A (manual promotion from console).

Command:

timeout 15 pg_isready -h <primary_ip> -p 5432 || echo "UNREACHABLE"

Expected: Output containing no response or UNREACHABLE within 15 seconds.

If you see: accepting connections — the primary is up. Stop here. Do not promote. Go to Fallback B (investigate application-level failure instead).

If the command takes longer than 15 seconds: The network is degraded. Note the time. Proceed to step 2 but expect all subsequent steps to take 3–5× longer than the time budget.

Time budget: Normal: 2 seconds. Degraded: 15 seconds. If it takes longer than 15 seconds, your SSH session may also be unstable—reconnect before proceeding.

Stop here if: You cannot establish SSH to the replica at all. Go to Fallback A.

That is more verbose than the original. It is also executable by someone who has never done a failover before, at 3 AM, over a bad link, without making a decision that destroys data.

Encoding Network Assumptions as Explicit Preconditions

The single most important change in the rewrite was making network assumptions explicit. The original runbook assumed the operator could reach the replica over a low-latency private link. That assumption was never written down, so it was never tested, so it failed silently.

In the rewritten runbook, every step that involves a network call has a precondition stating the expected network characteristics and what to do if they are not met. For the Lagos failover, step 2 now reads:

Step 2: Check replication status on replica.

Precondition: RTT to the replica is under 2 seconds. Test with:

timeout 10 ssh -o ConnectTimeout=10 replica_host 'echo OK'

If this does not return OK within 10 seconds, you are on a degraded link. All time budgets in this runbook are now multiplied by 5. If the command does not return within 30 seconds, proceed to Fallback C (promote blind, accept data loss, reconcile later).

This is uncomfortable to write. “Promote blind, accept data loss” is not something anyone wants to put in a runbook. But the alternative is worse: an operator who spends 11 minutes on a verification step while the primary is gone and the application is hard-failing, then makes a panicked decision with no structure at all. Explicit failure modes are not pessimism. They are honesty. Structured frameworks for operational risk management, like the NIST Cybersecurity Framework, make response and recovery explicit, versioned, and profile-based precisely because implicit knowledge breaks under stress. Your runbook should do the same.

Verification Steps That Degrade Gracefully

The original runbook’s verification step (“wait 30 seconds and verify”) had two problems. It did not say what to verify, and it assumed the verification would complete in 30 seconds. The rewrite addresses both.

For verification, every step now specifies the exact query and the exact expected output. For the failover, the verification is:

psql -h replica -p 5432 -c "SELECT pg_is_in_recovery();"

Expected: f (false). If you see t, the promotion did not work—go to Fallback D. If the query times out, the node is alive but overloaded. Wait 60 seconds and retry once. If it times out again, go to Fallback C.

For degradation, every time-bound step now has two budgets: a normal budget and a degraded budget, with the degraded budget typically 5× the normal. The runbook states that if any step exceeds the normal budget, the operator should note it and expect all subsequent steps to be similarly slow. This prevents the failure mode where an operator assumes the next step will be fast because the last one was, then makes a timing-based decision on bad data.

Testing Runbooks Over Throttled Connections

A runbook you have never executed under degraded conditions is a hypothesis, not a procedure. After the Lagos incident, I started testing every runbook over a throttled connection before it was considered production-ready.

The test setup is simple and crude, which is the point:

• Connect to the network you actually use during incidents. For us, that is a 4G APN with intentional throttling applied via tc qdisc on the test machine: 256 Kbps, 800ms RTT, 2% packet loss.
• Execute the runbook end to end, timing each step. Record where it exceeds the degraded budget.
• Have someone who did not write the runbook execute it. Watch where they hesitate, where they ask questions, where they improvise. Every hesitation is a missing precondition. Every question is a missing command specification. Every improvisation is a missing fallback.

The first time I ran the failover runbook through this test, step 2 took 9 minutes. The runbook said 30 seconds. The gap was not a bug in the runbook—it was an unstated assumption about the network. The test made the assumption visible. The rewrite made it explicit.

I also test runbooks with a second constraint: the operator cannot use a GUI. No dashboards, no web-based wiki, no browser tabs. Only a terminal and a local copy of the runbook as a plain text file. If the runbook requires a GUI to execute, it fails the test. This is not because GUIs are bad. It is because GUIs are the first thing to go when the network is degraded, and a runbook that depends on a Grafana dashboard is a runbook that cannot be followed when Grafana takes 45 seconds to load a panel.

Structuring the Document: Why I Do Not Use Wikis for Runbooks

Wikis encourage drift. Multiple editors, no linear structure, no review gate, no version that is “the” version. A runbook in a wiki is a pile of bullet points that someone edited once at 4 PM on a Tuesday and that someone else will try to follow at 3 AM on a Saturday. The two experiences are not the same document.

I draft runbooks as structured, versioned documents—plain text or Markdown in a Git repo, reviewed via pull request, with a single canonical version per incident type. The discipline of writing a runbook as a coherent document, not a wiki page that grew organically, forces you to think about narrative flow: does step 3 make sense if step 2 was degraded? Does the fallback path actually reconnect to the main path, or does it dead-end? These are questions that a bullet-point wiki does not force you to answer.

I have started drafting the narrative prose for new runbooks in a structured writing tool rather than a wiki. The act of composing a runbook as a document—with preconditions, decision trees, and fallback paths that must be internally consistent—is closer to writing a technical specification than maintaining a knowledge base. I use an AI novel writing app for the initial draft pass, not because it generates runbook content—it does not, and you should not let it—but because the structured outlining and chapter-based workflow forces the same discipline that a coherent technical document demands: every section has a purpose, every branch has a destination, every fallback has a name. The tool is for structure, not content. The content comes from incident experience, postmortem findings, and the throttled-connection tests described above. What the tool provides is the scaffolding that prevents the document from devolving into a wiki-style pile of loosely connected notes.

That same discipline applies to editorial structure: before publishing, editors need a way to test scattered notes become an argument readers can follow, which is where an AI novel writing app that fits the project can function as a planning aid rather than a substitute for domain evidence.

The key principle: a runbook should be a single, reviewable artifact with a version, an owner, and a test date. Not a living document that anyone can edit at any time. Living documents die during incidents.

The Postmortem Feedback Loop

Every runbook that is used during a real incident must be reviewed afterward. This is not optional. It is not a nice-to-have. The review answers three questions:

1. Did the runbook work? Did every step complete within its degraded budget? Did every fallback path get used, and did it work?
2. What assumptions did the runbook make that turned out to be wrong? Every unstated assumption that bit you during the incident gets written as an explicit precondition in the next version.
3. What did the operator do that was not in the runbook? Every improvisation is a missing step or a missing fallback. Add it.

This is the same feedback loop that postmortem culture in SRE practice prescribes, applied specifically to the runbook as an artifact. The postmortem is about the system. The runbook review is about the document that tells you how to operate the system. Both are necessary. A postmortem without a runbook review means you understand what went wrong but have not updated the procedure for next time.

Checklist: Runbook Readiness for Degraded Conditions

• Every step has a precondition. If the precondition is not met, the step says where to go instead.
• Every command includes explicit timeouts. No command relies on default timeouts.
• Every step specifies the expected output as a literal string or pattern, not “it should work.”
• Every step has a normal time budget and a degraded time budget (typically 5× normal).
• Every step has a “stop here if” condition that halts the procedure and triggers escalation.
• Every fallback path is named (Fallback A, B, C) and has its own steps, not just “figure it out.”
• The runbook has been executed end to end over a throttled connection (256 Kbps, 800ms RTT, 2% loss) by someone who did not write it.
• The runbook has been executed without a GUI—terminal only, local copy as plain text.
• The runbook has a version, an owner, and a last-tested date in the document header.
• After every real incident, the runbook is reviewed and updated within 72 hours.

A runbook is not documentation. It is a control system. Engineer it the way you would engineer anything that has to work when everything else is failing: with explicit assumptions, explicit failure modes, and explicit decision trees that a sleep-deprived human on a bad link can follow without thinking. If your runbook cannot be followed at 3 AM over 4G during a brownout, it is not a runbook. It is a wish.