I wrote my first real runbook at 2:14 AM, sitting on a plastic chair in a Lagos server room, listening to a diesel generator cough outside. The payment switch had failed. Again. The previous engineer’s documentation was a single text file: “Restart service. If still down, call Felix.” I was Felix. The service was down. And the file told me nothing about why restarting might work, what state the restart would leave the ledger in, or how to verify that we hadn’t double-posted transactions. That night, I learned that most runbooks are not technical documents. They are theater. They exist to satisfy an audit checkbox, not to guide a human being through a system failure under duress.
The problem is not that engineers are lazy. The problem is that we write runbooks as if the reader is a calm, well-rested colleague with perfect recall of the system architecture. In reality, the reader is you, at 2 AM, on your third cup of Nescafé, with a generator that might run out of diesel in forty minutes. Your working memory is shot. Your hands are shaking slightly from caffeine and cortisol. You need a document that does not just list steps. You need a document that tells a story.
Why Most Runbooks Are Documentation Theater
Walk through any engineering team’s runbook repository and you will find the same pattern: a flat list of commands, a few shell snippets, and a vague instruction to “check the logs.” These documents fail for three reasons. First, they omit context. They tell you what to run but not why you are running it, which means you cannot adapt when the expected output does not appear. Second, they assume a single failure mode. Real incidents are branching narratives. The database is slow because the disk is full, or because a replication slot is stuck, or because someone ran a schema migration during peak traffic. A linear runbook collapses at the first fork. Third, they are never tested. The team writes them during a calm sprint, files them away, and discovers they are useless only when the pager fires.
This is not a new observation. The Google SRE book dedicates entire chapters to emergency response and managing incidents, emphasizing that effective troubleshooting requires structured, context-rich documentation. The SRE approach treats runbooks as living artifacts that reduce mean time to recovery by guiding engineers through decision trees and system dependencies. But most teams outside the hyperscaler bubble do not have the bandwidth to maintain that level of documentation rigor. We need a lighter-weight method that still captures the narrative structure of a real incident.
The Narrative Structure of a Useful Runbook
Think of a runbook as a script. Not a screenplay with character arcs, but a tight technical script with clear scenes, character motivations, and branching plotlines. Every incident has a beginning, a middle, and—if you are lucky—an end. Your runbook should mirror that structure.
Scene One: Symptoms. What does the failure look like from the outside? Be specific. “Payment switch returns HTTP 503” is better than “service is down.” Include the exact error message, the monitoring dashboard URL, and the alert that fired. The goal is to confirm within thirty seconds that the runbook matches the incident. If the engineer has to guess whether this is the right document, you have already lost five minutes of generator runtime.
Scene Two: System Dependencies. Every service has dependencies. Your runbook must name them explicitly: the PostgreSQL primary, the Redis instance, the upstream NPSB (Nigerian Payment Service Bureau) connection, the VSAT link to the backup site. For each dependency, state how to check its health and what the expected healthy state looks like. During that Lagos outage, I wasted twenty minutes chasing a payment switch bug that was actually a failed NPSB TLS handshake because the runbook never mentioned the external dependency.
Scene Three: Decision Tree. This is where most runbooks collapse into a wall of text. Instead, structure the troubleshooting as a series of binary questions. “Is the database reachable? If yes, go to step 4. If no, go to step 7.” Each branch should have a clear exit condition. You are not writing a novel. You are writing a choose-your-own-adventure book where the stakes are real money.
Scene Four: Recovery Actions. For each leaf node in the decision tree, provide the exact commands to run, the expected output, and the rollback procedure if the recovery action makes things worse. Include the command to check the current state before you run the recovery. I have seen engineers restart a database replica without checking if it was still catching up, turning a five-minute outage into a six-hour resync.
Scene Five: Verification. How do you know the system is actually healthy? “Service returns 200” is not enough. For a payment switch, verify that a test transaction settles end-to-end. For a database, verify that replication lag is below your threshold. State the exact query or API call, and state the expected result. If the verification step fails, the runbook should loop back to the decision tree.
Writing the Script: A Concrete Example
Let me ground this in a real scenario. You are running a payment ledger on a single ARM board with 512MB RAM, using SQLite as the primary store. The board is in a rack in Lagos, powered by a generator that has been running for six hours. The alert fires: “Ledger write latency > 5 seconds.” You SSH in over a flaky 4G connection. What does a narrative runbook look like?
Symptoms: Alert name: ledger_write_latency_high. Dashboard: Grafana panel “Ledger Write Latency p99” at http://monitor.lagos.internal:3000/d/ledger. Expected: p99 < 500ms. Current: p99 = 8.2s. Check: curl -s http://ledger.lagos.internal:8080/health | jq .write_latency_ms.
Dependencies: SQLite database at /data/ledger/ledger.db. Disk: /dev/sda1 (ext4, 64GB eMMC). Power: generator phase A, monitored via ups.lagos.internal. Network: MTN 4G router at 192.168.1.1.
Decision Tree:
- Is the disk full? Run
df -h /data. If usage > 90%, go to Recovery A (disk cleanup). If not, go to step 2. - Is SQLite experiencing a write lock? Run
sqlite3 /data/ledger/ledger.db "PRAGMA busy_timeout;". If timeout is 0, go to Recovery B (set busy_timeout). If not, go to step 3. - Is the eMMC throttling? Run
cat /sys/class/thermal/thermal_zone0/temp. If > 80°C, go to Recovery C (thermal throttling). If not, escalate to on-call engineer.
Recovery A (Disk Cleanup): Check WAL size: ls -lh /data/ledger/ledger.db-wal. If > 100MB, run sqlite3 /data/ledger/ledger.db "PRAGMA wal_checkpoint(TRUNCATE);". Verify: df -h /data shows usage < 80%. Rollback: none; checkpoint is safe. If checkpoint fails, escalate.
Recovery B (Write Lock): Set busy_timeout: sqlite3 /data/ledger/ledger.db "PRAGMA busy_timeout=5000;". Verify: re-run health check, p99 < 500ms. Rollback: PRAGMA busy_timeout=0;.
Recovery C (Thermal Throttling): Check if fan is running: cat /sys/class/hwmon/hwmon0/fan1_input. If 0 RPM, manually start fan: echo 255 > /sys/class/hwmon/hwmon0/pwm1. Wait 5 minutes. Re-check temperature. If still > 80°C, reduce write load: systemctl stop ledger-reconciliation.timer. Verify: temperature < 70°C, p99 < 500ms. Rollback: systemctl start ledger-reconciliation.timer after temperature stabilizes.
Verification: Run end-to-end test: curl -X POST -d '{"amount": 100, "from": "test", "to": "test2"}' http://ledger.lagos.internal:8080/transact. Expect HTTP 201 and transaction ID. Check balance: curl http://ledger.lagos.internal:8080/balance/test2. Expect 100. If either fails, return to decision tree step 1.
This runbook is not beautiful. It is functional. It assumes the reader is tired, stressed, and working with limited tools. It does not require a separate wiki page to explain what a WAL checkpoint is. It embeds the context directly in the recovery step.
Why Narrative Structure Reduces MTTR
Cognitive load is the enemy of incident response. When you are on call, your brain is running a finite state machine with limited stack depth. A linear runbook forces you to hold the entire decision tree in your head, simulating branches mentally while also trying to remember which commands you have already run. A narrative runbook externalizes that state. Each step tells you where you are in the story, what you have learned so far, and what the next meaningful question is.
This approach aligns with the NIST Cybersecurity Framework, which treats incident response and recovery as structured functions requiring tested playbooks. NIST’s guidance emphasizes that operational procedures should be maintained and exercised, not filed away. A runbook that follows a narrative structure is inherently testable: you can walk through each branch during a tabletop exercise and discover where the script breaks.
During that Lagos outage, the generator failed at hour four. I had forty minutes of UPS runtime to either fix the payment switch or shut it down cleanly. The narrative runbook I wrote afterward—the one I just described—let me run through the decision tree in under ten minutes on a subsequent incident. The difference was not technical skill. It was that the document did not make me think about what to do next. It told me a story I had already rehearsed.
Maintaining the Script: Runbooks as First-Class Deliverables
A runbook is not a one-time artifact. It rots. Dependencies change, recovery commands drift, and the person who understood the original failure mode leaves the team. Treat runbooks like source code. Version them in the same repository as the service they document. Review them during code reviews. If a pull request changes the database schema, it must also update the runbook’s verification queries. If a new dependency is added, the runbook’s dependency section must reflect it.
I keep my runbooks in plain text, stored alongside the service configuration. No wikis, no Confluence pages that require VPN access when the VPN is down. A runbook.md file in the service repository, rendered by a simple static site generator, accessible over the local network even when the internet is unreachable. The format is deliberately constrained: Markdown with explicit section headers, no collapsible sections, no JavaScript. The document must be readable in less over an SSH session.
Writing a good runbook is closer to writing a technical script than writing documentation. You are crafting a sequence of scenes, each with a clear dramatic question: “Is the disk full?” “Is the database reachable?” “Is the thermal sensor lying?” The answer propels the reader to the next scene. This is why I find that tools designed for narrative structure—like a script writing app that forces you to think in scenes and beats—can clarify the mental model even for operational documents. You do not need Final Draft to write a runbook, but you do need the discipline of scene-based thinking. Every section must earn its place by answering a question the on-call engineer will actually ask.
The Decision Checklist: Is Your Runbook Useful or Theater?
Before you file a runbook as “done,” run it through this checklist. If you answer “no” to any question, the document is not ready for production.
- Can a tired engineer confirm in 30 seconds that this is the right runbook? The symptoms section must include the exact alert name, error message, or dashboard panel. If the engineer has to grep through a wiki search, the runbook has already failed.
- Does the runbook name every external dependency and how to check its health? If the payment switch depends on an NPSB connection, the runbook must include the command to test that TLS handshake. Do not assume the reader remembers.
- Is the troubleshooting structured as a decision tree with binary questions? “Check the logs” is not a decision. “Is the error log showing ‘connection refused’? If yes, go to step 4” is a decision.
- Does every recovery action include a pre-check command, the recovery command, expected output, and a rollback procedure? If you cannot roll back, state that explicitly and explain why.
- Does the verification step test the system end-to-end, not just the component you touched? Restarting the database is not a fix. A successful test transaction is a fix.
- Is the runbook stored in a place accessible when the primary network is down? If your runbook lives on a wiki that requires the VPN, and the VPN is what failed, you have a circular dependency.
- Was the runbook tested in the last quarter by someone who did not write it? Untested runbooks are documentation theater. Schedule a tabletop exercise. Break the system intentionally in a staging environment and hand the runbook to a colleague. Watch where they get stuck.
- Does the runbook include the name and contact method of the human who owns the service? Sometimes the script ends. The final scene of every runbook should be: “Escalate to [name] at [phone number]. State what you have checked and what the current system state is.”
The Generator Test
There is a final test I apply to every runbook I write. I call it the generator test. Imagine you are on call. The generator outside has been running for three hours. You have maybe ninety minutes of fuel left. The UPS is beeping. Your internet connection is a single 4G hotspot that drops packets whenever it rains. You are the only engineer awake within two time zones. Open your runbook. Can you follow it from symptom to resolution without once stopping to think, “What does this step mean?” If the answer is no, the runbook is not finished.
Most engineering advice assumes infrastructure that most of the world does not have. It assumes you can spin up a new instance, fail over to another region, or page a secondary on-call. Runbooks written under those assumptions are fragile. They break the moment the context shifts from “comfortable office with dual monitors” to “plastic chair in a server room with a dying generator.” Write your runbooks for the plastic chair. Write them as if the person reading them is you, at 2 AM, with everything on the line. That is the only audience that matters.