Write an incident response runbook for a web application

TL;DR — Build a runbook that answers four things fast: how to detect the incident, how to stop the bleeding, how to verify recovery, and who approves risky actions. The most common failure is writing generic prose instead of executable steps; fix that by putting exact commands, log queries, rollback actions, and expected output directly in the document. Reading time: ~5 min

Goal

When you are done, your repository contains a versioned incident response runbook for your web application that an on-call developer can execute during a real outage, including severity levels, contacts, diagnostic commands, containment steps, recovery actions, and verification checks.

Prerequisites

• Repository write access for the application and permission to merge to the default branch
• Access to your production environment: SSH, Kubernetes, container platform, or cloud console used by the app
• Access to logs, metrics, and error tracking used in production
• A shell with bash, git >= 2.39 — check with git --version
• A text editor available from terminal, for example vim, nano, or code
• The deployment identifiers at hand: service name, namespace, hostnames, load balancer name, database name, queue names
• The current on-call contacts and escalation path for engineering, product, and customer support
• One person who can review the runbook and one person who can participate in a 15-minute tabletop test

Steps

Step 1: Create the runbook file in the repo

Use a predictable path so responders can find it during an incident.

mkdir -p docs/runbooks
cat > docs/runbooks/incident-response-webapp.md <<'EOF'
## Incident summary
- Service: <service-name>
- Production URL: https://<hostname>
- Runtime: <kubernetes|vm|containers>
- Primary datastore: <postgres|mysql|redis|other>
- Queue/cache dependencies: <list>
- On-call channel: <chat-channel>
- Incident commander default: <role>

## Severity levels
| Sev | Trigger | Response target | Examples |
|---|---|---:|---|
| SEV1 | Full outage, data loss, active security event | 5 min | 5xx for all traffic, unauthorized access |
| SEV2 | Major degradation, partial outage | 15 min | checkout failing, elevated 5xx in one region |
| SEV3 | Minor degradation, workaround exists | 1 hour | delayed jobs, admin-only feature broken |

## Detection
- Primary alert source: <monitor name>
- SLI/SLO checks: <error-rate, latency, saturation>
- Health endpoints: <exact URLs>

## First 10 minutes
1. Acknowledge the page in <paging system>.
2. Start incident channel: <channel naming pattern>.
3. Assign roles: incident commander, comms, operations.
4. Freeze deploys:
   - <exact command or menu path>
5. Capture current state:
   - `date -u`
   - `git rev-parse HEAD`
   - <exact command to list running version>

## Triage commands
```bash
curl -sS -o /dev/null -w 'code=%{http_code} total=%{time_total}\n' https://<hostname>/health
curl -I https://<hostname>/
<log query command>
<metrics query or kubectl command>

Expected healthy result:

code=200 total=0.120
HTTP/2 200

Containment

• Disable bad rollout:
  • <exact rollback command>
• Remove instance from rotation:
  • <load balancer or scaling command>
• Block abusive traffic:
  • <WAF/rate-limit command or provider path>
• Rotate compromised secret:
  • <secret manager command>

Recovery

• Restore last known good version:
  • <exact deploy command>
• Restart workers:
  • <exact command>
• Clear stuck queue safely:
  • <exact command>
• Database failover or restore:
  • <exact command or documented path>

Verification

curl -sS -o /dev/null -w 'code=%{http_code} total=%{time_total}\n' https://<hostname>/health
curl -sS -o /dev/null -w 'code=%{http_code} total=%{time_total}\n' https://<hostname>/login
<command to verify error rate back to baseline>
<command to verify queue depth draining>

Communications

• Internal updates every: 15 minutes
• Customer status page owner: <role>
• Template:
  • Impact: <who is affected>
  • Start time: <UTC>
  • Current status: <investigating|identified|monitoring|resolved>
  • Next update: <UTC>

Exit criteria

• Error rate below <threshold> for 15 minutes
• P95 latency below <threshold>
• Queue depth below <threshold>
• No new critical errors in <error tracker>

Post-incident

• Preserve logs and timeline
• Open follow-up issue
• Schedule review within 3 business days EOF

You should see a new file at `docs/runbooks/incident-response-webapp.md`.

### Step 2: Replace placeholders with literal service values
Edit the file and replace every `<...>` token with real values from your environment.

```bash
grep -n '<.*>' docs/runbooks/incident-response-webapp.md

Success means grep prints the remaining placeholders you still need to replace; when finished, it prints nothing and exits with code 1.

Step 3: Add exact diagnostic commands for your stack

Replace the generic triage block with commands that work in your production environment. Use one of these patterns, not prose.

For Kubernetes:

perl -0pi -e 's#<log query command>#kubectl -n production logs deploy/web --since=10m | tail -n 100#; s#<metrics query or kubectl command>#kubectl -n production get pods -o wide#; s#<exact command to list running version>#kubectl -n production get deploy web -o jsonpath="{.spec.template.spec.containers[0].image}{"\n"}"#; s#<exact rollback command>#kubectl -n production rollout undo deploy/web#; s#<exact deploy command>#kubectl -n production rollout undo deploy/web#; s#<exact command>#kubectl -n production rollout restart deploy/worker#' docs/runbooks/incident-response-webapp.md

For a systemd service on VMs:

perl -0pi -e 's#<log query command>#journalctl -u webapp --since "10 min ago" -n 200 --no-pager#; s#<metrics query or kubectl command>#systemctl status webapp --no-pager#; s#<exact command to list running version>#readlink -f /srv/webapp/current#; s#<exact rollback command>#ln -sfn /srv/webapp/releases/<last-known-good> /srv/webapp/current && systemctl restart webapp#; s#<exact deploy command>#ln -sfn /srv/webapp/releases/<last-known-good> /srv/webapp/current && systemctl restart webapp#; s#<exact command>#systemctl restart webapp-worker#' docs/runbooks/incident-response-webapp.md

You should see the file updated with runnable commands instead of placeholders.

⚠️ Rollback, restart, queue purge, failover, and secret rotation can cause downtime or data loss if used blindly. Put the approval role directly next to each risky command, for example: "Approval: incident commander" or "Approval: DBA".

Step 4: Add one concrete symptom-to-action table

Append a table that maps common symptoms to the first command to run and the likely containment action.

cat >> docs/runbooks/incident-response-webapp.md <<'EOF'

## Symptom map
| Symptom | First command | Likely cause | First containment |
|---|---|---|---|
| All requests return 502 | `curl -I https://<hostname>/` | upstream app down | rollback or restart app |
| `/health` is 200 but `/login` is 500 | `kubectl -n production logs deploy/web --since=10m | tail -n 100` | app dependency failure | disable bad deploy, inspect auth/db |
| High latency with low CPU | `<queue-depth command>` | queue backlog or lock contention | scale workers or pause producers |
| Spike in 401/403 | `<auth log query>` | auth provider issue or bad secret | rotate secret or bypass optional auth path |
| DB connections exhausted | `<db connection query>` | leak or traffic spike | restart leaking app, raise pool carefully |
EOF

You should see a ## Symptom map section at the end of the file.

Step 5: Add a minimal incident flow diagram

Put the response order in a diagram so responders do not improvise sequencing.

cat >> docs/runbooks/incident-response-webapp.md <<'EOF'

## Response flow
```diagram
[Alert] ---- 1. page ----> [On-call]
                             |
                             v
                    [Incident Channel]
                             |
                             v
                    [Triage Commands]
                             |
          2. rollback/restart|3. status updates
                             v
                    [Containment]
                             |
                             v
                     [Verification]

EOF

You should see a `diagram` fenced block in the file.

### Step 6: Commit the runbook and open it for review
Version it with the app so changes ship with architecture changes.

```bash
git checkout -b docs/incident-runbook-webapp
git add docs/runbooks/incident-response-webapp.md
git commit -m "docs: add web application incident response runbook"

Success means git commit exits 0 and prints a commit hash, for example:

[docs/incident-runbook-webapp 1a2b3c4] docs: add web application incident response runbook
 1 file changed, 120 insertions(+)

Step 7: Run a 15-minute tabletop test against the document

Use a fake but realistic scenario and time whether another developer can execute the runbook without asking questions.

printf '%s
' 'Scenario: production web returns intermittent 502 after deploy; rollback required; customer login impacted.' > /tmp/incident-scenario.txt
sed -n '1,220p' docs/runbooks/incident-response-webapp.md

Success means the tester can answer, from the runbook alone: where to look first, how to stop deploys, how to rollback, how to verify recovery, and who sends updates.

Verify it works

Run these checks before merging:

grep -n '<.*>' docs/runbooks/incident-response-webapp.md
grep -nE 'ensure|appropriately|as needed|TBD|todo' docs/runbooks/incident-response-webapp.md

Expected result: both commands print nothing. The first exits 1 if no placeholders remain; the second exits 1 if no vague language remains.

Then verify the runbook has executable sections:

grep -n '^## ' docs/runbooks/incident-response-webapp.md

Expected output shape:

1:## Incident summary
8:## Severity levels
14:## Detection
18:## First 10 minutes
27:## Triage commands
39:## Containment
47:## Recovery
55:## Verification
62:## Communications
70:## Exit criteria
75:## Post-incident
79:## Symptom map
87:## Response flow

Finally, have a second developer use the runbook in a tabletop. It works if they complete the scenario without opening other docs and can state the rollback command and the verification command within 2 minutes.

Common pitfalls

Placeholders left in the file

Mistake: shipping <hostname>, <log query command>, or <exact command> tokens.

Symptom: during an incident, the responder stops to ask what service name or namespace to use.

Fix: run grep -n '<.*>' docs/runbooks/incident-response-webapp.md and replace every match before merge.

Generic language instead of commands

Mistake: writing "check logs" or "rollback the deployment".

Symptom: the runbook is technically complete but useless under pressure; responders improvise and skip steps.

Fix: replace each sentence with one literal command, for example kubectl -n production rollout undo deploy/web.

Missing verification criteria

Mistake: the runbook says how to restart or rollback but not how to prove recovery.

Symptom: incidents get marked resolved while 500s or queue lag continue.

Fix: add exact checks with thresholds, such as curl -sS -o /dev/null -w 'code=%{http_code} total=%{time_total}\n' https://app.example.com/health and "error rate below 1% for 15 minutes".

Risky actions have no approval owner

Mistake: secret rotation, queue purge, failover, or database restore are listed without an approver.

Symptom: either nobody acts, or someone acts and causes avoidable downtime or data loss.

Fix: append Approval: <role> on the same line as each risky command.

Runbook lives outside the app repo

Mistake: storing the runbook in a wiki that drifts from the deployed system.

Symptom: command names, namespaces, and hostnames are stale when the incident happens.

Fix: keep docs/runbooks/incident-response-webapp.md in the application repository and update it in the same PR as infra or deployment changes.

This article was written by an AI system and published pending human review. Verify anything you intend to act on.