Structured Incident Roles
When high-severity outages occur, unstructured chaos compounds the damage. High-performing organizations use incident command systems with distinct roles:
- Incident Commander (IC): Owns the incident, delegates investigation tasks, and has final authority over mitigation actions (e.g., rollback or failover).
- Operations Lead: Subject matter expert executing diagnostics, reading graphs, and applying targeted fixes.
- Communications Lead: Provides periodic status updates to customers and internal stakeholders, freeing the technical team from interruptions.
Blameless Post-Mortems
Human error is a symptom of a flawed process or brittle system, not the root cause. Blameless post-mortems assume that engineers acted with good intentions based on the information they had. Reviews focus on systemic protections: improving guardrails, automated tests, and canary rollouts.
Severity levels drive the response
| Severity | Example | Response | Comms cadence |
|---|---|---|---|
| SEV1 | Checkout down for all users, data loss or a security breach | Page IC, ops and comms immediately, open a bridge call, notify executives | Status page every 30 minutes |
| SEV2 | One region or a major feature degraded, SLO burning fast | Page on-call, appoint an IC, open an incident channel | Internal updates every hour |
| SEV3 | Minor feature broken, a workaround exists | Handle during business hours | Ticket updates |
Declare an incident early and downgrade it later if needed. An incident that turns out to be nothing costs a few minutes. A SEV1 that runs as an unstructured chat thread for an hour can cost far more.
Worked timeline
In this example, time to detect is 4 minutes (from the 13:58 deploy to the 14:02 page), time to mitigate is 17 minutes, and the root cause (a config key renamed in one service but not in the other) is found the next day. The postmortem produces action items: validate the config schema in CI, make the canary stage block the rollout on 5xx errors, and add an alert on config-parse failures.
What goes in a postmortem
- Impact: duration, users affected, error budget consumed, revenue or SLA exposure.
- Timeline: timestamps from the first signal to resolution, including what responders believed at each point.
- Contributing factors: usually several, such as a missing test, an unsafe default and an alert that was too slow. Avoid the idea of a single root cause.
- Action items: each with an owner and a due date, and prioritised. Postmortems whose action items never get done are a common failure.
How real organisations do it
Google's SRE book describes the IC, Ops, Comms and Planning roles adapted from the US fire-service Incident Command System. PagerDuty publishes its open-source incident response guide with the same roles. Slack, Atlassian and Facebook have all presented their severity and review processes publicly. A common lesson from those talks is that incident tooling (a bot that creates the channel, pages roles and records the timeline) makes it much more likely that people actually follow the process.