Postmortem Report From Incident Data

Set the rule before the next incident, so everyone knows when a postmortem is due. Google's SRE book lets anyone with a stake in an event ask for one, and lists these common postmortem triggers¹:

• Downtime or slowdowns users could see, past a set threshold
• Data loss of any kind
• An on-call engineer had to step in, for example with a rollback
• Resolution took longer than a set threshold
• Monitoring failed, which usually means a person found the problem, not an alert

Tie the rule to your severity levels. PagerDuty, which sells incident response software, writes a postmortem for every SEV-1 and SEV-2 incident². If yours are vague, an incident escalation matrix defines each level and who gets called.

Name one owner as soon as the incident closes. Google's SRE Workbook advises a single owner with several collaborators³.

Start with the timeline; everything else rests on it. PagerDuty's process makes it the main focus at the start² and wants every entry linked to its data, such as a graph or log search.

1. 1

   Collect every record

   Export the incident tool's log, alert history, chat channel, deploy and config changes, status page posts and support tickets.

2. 2

   Put it on one clock

   Convert every timestamp to one zone, usually UTC, and say which. Keep each original time beside its converted one.

3. 3

   Note what each time measures

   Alerts usually fire after a problem starts. In one published Google postmortem³, machines were pulled from service at 16:38, but pages began only around 17:10.

4. 4

   Source every row

   Link each entry to the alert, message or log line it came from, so anyone can settle a disputed minute.

5. 5

   Mark the gaps

   Where the record goes quiet, say for how long and what would fill it. Never bridge it with a guess.

Don't reduce the incident to one duration. The sample postmortem in Google's SRE book⁴ labels when the outage began, was mitigated and ended, and when the incident was declared and closed. Add the first alert, and each span measures one stage of the response.

Mark what made things worse. In the same sample, a responder shifted traffic back toward the other clusters at 15:32, and they began failing again a minute later. At 15:34 the team found an order-of-magnitude error in its whiteboard math for adding capacity.

Put numbers on the impact: failed requests, affected users, revenue, support tickets. Google's sample does it in one line: an estimated 1.21 billion queries lost, no revenue impact. The SRE Workbook faults a deliberately bad example whose only number was the duration³, adding that "a well-informed estimate is better than no data at all."

Separate the trigger, the event that set the failure off, from the causes that let it happen and spread. In Google's sample, the trigger was an 88-fold jump in traffic after a new sonnet was discovered. Its root cause: a resource leak on failed searches that, under that load, set off a cascading failure.

Atlassian, which sells incident management software, separates proximate causes from root causes⁵. Proximate causes directly led to the incident; root causes sit where "making a change will prevent this entire class of incident." Expect more than one. Richard Cook, a physician who researched patient safety, argued there is no isolated cause of an accident⁶, only contributors that cause it together.

Describe each action as the person saw it then. Google's SRE book says a blameless postmortem¹ assumes everyone involved "had good intentions and did the right thing with the information they had." Hindsight makes warning signs look clearer than they were, as Cook warned, so record what each responder knew and name roles, not people.

The Workbook's bad example shows how action items fail: every item had the same priority, two said "Improve" or "Make better," and only one had a tracking bug³. Without tracking, it warns, action items "are often forgotten, resulting in outages."

Give each item one owner, a priority, a ticket and an end state you can check. Prefer fixes to systems over reminders to people; as the Workbook puts it, "trying to change human behavior is less reliable than changing automated systems and processes."

Atlassian asks for at least one action that mitigates or prevents future incidents⁵, worded to pass three tests from Google's Sue Lueder and Betsy Beyer:

Then make sure they close. Atlassian gives each "priority action" a target of 4 or 8 weeks⁵, depending on the service, with reminders and reports. If a restore ran slower than planned, update your IT disaster recovery plan too.

Hold the review within days. PagerDuty's postmortem meetings usually last 15 to 30 minutes²: agree on the timeline, then discuss how the problem could have been caught, the customer impact and the action items.

Invite the incident commander, responders and service owners, then publish the postmortem widely. Google's SRE Workbook praises one circulated less than a week after the incident closed³.

Customers need a shorter note: when the problem started and ended, who was affected and whether they need to act. Save the word "outage" for a full outage; PagerDuty notes customers generally see it and assume the worst. The incident status page update template includes a resolution note for this.

Short on time? The tool at the top of this page puts your alerts, chat and logs on one clock and drafts the postmortem for you to check.