How to write a blameless postmortem

What does a blameless postmortem record?

Google SRE defines a postmortem as a written record of an incident. This record includes the incident's impact, the actions taken to mitigate the impact or resolve the problem, the root causes, and follow-up actions to prevent the incident from recurring. The purposes of writing it include documenting the incident, understanding the contributing root causes, and identifying preventive actions that will reduce the likelihood or impact of a recurrence.

In a blameless approach, the review focuses on the causes that contributed to the incident rather than accusing a person or team of wrongdoing. Google SRE assumes that everyone involved in the incident acted in good faith and did the right thing with the information they had. This approach treats the postmortem as an opportunity for the organization to learn, rather than as a tool for punishment.

According to Google SRE, in an environment of blame and shame, people may not bring problems to light for fear of punishment. A blameless review instead investigates why a person or team had incomplete or incorrect information. Systems and processes can be corrected to support the people who design and operate complex systems.

Which incidents require a postmortem?

Google SRE expects postmortems to be prepared after significant, undesirable incidents. Because the process costs time and effort, teams determine which incidents require this review. Google's example triggers include a user-visible outage or service degradation, data loss, and intervention by the on-call engineer. Intervention may involve rolling back a change or redirecting traffic.

Resolution time exceeding a defined threshold and a monitoring failure are also example triggers. Google SRE recommends defining postmortem criteria before an incident so that everyone knows when a review is required. Alongside objective triggers, any stakeholder can also request a postmortem for an incident.

Describing the incident and its impact

The Google SRE Workbook defines the role of a postmortem as helping the reader understand what happened, how the problem was mitigated, and how users were affected. It recommends using numerical information to show the impact consistently, especially for outages affecting multiple services. It states that when precise data is unavailable, an informed estimate is more useful than no data at all.

The impact section of the Workbook's positive example includes details from different perspectives. The example uses measurements such as cache hit rates, traffic levels, and the duration of the impact. Linking to locations containing relevant data gives the reader context and reduces uncertainty.

The actions taken during the incident are also part of the record. The Workbook notes that the Recovery Efforts section was left blank in the negative example it reviews. This section is often where answers to questions such as how the problem was mitigated and how users were affected can be found. In Google SRE's approach, completion of the impact assessment is also one of the review criteria.

Readers are not limited to the team where the incident occurred. The Workbook recommends providing context for technical terms in Background or Glossary sections. These sections were left blank in the negative example it reviews. In the positive example, the Glossary helps a broader audience understand the text.

The Workbook's positive example also addresses long chat transcripts and system logs. These are summarized in the postmortem, with links from the main text to their unedited versions. The same example includes explanations of graphs for readers unfamiliar with them.

Examining root causes without blame

The Google SRE Workbook lists identifying the root causes and the circumstances that triggered the incident among the important purposes of writing a postmortem. In the negative example it reviews, these topics are described in a short paragraph, but the lower-level details of the problem are not investigated. In the positive example, the review is not limited to the area where the failure occurred directly; it also addresses effects and system flaws involving multiple teams.

Blameless language focuses on what went wrong rather than who caused the incident. In the Workbook's positive example, no person or team is blamed in the Things that went poorly section. The Root cause and trigger section examines the cause and trigger of the incident. The actions also aim to improve the system rather than trying to change people.

The Workbook states that explicitly singling people out can lead team members to avoid risk for fear of being publicly shamed. The same fear can also lead to the withholding of information that is important for understanding the incident and preventing its recurrence. Google SRE advocates examining the contributing conditions rather than blaming or shaming individuals and teams.

Language is also part of this review. The Google SRE Workbook says that a postmortem should be a factual record, free of personal judgments and subjective statements. It calls for the postmortem to address different perspectives and be respectful of others. Dramatic language can distract from the main message; the Workbook recommends using verifiable data to support the seriousness of a situation.

Defining and tracking actions

According to the Google SRE Workbook, a postmortem without actions is ineffective. In the positive example it reviews, each action has an owner and a tracking number. Each action has been assigned a priority level, and whether it has been completed can be verified. The actions also include work intended to prevent the outage from recurring or reduce its impact.

The Workbook notes that most actions in the negative example are aimed at mitigating the current impact. It recommends also including preventive actions and fixes to reduce the likelihood of recurrence. In the same example, giving all actions equal priority makes it difficult to determine which to address first.

How an action is phrased also affects how the outcome is measured. The Workbook finds expressions such as simply improving something or making it better vague; these expressions make it difficult to understand and measure the criteria for success. In the positive example, the actions have a verifiable end state. Grouping actions from a major incident by theme also makes it easier to assign owners and priorities.

Ownership is addressed at two levels. The Workbook recommends a single point of contact responsible for follow-up and completion of the postmortem, along with multiple contributors. Actions should also have clear owners; according to the Workbook, actions with unclear owners are less likely to be resolved. Google tracks the closure of actions arising from postmortems by recording them in an issue tracking system.

In the Workbook's negative example, only one action has a tracking record. The Google SRE Workbook states that postmortem actions can be forgotten when there is no formal tracking process. In the tracking system Google uses, the closure of actions arising from each postmortem can be tracked.

Review and sharing

In Google SRE, teams share the first draft internally and ask a group of experienced engineers to assess it for completeness. This assessment checks whether important data about the incident has been collected and whether the impact assessment has been completed. Other example criteria include whether the root cause investigation goes deep enough, the suitability of the action plan, and whether the results are shared with relevant stakeholders.

Google SRE encourages regular postmortem review sessions to review every completed draft. In these sessions, open discussions and comments are resolved, ideas are recorded, and the text is finalized. When participants are satisfied with the content and actions, the postmortem is added to the place where the team or organization keeps records of past incidents.

The Google SRE Workbook also addresses involving incident participants in the writing process. When a postmortem is written by one person alone or by only one team, important factors that contributed to the outage may be overlooked. An open review process and communication plan can help prevent blaming language and perspectives from spreading within the organization.

After the initial review, Google SRE shares the postmortem with a broader engineering audience or on an internal mailing list. The aim is to reach people who could benefit from what was learned. In Google's approach, information that could identify an end user is also excluded from internal postmortems.

The Google SRE Workbook also lists cross-team reviews among sharing methods. In these sessions, one team describes an incident it experienced; other teams ask questions and learn from the incident. In the reading groups described by Google SRE, participants also discuss how the incident unfolded, what it taught, and what happened afterward.

Limits of the process

Google SRE notes that preparing a postmortem takes time and effort. According to the Workbook, the quality of a postmortem may decline when a team is overloaded with other work. A weak postmortem with incomplete actions also increases the likelihood of the incident recurring. The Workbook recommends tracking the writing and review process and ensuring that teams have enough time to implement the relevant action plan.

Not every step of writing a postmortem can be fully automated. According to the Google SRE Workbook, templates and tools can make writing and managing related data easier. This allows authors to spend time on tasks such as root cause analysis and action planning. Google SRE also emphasizes that the review should not be forgotten as a formality and that the identified actions must be put into practice.