Ninety minutes of checkout failures, and the postmortem was two lines: cache misconfiguration, config corrected. Both true. Neither explained why no alert fired for the first 41 minutes, or why the only person who understood that cache layer was on a flight. Six weeks later the same shape of failure came back, and that time I could not pretend it was bad luck.
What follows is the process I use now with teams here, written for the common Singapore setup: a small core team, some of it distributed across neighbouring time zones, carrying a service that customers notice immediately when it stops.
Step 1: Write the Trigger Down, and Hold the Review Inside 48 Hours
Decide in advance which incidents get a written postmortem. Deciding case by case always drifts toward writing them only when someone senior was annoyed, which teaches the team that the process is about politics.
A default that works for a team under about forty engineers:
- Any customer-visible outage over 15 minutes.
- Any data loss or data exposure, regardless of duration.
- Any incident that needed a manual fix in production.
- Any near miss that avoided customer impact by luck.
The near miss category is the one most teams skip and the one I would fight for. It carries nearly the same information as a real outage and costs nothing in trust, because nobody is bruised. A team that reviews near misses learns most of what it needs without having to fail first.
Then the timing rule: hold the review within 48 hours. Sooner and people are still defensive and tired. Much later and the details are gone, replaced by a tidy story everyone has rehearsed. Forty-eight hours is late enough for adrenaline to clear and early enough that someone can still say “actually I checked that dashboard first and it looked fine”, which is usually the most valuable sentence in the room.
Step 2: Build the Timeline Before Anyone Explains Anything
One person reconstructs the sequence from logs, alerts and chat history, with no interpretation at all. First anomalous metric. First alert, if any. First human action. Each hypothesis and when it was dropped. The fix. Confirmation of recovery.
Circulate it before the meeting, and start the meeting by correcting it rather than discussing it. This ordering is not cosmetic. If you open with explanation, the first senior person to speak anchors everyone else and the rest of the hour is spent elaborating their theory. If you open with a timeline, the gaps announce themselves. The 41 minute silence in our incident was obvious on a timeline and invisible in conversation, because in conversation the story began at the moment someone noticed.
Include the dead ends. A timeline that only contains correct actions is a document someone has already edited for dignity, and it hides the most expensive part of most incidents, which is time spent confidently investigating the wrong thing.
Step 3: Separate the Trigger From the Conditions That Allowed It
Name the proximate trigger once, in one sentence, then stop. “A cache configuration change removed a TTL.” Done. If the review ends there you have written our useless two liner.
The useful work is in the conditions, and there are almost always three or four:
- Detection: why did nothing alert for 41 minutes. For us, no alert existed on that path, because the path had never failed before.
- Knowledge: why did one person understand this component. That is a staffing and pairing answer, not a reliability one.
- Reversibility: why was rolling back harder than fixing forward. Usually because rollback was never tested.
- Ease of the wrong action: what made the mistake easy to make. A config field with no validation and no review requirement makes a bad change effortless.
Every one of those is fixable by changing a system. None is fixable by asking someone to be more careful, which is the action item teams write when they have stopped at the trigger.
Step 4: Ask the 1 Question That Removes Blame
For every human action in the timeline, ask: why was this reasonable at the time, with the information available then.
That phrasing does the work. It is not generosity, it is accuracy, because people genuinely do act reasonably on the information in front of them, and the information in front of them is something you control. If the answer is “because the dashboard showed green and the dashboard was measuring the wrong thing”, you have found a system problem and a fix. If you instead ask why someone did not notice, you get an apology, and an apology is not an input to anything.
Three rules keep this honest. The incident responder does not run their own review. No names in the written document, roles instead, since “the on-call engineer” reads differently from a name in a document that will be searchable for years. And if a genuine performance issue surfaces, it leaves the room and becomes a private conversation with a manager. Blameless describes the investigation, not the absence of accountability, and teams that blur those two things eventually abandon the practice after one bad quarter.
Our expert take
The fastest way to tell whether a team is genuinely blameless is to read a postmortem from six months ago and count the passive constructions. “The configuration was changed” means someone is still protecting themselves. A healthy document says what happened plainly and attaches the fix to a system, because nobody in that team is afraid of the sentence. You cannot mandate that with a template. It comes from how the first three reviews were run.
If incidents keep landing on one person
That is usually a hiring gap rather than a process gap. Tell us what your team carries and we will help you scope the role that fixes it, then shortlist pre-vetted engineers against that scope.
Get startedStep 5: Convert Findings Into Owned Actions, Capped at 5
Three constraints, and together they do more than any tracking tool.
- At most five actions. A list of twenty is a decision to do none of them, and everyone in the room knows it as it is being written.
- One named owner each, never a team. An action owned by a team is owned by nobody by Friday.
- Into the normal backlog, with normal prioritisation. A separate reliability list is where good intentions go to expire, because it is never the list anyone is judged on.
Classify each action as prevention, detection or response, and check you have at least one detection item. Teams reliably over-invest in prevention, which only addresses the failure that already happened, and under-invest in detection, which addresses the next one too. Our five from that incident were: an alert on the affected path, validation on the config field, a tested rollback, a second engineer paired into that component, and a line in the on-call runbook. Only the second was prevention.
| Action type | What it protects against | Typical mistake |
|---|---|---|
| Prevention | This exact failure recurring | Writing four of these and nothing else |
| Detection | Every failure of this shape, including new ones | Omitted entirely, then omitted again |
| Response | The duration of the next incident | Left as a runbook nobody has read |
| Knowledge | The single-owner risk behind most long outages | Treated as training, not as staffing |
Step 6: Publish It Where People Already Read
A postmortem in a reliability folder is a filing exercise. Put it where the team actually works, post the summary in the channel the incident was fought in, and write the first paragraph so a new joiner with no context can follow it.
That last constraint has a hiring benefit people underestimate. A readable archive of incidents is one of the fastest onboarding assets a team can own, because it teaches a new engineer how your system fails, which is the knowledge that otherwise takes nine months to acquire. We point new joiners at three postmortems in their first week, and it has done more than any architecture document. If you are also rewriting how people join the team, it pairs directly with onboarding remote developers.
Step 7: Review Quarterly, and Count the 2 Numbers
Once a quarter, pull every postmortem from the period and count two things.
- The share of actions that actually shipped. Below half, your process is theatre, however good the documents read. This is the number that embarrasses people into honesty.
- Incidents that repeated a known cause. The only real measure of whether any of this worked.
Then look for patterns across incidents rather than within them. Three outages all involving an untested rollback is a single finding that no individual review could have produced, and it is worth more than the three reviews separately. Our quarterly check is also where we notice when incidents are concentrating on one person, which is a staffing signal rather than a reliability one and belongs next to the on-call rotation design rather than in a retrospective.
What This Changes About Who You Hire
Once you run reviews this way, you start noticing that incident capability is a hiring attribute rather than a training outcome, and it is cheap to screen for. Ask for one specific incident the candidate was part of, then listen for three things.
Whether they give you a timeline rather than a conclusion. People who have done this properly remember the sequence, including the twenty minutes on the wrong hypothesis, and they volunteer that part. Whether they describe conditions as well as a cause: “the deploy broke it” is a trigger, while “the deploy broke it, there was no alert on that path and rollback had never been tested” is a system. And whether they assign fault to a named colleague, which tells you exactly what they will bring to your review culture and costs nothing to detect.
That is a ten minute conversation, no take-home required, and it predicts on-call behaviour far better than asking someone to design a monitoring system on a whiteboard. If you are building the wider loop around this, it sits alongside building a dedicated QA engineering team and the exit side of the same coin in running developer exit interviews.
The Mistake I Made Introducing This
I rolled it out with a template and a mandate, and the first three reviews were worse than the two line document they replaced. People filled in the sections defensively, wrote “the configuration was changed” in the passive voice, and produced eleven action items nobody owned.
What fixed it was not a better template. I ran the next review on an incident that was unambiguously my own fault, said so plainly in the room, and let the conditions discussion land on decisions I had made. The fourth review was honest, and every one since has been. If you are introducing this on a team that has been blamed before, that first review is the whole intervention, and it should be about an incident you own rather than one of theirs.
Frequently Asked Questions
Does blameless mean nobody is accountable for an outage?
Which incidents deserve a written postmortem?
How do I stop postmortem action items from being quietly abandoned?
What should I look for when hiring an engineer who will carry incidents?
Start with step 2 on your last incident
Rebuild one timeline this week and the gaps will choose your first actions for you. If the gap turns out to be a person-shaped hole in your team, we can help you fill it.
Get started