We Reported 99.95% Uptime and Still Lost the Account: the 7-Step SLO and Error Budget Policy That Fixed What Our Number Hid

A dashboard of time series charts on a monitor, illustrating the difference between a server uptime number and a service level objective measured on the user journey
Bryan

Bryan

Delivery & Offshore Teams Expert · November 12, 2026 · 12 min read

TL;DR

  • •The problem: we reported 99.95% every month and lost an account anyway. The number measured whether our servers answered. The customer was measuring whether a payment finished, and nobody had ever put a target on that.
  • •The method: 7 steps, and the only one that actually matters is step 5. A target without a stated consequence is a dashboard. Steps 1 to 4 just make step 5 defensible.
  • •The honest part: our first target was too strict, we suspended the policy in week six, and it took two quarters to get the credibility back. Step 6 exists because of that. The 4 things this will not fix are near the end.

The call was short. They had decided not to renew, and when we asked about reliability, the account manager read us a list of dates. We went and checked every one of them against our monitoring. The servers had been up on all of them. Our monthly report said 99.95 per cent, which was true, and it was true about something nobody on the customer side had ever cared about. We had spent two years getting very good at answering a question we had chosen ourselves.

What an Uptime Number Hides

An availability figure taken from infrastructure answers the question “did the service respond”. A customer is asking “could I finish what I came to do”. Those two questions give different answers more often than most teams realise, and the gap between them is where accounts get lost quietly.

Here is the shape of it. A payment flow touches your API, a queue, a ledger write and a third party. Each component is up 99.9 per cent of the time, independently. The journey through all four succeeds about 99.6 per cent of the time, and every individual dashboard looks green. Nobody is lying. Nobody is even careless. The measurement is simply taken at the wrong altitude, and the customer experiences the product at the altitude you are not measuring.

A service level objective fixes the altitude. An error budget policy is what makes the objective change behaviour rather than decorate a slide. In my experience teams get the first one roughly right within a month and never write the second one at all, which is why the whole method below is arranged to make step five possible.

Our expert take

In a distributed team the altitude problem is worse, and it is structural rather than cultural. If part of your team owns infrastructure and part owns product, each group measures what it controls, and both sets of numbers can be excellent while the journey across the boundary is poor. Nobody is accountable for the seam, because no dashboard describes it. The single most valuable thing an objective measured on the user journey does for a distributed team is to give the seam an owner.

Step 1: Pick the One Journey That Defines the Product

One journey. Not a service catalogue, not a tier list, not eleven objectives across four teams. Choose the thing which, if it fails, means the product failed regardless of what else worked.

For a payments product it is usually a transaction completing end to end. For a marketplace it is search returning results a buyer can act on. For an internal platform it is a deployment reaching production. Write it as a sentence a non-engineer would recognise, then trace every component it passes through and confirm you can actually observe the whole path. If you cannot, that instrumentation gap is your first piece of work and it is more valuable than the target.

Resist expanding to a second journey for at least a quarter. Teams that start with eight objectives end with zero, because nothing with eight owners has one.

Step 2: Write the Indicator as Good Events Over Valid Events

The indicator is a ratio, and both halves need a written definition. Good events divided by valid events, over a stated window.

The argument about what counts as good is the single most useful hour you will spend on this, and it has to happen now rather than during an incident. Is a payment that succeeded on the second attempt a good event? Is a request that returned in eleven seconds good, when your intent was two? Is a request from a customer who was rate limited for abuse a valid event at all? There is no universally right answer. There is only a written answer, which is what lets you compare this month to last month honestly.

Put the definition somewhere a new engineer will find it without asking, next to the query that computes it. An indicator whose definition lives in one person’s head drifts every time that person leaves.

Step 3: Set the Target From What the Business Will Defend

Here is where our first attempt failed. We picked 99.95 per cent because it sounded serious, discovered our actual journey delivered 99.6, burned the entire budget in the first fortnight, and suspended the policy in week six. A policy suspended in week six has taught your team that the policy is theatre, and that lesson takes two quarters to unlearn.

The disciplined sequence is the reverse. Measure the journey for a month first with no target at all. Take that number to whoever owns the customer relationship and ask one question: if we told a customer this is our commitment, would you defend it? The number they will stand behind, rather than the number engineering can brag about, is your target.

If you are a financial institution, note that this sits underneath a regulatory ceiling rather than replacing it. The Monetary Authority of Singapore sets a maximum unscheduled downtime of four hours in any twelve month period for each critical system, within its Technology Risk Management guidelines. That is a compliance limit for a defined set of systems, not a budget to spend, and your internal objective should be stricter and should cover journeys the regulator never named.

Step 4: Convert the Target Into Minutes and Publish It

Percentages are abstract. Minutes are a quantity people spend deliberately. Convert once and publish the conversion where the team already looks, which in practice means wherever your alerts land rather than a wiki nobody opens.

TargetBudget per 30 day monthWhat that feels like
99.5%About 3 hours 36 minutesOne bad afternoon, or a slow month of small failures
99.9%About 43 minutesOne serious incident consumes most of it
99.95%About 21 minutesA single deploy rollback can take half
99.99%About 4 minutesRequires architecture, not discipline

That last row is the one worth staring at. Four minutes a month is not a target you reach by being careful; it is a target you reach by removing single points of failure, which is a budget conversation rather than an engineering one. Teams that commit to four nines without funding the architecture produce an objective that is permanently breached and therefore permanently ignored.

What each extra nine actually costs youAllowed bad minutes in a 30 day month216 min99.5%43 min99.9%21 min99.95%4 min99.99%Figures are a 30 day month and are rounded. The jump from 99.9 to 99.99 is a factor of ten in allowance and usually a change in architecture, not in diligence.

Need the engineer who can instrument the journey

Tell us which journey defines your product and we will shortlist Singapore-based and remote engineers who have measured one end to end before, not just configured alerts.

Get started

Step 5: Write What Changes When the Budget Is Spent

This is the policy. Everything before it is measurement. If you only do one step from this article, do this one, and if you cannot get agreement on it, stop rather than shipping a dashboard and calling it a programme.

The consequence has to be specific, automatic and unwelcome enough to matter. The version that has worked for us: when the budget for the month is exhausted, feature releases pause and the next sprint is reliability work until the trailing window recovers. Not a conversation about whether to pause. A default that applies unless somebody overrides it under step six.

Two refinements worth adopting from the start. Use a trailing window rather than a calendar month, otherwise the first of the month becomes an amnesty and incidents cluster oddly near month end. And define a warning threshold, usually around seventy five per cent consumed, that triggers a conversation rather than a freeze, because a policy with only one state arrives as a surprise every time. The reference implementation most teams borrow from is the error budget policy chapter in the Google SRE workbook, which is worth reading for the structure even if your numbers look nothing like theirs.

The consequence also has to connect to how incidents get examined, or you will pause features and learn nothing. The handoff is from the freeze into a review process that is actually honest about causes, which we describe in our method for running blameless incident postmortems.

Step 6: Agree the Override in Advance, in Writing

Every error budget policy gets overridden. A regulatory deadline, a contractual launch, a customer commitment made before the policy existed. Pretending otherwise is how a policy dies: the first override happens informally, the team notices that the rule is negotiable, and it stops functioning as a default.

So decide the terms now. Name the single role that can suspend the freeze. Put a duration on the suspension, two weeks rather than indefinitely. And state what is owed afterwards, which should be a written note of what was traded and a reliability sprint on the other side. We learned this the expensive way: our first suspension had no end date and no cost attached, and the policy simply never came back.

Our expert take

The signature on the policy matters more than the wording of it. If the only name on the document is the engineering lead, the first conflict with a launch date resolves in favour of whoever outranks them, which is nearly everybody. In the teams where we have seen this stick, the product owner or the general manager co-signs, so the trade-off was agreed in principle in a calm week rather than argued in a loud one. If you cannot get that signature, you have learned something real about where reliability sits on the priority list, and it is far cheaper to learn it while drafting than during an incident.

Step 7: Review the Target Against Complaints, Not the Dashboard

Once a quarter, put two things side by side: budget consumed, and what customers and support actually raised. Four outcomes, and three of them are instructions.

  1. Budget burned, customers complained. The target and the journey are right. Keep both and do the reliability work.
  2. Budget burned, nobody complained. The target is stricter than the business needs. Loosen it. A target that cries wolf costs you the next real alarm.
  3. Budget intact, customers complained. The expensive one, and the one we were in. You are measuring the wrong journey. Go back to step one rather than tightening the number.
  4. Budget intact, nobody complained. Either it is working or the target is too loose to be informative. Check whether you have ever come close to the warning threshold. If not, tighten it until the policy occasionally has something to say.
The quarterly review, as four outcomesThree of the four quadrants are instructions to change somethingCustomerscomplainedCustomersquietBudget intactBudget burnedWrong journeyReturn to step 1. Do nottighten the number.Working as designedKeep the target. Spend thesprint on reliability.Possibly uninformativeIf you never near the warningline, tighten it a little.Target too strictLoosen it. A policy that crieswolf costs the real alarm.Compare the same quarter in both columns, and use raised tickets rather than sentiment, so the comparison is reproducible next quarter.
StepTime costThe failure it prevents
1. One journey, end to endOne afternoon plus instrumentationGreen dashboards and an unhappy customer
2. Indicator defined in writingOne hour of argumentMonths that cannot be compared honestly
3. Target the business will defendOne month of measuring firstA policy suspended in week six
4. Budget published in minutes30 minutesA percentage nobody reasons about
5. Stated consequenceOne meeting, one pageA metric mistaken for a policy
6. Override agreed in advance30 minutesAn informal override that ends the policy
7. Quarterly review against complaintsTwo hours per quarterOptimising a number nobody feels

The 4 Things This Will Not Fix

  1. An under-resourced on-call rotation. A budget tells you reliability is being consumed. It does not tell you that the same two engineers absorb every incident, which is a staffing problem with a predictable ending. The structure we use for that is in our method for building an on-call rotation in a Singapore engineering team.
  2. Architecture that cannot reach the target. If your journey depends on one database with no failover, no policy will produce four nines. The honest move is to set a target the architecture can hold and put the gap in a funding conversation rather than in a sprint.
  3. A team without the skills to do the reliability sprint. Freezing features is only useful if the unfrozen time produces something. If nobody on the team has done capacity work or failure injection before, the freeze becomes a quiet fortnight. That is a hiring question, covered in our guide to hiring a site reliability engineer in Singapore.
  4. A roadmap nobody will protect. If every quarter contains a commitment that would justify an override, you do not have a reliability problem, you have a planning problem, and step six will be invoked until it means nothing.

If you are standing this up at the same time as a wider build, the sequencing question is covered in our guide to custom software development in Singapore.

Frequently Asked Questions

Do we need an SLO if we already report uptime to customers?
You need one more if you report uptime, not less, because the two answer different questions. A contractual uptime figure is a liability threshold: cross it and you owe a credit. An objective is an engineering decision rule: approach it and you change what the team works on next week. The failure we see constantly is a team that hits its contractual number every month, pays no credits, and loses accounts anyway, because the thing measured is whether the infrastructure answered rather than whether a customer could finish the task they came to do. Build the journey number first and keep the contractual figure as a reporting obligation rather than a management tool.
What target should we start with?
Start with what your current system actually delivers on the journey from step one, then ask whether that is acceptable. This is the opposite of what most teams do, and it matters because a target invented in a planning meeting is usually either so loose it never triggers anything or so tight that the policy is suspended in month two and never recovers. Measure for a month first. If the journey succeeds 99.7 per cent of the time and nobody has complained, 99.5 per cent is a defensible first target with room to learn. If customers are already unhappy at 99.7, the number is not your problem and a stricter target will not fix it.
How does this interact with MAS requirements for financial institutions?
They operate at different levels and you need both. The Monetary Authority of Singapore sets a regulatory ceiling for financial institutions, including a maximum unscheduled downtime of four hours within any twelve month period for each critical system. That is a compliance floor you do not negotiate, and it is an annual figure for a defined set of systems. An error budget policy is an internal decision rule that should be stricter and should cover journeys the regulator has not named. Running the two as one number is a mistake in both directions: engineering starts treating a compliance limit as a budget to spend, and customer journeys outside the critical system definition end up with no target at all.
Who should own the error budget policy?
The engineering lead writes it and somebody outside engineering signs it, and the second part is not a formality. A policy whose only signature is engineering cannot survive its first conflict with a launch date, because the person who wants the launch outranks the person defending the budget. Where this has worked, the product owner or the general manager co-signs, so the trade-off was agreed in principle before the week it became real. If you cannot get a signature from outside engineering, you have learned something important about whether reliability is actually a priority, and it is better to learn it while writing a document than during an incident.

Start with one journey, not a programme

We help Singapore employers put a reliability policy in place that survives its first launch date, and shortlist the engineers who can do the work the freeze creates.

Get started