The call was short. They had decided not to renew, and when we asked about reliability, the account manager read us a list of dates. We went and checked every one of them against our monitoring. The servers had been up on all of them. Our monthly report said 99.95 per cent, which was true, and it was true about something nobody on the customer side had ever cared about. We had spent two years getting very good at answering a question we had chosen ourselves.
What an Uptime Number Hides
An availability figure taken from infrastructure answers the question “did the service respond”. A customer is asking “could I finish what I came to do”. Those two questions give different answers more often than most teams realise, and the gap between them is where accounts get lost quietly.
Here is the shape of it. A payment flow touches your API, a queue, a ledger write and a third party. Each component is up 99.9 per cent of the time, independently. The journey through all four succeeds about 99.6 per cent of the time, and every individual dashboard looks green. Nobody is lying. Nobody is even careless. The measurement is simply taken at the wrong altitude, and the customer experiences the product at the altitude you are not measuring.
A service level objective fixes the altitude. An error budget policy is what makes the objective change behaviour rather than decorate a slide. In my experience teams get the first one roughly right within a month and never write the second one at all, which is why the whole method below is arranged to make step five possible.
Our expert take
In a distributed team the altitude problem is worse, and it is structural rather than cultural. If part of your team owns infrastructure and part owns product, each group measures what it controls, and both sets of numbers can be excellent while the journey across the boundary is poor. Nobody is accountable for the seam, because no dashboard describes it. The single most valuable thing an objective measured on the user journey does for a distributed team is to give the seam an owner.
Step 1: Pick the One Journey That Defines the Product
One journey. Not a service catalogue, not a tier list, not eleven objectives across four teams. Choose the thing which, if it fails, means the product failed regardless of what else worked.
For a payments product it is usually a transaction completing end to end. For a marketplace it is search returning results a buyer can act on. For an internal platform it is a deployment reaching production. Write it as a sentence a non-engineer would recognise, then trace every component it passes through and confirm you can actually observe the whole path. If you cannot, that instrumentation gap is your first piece of work and it is more valuable than the target.
Resist expanding to a second journey for at least a quarter. Teams that start with eight objectives end with zero, because nothing with eight owners has one.
Step 2: Write the Indicator as Good Events Over Valid Events
The indicator is a ratio, and both halves need a written definition. Good events divided by valid events, over a stated window.
The argument about what counts as good is the single most useful hour you will spend on this, and it has to happen now rather than during an incident. Is a payment that succeeded on the second attempt a good event? Is a request that returned in eleven seconds good, when your intent was two? Is a request from a customer who was rate limited for abuse a valid event at all? There is no universally right answer. There is only a written answer, which is what lets you compare this month to last month honestly.
Put the definition somewhere a new engineer will find it without asking, next to the query that computes it. An indicator whose definition lives in one person’s head drifts every time that person leaves.
Step 3: Set the Target From What the Business Will Defend
Here is where our first attempt failed. We picked 99.95 per cent because it sounded serious, discovered our actual journey delivered 99.6, burned the entire budget in the first fortnight, and suspended the policy in week six. A policy suspended in week six has taught your team that the policy is theatre, and that lesson takes two quarters to unlearn.
The disciplined sequence is the reverse. Measure the journey for a month first with no target at all. Take that number to whoever owns the customer relationship and ask one question: if we told a customer this is our commitment, would you defend it? The number they will stand behind, rather than the number engineering can brag about, is your target.
If you are a financial institution, note that this sits underneath a regulatory ceiling rather than replacing it. The Monetary Authority of Singapore sets a maximum unscheduled downtime of four hours in any twelve month period for each critical system, within its Technology Risk Management guidelines. That is a compliance limit for a defined set of systems, not a budget to spend, and your internal objective should be stricter and should cover journeys the regulator never named.
Step 4: Convert the Target Into Minutes and Publish It
Percentages are abstract. Minutes are a quantity people spend deliberately. Convert once and publish the conversion where the team already looks, which in practice means wherever your alerts land rather than a wiki nobody opens.
| Target | Budget per 30 day month | What that feels like |
|---|---|---|
| 99.5% | About 3 hours 36 minutes | One bad afternoon, or a slow month of small failures |
| 99.9% | About 43 minutes | One serious incident consumes most of it |
| 99.95% | About 21 minutes | A single deploy rollback can take half |
| 99.99% | About 4 minutes | Requires architecture, not discipline |
That last row is the one worth staring at. Four minutes a month is not a target you reach by being careful; it is a target you reach by removing single points of failure, which is a budget conversation rather than an engineering one. Teams that commit to four nines without funding the architecture produce an objective that is permanently breached and therefore permanently ignored.
Need the engineer who can instrument the journey
Tell us which journey defines your product and we will shortlist Singapore-based and remote engineers who have measured one end to end before, not just configured alerts.
Get startedStep 5: Write What Changes When the Budget Is Spent
This is the policy. Everything before it is measurement. If you only do one step from this article, do this one, and if you cannot get agreement on it, stop rather than shipping a dashboard and calling it a programme.
The consequence has to be specific, automatic and unwelcome enough to matter. The version that has worked for us: when the budget for the month is exhausted, feature releases pause and the next sprint is reliability work until the trailing window recovers. Not a conversation about whether to pause. A default that applies unless somebody overrides it under step six.
Two refinements worth adopting from the start. Use a trailing window rather than a calendar month, otherwise the first of the month becomes an amnesty and incidents cluster oddly near month end. And define a warning threshold, usually around seventy five per cent consumed, that triggers a conversation rather than a freeze, because a policy with only one state arrives as a surprise every time. The reference implementation most teams borrow from is the error budget policy chapter in the Google SRE workbook, which is worth reading for the structure even if your numbers look nothing like theirs.
The consequence also has to connect to how incidents get examined, or you will pause features and learn nothing. The handoff is from the freeze into a review process that is actually honest about causes, which we describe in our method for running blameless incident postmortems.
Step 6: Agree the Override in Advance, in Writing
Every error budget policy gets overridden. A regulatory deadline, a contractual launch, a customer commitment made before the policy existed. Pretending otherwise is how a policy dies: the first override happens informally, the team notices that the rule is negotiable, and it stops functioning as a default.
So decide the terms now. Name the single role that can suspend the freeze. Put a duration on the suspension, two weeks rather than indefinitely. And state what is owed afterwards, which should be a written note of what was traded and a reliability sprint on the other side. We learned this the expensive way: our first suspension had no end date and no cost attached, and the policy simply never came back.
Our expert take
The signature on the policy matters more than the wording of it. If the only name on the document is the engineering lead, the first conflict with a launch date resolves in favour of whoever outranks them, which is nearly everybody. In the teams where we have seen this stick, the product owner or the general manager co-signs, so the trade-off was agreed in principle in a calm week rather than argued in a loud one. If you cannot get that signature, you have learned something real about where reliability sits on the priority list, and it is far cheaper to learn it while drafting than during an incident.
Step 7: Review the Target Against Complaints, Not the Dashboard
Once a quarter, put two things side by side: budget consumed, and what customers and support actually raised. Four outcomes, and three of them are instructions.
- Budget burned, customers complained. The target and the journey are right. Keep both and do the reliability work.
- Budget burned, nobody complained. The target is stricter than the business needs. Loosen it. A target that cries wolf costs you the next real alarm.
- Budget intact, customers complained. The expensive one, and the one we were in. You are measuring the wrong journey. Go back to step one rather than tightening the number.
- Budget intact, nobody complained. Either it is working or the target is too loose to be informative. Check whether you have ever come close to the warning threshold. If not, tighten it until the policy occasionally has something to say.
| Step | Time cost | The failure it prevents |
|---|---|---|
| 1. One journey, end to end | One afternoon plus instrumentation | Green dashboards and an unhappy customer |
| 2. Indicator defined in writing | One hour of argument | Months that cannot be compared honestly |
| 3. Target the business will defend | One month of measuring first | A policy suspended in week six |
| 4. Budget published in minutes | 30 minutes | A percentage nobody reasons about |
| 5. Stated consequence | One meeting, one page | A metric mistaken for a policy |
| 6. Override agreed in advance | 30 minutes | An informal override that ends the policy |
| 7. Quarterly review against complaints | Two hours per quarter | Optimising a number nobody feels |
The 4 Things This Will Not Fix
- An under-resourced on-call rotation. A budget tells you reliability is being consumed. It does not tell you that the same two engineers absorb every incident, which is a staffing problem with a predictable ending. The structure we use for that is in our method for building an on-call rotation in a Singapore engineering team.
- Architecture that cannot reach the target. If your journey depends on one database with no failover, no policy will produce four nines. The honest move is to set a target the architecture can hold and put the gap in a funding conversation rather than in a sprint.
- A team without the skills to do the reliability sprint. Freezing features is only useful if the unfrozen time produces something. If nobody on the team has done capacity work or failure injection before, the freeze becomes a quiet fortnight. That is a hiring question, covered in our guide to hiring a site reliability engineer in Singapore.
- A roadmap nobody will protect. If every quarter contains a commitment that would justify an override, you do not have a reliability problem, you have a planning problem, and step six will be invoked until it means nothing.
If you are standing this up at the same time as a wider build, the sequencing question is covered in our guide to custom software development in Singapore.
Frequently Asked Questions
Do we need an SLO if we already report uptime to customers?
What target should we start with?
How does this interact with MAS requirements for financial institutions?
Who should own the error budget policy?
Start with one journey, not a programme
We help Singapore employers put a reliability policy in place that survives its first launch date, and shortlist the engineers who can do the work the freeze creates.
Get startedRelated reading: Blameless incident postmortems · Building an on-call rotation · Hiring a site reliability engineer
