3 Outages, 1 On-Call Rotation Nobody Wanted — the 7-Step Method That Finally Landed Our First Singapore SRE

We took three outages in one quarter. None of them were catastrophic; all three were embarrassing, and all three were detected by a customer before they were detected by us. The engineering team was strong. The on-call rotation was four people who had never asked to be on it, working from a runbook last updated eleven months earlier.

The obvious fix was “hire an SRE”. We did that, badly, twice — two searches that produced no offer and a lot of wasted candidate goodwill. The third search worked. The difference was not the sourcing; it was that we finally did the preparatory work the first two searches had skipped.

Step 1 — Define what reliability means numerically, before writing the role

This is the step we skipped twice, and it is the reason both searches failed.

Site reliability engineering is not a seniority level or a tooling preference. It is a discipline organised around measurable objectives: a stated target for a user-facing journey, an agreed measurement window, and an error budget that creates a real trade-off between shipping and stability. Remove those and the role has no spine. The person you hired for reliability expertise ends up processing access requests and fixing pipelines, because in the absence of objectives, whatever is loudest wins.

Before you write the job description, write one sentence: “Checkout completes successfully for 99.9% of attempts, measured over a rolling 28 days.” It does not need to be the right number. It needs to exist, and someone outside engineering needs to have agreed to it.

Strong SRE candidates ask for this in the first call. When our answer was “we want to improve reliability generally”, the good candidates disengaged politely and the ones who stayed were the ones who did not know to ask. That is the entire story of our first two failed searches.

Step 2 — Decide whether you are hiring your first SRE or your fifth

These are different jobs and the market treats them as one, which is why so many placements fail at around month six.

Your first SRE is a change agent. Most of their work is social: convincing product managers that an error budget is real, persuading engineers to write postmortems without blame, negotiating what happens when the budget is exhausted. Deep technical skill matters, but it is not the constraint. The constraint is credibility and patience.

Your fifth SRE joins a functioning practice. They can be quieter, deeper, more specialised — capacity modelling, load shedding, incident tooling. They will be evaluated on technical contribution rather than organisational change.

Hiring a brilliant introvert as your first SRE is a well-documented way to waste twelve months and lose a good engineer. They will produce excellent analysis that nobody acts on, conclude that the organisation is not serious, and leave. We nearly did exactly this in our second search.

Step 3 — Source from operators, not from monitoring-tool keywords

Searching on observability tool names produces a pipeline of people who have configured dashboards. That is a different population from people who have been woken at 3am by a system they own and had to make a judgement call with incomplete information.

What has worked for us in Singapore, roughly in order of yield: engineers at regional platforms with genuine traffic — payments, logistics, ride-hailing, streaming — who have carried a real pager; backend engineers who drifted into reliability work because they were the person who kept fixing things, and who are often not searching under an SRE title at all; and people who have written publicly about an incident, which is a strong signal of the exact temperament you want.

Expect most of your pipeline to arrive through referral and direct approach. Good reliability work is quiet and satisfying, and the people doing it are rarely on the market. Plan a longer calendar rather than a wider net.

Step 4 — Screen with a postmortem review, not system-design trivia

This is the exercise that changed our hit rate, and it takes about forty minutes.

Write a two-page incident report for a realistic outage. Include a timeline, a stated root cause, and a set of action items. Then deliberately break it: make the stated root cause plausible but not actually consistent with the timeline, make one action item address a symptom rather than the cause, and leave detection latency conspicuously unmentioned. Hand it to the candidate and ask them to review it as if a colleague had written it.

What you are watching for is whether they attack the narrative or accept it. Strong candidates start pulling at the seam within a couple of minutes: the timeline says the errors started at 14:02 but the deploy was at 14:19 — what happened at 14:02? They ask how it was detected and how long that took. They notice that an action item adds an alert without addressing why the failure occurred.

Weaker candidates read the report as authoritative, agree with the conclusion, and suggest more monitoring. This is not a lack of intelligence; it is a lack of the specific scepticism the job requires. Across our third search, this exercise separated the field more cleanly than every other stage combined — and it has the pleasant property of being a realistic sample of the actual work.

Step 5 — Be specific about on-call before the candidate asks

Every experienced SRE asks about on-call, usually within the first two calls, and they are listening for evasion more than for the answer itself.

Have five numbers ready. How many people are in the rotation. How frequently each person is primary. How many pages fired in the last quarter, and how many were actionable. What the escalation path is at 3am. What compensation or time off attaches to it.

If your honest answer is uncomfortable — four people, weekly rotation, noisy alerts — say it plainly and follow it with what the hire is expected to change. Experienced candidates have seen bad rotations and are not scared of them; they are scared of employers who describe them vaguely, because vagueness reliably predicts that nothing will improve.

Our third search improved noticeably the moment we put the page statistics in the job description itself. It filtered out people looking for a quiet life, which was correct, and it signalled to everyone else that we were measuring the thing they would be asked to fix.

Step 6 — Account for regulated-sector constraints in the loop

A large share of Singapore’s engineering demand sits in or adjacent to financial services, and that changes reliability work in ways worth interviewing for explicitly.

Regulated environments impose real constraints on what an SRE can do: change windows, formal approval for production access, evidence requirements for incident handling, and reporting obligations with defined deadlines. An engineer whose entire experience is in an unregulated consumer product will design remediation processes that are technically excellent and procedurally unusable, and will find that out in month two.

Ask directly: what changes about your incident response when every production access has to be logged and justified, and when the regulator expects notification within a defined window? You are not looking for regulatory expertise — you can teach that. You are looking for whether they treat the constraint as a design input or as an obstacle to route around. The second answer is a genuine risk in this market.

This intersects with the evidence and retention obligations that surround AI-driven systems too, which we covered in our analysis of Singapore’s profession-wide AI fluency push. The same underlying skill — producing defensible evidence about system behaviour — is becoming a shared requirement across reliability and AI operations.

Step 7 — Close on error budget authority, not on title

Compensation opens the conversation. As of 2026, mid-level SREs in Singapore typically land at SGD 9,000–12,500 per month, and senior engineers with production reliability experience in regulated or high-traffic systems at SGD 14,000–19,000 and above. On-call arrangements vary widely and should be stated as a number, not a philosophy.

What actually closes a strong SRE is a different thing entirely: authority. Specifically, an answer to the question “what happens when the error budget is exhausted and product wants to ship anyway?”

If the answer is “we ship anyway”, say so. You will lose some candidates, and you will lose them before you have wasted six weeks. If the answer is “the SRE can call a freeze and it holds”, get that in writing from whoever would have to honour it, because the candidate will ask and a hedged answer is worse than a negative one.

Our successful hire told us afterwards that this was the deciding factor between us and a better-paying offer. Not the money, not the technology — the fact that we had already written down what happens when the budget runs out, and that the head of product had signed it.

A note on the regional pool

Reliability talent is tight across Asia-Pacific and the pressure is not evenly distributed. Teams recruiting in Dubai face a market where infrastructure demand is being pulled up by large government technology programmes, while employers in Tokyo compete with national-scale infrastructure investment for the same operator profiles. If your search stalls locally, a regional approach is realistic — but budget the pass and relocation calendar honestly rather than optimistically.

Frequently asked questions