Our 4-Person On-Call Rota Burned Out in 5 Weeks — the 7-Step Singapore Rotation I Build Now

The rotation that taught us this lasted five weeks. Four backend engineers, one week each, a paging tool that took an afternoon to configure, and a launch that had gone well enough to justify overnight coverage. By week five one engineer had asked to be taken off it, a second had quietly stopped acknowledging alerts before morning, and the most senior person on the team was having a conversation with a recruiter. Nothing about the schedule was unusual. The mistake was that we had built a rota without first deciding what was worth waking someone up for, and an undefined paging criterion inherits every alert anyone has ever written. Below is the order we build these in now.

Step 1 — Define What Actually Deserves to Wake Someone Up

Write the paging criterion before the schedule, because the schedule is downstream of it. Ours is deliberately narrow: a page means a customer-visible problem that requires action now, and that one engineer can act on alone. Three conditions, all of which must hold.

Test each existing alert against it. “Disk usage at 80 percent” fails the first condition and usually the second. “Error rate above baseline for five minutes” may pass all three. “Nightly batch job failed” almost always fails the second, because nothing is customer-visible until the morning, and it is the single most common alert we find sitting in an overnight rotation for no reason other than that someone created it once.

Everything that fails the test is not deleted. It becomes a ticket, a dashboard or a morning digest. The distinction you are drawing is not between important and unimportant, it is between now and not now, and conflating those two is what makes rotations unbearable.

Step 2 — Size the Rota Against the Headcount You Really Have

This is arithmetic, so do it on paper before you commit to anything in a tool.

Note the last row, because it is the failure nobody predicts: past roughly one week in eight, engineers are on call rarely enough that they lose familiarity with the runbooks and the tooling, and shifts start with fifteen minutes of relearning. The fix is not a smaller rota, it is periodic drills.

If the arithmetic tells you that you do not have enough engineers, you have three honest options: hire, reduce the coverage you promise, or pay properly for a heavier rotation. Choosing none of them and hoping is the option that ends in a resignation. If hiring is the answer, the role we would hire first is discussed in our guide to hiring a site reliability engineer in Singapore.

Step 3 — Settle Compensation and the Legal Position in Writing

This is the step Singapore teams most often skip, usually because someone assumes the statutory overtime framework covers it. For most software engineers it does not.

Part IV of the Employment Act — the part that governs hours of work, rest days and overtime at one and a half times the hourly basic rate — applies to non-workmen earning a basic monthly salary of S$2,600 or less, and expressly excludes managers and executives. Typical engineering salaries in Singapore sit above that threshold, which means on-call compensation is a contractual matter rather than a statutory entitlement.

The practical consequence is not that you owe nothing. It is that nothing arrives automatically, so anything you intend to provide has to be written down to exist. We document three components:

1. A standby allowance for carrying the pager, paid whether or not anything happens, because the constraint on your evening is the cost.
2. A callout provision for actually being woken and working, separate from the allowance.
3. Time off in lieu, with a deadline. Without a deadline it accrues and is never taken, which converts a benefit into a grievance.

Put it in the contract or in a policy document referenced by the contract, and confirm your own position against your employees’ actual classifications with a Singapore-qualified adviser. The thresholds are exactly the kind of detail people get wrong from memory.

Step 4 — Design the Escalation Ladder With Acknowledgement Timeouts

A rota with one name in it is a single point of failure wearing a process costume. People sleep through phones, lose signal in an MRT tunnel and occasionally have a genuine emergency of their own.

The ladder needs three rungs and, more importantly, automatic timeouts between them:

• Primary — the engineer on shift. Acknowledgement expected within a short window, typically five minutes overnight.
• Secondary — a named second engineer, paged automatically if the primary has not acknowledged within that window. Not a volunteer, not “whoever is around”.
• Management escalation — an engineering manager or head of engineering, paged if the secondary has not acknowledged within a further window. Their job is not to debug, it is to make decisions that cost money: wake more people, communicate with customers, accept a degraded mode.

The timeout is the entire mechanism. Without it the ladder relies on the sleeping person noticing they are asleep, which is not a control. Our companion piece on the UAE side goes deeper on the matrix itself — see building an on-call escalation matrix for a dedicated team in Dubai, which is the closest thing to a template we publish.

Step 5 — Write the Runbook Before the First Shift, Not After the First Incident

Every alert capable of paging a human needs a runbook entry with four things: the first diagnostic action, the known causes, the safe mitigation, and who to escalate to if the mitigation does not work.

Make the runbook link a required field on the alert definition. This is a small piece of process with a disproportionate effect, because it inverts the default: instead of alerts accumulating freely and documentation lagging behind them, an alert cannot exist until someone has thought through what a human should do at 3 a.m.

Keep entries short. A runbook that reads like a design document will not be opened during an incident. Four bullet points that a tired person can follow beats two pages of accurate context, and if the first diagnostic action does not fit on one line, the alert is probably firing on a symptom too far from the cause.

Then say the quiet part explicitly to the team: an alert with no runbook trains people to ignore alerts. Once an engineer has been paged three times by something they could not act on, the pager has lost its meaning, and you cannot restore that by asking people to try harder.

Step 6 — Enforce an Alert Quality Gate With a Noise Budget

Track pages per shift as the primary operational metric, split between office hours and overnight, and give it an explicit budget. Ours is no more than two pages per overnight shift. The number matters less than the fact that a number exists, because a budget converts a vague complaint into a threshold somebody owns.

When an alert breaches the budget, apply the rule that makes the whole system work: the alert is the defect, not the engineer’s tolerance. Three remedies, in order of preference — fix the underlying cause, raise the threshold so it fires on real conditions, or delete it. “Leave it and tell people to expect it” is not on the list.

Give the on-call engineer standing authority to silence a noisy alert during their own shift, with a required follow-up ticket. Teams resist this because it feels like letting people turn off the smoke detector. In practice it is the opposite: engineers who cannot silence noise stop reading alerts entirely, which is a silent failure rather than a logged one. The retention dimension of this is not incidental — we covered the broader pattern in our note on developer retention in Singapore, and on-call quality is one of the few operational variables that shows up directly in resignations.

Step 7 — Run the Handover and Review Pages per Shift Monthly

End every shift with a written handover: what paged, what was done, what remains open, and anything the next person should watch. Written, not verbal, and certainly not verbal across a time-zone boundary — a verbal handover between regions is a handover that did not happen.

Once a month, review three numbers with the team:

1. Pages per shift, overnight and office hours separately, trended over the previous months.
2. Time to acknowledge, which tells you whether the ladder is working or whether the secondary is quietly carrying the primary.
3. Proportion of pages that were actionable — the engineer did something other than acknowledge and go back to sleep. Below roughly three quarters, you have an alerting problem that is actively teaching the team to distrust the pager.

Feed the output back into steps 5 and 6. This is the step that gets dropped once things feel stable, and dropping it is how a healthy rotation degrades quietly for two quarters until somebody resigns and everyone is surprised.

If you are coordinating across regions, the pairing worth considering is Singapore Time at UTC+8 with a Gulf team four hours behind, which gives a genuine overlap window rather than a gap. The prerequisite is runbook parity: without it the second region escalates everything back to the first, which is worse than no handover because it wakes the same people with extra delay. Our colleagues’ guide to building a remote DevOps team in Abu Dhabi covers the staffing side of that second region.

Three Mistakes That Cost the Most

Building the schedule before the paging criterion. The rota is downstream of what deserves a page. Get the order wrong and you have industrialised your existing alert noise into a shift pattern.

Assuming statutory overtime covers on-call. For most Singapore engineers it does not, because Part IV is limited by salary threshold and excludes managers and executives. Unwritten compensation is non-existent compensation, and this surfaces at the worst possible moment: when someone is already unhappy.

Treating the first resignation as unrelated. On-call load is one of the few operational variables that shows up directly in attrition, and it does so with a lag of about two months. If a rotation started and someone senior left a quarter later, those two facts are usually the same fact.

FAQ — On-Call Rotations in Singapore Engineering Teams

Regulatory references: Employment Act (Singapore), Part IV — hours of work, rest days and overtime — which applies to non-workmen earning a basic monthly salary of S$2,600 or less and excludes managers and executives; overtime payable at 1.5 times the hourly basic rate where Part IV applies. Rota frequencies and noise budgets in this article are our operating practice, not standards. This is general information about engineering management practice, not legal advice — confirm employee classifications and entitlements with a Singapore-qualified adviser.