What Is On-Call? A Practical Guide for Engineering Teams

On-call in one picture

Imagine a line running from a broken system to a person who can fix it. On-call is the design of that line.

1. Something in production degrades or breaks.
2. Monitoring notices and produces an alert.
3. Routing rules decide which team or service the alert belongs to.
4. The schedule decides which person holds that responsibility right now.
5. A notification channel — phone call, push, SMS — interrupts that person.
6. The person acknowledges, which tells the system a human has it.
7. If nobody acknowledges in time, escalation moves the alert to the next person.
8. The responder opens the runbook, diagnoses, mitigates, and communicates.
9. Afterwards the team reviews what happened and changes something.

Every step is a place the line can break: a monitor that never fires, a routing rule pointing at a team that was reorganised away, a schedule with a gap at 02:00 on the first Sunday of the month, a phone in do-not-disturb, an escalation policy whose second step is a person who left. These failures are quiet. None of them announce themselves until the night they matter.

What on-call is not

It is easy to inherit the wrong idea of on-call from a previous job or from folklore. A few clarifications:

• On-call is not a night shift. The expectation is reachability and response, not continuous work. If your on-call person is working most nights, you do not have on-call — you have understaffed shift work with the wrong name on it.
• On-call is not a punishment or a rite of passage. Framing it that way makes people hide problems rather than report them.
• On-call is not a substitute for reliability work. If the same alert fires weekly, the rotation is absorbing a defect the codebase should have fixed.

The seven parts of a working system

A rotation with six of these is not six-sevenths of a system. It fails completely in the situation the missing part covers. Ownership without routing means alerts land in a channel nobody watches. Routing without escalation means one sleeping person is a single point of failure. Runbooks without review means the same incident is solved from scratch every time.

Start with ownership. It is the hardest of the seven and the most valuable: a service with no owning team produces alerts everyone assumes someone else is handling.

Primary and secondary

Most rotations use two roles.

Primary is the first person notified. They acknowledge, assess severity, mitigate, and decide whether to bring anyone else in. During a shift, primary work takes precedence over planned work — make that trade-off explicit, or the person carries both loads and does neither well.

Secondary exists because primary is a human being who will sometimes be in a tunnel, in surgery, or asleep through a phone on silent. Secondary is the answer to "what if primary does not respond", and sometimes also "what if this is bigger than one person".

Decide three things and write them down:

A common mistake is creating a secondary role with no defined trigger. If secondary is never paged automatically, they are not a safety net; they are a person with disturbed sleep and no function.

What happens after an alert fires

The minutes after a page have a shape worth rehearsing before you need it.

1. Acknowledge. This stops escalation and tells everyone a human is engaged. Acknowledge first, diagnose second.
2. Assess. Is anything user-facing broken? Is data at risk? Is it getting worse? These three questions produce a severity without a long taxonomy.
3. Stabilise. Mitigate before you understand. Roll back, fail over, disable the feature, drain the node. Root cause can wait; the outage cannot.
4. Communicate. Say what is broken, what you are doing, and when you will update next. Say it even if the answer is "still investigating" — silence forces other people to interrupt you to ask.
5. Escalate if needed. Escalating is a signal that the problem is larger than one person, not an admission of inadequacy. Set the norm early, or people will wait too long.
6. Hand over or close. If the shift ends mid-incident, hand over explicitly, in writing, with current state and next steps.
7. Record. Leave enough behind that the next person meets a known problem, not a new one.

Expectations and boundaries

On-call works when its terms are explicit. Vague expectations are absorbed by whoever is most conscientious, and that person burns out first.

Write down, before anyone's first shift:

• Response time. How long after a page should someone acknowledge? This sets the practical radius a person can travel from a laptop and a signal.
• What qualifies as a page. A page asserts a human must act now. Everything else is a ticket or a dashboard. If a signal cannot survive that test, it should not wake anyone.
• Shift boundaries. Start time, end time, time zone, and what happens at a handoff.
• Compensation or time back. Whether it is pay, time off in lieu, or reduced planned work, decide it rather than leaving it as an unspoken favour.
• The right to be unavailable. Illness, travel, emergencies. There must be a documented way to hand the phone over that does not require heroics.
• What happens after a bad night. If someone was awake from 02:00 to 05:00, "come in late" should be policy, not a favour they have to request.

A worked example: a two-person team

This is an illustrative example, not a real company.

Two engineers, Ada and Bo, run a payments integration for a small business. It matters — failed webhooks mean orders do not complete — but there are no other engineers to share the load. Round-the-clock cover with two people is not sustainable. A smaller, honest promise is:

• Business-hours cover is normal work. No paging is needed between 09:00 and 18:00; both are already watching.
• Out-of-hours paging is restricted to two conditions: the payment webhook consumer is down, and the error rate on order completion is above the agreed threshold for ten minutes. Everything else waits for the morning queue.
• Weekly alternating shifts. Ada takes the first week, Bo the second. One week on, one week off is a rhythm a person can plan around.
• Secondary is the other person, paged after ten minutes of no acknowledgement. With two people, the safety net and the rotation are the same two phones.
• A stated limit. The written policy says out-of-hours response is best-effort within thirty minutes, and that a single-person outage during annual leave is possible. This limit is shared with the business owner, in writing, so nobody is surprised by it during an incident.
• Two runbooks only. One for each paging condition, each about a page long, each ending with "if this does not work, escalate to the other person and post in the ops channel".

This setup is unglamorous and it holds, because the promise is small enough to keep. A small team's real choice is not between good cover and bad cover; it is between a narrow promise it can keep and a broad one it cannot.

Common failure modes

• Too many alerts. When most pages need no action, responders start treating all pages as probably nothing — including the one that was not.
• The unowned service. Alerts fire into a channel with no named team. Everyone assumes it is handled.
• The silent schedule gap. Nobody is on-call between the end of one rotation and the start of the next, or during a public holiday nobody accounted for.
• The escalation policy that ends nowhere. The last step is a person who left, a group with no members, or an email address.
• The untested channel. Routing is correct and the phone never rings.
• Runbooks written for people who already know. "Restart the service" is not a runbook if the reader does not know where the service runs.
• No feedback loop. Incidents recur because nothing changed after the last one.

What to do next

Work through this in an hour, today:

• List your production services and write one owning team beside each. Mark the ones you cannot answer — those are your first problem.
• List every alert that currently pages a human. For each, ask: does this require action within an hour? If not, downgrade it to a ticket.
• Name who is on-call this week and next. Put it somewhere shared, not in a private message.
• Define the escalation path: primary, wait time, secondary, wait time, final fallback. Confirm every named person is still employed and still owns the system.
• Send yourself a real test page, out of hours, on the device you will actually be carrying. Confirm it wakes you.
• Write one runbook for the alert most likely to fire, and have someone who has never touched the system read it back to you.
• Write down response time, shift boundaries, and what happens after a broken night. One page is enough.