On-call rotation best practices: the discipline that keeps a rota working
Handover ritual, overrides instead of edits, quiet hours and severity, secondary and catch-all levels, the monthly review, what to measure, and the compensation questions to settle.
Sandeep Sidhu · Founder, AlertKick
On-call rotation best practices come down to a handful of habits: hand over at a fixed time with a short written summary, cover absences with overrides and swaps rather than by editing the rota, only wake people for alerts that need a human now, always have a secondary and a catch-all escalation level, review the rotation monthly against real numbers, and decide the compensation and time-off policy explicitly. These are the operating discipline that keeps a rotation healthy after the initial setup enthusiasm has worn off.
Setting up a rotation takes an afternoon. Keeping one working for a year is harder, because the failure modes are slow: one person accumulates the holiday weeks, an alert that should have been tuned keeps paging, a shift is silently covered by the same colleague every time, and the roster drifts from what is written down. This post is about the practices that prevent that drift. For the setup itself, see how to create an on-call schedule and the on-call roster guide.
What should an on-call handover look like?
A handover is a ten-minute exchange at the scheduled handover time that transfers the state of the system from one engineer to the next. It should be written, short, and boring.
The handover time should be during working hours, at the start of the week for a weekly rotation, so that both engineers are at their desks. Monday 09:00 in the roster’s timezone is the common choice. A handover at Friday 17:00 means the incoming engineer starts with a weekend and no context.
What the outgoing engineer covers:
| Item | Why it matters |
|---|---|
| Open alerts and their current state | The incoming engineer will be paged about them next |
| Anything that fired more than twice | Candidate for tuning, or a problem that is not fixed |
| Temporary fixes made under pressure | Restarted services, resized volumes, disabled jobs |
| Silenced or suppressed alerts | Including when the suppression expires |
| Upcoming risky work | Deployments, migrations, maintenance windows |
| Anything that felt wrong | Slow dashboards, an unusual pattern, a hunch |
Post it in the team channel rather than in a direct message so that the secondary and the rest of the team see it too. Over time the handover thread becomes the best record of what actually happens to the service out of hours, which feeds the monthly review below.
Why should you use overrides instead of editing the rota?
An override changes only the named dates. Editing the rota rewrites every future shift.
Reordering or removing someone from the roster to cover a holiday regenerates the schedule from the next handover onwards. It can give someone an unexpected double turn. The schedule must be reversed when they return, which regenerates it again. Everyone’s calendar feed changes twice.
An override says “Ana covers Tom from 2 September to 4 September” and nothing else. The rotation underneath is untouched and resumes exactly as it was. A swap is two overrides paired so that both people give and take a shift, which keeps the load balanced over time. Edit the roster only when the membership genuinely changes: a joiner, a leaver, or a deliberate reorder. Roster management covers the mechanics.
Two related habits:
- Put the override in as soon as the absence is known, not the day before. Calendar feeds update, and the person covering can plan.
- Record who covered for whom. If the same person is always giving cover and never receiving it, the rotation is not as fair as the schedule suggests.
How should quiet hours and severity interact?
Only alerts that need a human within minutes should reach a channel that wakes someone. Everything else should wait for working hours or go to a channel that does not interrupt sleep.
This is a routing decision as much as an alerting one. The escalation policy decides which channels are used and in what order; the alert rules decide what severity each alert carries; and per-user quiet hours decide whether a push notification is delivered at all during certain hours.
A workable arrangement:
| Severity | Out-of-hours behaviour | Example |
|---|---|---|
| Critical | Pages through quiet hours, escalates to SMS or WhatsApp | Customer-facing service down, data loss risk |
| High | Pages, respects the person’s quiet hours on push and Telegram | Single node down with redundancy holding |
| Medium | Slack or email only, reviewed at handover | Disk at 80 percent with days of headroom |
| Low | Email digest | Certificate expiring in 30 days |
The trap is letting the middle of the table drift upwards. An alert that is Medium on paper but pages at night because someone raised its severity “just to be safe” is the beginning of alert fatigue. Quiet hours per user, with critical alerts allowed through, give each engineer a floor that the alert rules cannot erode, and put the burden on the team to classify alerts honestly.
Maintenance windows belong in the same discussion. A planned migration that will trip a dozen alerts should have a suppression window scoped to those hosts and that time range, not a series of manual acks at 02:00.
Why does every rotation need a secondary and a catch-all?
The primary will miss a page. The rotation must keep working when that happens.
The primary is the person on the roster. The secondary receives the alert if the primary does not acknowledge it within the first wait period. The catch-all is the final level, triggered when all prior levels fail. It ensures a missed page reaches someone with the authority to act on the process, not just the alert.
A minimal policy for an out-of-hours roster:
On trigger: push + Slack to current on-call
Level 1: wait 5 min -> Roster (current on-call), SMS
Level 2: wait 10 min -> Notify Person: secondary (their own preferences)
Level 3: wait 15 min -> Notify Person: team lead
Repeat: once
The secondary can be a second roster with the same members in a shifted order, ensuring the same person is never primary and secondary in the same week. Alternatively, it can be a Notify Person level pointing at a fixed backup. Escalation levels by default argues for setting this up on day one rather than after the first missed page. Escalation policies covers the level types.
Two rules for the catch-all: it should not be the same person as the secondary. If it fires more than occasionally, the waits or channels at the earlier levels need attention, not the catch-all.
How often should you review the rotation, and what should you look at?
Monthly, against numbers, with a written outcome.
The review catches drift. It does not need to be long.
- Was the load fair? Pages per shift, broken down by person. Two people with the same number of shifts can have very different months if one had the release week.
- Were the waits right? Time to acknowledge, and how many alerts escalated past the first level. If most alerts are acknowledged in two minutes, a five-minute first wait is fine. If many reach the secondary, either the primary’s channels are wrong or the wait is too short for the alerts involved.
- Which alerts fired most? The top five by volume are the tuning backlog. An alert that fired twenty times and was resolved without action every time should not page.
- Who covered for whom? Override counts per person, given and received.
- Who is joining or leaving? Onboarding stages, and whether the roster still has at least three solo members.
- What changed? New services, new alert sources, new integrations that may be routing to the wrong roster.
Record what was decided, even if the decision is “nothing changed”. The handover thread and the alert timeline together should answer all six questions without anyone having to reconstruct the month from memory.
What should you measure for an on-call rotation?
Five things, all of which come from the alert history rather than from a survey.
| Metric | What it tells you | What to do with it |
|---|---|---|
| Pages per shift, by person and severity | Whether load is fair and whether severity is honest | Rebalance, or retune the alerts |
| Time to acknowledge | Whether channels and waits are right | Adjust escalation waits or channel order |
| Time to resolve | Whether runbooks and access are adequate | Fix runbooks, grant access |
| Escalations past level 1 | Whether the primary is reachable | Check channel preferences, quiet hours, waits |
| Top alert sources by volume | Where the tuning effort should go | Tune, dedupe, or downgrade |
Two things are deliberately absent. Pages per person per year hides the release-week effect. Targets for time to acknowledge are guesses until the numbers have been looked at; measure first, then decide what good looks like for this service.
How should compensation and time off be handled?
Write it down. Apply it consistently. The specific answer depends on the organisation, the country, and the employment contract. This post does not recommend a figure.
Answer these questions:
- Is there a standby allowance for carrying the pager, regardless of whether it fires?
- Is out-of-hours work paid per page, per hour, or not separately at all?
- Does an engineer who is paged overnight start late the next day, and is that automatic or at their discretion?
- What counts as an out-of-hours page for these purposes? A Slack message that was ignored is different from an SMS at 03:00.
- Are on-call obligations part of the role, or opt-in, and what happens to the rotation if someone opts out?
The best practice is not any particular answer. It is that the team is not discovering the answer for the first time after a bad night, and that the alert history is detailed enough to support whatever policy is chosen. An unwritten policy tends to be fair to whoever complains loudest.
How AlertKick handles this
AlertKick rosters support overrides and swaps layered on the rotation. The rota is never edited for cover. The rotation resumes untouched when the override ends. Escalation policies have ordered levels with a wait in minutes each. Level types include Roster, Notify Person and Round Robin. Policies have a repeat count, and levels can be switched off without being deleted. Quiet hours are per user and enforced on push and Telegram. Critical alerts can be allowed through. Maintenance windows suppress alerts for a scope and time range. Acknowledging anywhere, whether Slack, the mobile app, Telegram or the web, stops escalation everywhere. Every notification is recorded on the alert timeline so the monthly review has real numbers. These features are on every plan, including Free. See on-call features.
Frequently asked questions
- What are the most important on-call rotation best practices?
- Hand over deliberately at a fixed time, use overrides and swaps rather than editing the rota, route only genuinely urgent alerts through channels that wake people, always have a secondary and a catch-all escalation level, review the rotation monthly, and measure pages per shift and time to acknowledge. Decide the compensation and time-off policy explicitly rather than leaving it to custom.
- What should an on-call handover include?
- Open alerts and their current state, anything that fired repeatedly during the shift, changes made under pressure that still need a proper fix, upcoming risky work such as deployments or maintenance windows, and anything the outgoing engineer silenced. It should take about ten minutes and happen at the scheduled handover time, not whenever both people are free.
- Should you edit the on-call rota to cover a holiday?
- No. Use an override for the dates and leave the rotation alone. Editing the member list or the rotation to cover one absence changes every future shift and usually gives someone two consecutive turns. An override covers only the dates named and the rotation resumes untouched afterwards.
- What is a catch-all escalation level?
- The last level in an escalation policy, reached only when every earlier level has failed to produce an acknowledgement. It targets a named person or a group rather than the roster, typically a lead or manager, and its purpose is to make sure a missed page is noticed rather than to fix the alert. If the catch-all fires regularly, the earlier levels need attention.
- How often should an on-call rotation be reviewed?
- Monthly is a good default. Look at pages per shift for each person, how long acknowledgement took, which alerts fired most, how many overrides each person gave and took, and whether anyone is about to join or leave. Adjust the roster, the alert rules or the escalation policy based on what the numbers say, and record the decision.
- What metrics should you track for an on-call rotation?
- Pages per shift, split by person and by severity; time from alert to acknowledgement; time from alert to resolution; the number of alerts that escalated past the first level; and the top alert sources by volume. These tell you whether the load is fair, whether the escalation waits are right, and which alerts are candidates for tuning.
- Should on-call be paid?
- That is a policy decision each organisation must make explicitly, and local employment law may constrain it. The options usually discussed are a standby allowance for carrying the pager, a per-page or per-hour rate for out-of-hours work, time off in lieu, or a combination. The important best practice is that the answer is written down and applied consistently, not that any particular option is chosen.
