Reducing alert fatigue: a practical guide for small on-call teams

Every on-call team knows the spiral. Alerts multiply, most turn out to be nothing, people start glancing at pages instead of reading them, someone mutes a channel “temporarily” - and then the real incident lands in the muted channel. The postmortem says the alert fired. It did. Nobody trusted it enough to look.

The standard response is exhortation: be more disciplined, triage harder, care more. It doesn’t work, because alert fatigue isn’t a discipline problem - it’s a design problem with a well-understood shape. Humans cannot sustain attention to a channel that’s usually wrong. If your alerts are mostly noise, your team will stop reading them; the only question is when. The fix is making the channel mostly right.

Start with an audit, not a resolution

Pull the last month of alerts and sort each into one of three buckets:

• Actioned: someone did something because of it.
• Acknowledged-ignored: someone looked and correctly did nothing.
• Unseen: nobody even looked.

Most teams find the actioned bucket is under 20%. That number is your channel’s trust score, and every rule below exists to raise it. Re-run the audit monthly; it’s the only metric of alert quality that can’t be argued with.

Four rules that do the actual work

1. Every alert must have an action. If the correct response to an alert is “note it and move on”, it isn’t an alert - it’s a metric with a megaphone. Demote it to a dashboard. The test: finish the sentence “when this fires, the on-call person should ___”. No answer, no page.

2. Alert on symptoms, not causes. Page on “the service is slow for users” and “disk will be full in 4 hours” - not on every individual condition that might someday contribute. Cause-level alerts are how one incident becomes forty pages, and forty pages are how the one that matters gets scrolled past.

3. Severity means “when may this interrupt a human”. Three levels are enough: wake someone (revenue or data at risk now), tell someone today (fix during working hours), file it (batch review). The discipline that matters: business-hours-only alerts for business-hours problems. A certificate expiring in three weeks is real work - it is not a 3 AM page.

4. Deduplicate ruthlessly, escalate deliberately. One incident, one page - not one page per host per check per retry. And escalation policies exist so a missed page goes to the next person, not to everyone louder. Broadcasting is how you burn a whole team on one incident.

The rotation is part of the fatigue

On small teams, the schedule does as much damage as the noise. Three patterns keep a rotation of three-to-six people sustainable: short shifts (a week is the maximum; longer breeds resentment, shorter breeds handoff chaos), real off-duty (off-rotation means notifications silenced - if people can’t trust the schedule to protect them, they never truly rest, which is fatigue by another route), and overrides without ceremony (swapping for a holiday should be one click, not a negotiation - friction in overrides quietly converts to resentment of on-call itself).

The prerequisite for all three is that alerts reliably reach only the on-call person. Alerts sprayed at a shared channel make everyone perpetually half-on-call - the most fatiguing schedule there is.

Where AI triage honestly helps (and where it can’t)

The rules above fix the alerts you designed. Security events break the model: a busy host emits thousands of them, no human-written ruleset classifies them all, and “just look at the stream” is the fatigue spiral with extra steps.

This is the one place automation genuinely changes the economics. AlertKick’s AI triage reads every eBPF security event in context - the process tree, the login that preceded it, the host’s history - explains it in plain English, and lets the routine stay quiet while the genuinely odd gets escalated. The same principle drives alert-to-event linkage: when a page does arrive, the story of why is attached, so triage takes the minute it should instead of the half-hour of dashboard archaeology that makes 3 AM pages so corrosive.

What AI can’t do is repeal rule 1. If a page has no action, no amount of intelligent summarisation makes it worth waking a human. Triage reduces the cost of judging each alert; only alert design reduces the count.

The compounding payoff

Teams that get this right don’t just page less - they respond faster, because every page carries earned urgency. That’s the loop worth building: trust makes response fast, fast response justifies the trust.

AlertKick ships the machinery for the whole list - sensible default severities, dedup, business-hours rules, rotations, overrides, and escalation, AI triage on every security event - so a small team gets the designed-alert experience without a quarter of tuning. Start free, run the three-bucket audit on your current setup, and see what a mostly-right channel feels like. Muted channels don’t protect anyone.