Alert Fatigue: How to Reduce Noise Without Hiding Risk

At a glance

Alert fatigue is not a volume problem on its own. It is what happens when the ratio of interruptions to decisions drops far enough that responders stop reading alerts carefully. Once that happens, the next real page arrives in a stream people have already learned to skim.

Two failure directions matter equally. Too much noise, and genuine pages are missed inside it. Too much filtering, and real risk sits behind a silence that looks like health. Judge every change below against both.

Symptoms worth taking seriously

• Responders acknowledge pages from the lock screen without opening them.
• A handful of alert rules produce most of the pages, and everyone can name them.
• Alerts are routinely closed with "no action needed" or "recovered on its own".
• There is a muted alerts channel that no team claims to own.
• Handover notes say "ignore the queue-depth alerts, they always fire".
• New joiners ask which alerts are real and receive a verbal answer.

The last one is the clearest signal. If knowing which alerts matter is tribal knowledge, the alerting configuration has stopped describing the system's actual risk.

The actionability test

Apply this to one alert rule at a time, and write the answers down. The value is in the written answer, not in the discussion around it.

The question: if this alert fires at 03:00 and nobody acts on it until 09:00, what is the worst realistic outcome?

Four supporting questions sharpen the answer.

1. Does a human make a decision? If the correct response is always the same fixed sequence of commands, automate the sequence and alert only when the automation fails.
2. Does the alert name an owner? An alert routed to a team that cannot change the underlying system is a notification, not a page.
3. Does the payload carry the first action? A responder should be able to start work from the alert body plus one linked runbook, without opening four tools to find out what the alert means.
4. Is it a symptom or a cause? Prefer alerting on user-visible symptoms and keeping cause-level signals as diagnostics. Ten cause alerts for one symptom is ten interruptions for one decision.

Page, ticket, or dashboard

Write the destination into the alert definition itself, as a field or label, so routing is reviewable in the same change that creates the rule.

Reducing volume without hiding risk

Deduplicate on a stable key. Alerts describing the same condition on the same object should collapse into one incident. Choose a deduplication key made of fields that do not change during the event — service, alert name, and the affected object — rather than fields that do, such as a timestamp or a pod identifier.

Group related alerts into one notification. When a dependency fails, everything downstream of it also fails. Grouping by service or by incident turns twenty simultaneous notifications into one page with twenty lines of evidence, which is the same information at a fraction of the interruption cost.

Require duration, not instants. A condition that must hold for several minutes before it pages removes most flapping. Where a threshold is crossed repeatedly, set separate firing and clearing thresholds so a metric hovering at the boundary does not produce a page each time it wobbles.

Auto-resolve what recovers. An incident that stays open after the condition clears trains people to ignore the queue. Auto-resolution keeps the record honest — and the count of alerts that auto-resolved before anyone acknowledged them is one of the more useful noise measurements available.

Attach ownership and context. At minimum, every paging alert should carry the owning team, the affected service or object, the current value against the threshold, a link to the runbook, and a one-line statement of user impact. If those fields cannot be filled in, the rule is not ready to page anyone.

Metrics worth reviewing monthly

An export of alert events is enough to calculate all of these. None of them requires a particular vendor.

• Pages per responder per shift, split into working hours and out of hours.
• No-action rate: the share of pages closed with no change made to any system.
• Top ten rules by page count, and the share of total pages they represent.
• Repeat rate: pages from the same rule and object within twenty-four hours.
• Auto-resolved before acknowledgement: pages that cleared before a human looked.
• Coverage checks: alerts that have not fired in six months, reviewed rather than assumed dead.

The last line is the guard against over-tuning. A cleanup that only ever removes alerts has no way of noticing what it broke.

Common failure modes

• Silencing instead of fixing. A suppression with no expiry and no owner becomes a permanent blind spot. Give every suppression an end date and a reason.
• Severity inflation. When everything is critical, severity carries no information. Count how many rules carry the top severity; if that is most of them, write down what each level means in terms of user impact and re-rate every rule against that definition.
• Alerting on every cause. Cause-level alerts multiply with infrastructure size; symptom-level alerts scale with user-visible behaviour. Keep the paging alert on the symptom and demote the causes it explains to dashboards.
• Cleanups without measurement. Without a before-and-after count, a cleanup is an opinion.
• Removing the test. Teams that stop testing the paging path after a cleanup discover the gap during a real incident. Re-test delivery on every route you changed, in the same week you change it.

A worked example

The following is an illustrative example, not a real organisation.

A four-person team owns a payments service. Over a month they receive roughly ninety pages. They export the alert events and find that three rules account for most of them: a per-pod memory alert that fires whenever a deployment rolls, a queue-depth alert with an instantaneous threshold, and a third-party latency alert that resolves itself within two minutes.

They apply the test. The memory alert fails question one — the response is always "wait for the rollout to finish" — so it moves to a dashboard. The queue-depth alert has real meaning but the wrong shape: they add a ten-minute duration condition and separate clearing threshold, and it becomes a ticket unless depth is still rising after thirty minutes. The third-party latency alert is a genuine symptom, so it stays a page, but they group it by provider and add auto-resolution.

Two rules are added, not removed: a heartbeat on the settlement job, which previously failed silently, and a symptom alert on payment success rate. The memory alert no longer pages at all, the queue-depth alert pages only when depth is still rising after thirty minutes, and the latency alert pages once per provider incident rather than once per spike. Two failure modes that used to be invisible now have alerts. Both counts — pages sent, and user-visible failures covered — are recorded so the next review has a baseline.

A 30-day cleanup plan

What to do next

• Export the last 90 days of alert events and count pages per responder per shift.
• Take the three noisiest rules and write the actionability answer for each, in full sentences.
• Move at least one rule off the phone this week, and record where it went and why.
• Add owner, impact line and runbook link to every rule that still pages.
• Put a recurring 30-minute alert review in the calendar, attached to the on-call handover.
• Re-test the paging path after the changes land, so the quieter system is also a working one.