Alert Fatigue Is a Discipline Problem, Not a Tooling Problem

It is 3am and your phone is buzzing again. CPU > 80% on web-04. You know before you open your laptop that web-04 is fine, that it always does this during the nightly batch, that in ninety seconds it will resolve itself. You silence it and go back to sleep. Three weeks later the same reflex silences the page that was actually the database running out of disk, and you find out at the standup instead of the pager.

That is alert fatigue, and it is not a tooling problem. You cannot buy your way out of it with a smarter alerting product, because the failure is in the alerts themselves and in the discipline that is supposed to prune them. Here is the thesis: an alert that does not map to an action a human takes right now is not an alert, it is a dashboard metric that learned to phone you. Every one of those you tolerate lowers the attention paid to the ones that matter, until the pager is background noise and a real incident gets the same shrug as the nightly CPU blip.

Why more alerts always makes it worse

The instinct when something slips through is to add an alert so it never happens again. Do that a few dozen times and you have built a machine that pages constantly, most of it noise, and the signal-to-noise ratio is now so low that the humans have rationally decided to stop reading. Alert fatigue is a self-reinforcing loop, and adding alerts feeds it.

The test every alert has to pass

There is one question that sorts real alerts from noise: when this fires, is there a specific action the on-call engineer takes right now? If the honest answer is "look at it and probably do nothing," it is not a page. It might be a useful graph, a ticket for the morning, or nothing at all, but it must not wake a human.

This is the difference between symptom-based and cause-based alerting. CPU > 80% is a cause with no guaranteed symptom; high CPU is often completely fine. Checkout p99 latency > 2s for 5 minutes is a symptom a user feels, and it maps to a clear action: investigate checkout now. Page on the symptoms your users experience, not on every internal number that might contribute to one.

Alert	Maps to an action?	What it should be
CPU > 80% on one host	No, usually self-resolves	Dashboard, not a page
Disk will fill in < 4 hours (trend)	Yes, free space now	Page
Checkout error rate > 2% for 5 min	Yes, investigate checkout	Page
Single 5xx spike, already recovered	No, nothing to do	Log / dashboard
Replica lag > 60s and climbing	Yes, reads are stale	Page
Nightly batch pins CPU (expected)	No	Delete or mute during window

How to actually cut the noise

Step 1: Pull the ack-with-no-action list

Most on-call platforms let you export alert history. Find every alert that fired in the last handful of incidents and was acknowledged or resolved without anyone doing anything. That list is your target. An alert that has paged ten times and produced zero actions is not protecting you; it is training your team to ignore the pager. The fix for it is deletion or a threshold change, and it is almost never "make it more sensitive."

Step 2: Set thresholds against your own baseline

The single biggest source of false positives is a threshold copied from a dashboard template or a vendor default without checking your service's actual variance. If your API normally swings between 200ms and 900ms at p99 depending on traffic, a flat "alert at 500ms" pages you every busy afternoon. Look at two to four weeks of real data, find the normal range, and set the alert comfortably outside it, on a duration window ("for 5 minutes") so a single scrape blip does not fire. A good alert fires when something is genuinely wrong, not when the service is merely busy.

Step 3: Alert on trends and burn rates, not instants

"Disk is 90% full" pages you at a moment that may be stable for weeks. "Disk will be full in under four hours at the current rate" pages you when there is something to do and time to do it. The same logic underpins error-budget burn-rate alerting: page when you are consuming your reliability budget fast enough to matter, not on every individual failed request. This collapses a class of instant-threshold noise into a handful of meaningful pages.

Step 4: Correlate symptoms into one incident

When the database gets slow, the API times out, the queue backs up, and the health check fails. That is one incident with four symptoms, and paging separately for each is how you get four 3am messages for a single root cause. Group related alerts by service and time window so the on-call sees one incident with its contributing signals attached, not a wall of notifications. This reduces perceived noise without reducing detection: you still catch everything, you just get told once. Doing it well depends on being able to correlate logs, metrics and traces around a shared identifier so the symptoms visibly belong to the same event.

Verification: how you know it worked

You are not measuring "number of alerts," you are measuring whether the pager is trusted. Track a few things over a quarter:

• Actionability rate: of pages that fired, what fraction led to a real action. Rising is good. If most pages still lead to "acked, nothing to do," you have not pruned enough.
• Pages per on-call shift, especially out of hours. If it is trending down while incidents are still caught, the pruning is working.
• Time to acknowledge. When the pager is trusted, ack times fall because people believe it. Rising ack times are an early sign fatigue is creeping back.

Review these in the same forum where you review incidents. Alert noise is a contributing factor to missed incidents, so it deserves the same standing as any other reliability problem, not a "we should clean that up someday" backlog ticket.

What people get wrong

"Add an alert so this never slips through again." The most natural reaction and the one that builds the fatigue. Before adding, ask whether an existing alert should have caught it and was ignored because of noise. Usually the answer is yes, and the fix is pruning the noise, not adding to it.

"Tighten the threshold so we catch it earlier." Tightening a noisy alert makes it noisier. If an alert fires without a corresponding action, the answer is a looser threshold, a duration window, or deletion, not more sensitivity.

"Route everything to a channel so nothing is lost." A firehose channel nobody reads is not coverage, it is the illusion of coverage. Informational signals belong on dashboards and in searchable logs, not in a stream you have trained yourself to scroll past.

"Buy a tool with better noise reduction." Correlation and grouping features help, but they cannot invent an action for an alert that has none. The tool cannot decide that CPU > 80% is not worth waking you. Only you can, by deleting it.

When it is still noisy

• One service generates most of the pages. It probably has a real reliability problem or a genuinely bad threshold set. Fix the service or rebuild its alerts from the symptom down, do not just mute it.
• Flapping alerts that fire and resolve repeatedly. Add hysteresis: require the condition to hold for a duration before firing, and a separate lower threshold before resolving, so it does not oscillate on the boundary.
• Nobody has time to prune. Put a recurring, owned slot on the calendar to review the ack-with-no-action list. Pruning that competes with feature work in an ad-hoc way always loses; make it a standing ritual with a name attached.
• Alerts fire but the runbook is empty. An alert with no documented action is a half-built alert. Either write the action or demote it to a dashboard. If you are starting from nothing, the same "what would I actually do" discipline applies to diagnosing a slow server with no fancy tooling.