Alerting That Does Not Cry Wolf: Monitoring Done Right

March 11, 2026•14 min read•

An alert fires at 3 AM. A dashboard shows CPU utilization is at 95%. The on-call engineer spends an hour investigating, only to find the high CPU was from a scheduled, non-critical background job. The service was, and is, perfectly healthy from a user's perspective. This is alert fatigue, a state where frequent, low-value alerts desensitize engineers, causing them to miss or delay responding to real incidents. The root cause isn't a lack of monitoring; it's the wrong kind of monitoring. Effective alerting isn't about watching every machine-level metric. It's about creating high-signal, low-noise notifications that are directly tied to user experience.

The solution is a philosophical shift: from monitoring causes to monitoring symptoms. Instead of paging an engineer because a server's CPU is high (a cause), you page them because users are experiencing slow responses (a symptom). This approach, popularized by Google's Site Reliability Engineering (SRE) practices, uses a framework of Service Level Objectives and error budgets to create alerts that are almost always actionable and significant. It transforms monitoring from a source of noise into a precise instrument for protecting user happiness and guiding engineering priorities.

The Core Philosophy: Symptoms vs. Causes

At the heart of modern monitoring is a distinction between two types of metrics: causes and symptoms. Understanding this difference is the first step to eliminating noisy alerts.

Cause-based monitoring tracks the health of individual components of a system. These are metrics like CPU load, memory usage, disk space, and network I/O. For decades, this was the standard approach. An alert would be configured with a static threshold, like ALERT IF CPU_usage > 90% FOR 5_minutes. The problem is that a cause does not always have a negative effect. High CPU might be acceptable during a data processing task. A full memory cache might be a sign of a healthy, well-performing system. Alerting on these metrics directly generates a high volume of false positives, conditioning on-call engineers to ignore pages.

Symptom-based monitoring focuses on the overall behavior of the service as experienced by its users. It answers questions like: Is the service available? Is it fast enough? Is it returning correct results? These are measured with Service Level Indicators (SLIs), which are direct measurements of service behavior. As Google's SRE handbook states, your monitoring should focus on alerting you about problems that are happening or are about to happen from the user's point of view. A system can be internally chaotic, with failing nodes and high resource utilization, but if users are not impacted, it does not warrant an emergency page.

This philosophy is formalized through a hierarchy of concepts: SLIs, SLOs, and Error Budgets.

Service Level Indicator (SLI): A quantifiable measure of some aspect of your service. An SLI is a direct measurement of a user-oriented property. Good SLIs are often expressed as a ratio of good events to total events. Common examples include:
Availability: The proportion of successful requests (e.g., HTTP 2xx or 3xx status codes) out of the total valid requests.
Latency: The proportion of requests served faster than a certain threshold (e.g., 300ms).
Quality: The proportion of requests that return correct, non-degraded content.
Service Level Objective (SLO): A target value for an SLI over a specific time window. An SLO is a promise to your users about the level of performance they can expect. For example, an SLO might be: "99.9% of homepage requests^[1] over a rolling 28-day window will be served successfully."
Error Budget: The mathematical inverse of the SLO (100% - SLO). An SLO of 99.9% implies an error budget of 0.1%. This budget represents the acceptable level of unreliability. It's not a goal to spend the budget, but an objective limit. If you have 10 million requests in a 28-day window, a 0.1% error budget allows for 10,000 failed requests. This budget provides a data-driven framework for balancing feature development with reliability work. As long as you are within budget, you can prioritize shipping new features. If you are close to exhausting it, reliability becomes the top priority.

Alerting in this model is not about crossing a static threshold. It's about the rate at which you are consuming your error budget.

Building a Symptom-Based Alerting System

Implementing an SLO-based alerting system involves four main steps. The goal is to configure alerts that fire only when the rate of errors is high enough to threaten the SLO for the entire window.

Define SLIs: Identify the critical user journeys for your application. For a typical web service, good starting points are the metrics captured by the RED method^[2]: Rate (requests per second), Errors (number of failing requests), and Duration (latency of requests).
Set SLOs: For each SLI, define a target and a window. A common window is 28 or 30 days, which smooths out weekly traffic patterns and provides a monthly view of reliability. Choosing the target (e.g., 99.9%, 99.5%) is a product and business decision. It should be realistic and reflect user expectations. Don't aim for 100%, as perfection is impossible and the pursuit of it leads to diminishing returns.
Instrument Your Code: Your application must export the necessary metrics. For latency SLIs, this typically means using a histogram^[3] to track the distribution of response times. For availability, a counter for total requests and another for failed requests is needed.
Configure Alerts on Error Budget Burn Rate: The key insight is to alert based on how quickly the error budget is being consumed. The "burn rate" is a multiplier indicating this speed. A burn rate of 1 means you're consuming the budget at a rate that will exhaust it precisely at the end of the SLO window (e.g., 28 days). A burn rate of 10 means you'll exhaust the monthly budget in just 2.8 days (28 / 10). High burn rates over short periods signify critical outages, while lower burn rates over longer periods indicate persistent but less severe problems.

This allows for multi-level alerting:

High-Severity (Page): A very high burn rate over a short window (e.g., an hour). This means a severe outage is in progress and will breach the SLO quickly if not fixed. Example: Alert if the error rate over the last hour implies the entire 28-day budget will be consumed in 24 hours.
Low-Severity (Ticket): A lower burn rate over a longer window (e.g., 6 hours). This detects slower-burning issues that need attention but aren't immediate emergencies. Example: Alert if the error rate over the last 6 hours implies the budget will be consumed in 7 days.

This approach automatically scales the alert's urgency to the magnitude of user impact.

A Concrete Example: Alerting with Prometheus

Let's walk through setting up SLO-based alerts for a web service using Prometheus^[4], a popular open-source monitoring system. Its query language, PromQL^[5], is particularly well-suited for these calculations.

Scenario:

Service: An API service (my-api).
SLI: The proportion of HTTP requests served in under 250ms.
SLO: 99.5% of requests over a 28-day period are served in under 250ms.
Error Budget: 0.5% of requests are allowed to be slower than 250ms.

First, the application must export a Prometheus histogram metric, http_request_duration_seconds, which buckets request latencies.

Next, we define alerting rules in a YAML file. We will set up two alerts: a critical page for fast-burning events and a warning ticket for slow-burning events.

The logic for a burn-rate alert is: error_rate > (burn_rate * error_budget).

groups:
- name: api-slo-alerts
 rules:
 # Define the error rate as a recorded rule for easier use.
 # This calculates the 5-minute rate of requests that were slower than 250ms,
 # divided by the 5-minute rate of all requests.
 - record: job:http_request_duration_seconds:error_rate_5m
 expr: |
 (sum(rate(http_request_duration_seconds_bucket{job="my-api", le="+Inf"}[5m])) - sum(rate(http_request_duration_seconds_bucket{job="my-api", le="0.25"}[5m]))) / sum(rate(http_request_duration_seconds_bucket{job="my-api", le="+Inf"}[5m]))

 # A similar rule for a 1-hour window.
 - record: job:http_request_duration_seconds:error_rate_1h
 expr: |
 (sum(rate(http_request_duration_seconds_bucket{job="my-api", le="+Inf"}[1h])) - sum(rate(http_request_duration_seconds_bucket{job="my-api", le="0.25"}[1h]))) / sum(rate(http_request_duration_seconds_bucket{job="my-api", le="+Inf"}[1h]))

 # --- Alerting Rules ---

 - alert: APILatencySLOBurnCritical
 # This alert fires if the error rate over the last hour is so high that
 # it would consume the entire 28-day budget in just 2 days.
 # Burn rate = 14 (since 28 days / 2 days = 14).
 # SLO = 99.5%, Error Budget = 0.005.
 # Alert threshold = 14 * 0.005 = 0.07
 expr: job:http_request_duration_seconds:error_rate_1h > (14 * 0.005)
 # The 'for' clause prevents flapping. The condition must be true for a sustained period.
 for: 5m
 labels:
 severity: page
 annotations:
 summary: 'High latency on my-api is burning through the 28-day SLO budget rapidly.'
 description: 'The error rate over the last hour ({{ $value | printf "%.2f" }}%) is high enough to consume the entire 28-day budget in 2 days.'

 - alert: APILatencySLOBurnWarning
 # This alert fires if the error rate over the last 6 hours is high enough
 # to consume the 28-day budget in 7 days.
 # Burn rate = 4 (since 28 days / 7 days = 4).
 # SLO = 99.5%, Error Budget = 0.005.
 # Alert threshold = 4 * 0.005 = 0.02
 expr: (sum(rate(http_request_duration_seconds_bucket{job="my-api", le="+Inf"}[6h])) - sum(rate(http_request_duration_seconds_bucket{job="my-api", le="0.25"}[6h]))) / sum(rate(http_request_duration_seconds_bucket{job="my-api", le="+Inf"}[6h])) > (4 * 0.005)
 for: 30m
 labels:
 severity: ticket
 annotations:
 summary: 'Latency on my-api is elevated, threatening the 28-day SLO.'
 description: 'The average error rate over the last 6 hours ({{ $value | printf "%.2f" }}%) will consume the 28-day budget in under a week.'

This configuration, defined as code in a rules file^[6], ensures that on-call engineers are only paged for events that truly threaten the service's reliability promise. The warning alert provides visibility into slower-moving problems that can be addressed during business hours.

The Role of Cause-Based Monitoring

A common misconception is that this new model makes cause-based metrics like CPU and memory obsolete. This is incorrect. Their role simply changes from being alerting triggers to diagnostic tools. When a symptom-based alert fires (the "what"), engineers turn to dashboards of cause-based metrics to investigate the "why".

An effective incident response workflow looks like this:

Alert: A page fires: APILatencySLOBurnCritical. The user-facing symptom is high latency.
Diagnose: The engineer opens a pre-built dashboard associated with the my-api service. This dashboard displays key cause-based metrics for all dependencies: CPU and memory of the API servers, database query latency, cache hit rates, network throughput, etc.

This is where frameworks like the USE Method^[7] (Utilization, Saturation, Errors) for resources and the RED method for services are invaluable. The USE method, developed by Brendan Gregg, provides a checklist for analyzing the performance of any resource (CPU, memory, disk). By examining these metrics after being notified of a real problem, engineers can quickly pinpoint the bottleneck without being distracted by noisy, low-level alerts during normal operation.

History and Evolution

The principles of modern monitoring evolved alongside the systems they measure. In the 1990s and early 2000s, systems were largely monolithic and ran on dedicated physical servers. Tools like Nagios^[8], first released as NetSaint in 1999^[9], dominated this era. They excelled at host-based checks: Is the server reachable? Is the disk full? Is the httpd process running? This was a cause-based model perfectly suited for the simpler architectures of the time.

With the rise of distributed, service-oriented architectures, this model broke down. The health of a single server no longer determined the health of the application. A service could be degraded even if all servers were "green." This complexity demanded a new approach.

The philosophy of symptom-based alerting was developed and refined internally at large-scale tech companies. Its public breakthrough came with the publication of Google's book Site Reliability Engineering in 2016. This book codified the concepts of SLIs, SLOs, and error budgets, providing a shared vocabulary and methodology for the entire industry.

This shift was supported by a new generation of tooling. Prometheus, created at SoundCloud starting in 2012^[10], was designed from the ground up for the dynamic, microservices-based environments of the cloud-native era. Its multi-dimensional data model and powerful query language made calculating SLOs and burn rates straightforward. Prometheus was the second project accepted into the Cloud Native Computing Foundation (CNCF)^[11] in 2016, cementing its role as a cornerstone of modern observability.

Tradeoffs and Decision Lens

Adopting SLO-based alerting is a powerful strategy, but it requires investment and has specific tradeoffs.

Costs and Complexity:

Implementation Effort: Defining meaningful SLIs and SLOs is a non-trivial task. It requires collaboration between product, business, and engineering teams to define what reliability means for each service and for the users.
Tooling: You need a monitoring system capable of storing high-cardinality time-series data and executing complex queries over time windows. Simple check-based tools are insufficient.
Cultural Shift: The entire engineering organization must understand and buy into the error budget concept. It must be respected as the mechanism for balancing reliability work against feature development. On-call engineers must trust that an SLO-based alert is important.

When to Use Symptom-Based Alerting:

For all user-facing services where performance and availability are critical.
For critical internal platform services upon which other teams depend.
In any complex, distributed system where individual component health does not reliably predict overall service health.

When to Use Other Alerting Methods:

Imminent Catastrophic Failure: For certain stateful systems, like a primary database, an alert on a cause might still be warranted. For example, an alert that a primary disk will be full in 4 hours is purely cause-based, but the consequence is so certain and so catastrophic that it justifies an exception. These should be rare and carefully curated.
Batch Jobs: For a non-critical batch processing job, a simple alert on job failure or completion may be all that is needed. An SLO is likely overkill.

A Decision Lens for Technical Leaders:
As a leader, consider these questions:

Is your on-call team suffering from burnout? Are alerts frequently acknowledged and then ignored? This is the primary indicator that your alerting strategy is broken.
Can you quantitatively answer the question, "How reliable was our service last month?" If the answer is a vague feeling instead of a number, you need SLIs and SLOs.
Do your teams argue about whether to build features or fix bugs? An error budget provides a clear, data-driven framework to resolve these debates.

The upfront investment in defining SLOs and re-instrumenting alerts pays long-term dividends. It reduces on-call toil, focuses engineering effort on what actually matters to users, and provides a common language for discussing and managing reliability.

Where We Are Today

SLO-driven monitoring is the established best practice for reliable software operations. Its adoption has been accelerated by cloud providers and the broader tool ecosystem.

Adoption: Major cloud platforms now offer first-class support for SLOs. Google Cloud's Service Monitoring^[12] is built entirely around this concept, and AWS provides guidance for implementing SLOs with its CloudWatch service. Commercial vendors like Datadog^[13], New Relic^[14], and Dynatrace^[15] have all made SLOs a central feature of their platforms.
Standards: While there is no formal ISO^[16] or IETF^[17] standard for defining SLOs, the terminology and methodology from Google's SRE publications serve as the de facto industry blueprint. To further formalize this, the OpenSLO^[18] project is working to create a vendor-neutral specification for SLOs, allowing them to be defined in a portable way.
Best Practices: A consensus has formed around how to succeed with this model. Start small with one or two critical services. Treat your alerting and SLO configurations as code, subject to version control and peer review. Continuously iterate on your SLOs; they are not static and should be reviewed quarterly or semi-annually to ensure they still reflect business goals and user expectations. By embracing this disciplined, user-centric approach, you can build an alerting system that commands attention, respects engineers' time, and truly protects the user experience.

Sources (18)