SLIs, SLOs, and Error Budgets: How to Measure Reliability

Fractional CTO · Founder of Injuria.ai
Balancing the push for new features with the need for stable, reliable systems is a core tension in software engineering. Teams that move too fast risk user-frustrating outages, while teams that are too cautious risk losing to competitors. This conflict often creates organizational friction between development teams, who are incentivized to ship features, and operations teams, who are incentivized to prevent failures. The framework of Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Error Budgets provides a data-driven, non-adversarial model for resolving this tension. It replaces subjective debates about risk with a quantitative system that aligns all stakeholders around a common goal: delivering a level of reliability that is good enough to keep users happy, while preserving a capacity for innovation and calculated risk.[14]
This framework originated within Google's Site Reliability Engineering (SRE)[1] discipline, a set of principles and practices for running large-scale systems.[15] Coined by Ben Treynor Sloss, SRE sought to bridge the gap between development and operations by applying a software engineering mindset to system administration problems. SLOs and error budgets became the central mechanism for making data-driven decisions about reliability, creating a shared language and accountability model for product developers and SREs alike.
The Core Components: SLI, SLO, and Error Budget
The system works by progressing from a raw measurement to a target, which then defines a budget for how often that target can be missed.
Service Level Indicators (SLIs): The Measurement
A Service Level Indicator (SLI) is a direct, quantifiable measure of a service's performance from the user's perspective. It's a metric, but a very specific kind of metric chosen to be a proxy for user happiness. A good SLI measures an outcome the user cares about, not just an internal system state. Choosing the right SLI is the most critical step; a poorly chosen SLI can lead a team to optimize for a metric that doesn't improve the user experience.[16]
SLIs are typically expressed as a ratio of two numbers: the number of good events divided by the total number of valid events. This format normalizes the measurement, making it comparable across different time periods and traffic levels.
Common types of SLIs include:
- Availability: Measures whether the service is usable. This is often called success rate. For a request-response service, this could be the proportion of requests that do not return a server error (e.g., an HTTP 5xx status code[2]).
- Latency: Measures whether the service is fast enough. It's not enough to measure average latency, as averages can hide long-tail problems that affect a significant number of users. A better SLI is the proportion of events that complete faster than a given threshold (e.g., the percentage of API requests that finish in under 300 milliseconds).
- Quality: Measures a more complex user-centric property. For a video streaming service, this might be the proportion of streams that play without stalls or rebuffering. For a storage service, this could be durability, measuring the probability that a stored file is not lost or corrupted.
An SLI must be clearly specified. A vague SLI like "request latency" is not useful. The SRE Workbook[3] distinguishes between an SLI specification (which is independent of measurement) and an SLI implementation (which includes where and how it is measured).
Service Level Objectives (SLOs): The Target
A Service Level Objective (SLO) is the target value for an SLI over a specified period. It's a formal statement of the desired level of reliability. An SLO is what the engineering team commits to achieving. It is a precise numerical target, not a vague goal like "make the service faster."
An SLO consists of three parts:
- The SLI: The metric being targeted.
- The Target: The desired percentage (e.g., 99.9%).
- The Compliance Period: The time window over which the target is measured (e.g., a rolling 30 days).
Putting it together, a well-defined SLO looks like this: "Over a rolling 28-day period, 99.9% of successful GET requests to the /api/v1/user/profile endpoint will complete in under 250ms."
Choosing the right target is a product decision, not just an engineering one. The goal is not to achieve 100% reliability. Perfect reliability is impossible and its pursuit is prohibitively expensive. It also stifles innovation, as any change, no matter how small, carries some risk of failure. The ideal SLO target is the point at which users are happy and perceive the service as reliable, but not so high that it prevents the team from shipping features. This is sometimes called the "happiness threshold." For many services, a 99.9% ("three nines") availability SLO is a common starting point, as it corresponds to a level of reliability that most users find acceptable.
Error Budgets: The Innovation Currency
An Error Budget is the mathematical inverse of an SLO. If your availability SLO is 99.9%, your error budget is 0.1%. This 0.1% is the amount of unreliability the service is allowed to experience over the compliance period without breaching its objective. It is a budget that can be "spent" on activities that risk failure.
Error Budget = 100% - SLO Target
This simple inversion is the key that transforms SLOs from a passive reporting metric into an active decision-making tool. The error budget provides a quantifiable, data-driven answer to the question, "Can we take this risk?"
The rules are simple:
- If you have error budget remaining: You can "spend" it. This means you can deploy new code, perform risky infrastructure maintenance, run experiments, or absorb small, unexpected outages without triggering an emergency.
- If you have exhausted your error budget: You have breached your SLO. The error budget policy dictates that all non-essential, risk-inducing work must stop. The team's priority shifts entirely to reliability. This could mean a temporary freeze on feature launches, a focus on fixing bugs that cause instability, or investing in infrastructure improvements. The freeze continues until the service has accumulated enough successful events over the rolling window to earn back some budget.
This mechanism creates a self-regulating feedback loop. A team that ships reliable code is rewarded with a larger capacity to innovate and move quickly. A team that introduces instability is forced to slow down and address the underlying problems. It removes emotion and blame from the conversation, replacing it with objective data.
A Concrete Worked Example: An E-Commerce Cart API
Let's walk through setting up and using an SLO for a critical service: the "add to cart" API endpoint in an e-commerce application.
1. Choose a User-Critical Journey and SLI
The most important action for this service is a user successfully adding an item to their shopping cart. A failure here directly translates to lost revenue.
- SLI Type: Availability.
- SLI Specification: The proportion of
HTTP POSTrequests to/api/v2/cartthat result in a non-5xx status code, as measured at the application load balancer. We exclude 4xx errors because they typically represent client errors (like bad requests[4]), not server failures.
2. Define the SLO
After consulting with the product manager, the team agrees that while perfect availability isn't necessary, the feature must be highly reliable. They decide on a "three nines" target.
- SLO: 99.9% of
/api/v2/cartrequests will succeed over a rolling 28-day window.
3. Calculate the Error Budget
- SLO Target: 99.9% (or 0.999)
- Error Budget: 100% - 99.9% = 0.1% (or 0.001)
Let's quantify this. Assume the service handles approximately 2 million "add to cart" requests every 28 days.
- Total allowed failures in 28 days:
2,000,000 requests * 0.001 = 2,000 failures
This budget of 2,000 failed requests is what the team can spend. This can also be expressed in time. A 28-day period has 40,320 minutes. The total allowed downtime is 40,320 minutes * 0.001 = 40.32 minutes. This means if the service goes 100% down, the team has about 40 minutes to restore it before breaching their 28-day SLO.
4. Measuring the SLI and Using the Budget
The team uses a monitoring system like Prometheus[5] to track the SLI in real-time. They can write a query using PromQL[6] to calculate the success rate.
# Calculate the 28-day success rate for the cart service
(
sum(rate(http_requests_total{job="cart-api", method="POST", path="/api/v2/cart", code!~"5.."}[28d]))
/
sum(rate(http_requests_total{job="cart-api", method="POST", path="/api/v2/cart", code!~"4.."}[28d]))
) * 100
Now, let's see the error budget in action:
- Week 1: The team deploys a new feature. The release is slightly buggy and causes 500 requests to fail before a fix is rolled out. The remaining budget is
2,000 - 500 = 1,500failures. The team proceeds as normal, but with heightened awareness. - Week 2: A planned database migration requires 10 minutes of downtime for the cart service. This downtime causes an additional 700 requests to fail. The remaining budget is
1,500 - 700 = 800failures. - Week 3: A major incident occurs. A bad configuration push takes the service offline, causing 1,000 failures in 20 minutes before it's rolled back. The budget is now
800 - 1,000 = -200. The SLO has been breached.
Consequence: The error budget is exhausted. The team's error budget policy is now active. The engineering manager declares a "reliability focus" period. All new feature development for the cart service is paused. The team's top priority is to conduct a postmortem on the major incident, implement the required fixes to prevent recurrence, and work on other known reliability risks from their backlog.[17] This freeze remains in effect until the 28-day rolling window moves forward and enough time has passed for the budget to become positive again.
The Decision Lens: Tradeoffs and Costs
SLOs are a powerful tool, but they aren't a silver bullet. Adopting them requires investment and organizational discipline.
When to Use SLOs:
- User-Facing Services: Any service whose performance is directly experienced by end-users is a prime candidate. This includes web frontends, mobile APIs, and other interactive systems.
- Critical Internal Platforms: An internal authentication service or a shared database platform can be a single point of failure for many other teams. Defining and adhering to SLOs for these platforms is crucial for overall organizational stability.
- Resolving Dev/Ops Conflict: When teams are deadlocked over release velocity versus stability, SLOs provide a neutral, data-driven framework to guide decisions.
When to Be Cautious:
- Early-Stage Products (Pre-PMF): For a startup searching for product-market fit[7], the primary goal is rapid iteration and learning. The cost of instrumenting and enforcing SLOs may outweigh the benefits. A simple alerting setup is often sufficient.
- Batch Processing Systems: For a system that runs a report once a day, a percentage-based availability SLO might not make sense. A simpler binary metric like "did the job complete successfully by 8 AM?" is often more appropriate.
- Lack of Organizational Buy-in: The most common failure mode for SLO adoption is a lack of enforcement. If leadership is unwilling to uphold the error budget policy and pause feature development when the budget is spent, the SLO becomes a meaningless number on a dashboard. It's an all-or-nothing commitment.
Costs and Failure Modes:
- Instrumentation Tax: You can't manage what you can't measure. Implementing accurate SLIs requires a mature observability stack (metrics, logs, traces). This is an engineering investment that must be made upfront.[18]
- Gaming the SLI: If not chosen carefully, SLIs can be gamed. For example, measuring API success at the load balancer is easy, but it misses errors that happen in an asynchronous background job triggered by the API. The SLI must reflect the full user experience.
- The Wrong Target: Setting an SLO that is too loose can lead to a poor user experience, while setting one that is too tight can cause constant fire-fighting and team burnout. It's often best to start with a reasonable, achievable target and iterate as you gather more data on user expectations.
Correcting Common Misconceptions
Several misunderstandings frequently arise when teams first encounter this framework.
Misconception: "SLOs are the same as SLAs."
This is incorrect. A Service Level Agreement (SLA) is a formal, often legally binding contract with a customer that specifies financial penalties (like service credits) if the agreed-upon service levels are not met.[19] SLAs are an external promise to customers. An SLO is an internal objective for the engineering team. Typically, an internal SLO is much stricter than the external SLA. For example, a service might have a 99.9% internal availability SLO to ensure it comfortably meets its 99.5% external availability SLA. Breaching an SLO triggers an internal engineering response; breaching an SLA costs the business money.
Misconception: "We should aim for 100% reliability."
This is a fallacy. As noted, the cost of each additional nine of availability grows exponentially. A 99% availability target allows for about 3.65 days of downtime per year. A 99.999% ("five nines") target allows for only 5.26 minutes. That extra reliability may require redundant infrastructure across multiple geographic regions, complex failover logic, and a drastic reduction in the pace of change, all of which come at a massive cost. The purpose of SLOs is to define "good enough," not to pursue perfection.
Misconception: "SLOs are only for SRE teams."
While the concept was popularized by SRE, its value extends to any team running a service. Product teams, platform teams, and DevOps-oriented teams can all use SLOs to manage their services effectively. The responsibility for a service's reliability, and therefore its SLO, should be shared between the product manager, developers, and operators.
The State of SLOs Today
What began as an internal Google practice is now a widely accepted industry standard for managing service reliability. Adoption is high in large technology companies and is growing rapidly in smaller organizations, thanks to better tooling.[20]
Tooling and Standards
There is no formal ISO[8] or IETF[9] standard for SLOs. However, the industry is converging on common practices. Major cloud providers have integrated SLOs as a first-class feature in their monitoring suites, including Google Cloud Operations[10] and Amazon CloudWatch Service Level Objectives[11]. Commercial observability platforms like Datadog[12] and New Relic also offer robust SLO tracking and alerting capabilities.
To promote interoperability, the OpenSLO[13] project was created. It provides a vendor-neutral specification for defining SLOs using a YAML format. The goal is to treat SLOs as code: they can be version-controlled, reviewed, and deployed alongside the application code they measure. This approach helps standardize how SLOs are defined, regardless of the underlying monitoring tool.
By adopting the principled framework of SLIs, SLOs, and error budgets, engineering organizations can move beyond the false dichotomy of innovation versus reliability. They can build a sustainable, data-driven culture where teams are enabled to make smart, calculated risks while being held accountable for the user experience they deliver.
Sources (20)
- Google SRE Book - Table of Contents
- MDN Web Docs - HTTP 5xx Server error responses
- The SRE Workbook - Implementing SLOs
- IETF RFC 7231 - HTTP/1.1 Semantics and Content (Client Error 4xx)
- Prometheus - Official Website
- Prometheus Documentation - Querying Basics (PromQL)
- Wikipedia - Product-market fit
- International Organization for Standardization (ISO)
- Internet Engineering Task Force (IETF)
- Google Cloud Operations (Stackdriver) - SLO Monitoring
- AWS Documentation - CloudWatch Service Level Objectives
- Datadog - Service Level Objectives
- OpenSLO Project
- Betsy Beyer, Chris Jones, Christof Leng, Site Reliability Engineering ("O'Reilly Media, Inc.", 2026)
- Niall Richard Murphy, Betsy Beyer, Chris Jones, Site Reliability Engineering ("O'Reilly Media, Inc.", 2016)
- Alexandra F. McCoy, SLIs and SLOs Demystified (Packt Publishing Ltd, 2025)
- Unai Huete Beloki, The Art of Site Reliability Engineering (SRE) with Azure (Springer Nature, 2025)
- Alex Hidalgo, Implementing Service Level Objectives (O'Reilly Media, 2020)
- Jamil Mina, Armin Warda, Rafael Marins, Digitalization of Financial Services in the Age of Cloud ("O'Reilly Media, Inc.", 2023)
- Vladyslav Ukis, Establishing SRE Foundations (Addison-Wesley Professional, 2022)