The Engineer’s Guide to Service Level Objectives (SLOs)

Imagine a world where engineers get more sleep, friction between engineering and management decreases, and your monitoring processes become more efficient and cost-effective. Sounds too good to be true, right? Well, it's possible with Service Level Objectives (SLOs).

Over the years, SLOs have become an essential tool in modern engineering because they provide a common language between business stakeholders and engineers to set aligned goals. When properly implemented, they improve your customer experience and overall system reliability—but getting started involves a bit more research than just jumping into setting targets. In this guide, we’ll dive into:

• The fundamentals of SLOs
• Their benefits
• Best practices for setting them up
• How to align them with business goals

What is an SLO?

SLOs are specific, measurable goals set for the performance and reliability of a service. They are typically internal targets that help ensure the user experience remains optimal and that contractual agreements with your customers are met. For example, an SLO might state that 99% of all user requests to a web service should be processed within 200 milliseconds.

They are part of the broader "service level" family, which also includes Service Level Indicators (SLIs) and Service Level Agreements (SLAs):

• SLIs are the metrics used to measure aspects of the service, such as latency, error rate, or availability.
• SLAs are formal agreements with customers that define the expected level of service and the consequences if these expectations are not met (i.e., the contractual agreements mentioned above). These consequences often include large fines and penalties.

Engineers and business folks often lack a shared language to discuss reliability—thoughtfully created SLOs, however, can bridge that gap, especially when tied to key business metrics and based on meaningful data.

Setting your team up for success with SLOs

In this section, we’ll go over a few best practices that will ensure your teams implement SLOs in a way that extracts the most benefit.

Think min, not max

SLOs should reflect performance minimums, not maximums: They’re about the bare minimum level of service needed to ensure a healthy user experience. Keep it simple and don’t overcomplicate things.

Get everyone aligned

One of the primary benefits of SLOs is that they provide a common language for both engineering and business teams. It's crucial to get everyone—from developers and SREs to platform engineers and even the business side—involved in discussing important user journeys. By aligning SLOs with business objectives, companies can ensure that engineering efforts directly contribute to business success.

Decide whether your SLOs should tie directly to SLAs or just act as early warnings. Then, choose the right user journeys to attach your SLOs to.

Leverage event-based SLOs

Traditional time-based SLOs can be limiting. Event-based SLOs, tied to observability, offer more granular insights. For instance, if a good customer experience means loading the homepage quickly, an SLI would qualify events with a request path of "/home" and measure if the duration is less than 100 milliseconds. Actionable SLOs, based on wide structured event data, lead to faster remediation and happier users.

Narrow it down to actionable SLOs

Teams getting started with SLOs should be careful not to SLO all the things. Less is more here, so make sure to choose wisely. If you’re thinking of implementing an SLO, ask yourself:

• Is it meaningful? Does it warn us in advance or only after customers are upset?
• Is it measurable? Can we express it clearly with common-sense metrics?
• Is it practical? Can we realistically meet the target?
• Once an SLO alert is triggered, is it actionable?
• When should it wake someone up, if at all?

Embrace the business benefits

Software development doesn’t happen in a vacuum. There needs to be dialogue and alignment between product-focused teams, platform/DevOps/SRE teams, and the organization at large. Teams should understand the company’s KPIs and how they contribute to them. SLOs provide a shared language and tools for both engineering and the business side to understand product performance. This helps in incident resolution, cost management, risk compliance, and even in prioritization efforts by giving strong signals of where to invest more in.

Supercharge SLOs with observability to reduce alert fatigue

Want to upgrade your SLOs and reduce alert fatigue while making developers happier? Tie SLOs to observability.

Observability gives SLOs access to a wide variety of data, the ability to ask new questions without adding instrumentation, and allows engineers to leave biases about previous incidents behind.

Traditional monitoring systems often lead to alert fatigue, where engineers are overwhelmed by too many alerts, many of which are noise. SLOs, when tied to key business metrics, can help reduce this noise by focusing only on what truly impacts the user experience and business outcomes.

Use customer-focused SLOs

Focus on what your customers care about, measure it, and alert on it. For example:

• The homepage loads in less than one second.
• No errors when submitting an order.
• Backend order processing completes in under 30 seconds.

SLOs help you think from the customer’s perspective, aligning service reliability with user expectations. The important thing about all of these is that it’s okay if they aren’t met 100% of the time. It’s expected, in fact; that’s why Honeycomb exists. We just need to find a threshold over a certain period of time that’s acceptable to us: if 99% of homepage renders take less than one second, we’re good.

Setting business goals with SLOs

Business leadership sets the strategic vision (as they should!), and other departments figure out how they can align in that direction. Collaboratively, everyone shifts and re-prioritizes work, organizational constraints become apparent, negotiations happen, and—together, ideally—the company arrives at a set of ambitious but achievable goals. The problem comes in when the negotiating parties aren’t speaking the same language.

SLOs can solve this problem. They provide a common language between business stakeholders and engineers for setting aligned goals. For example, a business goal might be “providing the fastest website experience possible.” Technically, this means a user should be able to load the homepage quickly. An SLI for this might look at events where the request path is "/home" and measure whether the duration is less than 100 milliseconds. The SLO target could be 99.9% success over a 30-day period.

Different parts of your business will have different targets. For example, at Honeycomb, we have different business goals for how we ingest and serve your data. We have very little tolerance for losing any of the events you send us, so our ingest API service has the strictest SLO target at 99.99%. The UI homepage is set somewhere in the middle with a 99.5% target. One of our less strict SLOs is around querying and requires to return results within 10 seconds, 99% of the time. But because we use rich observability data, we are also able to set different thresholds for other query types: a more expensive pattern can be given 30 seconds, or maybe customers with less data can expect their responses within five seconds instead—all within the same query SLO.

There’s more depth and detail to SLOs but, even in this high-level generalization, you can see how that approach enables explicit agreements with business stakeholders that determine critical paths and necessary investments, allows engineers to clearly understand priorities and how their work impacts business goals, and gives managers the tools needed to set expectations with both groups.

Because SLOs provide that ability, teams are eager to use SLOs when setting their yearly goals. Let’s see how that’s done in practice.

Case study: An OKR to implement end-to-end observability

Take a real-world example from a Honeycomb customer, a large US banking institution. They use Honeycomb to ensure application reliability for their finance management platform. When setting yearly goals, they planned to set an OKR around “implementing end-to-end observability” with key results based on SLO targets. However, they were advised to connect the objective to a business outcome, like better service reliability, and gather data to propose initial SLO targets and build consensus throughout the year.

The reliability objective became “a performant and available site experience for our users, which measurably increases retention and improves sales.” Key results for the next quarter included:

• Instrumentation coverage: 100% of user-facing requests through instrumented services.
• On-call adoption: 100% of on-call engineers use Honeycomb for troubleshooting.
• Identify, negotiate, and implement three to five SLOs with business and engineering buy-in.

When first implementing SLOs, it helps to take a quarter or two to gather data. You should understand how both planned and unplanned operations impact your SLO error budgets. In turn, that informs how service availability maps to business performance indicators. With that data, you can then negotiate targets with business stakeholders and engineering leaders. Doing that with three to five services is a pretty significant goal.

Remember that your SLO doesn’t need to be perfectly defined before those negotiations start. Identify the services most critical for your business outcomes, gather some early data, have those negotiations, then implement your SLOs and improve iteratively as you go.

Why not just use monitoring alerts?

At this point you may be thinking, aren’t monitoring alerts enough? Well, if you’re solely relying on monitoring alerts, then you’d have to know what will cause an eventual issue and set up an alert for it. What if the problem was caused by people using iPhone iOS 14.1 with a French language pack hitting out of Canada? Would you create an alert for all those conditions?

That’s where SLOs pull ahead: They’re not influenced by hindsight bias, and let you go on a debugging journey no matter where the issue is coming from.

Take the engineers at Honeycomb, for instance. We realized the power of SLOs during a partial degradation that our SLOs caught but our monitoring missed, because they were looking for two consecutive failed probes in a row. When you have a 1-2% brownout, two consecutive probes failing is not likely to happen. Luckily, our SLO immediately started burning and we quickly realized that our users were being affected.

How we handle SLOs

At Honeycomb, SLOs aren’t things we set up once and forget about—if an SLO stops making sense, we iterate on it. Generally speaking, here’s a short list of things we keep in mind for SLOs:

• We hold ourselves to an extremely high internal standard, but an SLO burning down faster than normal is merely a signal or alert that something is wrong. Reset the burndown chart if normal service has resumed, so it can start providing signal again. Teams might update the threshold or definition if the stakeholders agree a different level of reliability makes sense. We're also gradually shifting to burn rate alerts for pages, and budget alerts for prioritization. It’s worth repeating: SLOs should be an iterative process! Always keep improving.
• On-call has the authority to silence or disable a particularly noisy SLO they are being alerted by—or one that isn’t serving its purpose—as well as the responsibility to follow up with stakeholders and peers.
• There is no authoritative document other than the SLO itself within Honeycomb, though it usually has rich descriptions and is created with input from many stakeholders. A log is kept of burn alerts that are reset, and documentation maintained about our philosophy of SLOs.
• The SLOs that are internal or department-specific don’t need to have the same cross-team or customer-facing standards. However, everyone still needs disk alerts.
• How many SLOs, and what kind, is up to the team that owns it. We don’t have a minimum number of SLOs, and the SRE team plays an increasingly consultative role in their creation.

Conclusion

SLOs are a powerful tool for aligning engineering efforts with business goals, improving system reliability, and enhancing the user experience. By following the best practices outlined in this guide, organizations can set effective SLOs that drive meaningful outcomes and foster a culture of continuous improvement.

Implementing SLOs requires careful planning, collaboration, and ongoing monitoring, but the benefits are well worth the effort. With the right approach, SLOs can transform the way your organization manages and delivers reliable services. So, start small, gather data, and watch your team’s performance soar.

If you’d like to dive deeper into SLOs, read our guide: The Five Characteristics of a Good SLO.