The Five Characteristics of a Good SLO

Popularized by the Google SRE Book, service level objectives (SLOs) are helpful when setting internal service level agreements between engineering teams and business stakeholders. Unlike the familiar service level agreement (SLA), which sets external agreements between customers and the business, SLOs typically have much stricter measurables. They provide engineering teams a safety net (or buffer) in which they can identify and remediate issues before the user experience reaches unacceptable levels.

SLOs have become a popular concept within engineering teams because they contain several constructs that help teams prioritize work, set up more actionable alerts, reduce alert fatigue, better understand the customer experience, and negotiate agreements that align engineering priorities with business priorities.

However, not all SLOs are created equally: their benefits are only realized when implemented properly. In this guide, we’ll cover the essential characteristics for creating valuable SLOs, including:

1. Setting realistic targets
2. Mapping multiple user journeys to one SLO
3. Using predictive burn alerts
4. Configuring short-term and long-term burn alerts
5. Using events to calculate SLO targets

What is an SLO?

An SLO is an internal goal that serves as an indicator of overall service health. For example, an SLO could be that you require your website to render within 200 milliseconds 99.999% of the time. By observing and tracking the performance of your website load times, you can tell if you're on track to meet your SLO. If you're off track, you're likely facing an issue with performance that threatens to impact user experience.

SLOs are typically used in tandem with service level agreements (SLAs). Because they're internal goals, SLOs tend to be more strict than what's externally agreed to in the SLA. This strictness gives the provider buffer space to fix issues before they impact the SLA.

A service level indicator (SLI) is how you monitor your service to determine if it’s on track to achieve your SLO. In the example above about website render speeds, the SLI would be the page load times. By tracking the SLI, you'll know if you're meeting your SLO. If you're at risk of missing an SLO, the recommended practice is to stop the deployment of new features and focus all efforts on stabilizing the service to de-risk the miss.

Put simply, SLOs are a measure of service stability that informs how work should be prioritized. Nothing inherently connects the use of SLOs to observable systems, but engineering teams with observable systems can greatly benefit from using SLOs to monitor the health of their complex production environments and how customers experience them. SLOs prioritize measuring system health around issues important to the business. Observability helps calculate SLO targets with remarkable granularity, and it also helps teams quickly diagnose exactly why those targets might be in danger.

How do SLOs work with observability?

SLOs provide a way to measure system health in any system. They communicate clear agreements on performance thresholds that the entire organization can understand. The question, "Is our customer experience healthy?" can be quickly answered when tracked with SLOs.

However, SLOs only tell you whether your system is healthy or not. If your system is unhealthy, SLOs can’t tell you why or what may be causing that detrimental state. When paired with observability, you know whether your service is unhealthy, and if so, what exactly is causing that.

The rise in interest in observability is due to the fact that traditional monitoring approaches can't keep up with modern complex systems. These work well in closed systems, when you know what might go wrong because the system is predictable—but modern production environments are a distributed, unpredictable, complex web of virtual infrastructure, SaaS components, serverless functions, and container orchestration platforms.

This is further magnified by the fact that the performance of your production environment can change throughout the day depending on traffic load, the shape and size of underlying systems, new software deployments, or any number of other factors. This unpredictability is created by open and ever-changing systems, where you can’t know what might go wrong, and therefore, can't monitor for it.

Observability is necessary in complex environments because it uses a real-time analytical approach to detect where issues are occurring. With traditional monitoring, you have to know in advance that a certain type of failure might occur, then set up a probe that constantly checks for that specific failure. With observability, it doesn’t matter if you’d previously known this type of failure could occur: tools like Honeycomb help you use machine analysis to quickly comb through telemetry data to spot hidden patterns in any failure—whether it’s the first time you’ve seen it or the thousandth.

When you use SLOs in tandem with observability, you’re able to decouple the what from the where and why of a problem in your production environment. An SLO uses an SLI to calculate whether or not your system is healthy—it shows you that something is wrong in your system. Observability uses telemetry data to sift through billions of transactions, comparing them across thousands of dimensions to diagnose where performance issues are happening—it shows you exactly where something is wrong and why.

Five characteristics of a good SLO

SLOs are a subject with many considerations. We recommend reading Alex Hidalgo’s book, Implementing Service-Level Objectives, for an in-depth look at SLOs. That said, there are five characteristics every SLO should have in order to be effective. You must:

1. Set a realistic target
2. Map multiple similar user journeys to one SLO
3. Use predictive burn alerts
4. Configure short-term and long-term burn alerts
5. Use event-based calculations

1. Set a realistic target

An SLO needs to be stricter than the SLA, but not so strict that it's unachievable. If your SLO is unrealistic, you'll face a number of issues that negate its benefits.

Generally, strict SLOs are ones that require a >99.99% success rate. In the earlier website render time example, the website must render within 200 milliseconds 99.999% of the time. An SLO this strict can exhaust error budgets very quickly because the maximum allowable downtime is so small (approximately 26 seconds per month). Setting unrealistic targets can make your team constantly scramble to resolve issues instead of developing new features.

SLO targets should be negotiated between engineering leaders and business stakeholders. In general, strict SLOs should be reserved only for the most critical business systems that are infrequently updated. Ensure your SLO targets are kept to realistic timeframes. If they’re unrealistic, your service will always be unhealthy and the benefits of using SLOs will be unrealized.

2. Map multiple similar user journeys to one SLO

User journeys measure the reliability of your service. They typically center around measures like rates, types of errors, or durations. They should capture only key performance indicators, rather than all possible indicators. The goal with SLOs is to align engineering and business teams on outcomes that matter to the business, which typically means focusing on customer-impacting performance.

In our earlier example, page rendering speeds are considered vital to providing a good customer experience. When deciding which measures to include in a user journey, you should ask if that measure impacts your customers. As an example, latency could be impactful, but a high number of threads in your runtime wouldn’t. If it's not a direct measure of customer impact, it should not be included as a user journey.

Building on our earlier example, we may decide that rendering our homepage within 200ms is important, and also that requests to process shopping cart transactions succeed with a 200 status code 99.99% of the time. Above all else, the business believes that the homepage must load fast and that purchases must succeed. Rather than setting up two separate SLOs—one for the homepage and one for the shopping cart—both measures should be combined into the same SLO.

Combining similar user journeys into the same SLO keeps priorities focused and it will reduce alert noise. It clearly communicates which measures of customer experience matter to the business and are seen as vital to success. Different services or business units may have their own SLOs. Or, one service may have SLOs with varying targets. Having one SLO per user journey can lead to sprawling SLOs, which muddy your understanding of business priorities.

3. Use predictive burn alerts

Your alerts should tell you when you're on track to miss your SLO, not when you've missed your SLO. The ideal state is to identify and fix errors before they impact customers, not after you've received complaints.

SLOs have an error budget. In other words, there is an allowable number of failures before you miss your SLO target. The rate at which you exhaust your error budget is known as your error budget burn rate, or burndown rate. With SLOs, you look at burndown rate trends, not just your current state. Predictive burn alerts calculate when your error budget will be exhausted, given current trends. They will trigger alerts long before your error budget is exhausted, allowing you time to investigate and resolve issues.

The mechanics behind this are beyond the scope of this guide, but you can read more about how they work in the Observability Engineering book by Charity Majors, Liz Fong-Jones, and George Miranda.

4. Configure short-term and long-term burn alerts

Burndown rates can vary. You may get lots of errors happening all at once, or you may get a slow trickle of errors. In either case, something in your system is broken and you’ll want to know about it so that you can respond with plenty of time. You should have at least two types of predictive burn alerts:

A short-term burn alert looks at trends within the last hour. Based on the last few hours of rate changes, what’s likely to happen within the next hour? Short-term alerts keep you on top of immediate changes to service health. When these alerts go off, it's an "all hands on deck" situation because you're likely very close to exhausting your error budget and missing your SLO.

A long-term burn alert instead looks at trends within the last day. Based upon the last few days of rate changes, what’s likely to happen by tomorrow? Long-term alerts will keep you on top of more gradual changes. These alerts will help you avoid slower error creep, where errors slowly build up over time rather than all at once.

5. Use event-based calculations

SLOs with SLIs that use time-based measures (such as metrics or monitors) have much less accuracy and granularity than those using event-based measures—especially when calculating stringent SLOs where every second of availability matters.

Time-based SLIs (like metrics) count aggregate performance over a period of time. For example, during the past minute, did most requests to render my homepage complete in less than 200ms? If 49% of requests in that minute failed to meet the threshold, the metric will report 100% success during that last minute. If 51% of requests in that minute failed, the metric will report 100% failure for that minute. If thousands of requests were processed in that minute, you could easily be over-reporting or under-reporting SLO targets by hundreds of requests.

Event-based SLOs (like those calculated with logs or traces) count performance on a ‘per request’ level. If 100 requests to render your homepage failed to complete in less than 200ms then those requests are subtracted from the number of total requests. By considering each individual event, you’re able to more accurately track performance against your SLO targets.

For stringent SLOs, the error budget might burn through before a time-based SLO even registers something is wrong. In our website render SLO example, if you're in a state of very high traffic and you're using a one-minute time frame for time-based calculations, you could exhaust your error budget by the time that timeframe passes.

Event-based SLOs allow for deeper granularity and are closer to actual user experience. If you track the website load event in our website render SLO example, you'll know immediately if all of a sudden, a large volume of users' render speeds slow down.

Three real-world examples of great SLOs

How NS1 used Honeycomb SLOs to spark customer joy

NS1, a managed DNS provider, aimed to enhance its API's speed and reliability by leveraging Honeycomb for observability, with a focus on SLOs. SLOs helped NS1 identify and resolve issues proactively, ensuring minimal customer latency and uninterrupted service. By aligning their SLOs with customer outcomes, they set goals and alerts to monitor performance, enabling quick responses to potential issues before customers were affected.

NS1’s innovative use of Honeycomb SLOs focused not just on avoiding downtime but on maintaining high levels of customer satisfaction. Instead of just preventing customer pain, NS1’s SLOs aimed to prevent any reduction in customer joy. This approach allowed them to detect and address issues before they became noticeable to customers. For instance, by creating latency goals and burn alerts per endpoint and HTTP method, they could immediately identify performance regressions during deployments and take corrective actions.

The integration of Honeymarkers allowed NS1 to track deployments precisely, enabling the team to correlate changes with performance improvements. A notable instance was when they observed a significant performance boost after a deployment, thanks to a change in the tags implementation. This capability provided clear, visual insights into the system's state, and made it easier for both technical and non-technical team members to understand and address issues.

Overall, NS1’s adoption of Honeycomb for observability and SLO management resulted in substantial benefits, including faster issue resolution, enhanced performance, and increased customer satisfaction. By prioritizing customer joy and using tools like Honeycomb to maintain high service levels, NS1 demonstrated how effective SLOs and observability practices could drive superior customer experiences and better business outcomes.

How SumUp leveraged SLOs and tracing to improve service quality and strengthen customer loyalty

SumUp, a rapidly growing financial services company, faced challenges in maintaining service quality as its customer base expanded. The high volume of service alerts and complex system interactions made it difficult to pinpoint issues, leading to increased customer support contacts and potential churn. To address these issues, SumUp turned to Honeycomb to implement SLOs and enhance their observability practices.

By adopting SLOs, SumUp aimed to proactively identify and resolve issues before they impacted customers. The team experimented with Honeycomb's specialized SLOs and implemented a tech budget-based alerting system to manage the error burn rate. This approach allowed them to correlate data with system performance more effectively, moving beyond simple metrics and logs. The shift to using SLOs also required a cultural change, giving each team autonomy to address issues within their domain, fostering a more collaborative environment across the company.

Honeycomb's BubbleUp played a crucial role in helping SumUp visualize and analyze performance data. These tools enabled the team to track latency across various services and identify root causes of performance issues. For instance, they discovered that a database query was causing latency problems for a small percentage of customers, and by redesigning the query, they prevented potential impact on millions of users. This ability to quickly diagnose and resolve issues was instrumental in maintaining high service quality as the company grew.

The implementation of Honeycomb led to significant improvements in SumUp's incident response and overall service quality. SumUp's focus on observability and proactive issue resolution not only reduced customer churn but also enhanced collaboration across teams, ensuring that the company could continue to scale effectively while maintaining a high level of service quality.

How HelloFresh promoted the adoption of SLOs and observability

HelloFresh's platform team treats their work like it's a product in itself. They take the time to understand the needs of their developers, create a solution, then maintain and update that solution as time passes and needs change. When introducing SLOs and observability to their organization, the product team applied this same product-focused mentality.

When promoting the adoption of observability and SLOs, you increase the cognitive load of your developers in the short term in exchange for longer-term gains. Instead of dumping all the information and tools on them, HelloFresh's product team made the tools discoverable and referenceable in the wiki. They documented how to define an SLI and an SLO, as well as how to implement them using the automation the platform team produced. From there, they check in regularly to see how the documentation is going, how well their developers are using SLIs and SLOs, and they make updates when needed.

Try Honeycomb’s approach to SLOs

To see how SLOs could positively impact your business, create a free Honeycomb account or schedule a demo and we can walk you through it.