The CTO’s Guide to Predictable Observability Costs

If you're reading this guide, you probably agree that there's a Cost Crisis in Observability Tooling. In recent years, a class of tooling built atop the three pillars of metrics, logs, and traces—or traditional observability tools—has seen costs skyrocket at rates many multiples higher than your traffic increases. These scaling costs are radically disconnected from the value traditional tools can deliver. Most often, as costs go up, the value your business derives from these tools declines.

This guide unpacks why your observability bills skyrocket out of control (spoiler: this typically happens with metrics-based tooling). It examines how teams work to mitigate rising costs and shows why those measures lead to declining business value. In short, the metrics-based data model sets up a need to pay incrementally more for every level of detail you want to see. A seemingly benign requirement, that incremental cost typically leads to exponential growth in storage costs and layers on additional premiums for faster retrieval. It creates the type of hair-raising surprises that can suddenly add tens of thousands of dollars to your bill over the weekend.

We then show you how to ensure predictable observability costs. The guide introduces the strategy used by modern observability tools: a unified data model where adding additional details is free and retrieval is always fast. Your business can get better results while also lowering the cost curve. We cover concrete steps you can take today to get started in working your way out of the endemic observability cost crisis.

At Honeycomb, we believe the costs and value of observability should scale in alignment with your business growth. We hope this guide will get you started on your journey to reign in observability costs while also getting better answers from your observability tools, regardless of the vendor you ultimately choose.

The traditional approach to observability

When it comes to monitoring software, metrics-backed dashboards have long been the de facto approach adopted by the tech industry. Metrics are computationally cheap, fast, sparse, and the technology is decades old—therefore, stable. Most APM and RUM tools (like Datadog, Prometheus, Chronosphere, and more) are built using metrics primitives. Engineering teams are accustomed to instrumenting their software with metrics, deriving alerts from metrics, and using metrics-backed dashboards to debug their code.

You may have heard observability described as having metrics, logs, and traces of your system. Logs are often used for debugging, but they're too unruly and expensive to process en masse to use as a jumping-off point. Distributed traces are often perceived as too niche, and also often deemed unruly and expensive to use as a trigger for investigations. But metrics are cheap, fast, and easy to parse. Therefore, they're used on the front lines, heavily and with extreme prejudice.

What exactly is a metric?

Colloquially, “metrics” are often used as a generic synonym for telemetry data—i.e., this report provides various metrics at the class and method level.

In the context of software monitoring, a metric is a data type. The metric is a number, with tags (or “labels”) appended, which are then stored in a variety of formats—such as counters, gauges, and histograms—often written to a time-series database (TSDB). A TSDB is a collection of data points (numbers) gathered by time. The only type of index a TSDB has is an index by time, and the only type of queries you can run are point queries by time and range queries over time. A TSDB stores no relational or contextual data whatsoever: that's why they're so fast, cheap, and easy.

In short, metrics are summary data.

An abstraction that loses context

In every other part of the business, our tools are backed by relational databases or columnar stores because we understand that data is made valuable by context. We use valuable context everywhere else in the business, except when it comes to using traditional observability tools.

With metrics, all aggregation is performed at write time. That frontloads computation at write time and removes the need to process at query time. However, it does so at the cost of removing context and introducing an additional layer of abstraction and obfuscation.

An event in your system occurs, and any number of values are locally computed and recorded, including reporting into performance average buckets like 95th%, 99th%, 99.99th%, etc. When your engineering team queries for the 99th percentile latency across your fleet of app services, they get an aggregate of aggregates of the locally-computed 99th percentile latency over a rolling window across all instances. But if they want to consider an alternate dimension—let's say 99.95 percentile latency or the 85th percentile latency, or any other latency that was not computed at write time—they cannot.

That context is gone. Forever. There is no going back. Thus, we enter the most useful and valuable iteration of the metric: custom metrics.

The proliferation of custom metrics

If we lose context after a metric is written, the workaround is to capture more context at write time. Custom metrics solve for that.

Traditional observability tools often include metrics collection agents, like StatsD or DogStatsD. These agents automatically collect many system stats that are used to quickly churn out pretty graphs of things like system CPU, memory, disk space, etc., all with very little manual work required. But nearly all the practical value your business will derive from these tools will come from instrumenting your software with custom metrics.

In our previous example, your team may not have been able to see all the proper latency dimensions they needed. That's a simple enough problem to fix: they can add custom metrics to capture them. The team can customize the request_latency metric by adding the 99.95th and 85th percentile calculations.

What else besides latency matters to the business? To preserve customer experience, you'll probably want to track errors and HTTP status codes on any critical customer-facing service endpoint as custom metric dimensions. If your business has key strategic customers, you'll probably want to track performance for a particular user_id. If you deploy new software to production frequently, you'll also want to track things like build_id. And so on and so on.

For a moment, let's look beyond the fact that this approach requires your engineering teams to predict in advance all the possible dimensions they might need to see about any given problem they may encounter in the future.

Let's say your engineers know everything you'll need to measure in order to spot any issue that occurs. Add up all of these dimensions and you should know how many custom metrics you'll need, right?

When your engineers talk about adding custom metrics to their code, they are typically conceptualizing each line of instrumentation as a custom metric. Just as you probably did in order to calculate how many custom metrics you would need. When a metrics vendor says you get a couple hundred custom metrics for free, that sounds like a lot. Surely you won't need more than a thousand, maybe?

Unfortunately, that's not how time-series data works.

What matters for cost isn't how you instrument your code to capture those dimensions, it's how those dimensions get stored in your provider's TSDB. When metrics are stored in TSDBs, every unique combination of metric name and tag values generates another distinct time-series, also known as a custom metric. This gets a little complicated, and can vary by implementation or backing store.

That complication is exactly why your observability bills are skyrocketing unpredictably.

Calculating the cost of an custom metric

To unpack the problem of skyrocketing and unpredictable cost, let's dive deeper into the examples above by examining our TSDB storage footprint.

Let's say you submit a metric, request_latency, from five hosts with two tag keys; endpoint and status. You have a tiny starter application that only runs on four endpoints. You decide to only track two HTTP status codes, 200 and 500. You decide to submit it as a count metric.

5 hosts x 4 endpoints x 2 status codes = a footprint of 40 custom metrics for this one instrumented metric.

Now, let's say you operate at a moderate scale. Your app runs on about 1000 hosts, and you monitor 100 endpoints, or 5 methods and 20 handlers. There are 63 HTTP status codes in active use.

1000 hosts x 5 methods x 20 handlers x 63 status codes = 6.3 million custom metrics.

So far, the only thing your engineering team can do is look at a simple count of requests broken down by host, endpoint, and status code. By way of debugging capabilities, that's not much. Counts are fine, but what they're really trying to measure is latency.

statsd.histogram('request_latency.histogram', random.randint(0, 20), tags=[f'endpoint:{endpoint},status:{code}'])

If you submit request_latency as a histogram or distribution using nothing but the default aggregations max, median, avg, 95pc, and count, that's 31.5 million custom metrics.

In our previous example, we also wanted to compute the 99th, 99.5th, 99.9th, and 99.99th percentiles. For every percentile bucket you want to compute at write time, you add another multiplier. If you want to calculate 10 dimensions instead of 5, that's a footprint of 63 million custom metrics.

As an example, vendors like Datadog include about 100-200 custom metrics per host for free, depending on your plan. For every 100 ingested custom metrics over the allotment, you pay ten cents. In this example, that means you'd pay $63,000 per month just to collect barebones HTTP latency statistics. Every metrics tool and backing store will have a slightly different pricing model. Here, we cite Datadog because they are a market leader and reasonably representative for this type of tooling.

In this example, we haven't even tried to tag our metrics with anything particularly useful yet, like build ID or user ID. Now, let's reconsider the fact that this approach requires our engineering teams to predict in advance all the possible dimensions they might need to have available in order to debug any given issue. Even if you were able to calculate the footprint of all of those custom metrics, the next unforeseeable issue your engineers encounter in production could mean adding yet another dimension, and another, and another. Each with its own potential exponential effect on your observability bill.

With custom metrics, costs are hard to predict, and even harder to connect to value.

Why your observability bill spirals out of control

Calculating your metrics footprint is hard to do in advance, and it may change out from under you at any time. Engineering teams rely on policy documents, best practices, and expert code reviews to control costs, only to get bitten by seemingly unrelated changes made by infrastructure teams—or even autoscaling.

In the previous example, you have 1000 hosts and 20 handlers. Consider what happens to your bill when:

Your infrastructure team moves your app tier from 1000 xlarge EC2 instances to 4000 on-demand containers
Your on-call engineer needs to roll your entire app tier a few times inside of an hour, causing several thousand EC2 instances to spin up briefly before dying
Your team deploys new code that adds versioning to each handler
One of your engineers accidentally changes the value for histogram_aggregates or histogram_percentiles in a YAML config file, not realizing that this change will apply to all histograms
Several years ago, one of your rockstar engineers who is no longer with the company cleverly auto-generated a custom metric tag from AWS instance IDs, no one currently on your team realized how this was done, and overnight AWS instances switch to using a different string format

These are probably scenarios you recognize because they are unfortunately all too common. This is why a perception exists that observability costs are simply too high and too unpredictable. They're the reason you probably decided to download this guide in the first place.

For example, in the first scenario, your bill quadrupled to $252,000/month without a single line of application code changing, and without any change in server-side capacity. How can it be so easy to accidentally quadruple your bill while making it 0% easier to understand or debug your applications?

High costs are absolutely a problem. So is unpredictability. But what's worse is that often these costs are completely untethered from value. When your bill goes up, it should be a function of scaling up capacity and/or making your software and systems easier to understand.

How experienced teams practice cost control

So far, we've used simplified examples. Any experienced member of an observability engineering team would probably be irritated by the somewhat naive and profligate ways we have gone about cost estimation to this point.

In our earlier example, we arrived at a $63k/month estimate for request_latency by generating a unique time-series for the intersection of each host name, handler, method, and status code. Your engineering team would probably scoff and tell you that there's a much better way to wrangle costs. For example, they might rely on things like…

Creating status code buckets

Engineers versed in metrics do not generate a time-series per unique status code. They generally collapse status codes into five families—1xx, 2xx, 3xx, 4xx, 5xx—to cap cardinality. Your engineering teams can adjust to that, but it is not ideal. For example, there's valuable knowledge between knowing you're seeing a 502 (Bad Gateway) vs. a 504 (Gateway Timeout). One tells you you're connecting to the incorrect endpoint, while one tells you the endpoint is correct but currently operating incorrectly.

When debugging issues, sometimes a weird status code combined with a request string or an originating IP is all the information your teams need to understand an extreme outlier event. Smoothing over status codes is a cheap and easy upfront way to save money. But for large, complex businesses with distributed systems, that hack could be the difference between solving an issue immediately or explaining to shareholders why your quarterly revenue fell short.

Reaping “superfluous” tags and buckets

Remember how you had to predict all of the dimensions you'd need in advance? Do you really need to store them all? Why are you paying to store AVG latency or MAX? Do you really need to store 99, 99.5%, 99.9%, and 99.99%? Is median useful to the business? Storing some subset of these buckets is valuable, but nobody pays to store all of them, especially not broken down by every single endpoint or method and handler. It's simply too expensive and impractical to do so.

Does your engineering team really need to be able to check the latency of every endpoint or handler? As a manager reviewing skyrocketing observability costs, you've probably asked yourself these questions. Maybe some of these dimensions can be sacrificed on the altar of budgetary constraints. Reaping tags and pruning buckets is one of your best tactics for controlling costs. That's what we see happen in most engineering teams that use traditional observability tools. However, that also means constantly trying to remember which data has been useful recently, and predicting what data you think you can live without.

Working around host count

Metrics experts know all about the traps of host tags—in fact, this is how most engineers first learn about the concept of cardinality. It's common for teams to get started by tagging metrics with hostnames: clearly, you'd like to know exactly which host (or hosts) might be originating a particular issue. That type of tagging works great until you have about 100 hosts, at which point you run out of free tags and your bill starts to skyrocket.

To counteract this, your engineers have probably devised a ton of clever hacks and workarounds. Running larger instances, tagging with a host type prefix, scrapping hostnames entirely, etc. That works very well to control your costs. Unfortunately, that means losing sight of where in your infrastructure some issues originate. Instead, to see that, engineers might have to switch from debugging via the metrics-generated dashboards to using your logging tool instead because the logs still include hostnames. They end up context-switching between different tools and guessing at correlations, all to save a few bucks at the expense of longer service outages and unhappy customers.

The examples we used earlier could be worked around in order to save costs. But that's also kind of the point. An enormous amount of expertise gets front-loaded into creating and managing metrics, because it's so easy to have your bills spiral out of control. Managing metrics costs is a full-time job.

There are many well-known practices for controlling costs with metrics data. There is an even longer list of intricacies and techniques leveraged by experts, trying to walk the line between the ability to answer the questions that matter on one hand, and going bankrupt on the other. Your engineering team is likely familiar with many of these techniques, such as:

Deleting metrics, deleting tags
Deleting anything that hasn't seemingly solved a problem within recent memory
Prepaying for capacity or committed use
Polling less often, or expanding the window size
Disabling a lot of the defaults, especially for consolidated platforms
Setting ingestion volume controls
Setting caps on burst capacity
Configuring alarms to give you a heads up if your custom metrics footprint explodes
Configuring your retention policy
Disabling agents on some fraction of hosts or containers (metrics are pooled across hosts)
Ingesting all metrics, but only indexing a subset of them
Doing fancy tricks with streaming windows to shrink the number of active time-series in order to “support” high-cardinality metrics

This is a partial list of common workarounds. Some of these are fairly straightforward, but others are quite challenging or lead to deeper issues. For example, remember how we worked around the host count above? If you sacrifice tags like container ID, hostname, or host ID, you wind up getting metrics reported from several hosts that all look similar. Without that label, you now face collisions and duplicate records in your TSDB. If metrics collide, someone needs to decide whether to drop one of them (the lower metric or higher one?), or to merge them using sum, mean, average, or some other method. Your engineering team will need to have a pretty deep understanding of your data and use cases to determine which method is best a majority of the time.

These cost savings aren't free: they consume engineering cycles. All of these techniques eat up a substantial amount of ongoing engineering time. How many cycles in a sprint are dedicated to wrangling these costs? None of these techniques are a set-and-forget approach. They often require care and maintenance to shift strategically as your applications change. It is not uncommon for observability engineering teams to spend more time managing costs and cardinality than writing libraries or abstractions, consulting with teams, or otherwise adding value to the organization.

To some extent, all you reasonably can do to control the cost of custom metrics is watch closely and react fast. Maintaining control of your metrics bill is a sizable and neverending burden placed on your engineering teams.

This burden is likely a holdover from days when engineering was treated as a cost center rather than a value driver in organizations. Further, primary reliance on metrics is likely a holdover from a pre-cloud era when compute and storage were finite and terribly expensive. What if we didn't have to make future decisions about what we need at write time?

A shift is already well underway, bringing systems and application data into the relational fold. It also seems to be picking up speed thanks to soaring costs and the relentless explosion in underlying system complexity. Let's consider an alternate approach to metrics-based tooling: one that gives us better answers while making costs predictable and removing these holdover burdens from our engineering teams.

Shifting to a modern data model for observability

Metrics became the dominant data type used for monitoring during an era of simpler systems, when compute was limited and storage was expensive. Metrics make a tradeoff optimizing for storage and compute costs at the expense of losing context by simply summarizing. But these tradeoffs are hurting your business in today's world and they no longer make sense.

Rather than aggregating and summarizing at write time, what if you captured everything at write time and then decided what was relevant to see in the moment you needed to see it?

During any given investigation, that would allow you to decide whether you needed to see a high-level summary of overall system state or if you needed to zoom in on any particular set of details. You could compute any new percentile bucket on the fly. You could determine request latency for any particular user. You could see if latency went up for the new build ID that was just deployed, or if performance was just degraded for users in us-west-2, or if it was just for service requests hitting a particular MySQL secondary db.

In short, with full context, you could slice and dice your data arbitrarily, explore any combination of dimensions, zoom in, or out, or up, or down. Modern observability adds business context back into your telemetry data.

Shifting from traditional to modern observability

At this point, it's necessary to understand a few implementation details to paint a broader picture of how functionality changes. If traditional means observability 1.0 and modern means 2.0, let's look at a few key differences that enable giving your data full context during any given investigation.

In observability 1.0, you famously have three pillars of data types—metrics, logs, and traces. Telemetry of each type is collected for each system request, often stored in different databases, and is analyzed across many different tools: RUM, APM, logging, profiling, tracing, exemplars, alerts, SLOs, dashboards, etc. In this implementation, there's a fundamental separation of signal types. In other words, there is nothing connecting the data from tool to tool except the user who sits in the middle and must correlate between them.

In the age of generative AI, that user can now be artificial. Some vendors will even join all of these separate data types into a data lake, centralizing them in a common data store, and going so far as to insert artificial metadata between telemetry data points to hard link a correlation. But in any of those cases, the same fundamental problem remains: different tools collect different measurements and summaries of the exact same system conditions, only to later attempt making correlations between those differences.

In observability 2.0, you remove the degree of error introduced by artificial correlations. Instead, all system telemetry is collected as the same data type—a “wide event.” Every collection point gathers wide events, all wide events are written in a singular format to a singular data store, and every type of analysis tool reads from that one source of truth.

What exactly is a wide event?

In short, it is an arbitrarily wide set of structured key value pairs written as a JSON blob. In other words, they're very large structured logs.

{ "time":"2025-01-22T11:57:03-07:00", "level":"INFO", "authority":"10.0.0.3:63349", "duration_ms":123, "msg":"Served HTTP request", "path":"/super/slow/server", "port":80, "service_name":"slowsvc", "status":200, "trace.trace_id":"eafdf3123", "user":"foo" }

At write time, every measurement of your systems and applications is stored with all of its raw observed context complete: nothing is summarized, discarded, or artificially correlated. Each measurement contains a valuable level of context by default, but you will also derive even more value from all of the additional instrumentation your engineers add to append relevant business context.

Later, during any given investigation, you decide at query time if you need to see a summary of system state at a given time (i.e., a metric), a granular level of specific detail (i.e., a log), or an interconnected series of events (i.e., a distributed trace). You can get an aggregate view of overall system state, a detailed view of one specific system event, or any level of detail in between—all from the exact same data.

Observability 2.0 users have a completely reliable correlation experience between metrics, logs, and traces because there is no correlation to make.

Wide events typically include standard “out-of-the-box” data and additional “custom” data. For example, a wide event for a web service might contain standard data such as timestamp, source and destination IPs, proxy names, proxy handlers, method handlers, protocol, endpoint, status code, requested resource, object size, request length, user-agent, latency, and more.

Typically, your engineering team might want to see additional custom details, such as internal code variables used to make the request, user ID, shopping cart ID, device type, device ID, build ID, language pack, language internals, environment variables, raw database queries, normalized queries, the latency and contents of each of those queries, the latency and contents of any API request, the entire contents of /proc, every feature flag and its setting, or anything else that could possibly be useful in a later investigation. The more data you capture, the better!

Using metrics for their intended purpose

We should be clear: the cost problem in observability is not because metrics are bad or because these tools are overpriced. As anyone who regularly works with data can tell you, exploding costs and a high level of engineering overhead are just what happens anytime you use the wrong tool for the job. If you use the wrong data model, you suffer.

The cost crisis in metrics tooling has many visible aspects:

High bills
Price shocks and unpredictability
The maintenance tax on engineering teams tasked with cost management
The complexity involved in defining and curating metrics
The heavy lift of teaching every engineer how to contort their telemetry into the peculiar mental model of metrics and tags

But there are also deeper and far less visible costs. The absence of contextual data is connective tissue that can't be graphed because it doesn't exist. Its absence means engineering teams are reduced to inefficiently guessing, taking stabs in the dark, and relying on the experiences and tribal knowledge of your most senior engineers. The crisis in metrics tooling is less visibly experienced with statements like:

Why does it take us so long to figure out what happened?
Why can't we move faster?
Why don't we know which users are affected?
Why are customers reporting issues before we knew about them?

Metrics are great at what they do—cheaply and efficiently summarizing vast quantities of data. Because they're stripped of context, no individual metric can ever be connected or traced back to another metric from the same event, request, or session. Any relational data and context is discarded at write time. Context is what gives data meaning and power.

Metrics are the correct primitives to use in certain use cases, such as infrastructure—gathering statistics on low-level operating systems, drivers and hardware, or measuring performance on high-throughput devices like switches and network routers. In any system of sufficient scale, there are use cases where metrics collection can suffice.

Right now, most engineering teams use metrics-backed tools as a primary means to understand software and systems, and structured logs for a niche subset. That ratio needs to be reversed.

A cost model for wide events

If what you need to capture in a wide event is every possible bit of context, then it doesn't make sense to penalize users for capturing it. Many log-based observability tools charge you per GB of data ingested. In that model, if every single captured event was arbitrarily large, then your observability bill would also become arbitrarily large.

Observability 2.0 works because storage is cheap. Your observability tool's pricing model should align to incentivize collection of arbitrarily wide events. For example, Honeycomb charges on a per-event ingestion model. Therefore, whether your structured logs are each 1kb, 10kb, 100kb, or 1000kb in size, they each cost you the same exact amount to store and analyze.

Similarly, observability 2.0 also works because compute power is cheap and plentiful. At query time, your processing of those wide events must return fast results. If you collect billions of arbitrarily wide events, but it takes your query several minutes to return results, that also presents a user disincentive. For example, Honeycomb does not impose any additional fees to index data for faster performance: all data is always fast and queries across billions of rows comparing thousands of fields always return results within seconds, at no additional cost.

The Honeycomb pricing model isn't the only method for making observability 2.0 workable, but we present it here to demonstrate how incentives must align between engineering needs, observability use cases, and vendor pricing.

In this model, you can have as many “custom metrics” as you want, at no additional cost. You can have as much cardinality as you want, at no additional cost. You can have as many fast query results as you want, at no additional cost. Because Honeycomb charges based on the number of structured log lines (or “events”) and not on their size, that means you can make them as wide as you want—in fact, we encourage it!

If any detail might someday be valuable to you or your engineers, append it to a log line. It costs you nothing!

Later, you can ask questions that use that data, or about any other subset or combination of telemetry data and get answers quickly, at no additional cost. For example, you could decide to compute latency percentiles, min and max, avg and mean, or any dimension or view needed—without needing to set it up in advance or pay additional fees to get those insights. In this model, you eliminate the unpredictable nature and skyrocketing costs of traditional observability tooling.

Similarly, the wide event model helps control costs by reducing how many times you pay to ingest the same data. Rather than paying 3x, 5x, or more to ingest logs, metrics, and traces across multiple tools multiple times, you only pay for data ingestion once.

Managing and predicting modern observability costs at scale

Similar to using metrics-based tools, the wide events model may also present challenges at scale. Earlier, we examined a wide variety of workarounds various engineering teams use to wrangle exponentially increasing costs when using metrics.

With wide events, your observability bills go up as you scale up (your traffic grows, you add more services, more spans, etc.). When you scale, the value you get out of your telemetry goes up too, because your ability to trace becomes ever more fine-grained and powerful. Costs become predictable: no surprise crazy spikes, no fretting over what data types or how many possible values each key-value pair has, and no more relentless maintenance slog for your engineers. It's liberating and reclaims efficiency.

For larger scale use cases, there are also additional optional techniques that can further control costs. With wide events, controls focus on managing event volume: how many events should you pay your observability vendor to store?

Telemetry pipelines

A telemetry pipeline collects, processes, and routes telemetry data from various sources in your application stack. It typically sits between telemetry sources (application code, infrastructure, etc.) and telemetry destinations (observability tools, archival storage, additional analysis tools, etc.). In the context of controlling observability costs, telemetry pipelines allow you to choose which data is or isn't worth paying a premium to ingest and analyze.

For example, your app might be configured to log all authentication requests in order to meet security compliance requirements. When considering application performance, your app might authenticate as the same un-permissioned system user when issuing all service requests. In other words, knowing which user authenticated wouldn't help isolate performance issues, but it is useful for compliance reporting purposes.

A telemetry pipeline helps you route low-value observability data to destinations like inexpensive cold storage (e.g., Amazon S3, Azure Blob Store, etc.) or a compliance reporting tool, and saves costs by not sending it to your observability vendor. It reduces the amount of data you pay premiums for, while improving the quality of observability data and making it easier to analyze. Since they act as an intermediary collector, telemetry pipelines can also process telemetry bound for an observability platform in any number of ways, including masking sensitive data, appending additional tags, or making routing decisions based upon telemetry contents.

In a wide events model, charging solely on the number of ingested events, observability cost control becomes straightforward: decide which data is high-value, use a telemetry pipeline to route low-value data elsewhere, and measure/forecast high-value data volume to predict your total costs.

Sampling

Another cost mitigation technique is using telemetry sampling to refine your data collection. Past a certain scale, the cost of collecting, processing, and saving every single event in your system might outweigh the benefits. For example, let's say your app stack generates 10 billion events per month, all with the status message 200 OK. Do you really need to pay for all 10 billion of them? Or would it be just as effective to store 1 in 10 as a representative sample of every 200 OK event so that you know the actual shape and behavior of your apps, yet pay an order of magnitude less to store that knowledge?

Making intelligent sampling decisions is key for ensuring observability performance. In other words, it's generally best to make dynamic decisions about which data can be sampled based on several factors like current traffic volume, error rates, or telemetry content. Be selective and targeted with which events are okay to sample (e.g., 200 OK) and which should always be captured in full fidelity (e.g., 500 Internal Server Error). Indiscriminate, static, and one-size-fits-all sampling strategies often introduce more costs (via business impacts, like extended service outages) than they save.

Talk to your observability vendor to understand their approach when enabling sampling and how they let you control event volume without sacrificing analysis capabilities. Do they enable solutions for intelligent fine-grained controls, or are the techniques simplistic and indiscriminate?

Putting it all together

To illustrate concretely, we'll look at Honeycomb's approach to making your costs predictable while creating more value.

Honeycomb starts with data you already have—vendor-specific agents, OpenTelemetry, or various other data sources. Together, we build toward a telemetry data strategy that extracts usefulness and value for your teams, considering their ever-increasing data needs—while keeping costs manageable. Honeycomb's pricing model is simple: we charge based on the number of wide events you send per month, regardless of how large those events are. Our platform integrates with your existing stack, so you can collect, transform, and manage telemetry data without disruption.

With Honeycomb Telemetry Pipeline, your platform team can implement simplified data flows and operations, while your software developers gain peace of mind knowing their data is retained and easily accessible during on-call situations. That means you can focus on extracting meaningful insights while also controlling costs. Prioritize the structured logs that matter for real-time insights and send archival copies of less valuable logs to affordable storage, so you only pay for what matters.

We offer solutions like Refinery, an intelligent sampling proxy used by our largest customers to improve the value they get from their telemetry. Refinery adjusts sample rates to account for the volume of incoming data—prioritizing the capture of rare events over common occurrences.

Honeycomb also includes burst protection, a billing mechanism designed to mitigate cost surprises from unexpected bursts of traffic. Honeycomb calculates your average daily event volume and then, a few times a month, does not count events that exceed that limit. For example, if your daily event average is 30 million events, and today (within one day) you send 90 million events to Honeycomb, we do not count the excess 60 million events against your bill. Burst protection smooths out surprises from unexpected traffic surges.

We should note that Honeycomb's approach is not the only way to achieve the predictably techniques outlined in this whitepaper. But it does help illustrate how these concepts are put into practice in modern observability tools.

This packaging example illustrates ensuring predictable costs, but what about simultaneously providing more value to your business? To see that, let's look at real customer experiences.

Modern observability outcomes

With traditional observability tooling, engineering teams often flounder in the dark matter of imprecision and uncertainty—searching for answers, making best guess correlations between summary data, with any useful conclusions relying on the mercy of whatever insights the team predicted they needed in advance. But when full context is stored at write time and query time results are fast, the engineering experiences and business benefits this unlocks can feel downright magical. Listen to how customers describe that experience.

Duolingo stands out as a beacon of innovation and user engagement in the world of digital language learning. With millions of users worldwide, their platform is designed not only to teach languages, but also to create a fun and engaging learning experience. Duolingo manages vast amounts of data and user interactions daily. Experiencing rapid growth, the company was committed to delivering high-quality user experiences.

By switching to a modern observability platform with Honeycomb, the Duolingo team drastically sped up incident investigations and resolutions. Previously, when incidents arose, engineers had to piece together information from different sources in order to stitch together a narrative of the situation, often leading to long delays in resolution.

“We use BubbleUp a lot and are big fans. Everybody's favorite is the incident response dashboard—it's so fast and easy to investigate. More often than not, it's the go-to resource that lets us click straight into the trace we need, and we have our answer.”
— David Amin, Staff Site Reliability Engineer, Duolingo

Duolingo consolidated down from three separate tools into one single, powerful solution with Honeycomb. The Duolingo team reduced incident investigation times by an order of magnitude, accelerated feature release cycles, and deepened engineering team collaboration capabilities. They also report streamlining and reducing custom metrics by 30 million, resulting in a 16% savings in total observability tooling costs.

Getting started

We dived into a few Honeycomb-specific implementation details. But these concepts are vendor neutral, so let's take a step back.

You don't have to undergo a giant migration or change observability vendors just to stop hemorrhaging so much money, time, and engineering cycles. By now, chances are that you already know where to start. If metrics-backed tools are the foundation of traditional observability, then structured logs are the bridge to a modern approach. There are steps you can take today to start investing your engineering cycles more wisely.

Regardless of your interest in Honeycomb, if you're interested in building toward observability 2.0, there are a few places to start:

Structure your logs, if you haven't already.
Start shifting time, money, and engineering cycles away from proliferating metrics-backed tooling and shift toward logs and tracing.
Consolidate your logging into fewer, wider events, aka canonical logs. The wider the better. Context is everything.
Put OpenTelemetry on your engineering roadmap. This is the best way to defeat vendor lock-in and make sure any labor investments you make are reusable.
Limit investment in developing observability dashboards and interfaces unless they're exploratory, ones that will allow engineers to zoom in and out, up and down, or follow a trail of breadcrumb data to find answers.
Start pressing your vendors to consolidate data sources instead of charging you over and over to store telemetry in lots of disconnected data formats.
Invest in Service Level Objectives (SLOs) instead of monitoring for symptoms and triggering alert bombs. Debugging should start by monitoring service conditions that are relevant to your business. Make sure your SLOs are constructed from the same data your teams use to debug, instead of being yet another bespoke data source.

Observability 2.0 tooling consists of a single source of truth, the ability to visualize anything from a point in time event to a full end-to-end trace within the same interface, and quickly analyze any subset of data across as many dimensions as you like, containing as much high-cardinality data as you need to answer your questions.

You'll find many articles, talks, and threads on ways to reduce data cardinality, or how not to generate high-cardinality metrics to control debugging performance and cost. The problem with that approach is that high-cardinality data is the most important, useful, and identifying data you can capture. When your team sacrifices high-cardinality data, you sacrifice your ability to understand your software and customers in ways that are relevant to the business.

Remember

While you can derive metrics and summary statistics from wide events, the reverse is not true. Having the connective tissue of context in a tool where you can slice and dice, explore, and share has a tremendously democratizing effect, empowering everyone in your teams to become the best engineer in every corner of your systems. Our case studies are replete with that transformation repeatedly heralded by real teams shifting to observability 2.0.

With structured logs, the more dimensions you add, the wider each log line gets, and the more powerful your ability to correlate and tease out outliers becomes. The more densely and richly you can describe the experience your users have with your software, the greater your ability to pluck out rare conditions and describe unusual events.

It's incredible how such a small thing—formatting your data in a slightly different way—can unlock such sweepingly powerful sociotechnical waves of change. But it can, and it does.

Still in doubt? Let us show you how.

Ready to get started?

Start for Free Book a Demo