The Cost Crisis in Metrics Tooling

In my February 2024 piece The Cost Crisis in Observability Tooling, I explained why the cost of tools built atop the three pillars of metrics, logs, and traces—observability 1.0 tooling—is not only soaring at a rate many times higher than your traffic increases, but has also become radically disconnected from the value those tools can deliver. Too often, as costs go up, the value you derive from these tools declines.

This blog post struck a nerve. I heard from observability teams who spent the last year doing nothing but grappling with cost containment. I heard from engineers who scoffed at my anecdote about individual metrics that cost $30,000 per month and relayed hair-raising tales of metrics they shipped that each cost tens of thousands of dollars over the weekend.

I also received many questions. This material is dense, and not widely understood. In this companion piece I will take a slower, deeper dive into the cost models and tradeoffs involved with metrics-backed tooling, since that remains the load-bearing pillar of most teams’ toolkits. I’ll also make the argument that metrics are a niche power tool in our arsenal.

Metrics are a (very) mature technology

When it comes to monitoring software, metrics-backed dashboards have long been the state of the art. Metrics are cheap, fast, and sparse, and the technology is decades old, so the tooling (and integrations) are extremely mature. Most APM and RUM tools are built using metrics primitives. Tools like Datadog, Prometheus, and Chronosphere are what most engineers reach for to understand their systems. The mental model for using metrics is not especially intuitive, but for historical reasons it is by far the most widely understood.

Teams are accustomed to instrumenting their software with metrics, deriving alerts from metrics, and using metrics-backed dashboards to debug their code. Logs get used for debugging, but they’re too unruly and expensive to use as a jumping-off point, and traces are perceived as too niche, expensive, or heavily sampled. At most, teams have learned to jump from SLOs to dashboards, to logs, to traces, visually correlating data by timestamps and the shape of spikes or copy-pasting IDs from tool to tool.

It strikes me as odd that metrics are still the dominant data type for systems and application data. In every other part of the business, our tools are backed by columnar stores or other relational databases, because we understand that data is made valuable by context.

In my opinion, the present state is a holdover from the days when engineering was seen as a cost center. I believe a shift is already well underway, bringing systems and application data into the relational fold—and I believe it is picking up speed thanks to soaring costs and the relentless explosion in underlying system complexity.

What exactly is a metric?

The term “metric” is commonly used in two different contexts:

• The “metric” is a data type, a number with some tags (or “labels”) appended, which can be stored in a variety of formats, such as counters, gauges, and histograms.
• “Metrics” is also often colloquially used as a generic synonym for telemetry data.

Metrics (the data type) are traditionally stored in a time series database (TSDB), which is a collection of data points (numbers) gathered by time. The only type of index a TSDB has is an index by time, and the only type of queries you can run are point queries by time and range queries over time. It stores no relational or contextual data whatsoever.

All aggregation is performed at write time, not query time, including buckets like averages, 95th%, 99th%, 99.99th%, etc. If you query for the 99th percentile latency across your fleet of app services, you get an aggregate of aggregates of the locally-computed 99th percentile latency over a rolling window across all instances. If you want to query for the 99.95th% latency, or the 85th% latency, or any other latency that you did not compute at write time, you cannot.

When you install an agent, like StatsD or DogStatsD, it automatically ingests many system stats and churns out pretty graphs of CPU, memory, disk space, and the like with very little manual work. But nearly all the practical value you derive from these tools will come from instrumenting your code with custom metrics.

What is a custom metric?

Much like “metric,” the term “custom metric” has a colloquial meaning as well as a specific technical meaning. When an engineer talks about adding custom metrics to their code, they are typically conceptualizing each line of instrumentation as a custom metric. Which is why, when a metrics provider says you get a couple hundred custom metrics for free, that sounds like a lot!

Unfortunately, that’s not how time-series data works. When metrics are stored in TSDBs, every unique combination of metric name and tag values generates another distinct time-series, also known as a custom metric. This gets a little complicated, and can vary by implementation or backing store.

Let’s take a simple example from the Datadog custom metrics billing page.

Calculating the footprint of an example metric

statsd.increment('request_latency.increment', tags=[f'endpoint:{endpoint},status:{code}'])

Let’s say you submit a metric, request_latency, from five hosts with two tag keys, endpoint and status. You only monitor four endpoints on this tiny application and track two status codes, 200 and 500, and you decide to submit it as a count metric.

That comes out to a footprint of 40 custom metrics for this metric: 5 hosts * 4 endpoints * 2 status codes.

Now, let’s say you operate at a moderate scale. Your app runs on about 1000 hosts, and you monitor 100 endpoints, or 5 methods and 20 handlers. There are 63 HTTP status codes in active use.

We’re already up to 6.3 million custom metrics (1000 hosts * 5 methods * 20 handlers * 63 status codes) and the only thing we can do is a simple count of requests broken down by host/endpoint/status code. Oof. Let’s keep going. Counts are nice, but latency is what we’re trying to measure.

statsd.histogram('request_latency.histogram', random.randint(0, 20), tags=[f'endpoint:{endpoint},status:{code}'])

If you submit request_latency as a histogram or distribution using nothing but the default aggregations max, median, avg, 95pc, and count, that’s 31.5 million custom metrics. You also want to compute the 99th, 99.5th, 99.9th, and 99.99th percentiles, right? Well, for every percentile bucket you want to compute at write time, you add another multiplier. If you want to store 10 buckets instead of 5, that’s a footprint of 63 million custom metrics.

A Datadog account comes with 100-200 custom metrics per host, depending on your plan. For every 100 ingested custom metrics over the allotment, you pay ten cents. That means you’d pay $63,000 per month just to collect barebones HTTP latency statistics. Keep in mind, we haven’t even tried to tag our metrics with anything really useful yet, like build ID or user ID.

Costs are hard to predict, and harder to connect to value

One of the challenges with metrics is that calculating your metrics footprint is hard to do in advance, and may change out from under you. Engineering teams rely on policy documents, best practices, and expert code reviews to control costs, only to get bitten by seemingly unrelated changes made by infrastructure teams—or even autoscaling.

In my previous example, you have 1000 hosts and 20 handlers. Think about what happens to your bill when:

1. Your infrastructure team moves your app tier from 1000 xlarge EC2 instances to 4000 on-demand containers
2. Your on-call needs to roll your entire app tier a few times inside of an hour, causing several thousand EC2 instances to spin up briefly before dying
3. You deploy some new code that adds versioning for each handler
4. You change the value for histogram_aggregates or histogram_percentiles in your YAML config file, not realizing it will apply to ALL histograms
5. You auto-generate a tag based off an AWS instance tag, which changes overnight to a different string format

For example, in #1, your bill quadrupled to $252,000/month without a single line of application code changing, and without any change in server-side capacity. How can it be so easy to accidentally quadruple your bill while making it 0% easier to understand or debug?

High costs are a problem, yes, and so is unpredictability. But the worst part is when costs are so untethered from value. When your bill goes up, it should be a function of scaling up capacity and/or making your software and systems easier to understand.

How experienced teams control costs

Any experienced member of an observability engineering team is probably bouncing up and down by now, fully irritated by some of the naive and profligate ways I have gone about my cost estimation. I arrived at my $63k/month estimate for request_latency by generating a unique time-series for the intersection of each host name, handler, method, and status code. Experts would not have done it this way!

• Status code buckets. Engineers versed in metrics do not generate a time-series per unique status code. They generally collapse status codes into five families—1xx, 2xx, 3xx, 4xx, 5xx—to cap cardinality. People adjust to this, but it is not ideal. It really matters whether you’re seeing a 502 or a 504. That’s valuable information.

Sometimes, a weird status code combined with a request string or an originating IP is all the information you need to debug an extreme outlier event. Smoothing over status codes is a cheap and easy upfront hack, but for large, complex distributed systems, it ends up costing you in the long run.

• “Superfluous” tags and buckets. Why are we paying to store AVG latency or MAX? Do we really need to store 99, 99.5%, 99.9%, and 99.99%? Is median useful to us? Storing some subset of these buckets is valuable, but nobody pays to store all of them, especially not broken down by every single endpoint or method and handler.

Do we really need to be able to check the latency of every endpoint or handler? Maybe some of them can be sacrificed on the altar of budgetary constraints. Reaping tags and pruning buckets is one of your best tactics for controlling costs. This means constantly trying to remember which data has been useful recently, and predict what you think you can live without.

• Host count. Metrics experts know all about the traps of host tags—in fact, this is how most of us first learned about cardinality. Everyone starts off tagging metrics with hostnames. It works great until you have about 100 hosts, at which point you run out of free tags and your bill starts to skyrocket.

To counteract this, engineers have devised a ton of clever hacks and workarounds. Running larger instances, tagging with a host type prefix, scrapping hostnames entirely, jumping to your logging tool, etc. Most people aren’t shelling out millions of dollars just to keep their host tags.

Experienced observability engineers would not have done things the way I did, but that’s kind of my point. An enormous amount of expertise gets front-loaded into creating metrics, because footguns abound.

Managing your metrics bill is a full-time job

There are many well-known practices for controlling costs with metrics data. There is an even longer list of intricacies and techniques leveraged by experts, trying to walk the line between the ability to answer the questions that matter on one hand, and going bankrupt on the other. A very partial list of these are:

• Deleting metrics, deleting tags
• Deleting anything that hasn’t seemingly solved a problem within recent memory
• Prepaying for capacity or committed use
• Polling less often, or expanding the window size
• Disabling a lot of the defaults, especially for consolidated platforms
• Setting ingestion volume controls
• Setting caps on burst capacity
• Configuring alarms to give you a heads up if your custom metrics footprint explodes
• Configuring your retention policy
• Disabling agents on some fraction of hosts or containers (metrics are pooled across hosts)
• Ingesting all metrics, but only indexing a subset of them
• Doing fancy tricks with streaming windows to shrink the number of active time-series in order to “support” high-cardinality metrics

Some of these are fairly straightforward, but others are quite challenging or lead to deeper issues. For example, most people eventually end up having to sacrifice tags like container ID, hostname, or host ID. But if you drop that label, you now face collisions and duplicate records in your TSDB. You’ll need to decide whether to drop one (the lower or higher?) or merge them using sum, mean, average, etc. You need a pretty deep understanding of your data and use cases at this point.

And all of these techniques eat up a substantial amount of ongoing engineering labor. None of them are “set and forget.” From what I hear, it’s normal for observability engineering teams to spend more time managing costs and cardinality than writing libraries or abstractions, consulting with teams, or otherwise adding value to the organization.

To some extent, all you reasonably can do is watch closely and react fast. Maintaining your metrics bill is a sizable, neverending tax on engineering teams.

Using metrics for their intended purpose

I want to be clear that this is not because metrics are bad or because these tools are overpriced. As anyone who works with data can tell you, exploding costs and a high level of engineering overhead is just what happens anytime you use the wrong tool for the job. If you use the wrong data model, you suffer.

Metrics are great at what they do, which is to cheaply and efficiently summarize vast quantities of data. Metrics are a mature technology, and decades of top-tier engineering labor have gone into building rich, featureful, easy-to-use metrics-backed tools. If you use them to summarize system data, they are quite cost effective.

If there exists such a thing as a general purpose, universal data type, it would be wide, structured events. Metrics are a niche format. They are a specialized tool for specialized use cases.

The hidden drag on engineering teams

The cost crisis in metrics tooling has many visible aspects:

• High bills
• Price shocks and unpredictability
• The maintenance tax on engineering teams tasked with cost management
• The complexity involved in defining and curating metrics
• The heavy lift of teaching every engineer how to contort their telemetry into the peculiar mental model of metrics and tags

But there are deeper costs, ones we have to struggle to see at all. The connective tissue that doesn’t exist and can’t be graphed, the absence of which means engineers are reduced to inefficiently guessing, stabbing in the dark, and relying on the memories of the engineers who have been there longest.

This is the dark matter of engineering; the answers to frustrating, unanswerable riddles like, “Why does it take us so long to figure out what happened?” and “Why can’t we move faster?”

This is one of the most challenging things to try and explain to engineering teams who have never worked with richer tooling, with telemetry data that can support and illuminate fast feedback loops and scalpel-precise understanding of the code you write. It’s a bit like Plato and the Allegory of the Cave, or (less dramatically) trying to explain to someone with poor vision what it’s like to wear glasses. But I’ll try.

The hard, hard limits of metrics

A fundamental, definitional fact about metrics is that each one is a number stripped of its context. No individual metric can ever be connected or traced back to another metric from the same event, request, or session. All that relational data and context got discarded at write time:

• You can’t compute a new percentile bucket on the fly
• You can’t ask for the request latency for a particular user
• You can’t ask if latency has gone up in the build ID that just got deployed, or if the outliers are all the requests coming from us-west-2 or hitting a particular MySQL secondary
• You can’t slice and dice or explore your data, or zoom in or out or up or down

When we say that monitoring and observability 1.0 tools are about known-unknowns, this is what we mean. You have to predict and gather every possible combination or permutation of conditions you may someday want to ask in advance, collect those metrics, and then plot them on static dashboards. If you knew the question in advance, if you could predict it, you can ask it. If you can’t, you’re out of luck.

You can’t predict what data you’re going to need. It is not possible. Once upon a time, our systems failed in more or less predictable ways, but those days are long gone.

If there’s anything more important than metrics themselves, it’s the connectedness between them. Context is what gives data meaning and power.

The observability 2.0 cost model is very different

The difference between observability 1.0 tools and observability 2.0 tools begins with how the data gets collected and stored on disk. Observability 1.0 famously has three pillars—metrics, logs, and traces—and stores the telemetry data for each request across many different tools: RUM, APM, logging, profiling, tracing, exemplars, dashboards, SLOs, etc, with nothing connecting the data from tool to tool except the poor engineer sitting in the middle.

With observability 2.0, you have a single source of truth: arbitrarily-wide structured logs, either one per request, per service (canonical logs), or per trace span. Tracing is not a separate tool—you just have the ability to visualize the same events over time.

You can have as many “custom metrics” as you want, at no additional cost. You can have as much cardinality as you want, at no additional cost.

With Honeycomb, at least, we charge based on the number of structured log lines (or “events”). You can make them as wide as you want—we encourage it! The wider your events, the richer your context, and the more powerful and magical your ability to pluck outliers and correlations out of the air.

As your bill goes up (because you scale up, or you add more spans), the value you get out of your telemetry goes up too, because your ability to trace becomes ever more fine-grained and powerful. Costs are predictable: no crazy spikes out of nowhere, no fretting over what data types or how many possible values each key-value pair has. It’s liberating.

Store everything you want. For web services, consider starting with source and destination IP, proxy name, proxy handler, user identifier, user ID, user name, auth string, timestamp, method, handler, endpoint, status code, object size, protocol, requested resource, HTTP protocol, request length, latency, shopping cart ID, raw queries, normalized queries, internal variables, the entire contents of /proc, the latency and contents of every query, HTTP or API request or other network hop, language pack, device type, device ID, build ID, language internal stats, user-agent strings, language internals, environment variables, every feature flag and its setting, and on and on. Pack it in. Pack it all in!

If a detail occurs to you that might someday be valuable, append it to a log line. It costs you nothing! Then ask any question you want, about any subset or combination of telemetry data. Do prefix searches and regex searches. Compute percentiles, min and max, avg and mean—whatever you want.

Structured logs are the bridge to observability 2.0

We just dove into a bunch of Honeycomb-specific implementation details, but these concepts are vendor-neutral, so let’s take a step back.

You don’t have to do a giant migration or change observability vendors just to stop hemorrhaging so much time and engineering cycles. Chances are, you already know where to start. If metrics-backed tools are a bridge to our past, structured logs are the bridge to our future. There are steps you can take today to start investing your precious labor more wisely.

For those of you who aren’t Honeycomb customers or Honeycomb-curious, but are interested in building towards the future of observability 2.0, here’s where to start:

1. Structure your logs, if you haven’t already.
2. Start shifting your time, money, and engineering cycles away from metrics-backed tools and towards logs and tracing.
3. Consolidate your logging into fewer, wider events, aka canonical logs. The wider the better. Context is everything.
4. Get OpenTelemetry on your roadmap. This is the best way to defeat vendor lock-in and make sure your labor is reusable.
5. Only invest in dashboards and interfaces that are exploratory, ones that will allow you to zoom in and out, up and down, or follow the trail of breadcrumbs.
6. Start pressing your vendors to consolidate data sources instead of charging you over and over to store telemetry in lots of disconnected data formats.
7. Invest in Service Level Objectives (SLOs). Instead of debugging from pager bombs and symptoms, your debugging should start here. (Make sure your SLOs are constructed from the same data you use to debug, instead of being yet another data source.)

Observability 2.0 tooling consists of a single source of truth, the ability to visualize over time as a trace, and as many high-cardinality dimensions as you want. You can derive metrics and summary statistics from wide events; the reverse is not true.

Having the connective tissue in a tool where you can slice and dice and explore and share has a tremendously democratizing effect, bringing everyone up to the level of the best engineer in every corner of the system.

High cardinality is not the enemy! It’s your friend and ally

There are so many articles, talks, and threads on how to reduce your cardinality, or how not to have high-cardinality metrics. There’s just one tiny problem with this: high-cardinality data is the most important, useful, and identifying data of all.

When you sacrifice high-cardinality data, you sacrifice your ability to understand your software.

With structured logs, the more dimensions you add, the wider each log line gets, and the more powerful your ability to correlate and tease out outliers becomes. The more densely and richly you can describe the experience your users have with your software, the greater your ability to pluck out rare conditions and describe unusual events.

It’s incredible how such a small thing—formatting our data in a slightly different way—can unlock such sweepingly powerful sociotechnical waves of change. But it can and it does.

The sociotechnical consequences of better tooling

Engineering teams founder in the dark matter of imprecision and uncertainty—but when that connective tissue gets preserved, when you use the right data tool for the job, the experiences this unlocks can feel downright magical.

People don’t yet widely grasp that observability 2.0 tooling is a prerequisite or a force amplifier for most transformative modern software engineering practices.

For example, let’s look at feature flags. You can use feature flags with metrics and observability 1.0 telemetry, and you will benefit from doing so. You can use feature flags to deploy new code to production without turning it on for everyone at once—awesome!

But feature flags with observability 2.0 tooling—high cardinality-data, wide events, rich context, traceability—is next level. You can instrument every request with each feature flag and its setting as a key-value pair. You can break down and group by the value of any individual flag, or combination of flags—or combination of flag(s), userids, environments, build IDs, etc.—and examine the precise effects of each flag variable. You can tell at-a-glance whether turning the flag on resulted in higher latency, different error codes, more compaction, different user inputs or behaviors—whether anything at all changed.

This is what it takes for engineers to move swiftly, with high confidence. The ripple effects of the right tooling are massive and self-amplifying. Other commonplace consequences are:

• You can inspect the impact of your deploys, canaries, and feature flips with scalpel-like precision, down to the differences in individual requests.
• Most teams end up deleting 90-95% of paging alerts while increasing customer happiness.
• You actually find problems before your customers do. Reliably.
• You can find problems with AWS before AWS does. Reliably.
• Costs are predictable and aligned with value.

When the connective data exists in a tool that everyone can access and explore, instead of locked away in the heads of a few senior engineers, anyone can be a master debugger.

Is there a role for metrics in the modern stack?

Of course there is. Metrics are, and will always be, the correct primitives in certain use cases, such as infrastructure:

• Low-level operating systems, drivers and hardware.
• Super high-throughput devices like switches and network routers.

In any system of sufficient scale, there will be some use cases where only metrics will do. And metrics will always play a starring role in aggregating and aging out data.

But right now, most teams use metrics-backed tools as a primary means to understand software and systems, and structured logs for a niche subset. That ratio needs to be reversed.

Metrics are a bridge to the past

Metrics aren’t going away anytime soon, but they are a bridge to our past. Metrics were dominant when systems were simple and the cost of storage was high, neither of which have been true for a long time.

I urge observability teams to acknowledge how much time and energy they pour into the work of curating and culling metrics, and how little they get in return. Every day, they go to war with your data model. None of the work does a thing to move the needle forward on their business objectives, and it holds them back from adopting or fully leveraging other modern software engineering practices.

Software engineering teams that can truly, effortlessly understand their code are in a better position to do everything else they want to do. It’s time to cut our losses.