The Bridge From Observability 1.0 to Observability 2.0 Is Made Up of Logs, Not Metrics

Observability tooling can be loosely grouped into two generations: 1.0 and 2.0. If you've heard the famous saying that “observability has three pillars: metrics, logs, and traces”—that's observability 1.0 for you. Observability 2.0 tools, on the other hand, have one unified source of truth: arbitrarily-wide structured log events.

If you'd like to read more about the technical distinctions and sociotechnical ramifications between the generations, I've written extensively about it here. At a glance:

Observability 1.0 tooling vs. Observability 2.0 tooling

• How it's stored — 1.0: In many tools, in multiple formats: metrics, logs, traces, RUM, APM, profiling, tracing, structured and unstructured logs. 2.0: In wide, structured events, sometimes called canonical logs, with trace/span IDs. From this single unified source, you can derive any subset of metrics or logs, and you can visualize them over time as a trace.
• How you use it — 1.0: You need to know in advance what to search for, or which metrics to collect; data is aggregated at write time. Dashboards are configured in advance. 2.0: Exploratory and open-ended. You gather detail prolifically and store raw events; data is aggregated at read time, allowing you to slice & dice at will.
• Cost model — 1.0: You pay for many tools, so cost skyrockets; as cost goes up, value goes down. 2.0: You pay to store your data once. As cost goes up, value goes up.
• Use case — 1.0: Good for known-unknowns, and simpler systems that fail repeatedly in predictable ways. 2.0: Good for unknown-unknowns, and complex systems that fail in chaotic and unpredictable ways.
• Cardinality — 1.0: Cardinality (aka “custom metrics”) is prohibitively expensive at scale. 2.0: Handles high-cardinality and context-rich data with ease.

Observability (or “o11y”) 2.0 tools, like Honeycomb, Lightstep, and Baselime, are built off a different data model than o11y 1.0 tools are. Instead of having multiple pillars in many different data formats, they have a single unified source of truth. They are easier, faster, and more cost effective; they require far less ongoing engineering toil to maintain.

Instrumentation requires fewer painful tradeoffs between how valuable the data is and how costly it is to store or query. There are no limitations for custom metrics or cardinality. They are, however, admittedly much younger and less mature than o11y 1.0 solutions.

Giants of the o11y 1.0 generation, like Datadog and Prometheus, have billions of dollars in market capitalization and a user base of tens of millions. The tooling is robust and mature; you can Google for any set or subset of technologies and instantly receive a command to copy-paste that will generate a bunch of shiny graphs. They are fully featured and well understood by engineers at large. These are terrific tools, with literally hundreds of engineering-years of development and expertise behind them, and if your systems tend to fail in predictable ways, you can probably afford to stick with them.

The future of telemetry looks like observability 2.0

However, for many of us, o11y 1.0 tools are on an unsustainable trajectory in terms of cost and complexity. The cost multiplier is unyielding when you have to pay to store every single request in many places, in multiple formats. The costs multiply down the line, as engineers are forced to jump from tool to tool—metrics, to logs, to traces, to profiling, to RUM, to APM, to dashboards, to SLOs—using educated guesses to fill in the blanks.

For most of us, the complexity of our systems continues to skyrocket, meaning that trying to solve our problems by guessing and jumping from dashboard to dashboard is just not a viable solution. Our systems no longer fail in predictable ways, and just about every time we get paged, it should be about a novel issue (which then gets fixed). This is the only way to scale the operational load of increasingly complex systems; by eliminating noise and improving system resilience. The future of telemetry looks like o11y 2.0 because the future of complex systems is one where you have a long, thin tail of exceptionally rare states that almost never happen, except for when they do.

These models are incompatible—you can't have both one source of truth and many, you can't build for both observability 1.0 and 2.0. Therefore, the burning question on many minds right now is this: how do we get there? How can we go from where we are right now, in our respective multi-pillar worlds, to the glorious future of observability 2.0 where costs are predictable and aligned with value, not an exponential factor of our request growth—and where rich, detailed context about user experience doesn't break the bank?

Ultimately, this is a question about how we're going to equip our engineering teams to grapple with many successive waves of change. Observability 1.0 may have been centered around bugs, errors, and downtime, but o11y 2.0 is the cornerstone of fast feedback loops. It's what unlocks every team's ability to move swiftly, with confidence, because they're able to understand the consequences of their code and make course corrections instantly.

It's like putting your glasses on before you go whizzing down the freeway. The velocity of each engineering team is limited by how quickly and precisely they can understand the consequences of their code on their users. For complex reasons both technical and sociotechnical, the architecture of o11y 2.0 tooling unlocks the capability for teams to collectively move much faster than they could with o11y 1.0 tools.

All of this is increasingly well understood and widely discussed by industry leaders and observability experts writ large. The question is thus less about whether or not o11y 2.0 tooling will emerge, and more about how do we get there?

First steps, next steps, and where we're going

In this piece, I want to unpack the underlying shifts in the landscape. I want to help you understand the changing role played by metrics, and the rise of rich, context-dense structured data.

I want to equip you with some tools to help identify which tools and technologies are actually a sucking black hole that will slowly slurp away your time and treasure, and which tools and technologies will be force multipliers as you look to the future building happy, efficient, consistently high performing engineering teams.

And then I want to give you some concrete, actionable steps you can take to start making your telemetry more valuable and cost-effective.

All of the advice I will give here is completely vendor neutral, and none of it is hard. It can be done incrementally, bit by bit. You just need to have a vision for where you're going and why. Fortunately, every step along the way will pay off in making your systems more usable and understandable, which makes it easier for engineers to move fast and collaborate with each other. You can do this!

Especially as AI-enhanced software becomes more and more of a thing, it becomes even more important that we ensure our code can be readily understood by the engineers who are responsible for it.

Metrics have dominated software telemetry since its inception

It's hard to overstate just how dominant metrics have historically been when it comes to the tools we use to understand our software and our systems. Name a market leader or a well-known tool—Datadog, Stackdriver, Chronosphere, Prometheus, Nagios, Graphite, Ganglia—chances are, it was built on metrics.

RUM tools were built using metrics. APM tools were built using metrics. Metrics have powered the last thirty years of monitoring tools and technologies. We've ridden the metric bus about as far as we can possibly take it… and maybe then some.

It's not hard to understand why metrics have reigned supreme. For most of the past 40 years, hardware was exceedingly costly; metrics are tiny, fast, and cheap. Architecture, meanwhile, was relatively simple and static. Our systems tended to fail repeatedly in predictable ways, so it was possible to predict up front which metrics would be useful, and define your dashboards, queries, indexes, etc in advance.

Metrics are good for many things

Over the years, I've talked a lot of smack about the limitations of metrics, but metrics are far from useless. They're actually an incredibly powerful tool in the data universe. Metrics are the right tool for the job when used for:

• Summarizing vast quantities of data
• Helping you age out the granularity of your data over time, to retain shapes and summaries while managing costs (à la RRDtool)
• Counters, gauges, and other distinct data types
• Extremely high traffic volumes
• Infrastructure

In most cases, metrics are most powerful when they are derived from richer data. SLOs are the ultimate example of this. An SLO is typically a composite metric that your teams agree represents a good-enough user experience, like: “the number of query requests that return successfully in under 15 seconds”. An SLO is a powerful tool, and it is especially powerful if you can click on it and zoom in and interrogate it further. Like: “of the queries that violated this SLO, how many returned successfully in 15-60 seconds? How many returned immediately with a 500, 502, or 504? Of the queries that violated this SLO, were they dominated by any one particular app, user, or normalized query?”

If you can't derive your metrics from richer data that you can then link to and explore, they are less valuable to you. They still have some value, especially at extremely large scale (though typically much larger than people assume).

But in 2024, it remains the case that most companies use metrics to power most of their telemetry workloads—maybe 80/20 metrics/structured logs, maybe 70/30. There will always be a place for metrics in the data ecosystem. But my belief is that for most companies, these ratios should invert.

Rich, structured data is the workhorse for modern systems, and metrics are the niche tool—not the other way around.

A lot of this shift can be traced back to the receding importance of infrastructure in most engineering organizations' baskets of use cases, and the emerging dominance of application telemetry.

Metrics are the right tool for infrastructure

When I first got started in tech, every software company had a dedicated ops team and rented out space in a data center. I literally remember calling a cab to take me to the colo at 3 a.m. to flip the power switch on a MySQL primary. (Times were tough!)

Every tech company was an infrastructure company, because they had to be. It makes sense that metrics emerged as the dominant data type from this era. Not only because hardware was expensive, but also because metrics are the natural utility player for infra monitoring.

I think this holds true today: metrics are still the right tool for infrastructure monitoring. It's just that infrastructure is becoming a smaller portion of what each tech company needs to care about. Infrastructure is becoming a specialist discipline, one that almost every company ends up outsourcing to AWS, Google, and a variety of third-party providers, which frees them up to spend the majority of their engineering cycles and focus on their own software, the code they write.

Here's how I define infrastructure:

Infra is the software you have to run in order to run the software you want to run.

The software you want to run is the software that defines you as a company—your crown jewels. Every engineering org draws this line somewhat differently, based on factors as varied as your business differentiators, how things have historically been done, and the talent you have in house (more and more companies are using “platform engineering” teams to own and encapsulate that line of abstraction).

And then, there's the code you have to run to support the code you build yourself. For some companies, this consists of a handful of libraries and a backing database. Other companies still run their own data centers. But just about every company I've ever heard of has had some category of software they don't write or own themselves, but nevertheless depend on. That's infrastructure—and infrastructure code has a very different set of criteria when it comes to instrumentation and telemetry than your crown jewels do.

• Who writes and owns it? — Your code: your own engineers. Infrastructure code: other people's engineers, as much as possible.
• Rate of change (how often do you deploy it?) — Your code: days, hours, or even minutes. Infrastructure code: weeks, months, or years.
• How deeply do you need to understand it? — Your code: intimately—the success of your business is bound up in the quality and success of your software. Infrastructure code: honestly, as little as possible; good infra serves as an abstraction.
• How do you instrument it? — Your code: deeply, richly, so you can understand the unique experience of each user, and the delta of each change event. Infrastructure code: summary statistics.
• How much value do you derive from your investments in it? — Your code: hopefully a lot, although this depends on the team and the work. Infrastructure code: infra sits below the value line (it's a cost center).

To start with, the rate of change is very different. Your crown jewels may change every few minutes, hours, or days, while infrastructure code is typically updated on the order of weeks, months, or even years. The whole point of infra code is that you need it, but you don't want to have to mess with it. You only need it to exist and work well enough—it doesn't provide any core differentiators for you.

There's a value line here. In theory, the more time and energy you invest into your own code, the more value you can create for your users and your business. The more time and energy you invest into your infrastructure, the more… time and energy you've spent propping things up to be just good enough. Infrastructure is valuable, but it will always be below the value line.

The health of your system vs the happiness of your users

When it comes to your users, their experience matters. If you're an ecommerce site, shaving milliseconds off your latency can translate directly into millions of dollars. If you're a delivery company, you need to be able to reliably report the status of every single parcel in your system. If you're a payments API, you need to know exactly what happened to the money after each timeout.

This often manifests itself in errors, crashes, downtime, and bugs, but MTTR and MTTD are not the entirety of what observability is about. Observability is about making sure that your code is doing what you want it to do, and understanding how your users are interacting with it. You need to be able to see how each user is interacting with what you've done, what they're leaning into, what paths they decline to take, what brings them joy. These feedback loops are what give you confidence in your work, and help you decide what next steps to take—what to build more of, what to lean into, what to experiment further upon.

That's why we instrument our code. That's why context matters so much: so we can explore the unexpected and understand our code and our systems holistically.

The health of your system and the happiness of your users are two distinct concerns, and they aren't necessarily connected. Who among us has not shipped a diff that caused user experience to degrade across the board, or even made the system seem completely down for a subset of users, while meanwhile, on the backend, all server health statistics are a healthy green? The converse can happen too; it's possible for everything to seem fine to your users, all requests succeeding within the designated SLO range, even though processes are failing to restart, capacity is slowly shrinking, CPU is high, and memory is swapping.

User experience and system health can affect each other, but they are distinct concerns:

• For user experience, we require rich telemetry with lots of densely packed context, and we need it to be aggregated around the request so we can trace it and answer interesting questions about how it relates to other requests and patterns.
• For system health, we only need to aggregate around the health of the system, so we can answer simpler questions about whether it's “up” or “down,” good enough to serve the request or not good enough to serve the request. CPU statistics, memory % utilized, and so forth.

In a nutshell, this is why metrics are sufficient for infrastructure monitoring, but wide, traceable, structured log events are necessary for the code you own and write yourselves.

Metrics are numbers stored in time series, with no relational data

In engineering circles, we use the term “metrics” to mean two different things. The first is a generic, small-m metric, which is a synonym for any and all telemetry. The second is a specific, big-M Metric, which refers to a single number with tags appended to it, which can be a counter, gauge, etc., which is stored in a time-series database. When we say “metric” in this piece, we mean the latter type.

Each metric is a number with tags appended to it. It stores no relational data whatsoever. You may store hundreds or thousands of different metrics, but you have no way of detecting after the fact whether any given metric describes the same request or event as any other given metric.

Metrics-backed tools have no context, and therefore they grant you no ability to identify outliers or correlations.

You can't understand your systems without context

Metrics are tiny, cheap, and fast—you can't get much smaller than a single number.

Metrics are stunningly good at summarizing vast quantities of data, but they are not good at helping you understand complex systems. Understanding and exploring complex systems requires the ability to spot outliers and trends in the data, all of which requires context. Data is made valuable by context. And metrics can't store any context in time-series databases. In fact, you can't store any relational data at all in a TSDB.

You also cannot understand complex systems without high-cardinality data... All of these dimensions need to be not only stored, but capable of being queried, sliced and diced, combined, mix-and-matched, and zoomed in and out of. You need to be able to take a scalpel to your system—to ask, “When this thing happened, what else was happening?”

You can't understand your systems without high-cardinality data

The mathematical definition of cardinality says that it refers to “the number of unique items in a set.” High cardinality means lots of unique items; low cardinality means few unique items.

For example: if you have a collection of 100 million users, the highest possible cardinality will be any field with a unique identifier. Your social security number, for example. First, middle, last names will be high cardinality, but slightly less so, because presumably some people have duplicate names. A selector that allows you to pick status code would have a cardinality of 63 possible values.

Why does this matter? Because metrics can only handle low-cardinality data. But the higher the cardinality, the more useful and identifying the data is. You can't hope to understand complex systems with only low-cardinality dimensions with a few possible values. Consider the following:

• Container ID
• Shopping cart ID
• User ID
• App name
• Build ID
• Normalized query
• IP address
• Method
• URL
• Referrer
• Browser string
• …and on and on and on

All of them are very high-cardinality dimensions, and all of them are necessary for understanding even minimally complex systems.

Fighting cardinality is a constant battle with metrics-based tools. And even low-cardinality data (e.g., 100 unique values) gets prohibitively expensive really fast.

You can't understand your systems without custom metrics

If you're used to living in a metrics-backed world, you're probably obscenely familiar with the eternal tug of war over custom metrics: budget vs data, detail vs cost. It's not unusual for entire observability engineering teams to spend an outright majority of their cycles trying to manage the push-pull of cost vs custom metrics, trying to gather just enough detail to understand their systems, without so much detail that they go bankrupt, and trying to jump on spikes early when the shape of the data changes out from under them.

It's not unusual. But it should be. The term “custom metrics” is just a weird made-up vendor term for what it really means: cardinality.

Being numbers, metrics are also incapable of storing high-cardinality data—or rather, the price of doing so is extortionately high. You have to store every single combination of a metric + tag value in a unique time series. This is another way of saying that you pay for cardinality. When Datadog says you can have 100 free custom metrics, that means you can have 100 free cardinality. After that, you will pay for every new data value that enters your system.

Logs let you pack up rich context and relational state

The concept of “logs” is a famously messy one. It can refer to almost any data that comes flying out of any piece of software, in any shape or format. So we need to get clear about what we're trying to say.

For years, at Honeycomb, we avoided using the term “log” altogether. Why? When someone says “logs,” you assume a bunch of things about the data: that it's unstructured strings, that they are emitted willy-nilly through the execution of the request, etc. None of this is true of the requests we accept at Honeycomb, so we called ours “events.” In more recent years, the term “canonical logs” has been proposed to describe wide, structured logs that roll up lots of data emitted over the course of a request into one log line. Terrific.

The critical thing about logs is that they can emit multiple pieces of data, all bundled up together in one unit. A metric can only ever be a number, a timestamp, and some tags, but a log can have many things—host name, IP address, method, referrer, etc.—all joined together in a single unit to provide the context that describes a request.

A metric can only ever be a point on a line, but logs can build rich, deep webs of relationships between events. You can never correlate any two metrics, which is why debugging with observability 1.0 requires so many outright guesses and leaps of faith.

Metrics are a bridge to our past; logs are the bridge to our future

But not just any logs. When it comes to observability, yes, it is true that pretty much any logs will be better than pretty much any metrics. But you can make life tremendously harder or easier for yourself depending on how you structure your logs.

If there's one fundamental truth I hope you take away from my whitepaper, it's this: The bridge from observability 1.0 to observability 2.0 is made up of logs, not metrics.

Metrics are a bridge to our past—a past where the LAMP stack reigned, architecture was dead simple, systems failed in repeatable ways, and physical hardware was extremely costly. Infrastructure was a large percentage of the work every engineering organization had to do.

Metrics still play many important roles in the future—but they are not the workhorse of complex systems telemetry. Wide, structured log events are the way we get to a better tomorrow. If you want to get there, here are some practical tips.

Step 1: Begin reallocating resources away from metrics-based tools and towards logging-based tools

Your first step on the path to observability 2.0 consists of identifying all the time, energy, money, headcount, and engineering cycles you're currently pouring into the black hole of metrics-backed tools, and look for ways to begin reallocating them to tools backed by logs and structured data.

I know this can seem a bit daunting; the landscape is saturated with metrics-backed tools—RUM, APM, dashboards, monitoring, observability 1.0 tools, and so on. The market leaders are all metrics-backed tools. But they are a bridge to the past. The fact that they can't handle relational data, or high cardinality, or high dimensionality, or key-value pairs, or context, or anything that gives you the ability to identify patterns and correlations means they are simply a dead end. The cost curve alone is absurd—trying to store even medium-cardinality data will bankrupt you. Furthermore, it is really freaking hard:

• The amount of knowledge you have to have in order to instrument your code using metrics is absurd, and footguns are everywhere.
• The amount of expertise it takes an observability engineering team to manage cardinality and costs on an ongoing basis is also absurd.
• Many observability engineering teams already spend the outright majority of their time managing costs (and the cost vs cardinality tradeoffs) on a weekly basis.

The way you are doing it now is the hard way. It will shock you how easy it is to instrument and understand your code using observability 2.0.

Step 2: Structure your logs, if you haven't already done so

In 2016, when we first started asking around, very few teams had structured application logs. People knew they should have structured logs, and felt bad about themselves if they hadn't, but most teams still hadn't made the leap.

Happily, the industry has come a long way since then. Most teams these days have structured logs; in fact, a plurality of teams we talk to have started rolling out OpenTelemetry, which is fantastic!

Is OpenTelemetry a prerequisite for observability 2.0? No, it is not. There are lots of benefits OpenTelemetry brings to your org—first and foremost, the lack of vendor lock-in. You should only have to instrument your code one time and then observability vendors should compete for your business based on how amazing and effective they are, not how trapped you are behind their walled garden.

Step 3: Issue fewer, wider structured log lines

Read up on canonical log lines. The idea here is that instead of firing off lots of tiny niblets of data while the request is executing, you emit one log event at the end—right before the request is ready to exit or error out of the service—containing all the important information.

Instead of logging out something like this:

[2019-03-18 22:48:32.990] Request started http_method=POST http_path=/v1/charges request_id=req_123
[2019-03-18 22:48:32.991] User authenticated auth_type=api_key key_id=mk_123 user_id=usr_123
[2019-03-18 22:48:32.992] Rate limiting ran rate_allowed=true rate_quota=100 rate_remaining=99
[2019-03-18 22:48:32.998] Charge created charge_id=ch_123 permissions_used=account_write team=acquiring
[2019-03-18 22:48:32.999] Request finished alloc_count=9123 database_queries=34 duration=0.009 http_status=200

We might log something like this:

[2019-03-18 22:48:32.999] canonical-log-line alloc_count=9123 auth_type=api_key database_queries=34 duration=0.009 http_method=POST http_path=/v1/charges http_status=200 key_id=mk_123 permissions_used=account_write rate_allowed=true rate_quota=100 rate_remaining=99 request_id=req_123 team=acquiring user_id=usr_123

The fewer log lines we emit, the less we get charged. The wider those log lines are, the more powerful our ability to identify outliers and rare events becomes. If the request spans many log lines, we have to do some wizardry to reconstitute the data from each request into events before we can be sure that we're identifying the real true full context of each request. It's time consuming, painstaking, laboriously manual work. It's so much easier to do with fewer, wider, richer log events.

Tag each request with a unique request ID, naturally, and propagate it throughout your stack.

Step 4: Feed that stream into some visualization layer that supports high cardinality, showing traces over time, etc.

I'm not going to get prescriptive here. Obviously I have my own opinions on what the best observability 2.0 tooling out there is. But you can feed wide, structured logs into all kinds of tools, from open source log aggregators to Tableau. The point is, once you have your data structured in this way, you have a single source of truth per request.

From that wide, structured log event, you can:

• Derive metrics and dashboards.
• Derive SLOs.
• Derive traces and display events over time.
• Identify outliers and rare intersections of events.
• Find out immediately when five impossible things happen at once.

You don't have to worry about whether you pre-configured your telemetry to calculate and store the 99th, 99.5th, 99.9th, 99.99th, and 99.999th percentiles… you can simply calculate the percentile you want over the time range you need.

You don't have to worry about whether or not you defined a metric for the specific intersection of 504s, GETs, and requests that hit the /payment endpoint… you can simply gather all the status codes, http methods, and http urls, then slice and dice to get the specific combination you need.

It feels like magic, let me tell you.

A trace is a log is a span is a log is a trace is a…

What's the difference between a trace and a log? Nothing, necessarily.

In the past, logs were mostly unstructured and traces were highly structured, so these were stored separately. But a trace is just a group of structured logs, correlated by trace ID, which can be visualized over time. A span is just a structured log with a trace ID and a span ID, which helps you preserve relationships and ordering between events.

There is no good reason for you to be storing rich, structured telemetry about every request that enters your system into two different buckets, “logs” and “traces.” It's expensive, wasteful, and incredibly confusing. Having multiple buckets like this is somehow worth less than the sum of its parts.

OpenTelemetry is doing the Lord's work for the world by standardizing the idea that your logs can be correlated by traces, thus allowing you to group all this related data together instead of dispersing it into separate, disconnected buckets.

• You should be able to slice and dice your telemetry, zoom in to the level of raw requests, zoom out to the big picture… and visualize it over time as a trace.
• You should be able to derive your SLOs from this data. You should be able to derive and store arbitrary summary metrics about your user requests and experience, which you can then age out over time.
• You should be able to visualize your infrastructure health metrics—the distinct set of metrics that are aggregated around the health of the system, rather than around the user's experience—as an overlay across your telemetry.
• You should also be able to append any of your infrastructure metrics (CPU, mem, etc.) to each wide, structured log trace event, so you can include that data in your slicing and dicing.

Even if you can carefully pre-define join points that allow you to jump back and forth between logs, metrics, and traces, as some providers will allow you to, it's still objectively worse (especially for your budget!) than having all that context bundled together, correctly structured.

Wide, rich logs have a storied history

Honeycomb was founded in 2016. We spent a lot of time over the first year or two messing around with how to recommend people instrument their code, and eventually arrived at something very similar to what Brandur Leach dubbed “canonical logs” circa 2019. Years later, I found out that Amazon had actually been using this concept for debugging their own AWS services for years.

Amazon calls them “request logs,” which makes them impossible to search for. But I suggest spending some quality time with their documentation on how they do log lines internally, because there is some seriously good stuff here. For example:

• Emit one request log entry for every unit of work
• Emit no more than one request log entry for a given request
• Break long-running tasks into multiple log entries
• Record details about the request before doing stuff like validation
• Synchronize clocks
• Emit zero counts for availability metrics

And under “what we log…”

• Log the availability and latency of all of dependencies
• Break out dependency metrics per call, per resource, per status code, etc.
• Record memory queue depths when accessing them
• Add an additional counter for every error reason
• Log important metadata about the unit of work
• Avoid putting overly sensitive information in logs
• Log a trace ID and propagate it in backend calls
• Log different latency metrics depending on status code and size
• Include the corresponding request ID

One of the places I differ with their guidance is that I don't think it's worth building in support for log levels or more verbose warning logs. If you're using the right backend, it should be effectively free to append more dimensions to your existing log events (Honeycomb charges by number of events; it's free to add more and more dimensions to your existing log events, because we want to incentivize people to append lots of context). If it's costly for you to append more dimensions, you might have to do that, but I think it sends you down the wrong path. Try not to have log levels.

Context should be cheap; that's kind of the whole point. You don't know what you're going to need to know in the future, so you should append anything you think might someday be useful.

Huge thanks to David Yanacek for writing these awesome docs!, and to David Behroozi for another excellent piece of writing on application logs vs request logs.

Observability 2.0 is so much faster, easier, and cheaper than 1.0

We have stopped “seeing” all the energy, time, and money we sink into getting our metrics-backed systems to keep limping along, because we have collectively been using these tools for the past three decades. It's hard to unsee. It's hard to see how hard it is to use the tools we use now, because we've been using them for so long. But the costs are still being paid—by engineers, by managers, by customers.

The hidden costs of metrics-backed tools are paid every time someone…

• watches yet another talk, or reads yet another article on “How to Try Not to Have High-Cardinality Data in Your Telemetry”
• accidentally adds a super valuable tag, like container_id or build_id, which then costs the company $50k over a single weekend
• blows an afternoon trying to track down which build_id is when the errors started happening… something they could have identified instantly with context-aware data
• manually pipes logs into Tableau or MySQL or a spreadsheet (yes, really) just to be able to identify outliers, or ask questions like, “What happened to the 20% of users whose requests got 20% slower?”
• gets paged on a weekend, because they couldn't interrogate a trickle of errors from their last deploy until it eventually took the entire site down

These costs get paid every single day. The cost of all the things we can't see are far bigger than we realize, and skyrocketing.

Your ability to move swiftly, with confidence, is rooted in the quality of your observability

The complexity of our systems is ratcheting upwards faster than ever. These days, any time the pager goes off, it's likely to be something completely different. Every time you deploy your code, you face a complex intersection of services, user patterns, infrastructure, business problems, and the code you're trying to write to solve them. You need a scalpel to pick it apart and understand what's happening.

If you want to protect your team from complexity spiraling out of control, you have to invest in your observability. You have to look for opportunities to switch away from metrics-backed tools towards logging-backed tools, structure your data, and emit fewer, wider logs.

Toss all the context into your logs. Anything that you think might be useful someday—toss it in. Stuff it in S3 if you must—just put it somewhere in its raw, unaggregated original format. Someday, it could very well save your ass. Your future self, and generations of engineers to come, will thank you.

“Honeycomb makes it effortless to understand our code and extract the necessary information. This empowers our team to iterate faster, complete tasks, and avoid the frustration of digging through logs to find that needle in the haystack. Honeycomb enhances developer satisfaction and quality of life by giving us a single source of truth for our observability and allowing us to rapidly pinpoint issues. These intangible Honeycomb benefits are significant and have helped truly transform our dev approach.”
— Michael Garski, Director of Platform Engineering, Fender