How OpenTelemetry and Semantic Telemetry Will Reshape Observability

OpenTelemetry has a problem. It’s a subtle one, admittedly, but it’s holding you back from realizing its true value and potential. The problem with the problem, though, is that it’s a hard problem to even see. I think a lot of this is because OpenTelemetry’s mission is unhelpfully broad. “High-quality, ubiquitous, and portable telemetry to enable high-quality observability” is the stated goal of the project, but what does that even mean?

I’ve spoken with thousands of developers over the past five years as OpenTelemetry has grown from an idea to an emerging standard, and many of them would give me different answers and perspectives on what OpenTelemetry is. To some, it’s a convenient way to get distributed tracing data out of their microservices. To others, it’s a framework for building internal observability pipelines. Others still focus on integrating its API into their libraries in order to provide metrics and logs to their end users.

These are all a part of what OpenTelemetry does, but they don’t really get at what it is. In this whitepaper, I’m going to step back and reveal the real purpose of OpenTelemetry, the causes and solutions to the problem that it presents, and offer a rubric for evaluating observability platforms and their support for OpenTelemetry.

Defining how systems speak

Logically, telemetry is a way for a software system to express what it’s doing to a human operator. This is something you’ve probably experienced in your career—from your first ‘Hello, World!’ to your millionth alert('ran to here'), you’ve used telemetry to better understand what’s actually happening in your code. Without telemetry, we’re quite literally in the dark. The only way to understand how a piece of software is working is to look at its outputs, and the end user-facing outputs usually aren’t that useful when you’re trying to track down a bug or improve system performance across thousands or millions of users.

OpenTelemetry is inspired by the trend in cloud-native software systems over the last ten or so years of defining how things are done, independent of their exact operation. What do I mean by this? Let’s take a look at two examples: Kubernetes and Amazon S3.

Kubernetes does a lot of things, but I think if you asked most people, they’d tell you it handles container orchestration. It helps you run an application by handling common concerns, like resource limits and restarts. It makes sure that the code you give it is running as reliably and efficiently as possible by allowing it to automatically scale up or down. Most importantly, though, is that it provides a universal API for doing all of these things, along with a set of semantic concepts like pods, nodes, or ingresses that aren’t tied to one specific service provider or underlying technology.

Similarly, Amazon S3 is a blob storage service that allows for efficient and inexpensive storage and transfer of files. Its API is, effectively, universal as well—many other companies and providers offer their own blob storage services that use the S3 API, and S3 semantics, so that you can easily move from one to the other based on your needs.

With this in mind, let's recast what OpenTelemetry is doing for you. Rather than just providing a way to create telemetry, its mission is to define a universal standard about how systems speak. It’s not enough to just give developers and operators a convenient API for writing their own telemetry, we need to provide easy answers to what telemetry is important and how that telemetry should be represented. The real end state of OpenTelemetry is one where it’s mostly invisible, because it’s all under the surface, silently providing a scaffold of semantic data that represents the work being done by software services.

What is semantic data, and why is it important?

Modern software is complex by nature. Even traditional n-tier web applications, if deployed to the cloud or using managed services, rely on a wide variety of external and internal dependencies for storage, database, caching, and more. The organizations that build and maintain these applications are also increasingly complex. Distributed teams, AI-assisted code generation, and other novel forms of work have reshaped the way we conceptualize a ‘software organization.’ In order to build and operate software—not just at scale, to operate it at all—we need a lingua franca of what the software is doing, and how it’s built. In addition, we need this framework to exist independently of a single language, tech stack, cloud provider, or runtime. Modern software is polyglot, even if the part you touch is not.

Historically, developers have built the telemetry data they need to understand their systems ad-hoc. The level of abstraction for telemetry, generally, was at the provider or author level—whoever was running your code would provide logs, metrics, and traces for it, potentially with some added telemetry about your business logic that you created. If you built on a framework, then the framework authors would generally provide logging and metric facades for you to plug into. Large enterprises and other software-focused companies often build their own frameworks that standardize this process, more or less.

The problem with the status quo is that this is the wrong level of abstraction. The people that can write actual descriptive telemetry about what their code is doing are the people who write the code. Offloading this to downstream consumers leads to inconsistent, and often incompatible, semantics. If I write a web server and decide to add structured logging, but pick different names for the same concept than you, then our telemetry is not semantically compatible with each other. These collisions aren’t limited to concept names; if I decide that my server should emit a counter metric for connections and yours only emits logs, then our telemetry semantics are also incompatible. These collisions are responsible for a great deal of pain in the industry.

When telemetry collides, consumers have to make a decision. Do they exist in a split-brain world, where you have to look at metric foo for service x, but logs bar for service y, to understand the same thing? Generally, no—this is silly. However, one of these options needs to win. This results in sprawling, complex, and brittle pipelines that read and parse telemetry data, reformat it, and emit it for monitoring and observability purposes. These conversions decrease the availability of telemetry data, and are frequently lossy—you can’t reverse them to get back the original data, unless you preserve both (which is often viewed as wasteful, especially considering the volume of telemetry data a system can emit).

Similarly, when names collide, work must be done to normalize them or to teach engineers about which name means what where. This stymies more advanced data analysis as well, as inconsistent names across services means you can’t really aggregate across that name over your entire system. It forces you to look at services in isolation, and lends itself to highly siloed analysis methods.

Semantic telemetry aims to address this in two primary ways:

1. It provides a highly structured telemetry schema that codifies the structure of signals themselves.
2. It provides semantic conventions for common metadata attached to those signals.

What’s important to consider is how these are not isolated elements of a telemetry system. Consistent schemas for data are useful, but without consistent metadata, that utility extends only to ingestion and analysis. Conventions for metadata are useful, but without a consistent schema it’s only useful for indexing, aggregation, and contextual links. You can almost think of it as the difference between trying to communicate in a foreign language by focusing just on comprehending grammar or words. While communication is possible, it’s hardly fluent.

The tradeoff triangle in observability

All system design is, at the end of the day, about tradeoffs. Observability systems are no different. There are three competing priorities in this case:

Availability — How much telemetry data is preserved for analysis, and how detailed is it?

Recency — How long does it take for telemetry data to become analyzable, and how do you store it?

Speed — How long does it take to query and analyze the data, or be alerted to issues it reveals?

Fundamentally, all observability tools must balance these priorities by trading between them. When you focus on one, you invariably trade away the others. For example, you may desire to maximize availability by storing every piece of telemetry emitted by your system. Doing so requires you to determine tradeoffs between recency and query speed. Trying to maximize all three is nearly impossible, especially at scale.

Computers and databases are very fast, but large organizations generate on the order of hundreds of petabytes of logging data each day; the cost of maximizing all three of these axes would be astronomical. It’s important to note that none of these things are ‘bad’ or ‘good’ on their own. It’s almost always the case that you will want to throw away noisy or purely duplicative data; health check logs usually aren’t interesting, for instance. Crash reporting from mobile clients can be delayed due to network connectivity issues, but your ability to act on them is limited by many factors (e.g., App Store review times), so it’s less of a problem if data isn’t up-to-the-second accurate. Query speeds need to be fast when you’re firefighting an incident, but less so when you’re analyzing trends in CI/CD pipelines over months.

What is consistent, though, is that these three tradeoffs impact your entire observability system, its design, and how well it serves you and your customers.

How does semantic telemetry address these tradeoffs?

Part of the reason that this triangle exists is because, historically, we have had limited tools available to control the inputs to observability systems. Different languages and technologies have concomitantly developed their own ideas and implementations of logging or metric exposition formats, their own ideas about attribute naming, and even their own ideas about what type of telemetry certain system components should emit, and why.

If you’re a big enough company, or have deep pockets, perhaps this isn’t a concern. You can afford to hire and staff an observability engineering team that writes instrumentation for your entire system and normalizes third-party telemetry data to meet your specifications. If you’re a commercial provider of observability tools, then you as well can hire instrumentation engineers to write collection agents or instrumentation libraries that emit normalized data. It does leave a lot to be desired, though, if you’re not in either of those camps—and, really, isn’t this data a commodity anyway?

Semantic telemetry starts from a fairly straightforward assumption. During operation, a software system will emit events. There are countless numbers and types of these events: locking a mutex, making a database call, receiving an HTTP request, writing a file, and so forth. In isolation, these events are not terribly interesting. However, each event is part of something bigger—a request, a job, an API call. Those abstractions turn into larger, more complex abstractions: adding an item to a shopping cart, or calculating the number of products in a warehouse. This is the act of software development in a nutshell, solving problems for people in code by breaking those problems down and solving them.

Developers understand this loop, because it’s at the core of what they do. It stands to reason, then, that the developers of software are the best-placed people to describe what those events mean through telemetry.

Semantic telemetry gives developers a schema for how to translate these events into signals. These signals are things like metrics, or traces. A signal is self-describing, and self-documenting: its definition defines its scope, its boundaries, and its properties. Signals utilize a store of semantic conventions used for annotation and definition of common properties across a type of event. For example, all HTTP servers have some commonality in how they respond to an incoming request. All database clients have some shared concept of what a ‘query’ is. Semantic conventions give us a way to express those shared properties across language, framework, runtime, cloud provider, and on.

This combination alone is extremely powerful, because it reduces the amount of duplicate telemetry data a process emits. It allows developers to define, through code, the functionality and behavior of their framework or library. It also offers solutions to our tradeoff triangle: you can patch up tradeoffs by leveraging other signals. For instance, you can improve availability by ensuring that your signals are layered, giving you 100% accuracy and availability of high-level pre-aggregated metrics while also ensuring that lower-level signals (such as logs or traces) are preserved, since you know what telemetry is being emitted and what it means.

This opens the door to a dramatic reshaping of the observability landscape—one where less time is spent on re-implementing commodified telemetry data and more on building optimized databases, visualizations, and analytical tools on the back of semantic telemetry data.

OpenTelemetry is semantic telemetry

If you look at the design and priorities of OpenTelemetry, it should become clear how these ideas are intertwined. The OpenTelemetry API and Protocol provide consistent, vendor/platform-agnostic universal standards for how to translate events into signals. In addition, OpenTelemetry is not only defining semantic conventions for metadata, but tools and registries to extend those conventions.

The other motivating reason for OpenTelemetry’s design, though, is to provide correlation and context. If all we needed, as an industry or developers was a specification and standard, then we wouldn’t be in the situation we’re in today with dozens of proliferating, incompatible quasi-standards. In order to leverage semantic telemetry, you need to be able to accurately and concretely define the relationships between telemetry signals, the system components that emit them, and the events they represent. Remember, signals are just abstractions over work being done in your software—but only being able to correlate those abstractions on dimensions like time or host name means you’re still stuck digging through a proverbial haystack looking for needles.

OpenTelemetry’s context layer allows for explicit links to be made between disparate signal streams, which is a boon to observability tooling. Rather than having to guess at what’s interesting, you can leverage features like exemplars or trace context in logs to answer questions like, “For this specific transaction, what were the associated logs?” or “For my p99 API performance on this route, which non-error traces were emitted, and what did they have in common?”

One of the difficulties that a lot of people have with this vision, though, is that it’s not an easy one to grasp. The data alone does not give you the ability to ask these sorts of questions. It is necessary, but not sufficient.

When you get down to it, OpenTelemetry is just a bunch of wide events

Honeycomb, at its core, has long advocated for the adoption of ‘wide events.’ A wide event is mostly what it sounds like: an event with many arbitrary fields. The Honeycomb storage and query engine are optimized to process this sort of data. Often, people assume this means Honeycomb can’t handle metrics, or logs, or other sorts of semantic telemetry. This, however, is not accurate.

Indeed, OpenTelemetry signals are nothing more than wide events with a schema. Take, for example, the wire representation of a metric in OpenTelemetry. Each message contains a collection of resources that apply to all metrics in the report, then the scope(s) of those resources, then a list of measurements and the attributes that apply to them, along with each data point for the reporting interval. These reports can be decomposed into many smaller events, written to our column store, and then queried and plotted just like they could be from a time series database. We’re hardly the only people to realize this—open-source column stores have begun to add similar optimizations around time series data.

That covers the shape of the data, but what about its contents? This is another area where OpenTelemetry provides tools and options to ‘widen’ your data through semantic conventions. Now, as I said above, the ones that the project defines are intended to cover the scope of the project itself; describing how software systems interact. However, there are a lot of other conventions for your software that are going to vary from organization to organization, team to team, project to project. OpenTelemetry is also building tools and techniques that you can use to define these conventions and scaffold them into your existing telemetry.

Let’s take a straightforward example: a customer identifier. It’s almost certain that you have one way (or perhaps many) to identify users in your software, but there is some canonical representation. Perhaps it’s a UUID, perhaps it's an email address, perhaps it's some arcane string that’s been preserved for years due to tech debt and legacy systems. OpenTelemetry’s semantic convention tooling allows you to define a single constant and provide migrations to and from this data through configuration, then apply that configuration to existing sources of telemetry in order to normalize this data.

In addition, you can use these user-created conventions as part of new development work, ensuring consistency across both old and new services. Since these schema files can be processed programmatically, they can also be used for discovery of emitted telemetry, cataloging of what services produce what, and documentation purposes.

This is a core piece of the OpenTelemetry ecosystem, albeit one that is mostly ‘behind the scenes.’ However, these tools do exist today, and thinking about how they interact with your telemetry and observability practice is the difference between using and truly adopting OpenTelemetry, semantic telemetry, and observability.

Why traditional tooling isn’t enough

To unlock the value of semantic telemetry, tools must rise to the challenges and opportunities that they present. This is where the current thinking and design behind proprietary—and even open-source—analysis tools falls apart. Let’s discuss a few examples.

Semantic telemetry via OpenTelemetry allows you to drastically increase the availability of telemetry data for analysis by way of distributed context, as mentioned earlier. What this requires, though, is a different approach to well-worn topics such as sampling. Traditional tail-based sampling approaches generally require processing of each message for a given signal—for example, buffering and processing all spans in all traces in order to preserve a fraction of the whole. This approach is highly inefficient, though.

OpenTelemetry proposes, and enables, a model where tail-based samplers can utilize different signals to indicate sampling decisions. For example, utilizing a metric signal (such as a histogram of API latency and errors) to preserve a class of traces, regardless of the trace metadata itself. This approach also allows for unique approaches to sampling reservoirs and buffers. Rather than having to buffer a significant chunk of telemetry in-memory, higher resolution data can be stored closer to its source and then fetched just-in-time in response to sampling decisions. This helps ameliorate the tradeoffs between availability and recency, at the cost of query speed. However, such approaches can result in drastic cost savings.

Telemetry design and implementation is also aided by this semantics-driven approach. Rather than requiring internal teams to build and maintain instrumentation frameworks, a config-driven approach that builds off telemetry schemas can allow an implementer to understand where existing instrumentation exists, such as in gRPC or HTTP libraries, and then modify that telemetry thru parsing rules or custom code injection. This also can benefit downstream consumers of telemetry data, by allowing them to override or otherwise transform telemetry as it passes from producers to consumers. Rather than having to adapt to different distributions or wrappers around OpenTelemetry, analysis tools can ship config files that perform the necessary transformations required for dashboards or other workflows to function properly.

What is shared between all of these scenarios, though, is a recognition that the tools that we have today are insufficient for these purposes. Embracing this vision requires not just a different way of thinking about observability and telemetry data, but a commitment to engage with the OpenTelemetry community and build new functionality.

Let me be expressly clear for the people in the back: these tools don’t exist yet. It’s a chicken-and-egg problem. If someone is trying to sell you an ‘OpenTelemetry-native’ platform, you need to dig in and see if they’re really committed to building new functionality on top of the innovations that OpenTelemetry provides, or if they’re just selling you warmed-over “three pillars” style observability with no new answers.

How do you gauge which tools are in the best place today to successfully navigate this transition and transformation? I think there’s a few guiding principles:

• Is the tooling agnostic to siloed, signal-based workflows?
• Does the tooling provide the ability to quickly query and analyze data across many dimensions?
• How involved is the organization in open observability, and how are they contributing to OpenTelemetry?

These are not necessarily simple questions to answer, but if you’re thinking about the long tail of observability, they’re important ones.

The next five years are going to look very different from the last; be prepared, stay curious, and always move forward. OpenTelemetry has demonstrated that telemetry data itself is a commodity, while observability is anything but.