OpenTelemetry makes it easy to produce and transmit any type of telemetry. In production environments, this often means deploying the OpenTelemetry Collector as an intermediary to process, enrich, and route telemetry data. As systems scale, so does this infrastructure—sometimes to hundreds or thousands of Collectors spread across environments. The difficulty swings from collecting telemetry to managing the telemetry infrastructure: keeping configurations in sync, understanding agent health, rolling out changes safely, and knowing what’s actually running in production all become operational challenges.

That’s the problem the OpenTelemetry community set out to solve with the OpenTelemetry Agent Management Protocol (OpAMP). OpAMP is a vendor-neutral, OpenTelemetry-first specification for managing telemetry agents at scale. It defines how agents report their status and capabilities, how they receive configuration updates, and how management systems maintain a control plane over running agents.

In this post, I’ll explain what OpAMP is and why OpenTelemetry created it, then dive into how we use it to power Honeycomb Telemetry Pipeline.

Why did OpenTelemetry Create OpAMP?

OpAMP is strictly a specification that defines how agents, like the Collector, communicate with a remote server. It defines a control plane—it is not a specification for creating or transmitting telemetry.

Before OpAMP, the Collector could be managed via a few, limited ways, such as Helm or the OpenTelemetry Operator. However, both of these solutions stopped at deploying the Collector; they did not interact with the running Collector to provide any feedback loops on its status, nor did they update it.

OpAMP exists to provide management to running agents, like the Collector. The community wanted an open-source, OpenTelemetry-first solution for agent management that could apply to the Collector (or any other agent). OpAMP enabled the Collector to support remote configuration reloading, health and configuration reporting, version discovery, connection management, package management, and future extensibility. And since it is not Collector-specific, it can apply to any agent—more on that later.

How does OpAMP work?

OpAMP is made up of two parts: a server and a client. The server is the manager; it acts as the control plane orchestrating all the clients. The client is the agent in the data plane, like the Collector.

(sourced from https://opentelemetry.io/docs/collector/management/)

Communication between the client and the server always starts with the client contacting the server to announce its presence. Typically, this connection is upgraded to WebSocket, but the specification defines how the server and client can communicate over plain HTTP as well.

Once the connection has been established, the client and server are free to send each other messages at will.

This free communication model is what empowers many features in OpAMP. The client can report to the server and ask it for information. For example, it is expected that the client reports its health, effective config, remote config status (whether it successfully applied or not), available components, but the client may also ask for connection settings. Similarly, the server may offer the client new configuration, settings, or packages and it may also pass commands, like a command to restart.

What types of actions the client or server can take in response to a message is determined by capabilities. In messages, both client and server always tell each other what features they support. If the client connects to the server and does not specify the remote configuration capability, that means it should not send it any remote configuration messages. If any messages are sent that conflict with the client’s or server’s capabilities, they may be ignored.

In the end, OpAMP is really a set of instructions for how to send messages back and forth between clients and the server.

With those rules in mind, let’s review how Honeycomb implemented OpAMP to facilitate agent management in Honeycomb Telemetry Pipeline.

How Honeycomb implemented OpAMP’s client and server

OpAMP is integral to the Pipeline Builder in Honeycomb Telemetry Pipeline. When customers install the pipeline installation connected to Pipeline Builder, it’s OpAMP that lets us distribute configuration to all their clients, and it is OpAMP that lets all the clients report statuses and their usage back to Honeycomb.

For the server, we utilized OpenTelemetry’s OpAMP Go implementation in a custom application named Beekeeper. This application knows how to behave as an OpAMP Server, how to communicate with Honeycomb, and contains all the business logic for how we manage the pipeline installation remotely.

For the Collector, we used the OpAMP Supervisor to turn the Collector into an OpAMP client. It’s part of the OpenTelemetry Collector SIG and is the community’s current solution for how to use OpAMP with the Collector.

But the most interesting piece is how we’re using OpAMP with Refinery. Refinery is open-source, but not part of the OpenTelemetry project. We needed it to behave as an OpAMP client when run as part of Pipeline Builder. One of the best parts of OpAMP is that it’s not specific to any one piece of software, or even to the OpenTelemetry project. This meant that we were able to implement OpAMP support in Refinery, making it compatible with any OpAMP server, not just our own. If you’re looking for a good non-Collector client example, check out Refinery.

OpAMP powers the management layer of Pipeline Builder’s pipeline. It allows us to distribute configuration changes, observe agent health in near real-time, and understand how telemetry is flowing through a customer’s pipeline—all without requiring customers to manually redeploy or reconfigure agents.

By building this on top of OpAMP, Pipeline Builder can manage a heterogeneous fleet of Collectors and Refinery instances using a single, standardized control plane.

Conclusion

OpAMP provides the missing control plane for operating telemetry infrastructure at scale. By standardizing how agents are managed, it makes it possible to safely configure, observe, and operate large fleets of telemetry agents.

We chose OpAMP instead of writing our own proprietary protocol for the same reason we tell all our customers to use OpenTelemetry: we believe OpenTelemetry is the backbone of good observability. By choosing OpAMP, we get to use a system specifically designed for managing Telemetry Pipelines—and specifically designed for use around OpenTelemetry.

We’re already all in on OpenTelemetry. We believe in using and contributing to OpenTelemetry, which is why we contribute to OpAMP as an end user. We continue to engage in the community because we believe being invested in OpenTelemetry means better observability for anyone who wants it. As telemetry pipelines become increasingly critical infrastructure, having a shared, open control plane like OpAMP is key.