Mix & Match! Tracing Header Interoperability Between OpenTelemetry and Beelines

We’ve released support for tracing header interoperability in all of our Beelines. This means you can now mix and match distributed services instrumented with Beelines or with OpenTelemetry, and your traces will be preserved in Honeycomb!

Instrumentation: You Have Options

Since their introduction, Beelines have been a popular way for people to instrument their applications and send data to Honeycomb. Beelines provide auto-instrumentation and automatic tracing support with minimal code required. Tracing can be used to visualize a customer journey through your application, and in the case of microservices and other distributed architectures, across service and process boundaries. When one service makes a request to another, the Beeline will wrap the outgoing call and insert a header that includes information used to assemble spans into traces on the Honeycomb backend. This all happens automatically when using a supported framework and HTTP client.

Beelines aren’t the only way to instrument an application. Since the project started, we’ve been fans and active participants in the OpenTelemetry project, a collaboration among vendors and the open source community. If you’re new to it, OpenTelemetry is a collection of open source tools, APIs and SDKs used to export telemetry data from your application. There are SDKs for a large number of languages and auto-instrumentation for a lot of popular frameworks and libraries. The project is a huge benefit to engineers who want to add instrumentation to their applications, and we know a lot of Honeycomb customers love it. As the OpenTelemetry project gets closer to general availability, we’ve been working hard to make the Honeycomb experience better for people who use OpenTelemetry to instrument their applications.

Some Context About Trace Context

Some Honeycomb users have expressed interest in either migrating completely to OpenTelemetry, or having a mix of Beeline and OpenTelemetry instrumented services. Migrating tracing libraries can be a headache - if the two libraries don’t share the same wire format, they will emit different trace context headers, resulting in broken traces in tools like Honeycomb and an incomplete view of critical customer journeys. Beelines use a custom, Honeycomb-specific trace header (called X-Honeycomb-Trace) and OpenTelemetry uses the format defined in the W3C Trace Context open standard.

We really want Honeycomb users to be able to use the instrumentation libraries of their choice, so we’re extremely excited to announce trace header interoperability in all of our supported Beelines. Using trace header interoperability, you can now mix and match Beelines and OpenTelemetry in a single system. Each Beeline now provides support for trace header parsing and propagation hooks, as well as canned marshal and unmarshal functions for Honeycomb- and OpenTelemetry-compatible trace headers.

We chose a hooks interface instead of an arguably simpler configuration option, because deciding how a particular service parses or sends trace context headers might not always be straightforward. A service may want to parse headers of a specific kind only when they come from another service, or when the request comes from a specific network block. Similarly, on outgoing requests, a service may need to make decisions about what format of header to include depending on the destination. Sometimes a service may wish to not include a header at all, as is often the case when calling a third party service. These use cases are all supported by a hooks interface.

Using Hooks

To demonstrate, here’s how a service instrumented with our Go Beeline would parse incoming trace headers from an OpenTelemetry instrumented service using the W3C Trace Context header format:

import (
   "net/http"
  
   "github.com/honeycombio/beeline-go/propagation"
   "github.com/honeycombio/beeline-go/wrappers/config"
   "github.com/honeycombio/beeline-go/wrappers/hnynethttp"
   beeline "github.com/honeycombio/beeline-go"
)

func main() {
   beeline.Init(beeline.Config{
       WriteKey: "YOUR_API_KEY",
       Dataset: "YOUR_DATASET",
      
   })
   defer beeline.Close()
  
   ...
  
   http.ListenAndServe(":8080", hnynethttp.WrapHandlerWithConfig(muxer, config.HTTPIncomingCOnfig{
       HTTPParserHook: parseW3CHeaders
   }))
}

func parseW3CHeaders(r *http.Request) *propagation.PropagationContext {
   headers := map[string]string{
       "traceparent": r.Header.Get("traceparent"),
   }
   ctx := r.Context()
   ctx, prop, err := propagation.UnmarshalW3CTraceContext(ctx, headers)
   if err != nil {
       ...
   }
   return prop
}

The only addition here beyond configuring the Beeline is the parseW3CHeaders function and the wrapper function used to wrap the HTTP muxer. If your service needs to send headers in a format other than the Honeycomb header format, you can use a propagation hook when wrapping the HTTP client:

import (
   "net/http"
  
   beeline "github.com/honeycombio/beeline-go"
)

func propagateW3CHeader(r *http.Request, prop *propagation.PropagationContext) map[string]string {
   ctx := r.Context()
   ctx, headers := propagation.MarshalW3CTraceContext(ctx, prop)
   return headers
}

func main() {
   client := &http.Client{
       Transport: hnynethttp.WrapRoundTripperWithConfig(
           http.DefaultTransport, config.HTTPOutgoingConfig{HTTPPropagationHook: propagateW3CHeader}
       ),
       Timeout:   time.Second * 5,
   }
   req, _ := http.NewRequest("GET", "https://int-srv.local/", nil)
   res, err := client.Do(req)
}

Here the only addition is the propagateW3CHeader function and the wrapper function used to wrap the default HTTP transport struct.

These examples are in Go, but similar hooks are available for Python, Node.js, Ruby, and Java.

Beelines + OpenTelemetry = <3

The addition of trace header interoperability is a big step towards making Beelines and OpenTelemetry work together seamlessly. We’re also hard at work on solutions for making sure that sampling can be applied consistently and that OpenTelemetry is easier to use with Honeycomb. OpenTelemetry is a great instrumentation ecosystem and we’re really excited to provide people paths to adopt it into their own systems.

Getting started with tracing doesn't have to be a lot of work, whether you're using Beelines, OpenTelemetry, or both. Watch this recent episode of Raw & Real: The Tracing You Deserve So You Can Observe.