Action Recommended: OpenTelemetry HTTP Attributes Breaking Changes

Earlier this year, the folks working on OpenTelemetry launched an effort to stabilize HTTP Semantic Conventions. In November 2023, OpenTelemetry announced that HTTP Semantic Conventions were stable. They accomplished this by merging the existing HTTP Semantic Conventions with the Elastic Common Schema HTTP attribute conventions. The benefits in doing so are numerous: 

• Stabilizes the format of data for the most-used OpenTelemetry instrumentation libraries
• Makes OpenTelemetry compatible with any system that also uses the ECS HTTP schema
• Provides a way for all HTTP instrumentation libraries (the most popular libraries in OpenTelemetry) to offer stable, 1.0-quality releases

However, there are many breaking changes in these attributes. Most attributes were renamed or removed, new ones were added, and the http.target attribute was split into two attributes, url.path and url.query.

Who will this affect?

Later this month (December 2023), Java and .NET will release instrumentation versions that emit the new, stable values for HTTP by default. Up until now, they had supported emitting them behind an environment variable, but that will no longer be an option—only new values will be emitted. 

• If you have Java or .NET services, you should prepare for the change 
• If you do not have Java or .NET services, you won’t be affected by these changes yet 
• If you have a mix of languages that includes Java or .NET, you may want to hold off on upgrading until the changes are consistent in all instrumentations (via library or auto-instrumentation agent) relevant to you, then coordinate a release

As of now, no other languages have planned library changes (Node, Go, Python, Ruby, etc.), but most likely will in the future.

What’s changing?

The following attributes will change:

What can break?

Let’s say you update your HTTP instrumentations or Java auto-instrumentation agents. In that case, the following things may break:

• Triggers or SLOs

  • SLO tracking latency with an SLI that checks for http.status_code or a trigger with a GROUP_BY on http.route

• Refinery rule conditions and field lists

  • Conditions or fields that rely on http.status_code, http.route, or http.method

• OpenTelemetry Collector configurations

  • Processors like the transform, filter, or redaction processors that look for HTTP attribute fields

• Boards and saved queries

  • Any Boards and associated queries that use HTTP attributes in the VIZUALIZE, WHERE, or GROUP_BY sections of the query builder

• Derived Columns not used by SLOs

  • These might impact your saved Boards and queries, but look for Derived Columns that use conditionals or concatenation or regex functions against HTTP attribute columns

The consequences of this change can include the following:

• Collector configurations fail, leading to specific data being dropped or included unexpectedly
• Sampling rules with Refinery fail, leading to a large and unexpected uptick in events sent to Honeycomb
• Queries are no longer accurate
• Triggers send an alert unexpectedly, or fail to send an alert when they should have
• SLOs stop accurately counting events, leading to incorrect burn or burn rate alerts

We highly recommend that you plan to make changes in any of the above applicable areas before HTTP instrumentations proliferate.

What can you do about it today?

First, if you update HTTP instrumentations or auto-instrumentation agents to the latest version, do so carefully. To mitigate the impact of updating your HTTP instrumentations, you should audit them for any HTTP attributes. We recommend the following, in this order:

1. Use the OTEL_SEMCONV_STABILITY_OPT_IN environment variable at the application/service level to double-send old and new values

   • This is relevant for Java today

2. Check your OpenTelemetry Collector configuration

   • Consider using the transform processor to write both new and old attributes if you’ll be sending new attributes in at least one service

3. Check your Refinery configuration

   • Consider writing duplicate sampling rules, if applicable
   • If you are an Enterprise customer, reach out to your account team to get help

4. Check your triggers and SLOs and how you alert

   • Be aware that alerts may or may not fire unexpectedly, and adjust these according to the new attribute names if you plan to take the new attributes

If you have not taken any updates to HTTP libraries or auto-instrumentation, you do not need to make changes yet.

Consider centralizing how you update OpenTelemetry libraries and/or agents. If anyone in your organization pushes an update, they may inadvertently break things in Honeycomb. By centralizing your telemetry instrumentation and version management, you can prevent this scenario.

What’s OpenTelemetry doing about it?

The migration plan section of the OpenTelemetry blog post contains details about how OpenTelemetry will handle a migration period.

Libraries that update to the stable conventions are encouraged to introduce an environment variable that will let users control which attributes are emitted. They are also encouraged to continue to patch older versions for six months.

However, it’s important to understand that this period will not be uniform across all OpenTelemetry supported languages. Different languages will introduce stable conventions at different times, so if you have a multi-language system, you may need to hold off migrating until every HTTP instrumentation you use emits the new values.

What’s Honeycomb doing about it?

This notice is the first of several communications we will make. As more relevant updates happen to OpenTelemetry, such as a prominent instrumentation updating to the new semantic conventions, we will update you and provide refined guidelines on how to take action.

Thank you for your patience!

We’re working out how to best handle this transition. Although we’re excited about the HTTP attribute Semantic Conventions becoming stable, we’re very aware that the migration to these stable attributes may be painful. We will continue to notify you of this change and will provide updates when appropriate.