Ask Miss O11y: Observability Without Manual Tracing
By Jessica Kerr | Last modified on February 7, 2022I think my biggest question is: How do I introduce observability without manually weaving tracing-related code throughout my codebase since that adds a lot of noise?
TL;DR: Use auto-instrumentation from OpenTelemetry. Traces will happen. Then your code can use global library functions to customize those traces with your specific important data.
Manual tracing
Not so long ago, getting traces meant explicitly writing the code to emit spans with context fields attached. That meant passing the trace context down, down, down through all functions that might want to make a span or add data to a span. And you had to explicitly create all the spans you wanted to see. That can look like a lot of clutter.
Here are three functions that produce two manual spans (in JavaScript on Node.js; some abstractions not included). The tracing code is highlighted in orange. Please donât read this mess! Just notice how much code is orange.
The orange tracing code is scattered all over the place. Parameters passed, spans started, statuses set, spans ended. Even functions that donât do any tracing (like the middle one) have to pass the context along.
This obscures the real work of the functions. Observability is supposed to make it easier to see whatâs happening in our code, but this makes it harder!
This code creates distributed tracing that looks like this. Each incoming and outgoing request gets a span.
Auto-instrumentation
Fortunately, the days of light are upon us. For applications that use standard web frameworks, all this instrumentation can be magicked in with OpenTelemetry auto-instrumentation.
Hereâs what that same code looks like using auto-instrumentation for `http` and Express:
The only tracing code here pulls a span out of global space and adds an attribute particularly relevant to this application. This is not clutter! I think it adds valuable information to the codeâit expresses the significance of this variable.
The trace contains even more information! There are spans from some middleware that I didnât know was running. Bonus: Learn about your libraries from the traces they supply.
OpenTelemetry is legit
OpenTelemetry is an open-source project with hundreds of contributors, including companies like Honeycomb. The libraries are built to an industry standard so that any telemetry you add is portable between tracing and observability products.
Tracing is a cross-cutting concern, so itâs fair to use global variables, bytecode manipulation, and other language-runtime hooks to track context and make tracing functions available throughout your code.
- In Node.js, autotelemetry intercepts module load to monkeypatch functions.
- Rubyâs module-prepend and open classes lets libraries enhance each other..
- In Go, no such shenanigans are supported, so wrap library calls explicitly. The standard Context parameter holds tracing context.
- In Java, change no code at all! A runtime parameter can do it.Â
- In .NET, the standard ActivitySource API works for autotelemetry.
- Python has auto-instrumentation for lots of frameworks. Iâm guessing they monkeypatch them.
Iâd call some of these âhacksâ if I implemented them myself. However, in a well-supported, carefully vetted library like the OpenTelemetry ones, I call them foundations.
Get traces for network requests, database calls, and more. Check whatâs available for your language and framework of choice at the OpenTelemetry registry.
With auto-instrumentation doing most of the work, tracing code can make our domain logic more clear instead of less, by emphasizing whatâs important. Then, in production, those same lines give us the data that shines through with insight in a product like Honeycomb.
Check out our OpenTelemetry documentation and sample apps for auto-instrumentation, and sign up for a free Honeycomb account today to try it out for yourself!
Related Posts
Defensive Instrumentation Benefits Everyone
A lot of reasoning in content is predicated on the audience being in a modern, psychologically safe, agile sort of environment. Itâs aspirational, so folks...
Start with Traces, not with Logs: How Honeycomb Helped Massdriver Reduce Alert Fatigue
Before Massdriver, Dave worked in product engineering where he was constantly bogged down with DevOps toil. He spent his time doing everything except what he...
Infinite Retention with OpenTelemetry and HoneycombÂ
Honeycomb is massively powerful at delivering detailed answers from the last several weeks of system telemetry within seconds. It keeps you in the flow state...