Intro to o11y Topic 9: Filtering for Root Spans in Honeycomb

Jessica Kerr [Developer Advocate|Honeycomb]:

Let’s take a few minutes to go deep on one important subtlety of Honeycomb queries: filtering for root spans. Here we are on the latency graph. This is a heatmap of duration, which counts events by timestamp and duration. Which events are counted? Is it everything received by Honeycomb? The next part of the query definition determines which events are included. It is not all of them; it is only events where trace.parent_id does not exist.

This particular filter has a special meaning. I mean, it literally means what it says, that the field trace.parent_id does not exist on the event, but that means that we’re looking only at the root span of a trace. To see why, look at this trace. All these spans are related to each other, and Honeycomb can knit them together using the fields that start with trace. 

Cool trick, this box filters the list of fields. Here I’m seeing only those that include trace dot in the name or value. These fields are for the currently selected span. I have the root span selected. And it has fields for trace.trace_id and trace.span_id. They identify the overall trace and this span in particular. Every other span has three fields defining its place in the trace hierarchy. There’s the same trace.trace_id, another unique trace.span_id, and a trace.parent_id. That trace.parent_id, I can select it and copy it.

And then search these spans for that value. Cool trick, this search highlights all spans with a field name or field value containing this text. This span here has that value in its span ID. The span right under it has that value in its parent ID. So this field points to the parent span. Every single span has a trace.parent_id, except this one, the root span.

Therefore, when we filter by trace.parent_id does not exist, this one span represents the entire trace. And the trace represents one incoming request from outside. And its duration represents the latency of that request, as measured by the server.

In this graph, then, we’re looking at the latency of requests. Not every event, not every span, but every root span. If we go back to the home page and click into the graph of total requests, this query is looking at a count over time where trace.parent_id does not exist. That means it is counting root spans, and that means it is counting requests, at most 58 per second.

If I want to count every event sent to Honeycomb – every individual span, I can remove this filter. I click on the where clause here, and then click x to remove this filter. And then I push “Run query” to send this new query to Honeycomb. These results go up to 6000. Our application generated 6000 events 58 incoming requests. It’s been busy!

I can click the back button to return to the previous query. Now you know all about “trace.parent_id does not exist.” use it in your queries when you want to count requests, when you want to count each trace once.

Transcript