Intro to o11y Topic 13: Refining a Trace in Honeycomb

Jessica Kerr [Developer Advocate|Honeycomb]:

Here I am at this long trace view and I have the index field showing. Wow, there’s a lot in here! There must be something else after this request, but how can I see it? When I hover over the span I can see three dots over on the left. Click that, and a menu appears. Hmm, let’s collapse spans with this name, maybe

Ah! Here’s the next span, the other part. Hmm, I can’t see the index anymore. To undo that, I go back to the three dots and expand spans with this name. Now let’s try this one that does show the index. Ack! Too much! Hmm, maybe if I “expand spans at this depth.” I do want the root span expanded That’s better! OK, I see that this call to fib of 9 sent a request for fib 8 and fib of 7. And inside here… And the request to fib of 8 sent a request for fib 7 and fib 6. And this other request for fib of 7 sent a request for indexes 6 and 5

Wait, here’s a request to fib of 6, and here’s another request to fib of 6. How many times does fib of 6 get called in this request? I can see here that it found index=6 inside the http.url It found six spans in this trace that include the text index=6 in some field name or value. And they’re highlighted… wait. 1 2 3 4, I see five. And this one isn’t 6, it’s 7. Oh wait, this is saying that somewhere in these 203 collapsed spans there’s something with index = 6. Let me expand those again. OK, now we see that these two spans are highlighted. This one is the client call out to index=6

And this one is the server receiving that call. So there’s two spans highlighted per call to index of 6 within this overall request for index of 9. Six spans found, two spans per call, there are three calls to index=6 in one call to index=9. What about 5? It found 10 spans with index=5, so five calls to index=5 in one call to index=9. This seems like a key question – how many duplicated internal calls are happening in each external call from the front end? 

Transcript