Training Videos Debugging

Using Query Builder for Multi group bys

In this video, we will leverage Honeycomb’s Query Builder to isolate performance issues by driving down to multiple groups. We will start with a high-level group by and iterate by adding additional group bys until we isolate an exact database operation and observe the performance for the average and the outliers.


Brian Langbecker [Solution Architect|Honeycomb]:

The query builder. Let’s go into our query builder and look at one of the coolest features, which is the ability to do multiple group bys. I’m going to click on my query builder over here. As it’s coming up, I’m going to jump over here and go to the last 14 days, because I want to look at some patterns. And I’m going to jump over here and type the word count. I’m going to do the average of my duration, or latency as it’s known, and I’m going to do it P95 of my duration. I’m going to jump over here and do my first group by, and I’m going to group by the service name because I want to break it down by the service name.

Now, once it comes up, you’re going to see all these lines. I’m going to press M on my keyboard to get rid of those. Those are deployment markers, and now I can see everything broken up by my various services. I can see a big variance between my typical front end website and my P95. It may pay to explore that. I can even see my cart down here and it almost doubles, and I definitely want to explore that. But the reason I’m doing all these queries is I want to look at the database calls in my system. Now, if I jump up here and I type DB name and I select exists, it’s going to filter that list from that large list below down to a smaller list that is just related to a certain number of services.

If you noticed, a number of my services disappeared because they don’t have a database here, but I can still see big differences. I can see that on my product catalog, it gets close to doubling in size. Recommendation engine goes up by about 40%. My checkout goes up by about 40%. That’s really interesting, but I’d like to explore it a little bit more. And I’m going to jump over here and I’m going to add that database name. While I’m here, I’m going to add the database statement because I want to look at the individual calls that are going on in the system here.

Now, I can clearly see the calls that are taking the most in terms of the count and the biggest differences that are going on in the system. I see this top level one. It’s actually pretty darn fast, but this one below it, seems to be a little bit on the slow side. If I go down to this one, this checkout, this is really slow. My average is 123 milliseconds, pretty fast, but when I hit my P95, it is almost 40% bigger. As I go up, my customers keep getting really poor performance in terms of the checkout process. Nothing like ordering a few things on the checkout process.

What have we looked at here and what have we learned? We’ve learned that we can do a visualization. We’ve learned that we can filter it down to things I care about. But more importantly, I have the ability to group by not only one thing, not only two things, but as many things as you want. This allowed me to see my services, my individual database calls that are going on in a system like the shipping one, which has two calls or actually three calls, and how much time these things are taking.

If you see any typos in this text or have any questions, reach out to