Standing on the Shoulders of Giants (and Reusing Their Queries)
“What was the query that Erica ran last week?” mused Scott as he tapped rhythmically on the desk. “There was this spike, and the table explained what was going on. What was it?”
Here at Honeycomb, we spend lots of time thinking about how to help our users be more awesome at unearthing insights from their data so they can solve production issues in real-time. We think a lot about how to make running a query easy, how to guide users to wield our Query Builder effectively to find the needles in the haystacks of data that they send us. In doing this, we’ve found a common scenario: often, the answer to the question you have right now about your infrastructure is a query that a co-worker ran yesterday, or one you yourself ran last week.
Over the last few months, we have done a lot of work to understand what our users are trying to achieve when they are interacting with Honeycomb. We’ve found that users are typically motivated by investigating an incident, debugging a problem, and quite often by curiosity alone–a desire to understand more about the innards of the complex service that they are working with.
A typical Honeycomb story starts like this:
“Hmm, Tammy just filed a ticket that a customer called in to say that the pages are taking forever to load. It wasn’t like this the last time we looked, I wonder what happened?”
“I just made a change to the function to use a slightly more expensive algorithm to produce a more accurate estimation, I wonder how it affects our users’ experiences?”
When Erica receives a ticket from Tammy, she could go directly to the Query Builder page and create a search, conjure up the set of verbs and nouns, examine the wriggly line that Honeycomb produces and dive into the details. But more likely, she might recall that this sounds like a similarly-shaped problem to one she successfully resolved last week–and go through her historical backlog of Honeycomb queries to find that one to start with (kind of like paging through your Google search history to find that one page you were looking at….back then.)
What we’ve seen here at Honeycomb is that while problems might come in all shapes and sizes, the symptoms—and the kinds of questions you need to ask to begin figuring out what’s going on—are not so variable.
In this particular case, Erica looks first at the latency numbers across the various pages that are serviced by this system. From there, she notices that there has been an increased occurrence of the bundling endpoint being called. She also notices that the endpoint is being called by a single client and in an unusually regular rhythm. She looks into it more, and realizes that this endpoint is rather expensive to call, and the client scripting it with a cron job is probably causing other users on the service to see degraded performance.
It all starts with a why, and then another why, and another why, and voila!
More recently, we’ve been making changes to our Query page to further differentiate the question from the data that helps answer the question. We think Queries are reusable entities that should be easily reachable in your back pocket, and in our case, stored safely in Boards. When you craft a query that is the starting point to a series of useful dissections, or one that succinctly points out a performance characteristic, give it a name to help you remember what it does. Add a description to remind yourself what the baseline numbers look like. Consider saving it to a Board for reuse. You’ll thank yourself later.
Connecting an archipelago
It doesn’t just stop there.
A couple of weeks later, Erica’s team mate Scott was investigating an issue that looked awfully similar to the one that Erica was investigating. He distinctly remembered looking at the query that was shared into the #eng slack channel. It was titled latency endpoint something something…so he pulled up Honeycomb and did a quick search for the first two words. One of Erica’s queries showed up, and Scott used that as a starting point into his own investigation.
Giving queries name and descriptions is not just a great way to help annotate the what does it do and why does it do that of a query, it also helps connect your work in Honeycomb with work that other people can do. We rarely work alone, and in teams, most of us will share the responsibility to make sure that the system that we are looking after is as healthy as possible. We share guardianship duties, and thus by sharing the tools that make each one of us more effective, we make everyone more so.
Back to our story: Scott was ecstatic that he was able to solve his problem so quickly by leveraging Erica’s work. And, since both he and Erica had relied on this exact query to identify and solve problems, he also realized this query might be useful for other teammates in the future. The way to do this, he decided, is to store all these useful queries in a single place. Scott thus created a Board, named it useful queries, designated it as a List Board, and added the current one to the Board. Why not add in a description about what to look for? he thought. And so begins an excellent tradition of sharing and expanding brains.
The question, marked
Building queries can be kinda hard. There, we said it. While we take pride in making it as simple as possible to construct a query, we recognize that building one that produces the right intersection of data requires significant creative energy. We believe that the cycle of creating queries, giving them meaning, collecting and sharing them with others is equally effective at helping everyone level up.
As we strive to help people get better at exploration and thereby discovering more pertinent things about their systems, we will continue to make it easier to plumb the depths of all the graphs that have already been generated, all the questions that have already been constructed and translated into the Query Builder — and make it easier to share, save and collect useful queries.