Product Updates  

Speeding Things Up So Your Queries Can Bee Faster

By Ian Wilkes  |   Last modified on January 11, 2019
Honeycomb strives to be a fast, efficient tool; our storage back-end satisfies the median customer query in 250ms (and the P90 in 1.3 seconds). Still, every system has its limits, and customers with large datasets know that querying over a long time range, grouping by high-cardinality columns, building complex derived columns, and throwing a quantile or heat map into the mix can lead to some pretty slow queries. If this sounds familiar: good news! Over the past month we've shipped a variety of improvements to attack long-running queries, and if you've got a query which has been troublesome for you in the past, now's a good time to give it another try. (And if you still have trouble, please let us know.)


cartoon bee going fast

Faster merging of query results

The biggest change we've made is implementing fully distributed merge for multi-group, multi-node queries. Previously, queries with hundreds of thousands of result groups, or more, (computed before applying the query's Limit) could bottleneck merging intermediate results from multiple storage nodes, especially in large datasets spread across a lot of hosts. This system has been thoroughly reworked, distributed, and optimized, allowing for far faster merging across as many machines as your dataset has available. In extreme cases, this has led to speedups of 5X or more.

Faster derived columns

In addition, derived columns have gotten some much-needed love with speedups across the board, largely by eliminating wasted steps and unnecessary intermediate values. While not instantaneous, these runtime calculations are now reasonably efficient and should deliver at least an approximation of the performance of raw data columns. Fair warning though, REG_MATCH, REG_VALUE, and UNIX_TIMESTAMP are still, by their nature, relatively expensive.

Faster quantiles

The code we use to compute quantiles, always the most expensive aggregate function, has also been improved, approximately doubling performance. Pro tip: all quantiles for any given column are computed from the same data, so if you add one to your query, adding more on the same column won't slow things down.


Faster storage

If you're not doing anything fancy which would benefit from the above, that's fine too: the core storage engine has been updated to read large volumes of data even faster than before. A key change here, and common to all of the above improvements as well, is largely eliminating the allocation of garbage memory during query processing.
Honeycomb's back-end is written in Go, which makes it relatively straightforward to write performant code, and also makes it extremely easy to allocate heap memory incidentally. Third-party libraries can be quite profligate in this regard, and even the standard library is, in some cases, surprisingly ready to put short-lived data on the heap. These allocations are not free, and it's not uncommon to profile something in Go and find that allocation is the single largest consumer of CPU time, followed or sometimes even exceeded by garbage collection of those same allocations. So, we've taken an axe to everything which abandoned heap memory in mid-query, and left the garbage collector with very little to do, saving a lot of time for big queries.
So, go forth and calculate! We'll continue working to make your queries as fast and reliable as possible.


New to Honeycomb? Try out a real-world, live data scenario with no strings attached:

Related Posts

Product Updates   Observability  

Introducing Honeycomb for Frontend Observability: Get the Data You Need for Actionable Customer Experience Improvements

Honeycomb for Frontend Observability gives frontend developers the ability to quickly identify opportunities for optimization within their web app. This starts with better OpenTelemetry instrumentation,...

Product Updates   Observability  

Introducing Relational Fields

Expanded fields allow you to more easily find interesting traces and learn about the spans within them, saving time for debugging and enabling more curiosity...

Product Updates   Instrumentation  

Safer Client-Side Instrumentation with Honeycomb's Ingest-Only API Keys

We're delighted to introduce our new Ingest API Keys, a significant step toward enabling all Honeycomb customers to manage their observability complexity simply, efficiently, and...