Honeycomb strives to be a fast, efficient tool; our storage back-end satisfies the median customer query in 250ms (and the P90 in 1.3 seconds). Still, every system has its limits, and customers with large datasets know that querying over a long time range, grouping by high-cardinality columns, building complex derived columns, and throwing a quantile or heat map into the mix can lead to some pretty slow queries. If this sounds familiar: good news! Over the past month we’ve shipped a variety of improvements to attack long-running queries, and if you’ve got a query which has been troublesome for you in the past, now’s a good time to give it another try. (And if you still have trouble, please let us know.)
Faster merging of query results
The biggest change we’ve made is implementing fully distributed merge for multi-group, multi-node queries. Previously, queries with hundreds of thousands of result groups, or more, (computed before applying the query’s Limit) could bottleneck merging intermediate results from multiple storage nodes, especially in large datasets spread across a lot of hosts. This system has been thoroughly reworked, distributed, and optimized, allowing for far faster merging across as many machines as your dataset has available. In extreme cases, this has led to speedups of 5X or more.
Faster derived columns
In addition, derived columns
have gotten some much-needed love with speedups across the board, largely by eliminating wasted steps and unnecessary intermediate values. While not instantaneous, these runtime calculations are now reasonably efficient and should deliver at least an approximation of the performance of raw data columns. Fair warning though,
are still, by their nature, relatively expensive.
The code we use to compute quantiles, always the most expensive aggregate function, has also been improved, approximately doubling performance. Pro tip: all quantiles for any given column are computed from the same data, so if you add one to your query, adding more on the same column won’t slow things down.
If you’re not doing anything fancy which would benefit from the above, that’s fine too: the core storage engine has been updated to read large volumes of data even faster than before. A key change here, and common to all of the above improvements as well, is largely eliminating the allocation of garbage memory during query processing.
Honeycomb’s back-end is written in Go, which makes it relatively straightforward to write performant code, and also makes it extremely easy to allocate heap memory incidentally. Third-party libraries can be quite profligate in this regard, and even the standard library is, in some cases, surprisingly ready to put short-lived data on the heap. These allocations are not free, and it’s not uncommon to profile something in Go and find that allocation is the single largest consumer of CPU time, followed or sometimes even exceeded by garbage collection of those same allocations. So, we’ve taken an axe to everything which abandoned heap memory in mid-query, and left the garbage collector with very little to do, saving a lot of time for big queries.
So, go forth and calculate! We’ll continue working to make your queries as fast and reliable as possible.