The New Rules of SamplingBy Rachel Perkins | Last modified on May 9, 2019
One of the most common questions we get at Honeycomb is about how to control costs while still achieving the level of observability needed to debug, troubleshoot, and understand what is happening in production. Historically, the answer from most vendors has been to aggregate your data--to offer you calculated medians, means, and averages rather than the deep context you gain from having access to the actual events coming from your production environment.
This is exactly what it sounds like--a poor tradeoff for performance. With classic metrics and APM tools, you can never again get back to the raw event source of truth, which means you'll regret that choice when debugging a complex, distributed system. When you’re working with metrics, the data must be numeric, and any other type of data must be stored as metadata either attached to the datapoints themselves or out-of-band in some way (“tags”, “dimensions”, etc), AKA: more limits on what you can store and retrieve.
Honeycomb's answer is: Sample your data.
But, you say, sampling means I'm throwing away some (or a lot) of my data. How is that OK? I won't know what I am not seeing, right?
What if you had more flexibility? What if sampling offered a greater breadth of options than just "send a percentage of my data"?
Find out what's possible in The New Rules of Sampling.
Author’s Cut—A Sample of Sampling, and a Whole Lot of Observability at Scale
In this post, we’re moving from the foundations of observability to things that become critical when you start practicing observability at scale. Tools like sampling...
Honeycomb Pro: Now With Metrics & SLOs
Honeycomb Pro is about to get even better. Starting today, all Pro accounts have access to Honeycomb Metrics and two Service Level Objectives (SLOs), previously...
The Truth About “MEH-TRICS”
A long time ago, in a galaxy far, far away, I said a lot of inflammatory things about metrics. “Metrics are shit salad.” “Metrics are...