Dogfooding for Deploys: How Honeycomb Builds Better Builds with Observability
Observability changes the way you understand and interact with your applications in production. Beyond knowing what’s happening in prod, observability is also a compass that...
Observations on ARM64 & AWS’s Amazon EC2 M6g Instances
At re:Invent in December, Amazon announced the AWS Graviton2 processor and its forthcoming availability powering Amazon EC2 M6g instances. While the first-generation Graviton processor that...
Using Honeycomb to remember to delete a feature flag
Feature flags are great and serve us in so many ways. However, we do not love long-lived feature flags. They lead to more complicated code,...
Instrumenting Lambda with Traces: A Complete Example in Python
We’re big fans of AWS Lambda at Honeycomb. As you may have read, we recently made some major improvements to our storage engine by leveraging...
Honeycomb SLO Now Generally Available: Success, Defined.
Honeycomb now offers SLOs, aka Service Level Objectives. This is the second in a set of of essays on creating SLOs from first principles. Previously,...
From "Secondary Storage" To Just "Storage": A Tale of Lambdas, LZ4, and Garbage Collection
When we introduced Secondary Storage two years ago, it was a deliberate compromise between economy and performance. Compared to Honeycomb’s primary NVMe storage attached to...
Incident Report: Running Dry on Memory Without Noticing
On November 6, 2019, we intermittently rejected 1-3% of customer telemetry data at ingest for four periods of 20 minutes each. The trigger of the...
Working Toward Service Level Objectives (SLOs), Part 1
In theory, Honeycomb is always up. Our servers run without hiccups, our user interface loads rapidly and is highly responsive, and our query engine is...
Never Alone On Call
Does your organization have an on-call rotation? Several members of the Honeycomb engineering team recently hosted a live webcast about why they never feel alone...
All Together Now: Better Debugging With Multiple Visualizations
"Nines don't matter when users aren't happy" is something you may have heard a time or two from folks here at Honeycomb. We often emphasize...
Understand Your AWS Cost & Usage with Honeycomb
First published in August 2019. AWS bills are notoriously complicated, and the Amazon Cost Explorer doesn’t always make it easy to understand exactly where your...
Treading in Haunted Graveyards
Part 1: CI/CD for Infrastructure as Code At Honeycomb, we've often discussed the value of making software deployments early and often, and being able to...
Incident Review: You Can't Deploy Binaries That Don't Exist
Between 22:50 and 22:54 UTC on July 9, our capacity to accept traffic to api.honeycomb.io gradually diminished until all incoming requests started to fail. 8...
Automating Collection of Troubleshooting Data with Triggers: a How-To Guide
Everyone wants to be more efficient -- to spend less time on the tedious things, and more time on the things that move the needle....
Stop Your Database From Hating You With This One Weird Trick
Let's not bury the lede here: we use Observability-Driven Development at Honeycomb to identify and prevent DB load issues. Like every online service, we experience...