Software Engineering

Incident Report: Running Dry on Memory Without Noticing

On November 6, 2019, we intermittently rejected 1-3% of customer telemetry data at ingest for four periods of 20 minutes each. The trigger of the incident was a slow memory leak that manifested over…

Making Instrumentation Extensible

Observability-driven development requires both rich query capabilities and sufficient instrumentation in order to capture the nuances of developers’ intention and useful dimensions of cardinality. When our systems are running in containers, we need an…

A New Bee’s First Oncall

I’m Honeycomb’s newest engineer, now on my eighth week at Honeycomb. Excitingly, I did my first week of oncall two weeks ago! Almost every engineer at Honeycomb participates in oncall, and I chose to…

Spooky Tales of Testing In Production: A Recap and Lessons Learned

The Core Analysis Loop, or: Heatmaps Make Analysis Better

In this blog miniseries, I’m talking about how to think about doing data analysis, the Honeycomb way. In Part I, I talked about how heatmaps help us understand how data analysis works. In Part…

Heatmaps Make Ops Better

In this blog miniseries, I’d like to talk about how to think about doing data analysis “the Honeycomb way.”  Welcome to part 1, where I cover what a heatmap is—and how using them can…

Level Up with Derived Columns: Two Neat Tricks That Will Improve Your Observability

When we released derived columns last year, we already knew they were a powerful way to manipulate and explore data in Honeycomb, but we didn’t realize just how many different ways folks could use them.

Level Up With Derived Columns: Bucketing Events For Comparison

When we released derived columns last year, we already knew they were a powerful way to manipulate and explore data in Honeycomb, but we didn’t realize just how many different ways folks could use them.

Level Up With Derived Columns: Understanding Screen Size (With Basic Arithmetic)

When we released derived columns last year, we already knew they were a powerful way to manipulate and explore data in Honeycomb, but we didn’t realize just how many different ways folks could use them.

Observability: A Manifesto

Everybody and their freaking grandpa is now claiming to do observability, not stodgy old monitoring. Fine, great. Nice to be trendy I guess. But are they? What’s the difference? You all know how I…

Resolving High CPU Usage in Kubernetes With Honeycomb

At Honeycomb, we’re excited about Kubernetes. In fact, we’re in the early stages of moving some of our services to k8s. Tools like kops have made getting started with k8s easier than ever. But…

How Honeycomb Uses Honeycomb, Part 4: Check Before You Change

This post continues our dogfooding series from How Honeycomb Uses Honeycomb, Part 3: End-to-end Failures. As Honeycomb matures, we try to roll out changes as smoothly as possible to minimize surprise on the part…

Part 4/5: Everyone is a DBA

DBAs may be the last remaining priesthood in our industry. But software engineers and operations engineers are increasingly finding themselves responsible for precious company data, and DBAs are increasingly adopting generalist skillsets and best…

Part 2/5: Dear Software Engineers

Observability is not a thing for operations or some other team to care about. Software engineers, you are increasingly the primary owners of your own services … and this is a great thing. Developers,…