Software Engineering

“Why Are My Tests So Slow?” A List of Likely Suspects, Anti-Patterns, and Unresolved Personal Trauma

If you get CI/CD right, a lot of other critical functions, behaviors, and intuitions align to be comfortably successful and correct with minimal effort. If you get it wrong, you will spend countless cycles chasing pathologies. It’s like choosing to eat your vegetables every day vs choosing a diet of cake and soda for fifty years, then playing whack-a-mole with all the symptoms manifesting on your poor, moldering body.

Exploring AWS Costs Beyond the Service Level

This post will talk about using a derived column to directly connect individual customer experiences to the cost of providing that service with AWS Lambda. By leveraging these tools, we can better understand when our product is used in costly ways.

The power of asking questions

This is a guest post by Vlad Ionescu. Vlad Ionescu jokingly describes himself as a “Professional mistake avoider” which is a better way of saying “Independent Consultant.” His work focuses on getting companies to…

On the Brittleness of Dashboards

Dashboards are one of the most basic and popular tools software engineers use to operate their systems. In this post, I’ll make the argument that their use is unfortunately too widespread, and that the…

How We Define SRE Work

At the time of writing this post, I have officially been at Honeycomb for one year as a site reliability engineer (SRE). I had shared my initial experiences and impressions in this post and…

Incident Report: The Missing Trigger Notification Emails

On November 18, between 00:50 and 00:56 UTC, an update was deployed that improved Honeycomb’s business intelligence (BI) telemetry available from our production operations environment. Contained within that update was a defect that escaped…

Incident Resolution: Do You Remember, the Twenty Fires of September?

From September to early October, Honeycomb declared five public incidents. Internally, the whole month was part of a broader operational burden, where over 20 different issues interrupted normal work. A fraction of them had…

Game Launches Should Be Exciting for Your Players, Not for Your LiveOps Team

This blog was co-authored by Amy Davis.  The moment of launching something new at a game studio (titles, experiences, features, subscriptions) is a blockbuster moment that hangs in the balance. The architecture—distributed and complex,…

Lessons Learned From the Migration to Confluent Kafka

Over the last few months, Honeycomb’s platform team migrated to a new iteration of our ingest pipeline for customer events. Our migration to this newer architecture did not go too smoothly, as can be…

The Future of Developer Careers

While JavaScript frameworks come and go, a change has been brewing over the last several years that will permanently change what it means to be a modern developer: how our code goes from our…

Incident Review: Meta-Review, August 2020

Every once in a while, teams or systems hit an inflection point where enough things change at once and the pattern of incidents shifts. We found ourselves at an inflection point like that last week.

Getting At The Good Stuff: How To Sample Traces in Honeycomb

(This is the first post by our new head of Customer Success, Irving.) Sampling is a must for applications at scale; it’s a technique for reducing the burden on your infrastructure and telemetry systems…

From “Secondary Storage” To Just “Storage”: A Tale of Lambdas, LZ4, and Garbage Collection

When we introduced Secondary Storage two years ago, it was a deliberate compromise between economy and performance.

Incident Report: Running Dry on Memory Without Noticing

On November 6, 2019, we intermittently rejected 1-3% of customer telemetry data at ingest for four periods of 20 minutes each. The trigger of the incident was a slow memory leak that manifested over…

Making Instrumentation Extensible

Observability-driven development requires both rich query capabilities and sufficient instrumentation in order to capture the nuances of developers’ intention and useful dimensions of cardinality. When our systems are running in containers, we need an…

A New Bee’s First Oncall

I’m Honeycomb’s newest engineer, now on my eighth week at Honeycomb. Excitingly, I did my first week of oncall two weeks ago! Almost every engineer at Honeycomb participates in oncall, and I chose to…

Spooky Tales of Testing In Production: A Recap and Lessons Learned

The Core Analysis Loop, or: Heatmaps Make Analysis Better

In this blog miniseries, I’m talking about how to think about doing data analysis, the Honeycomb way. In Part I, I talked about how heatmaps help us understand how data analysis works. In Part…

Heatmaps Make Ops Better

In this blog miniseries, I’d like to talk about how to think about doing data analysis “the Honeycomb way.”  Welcome to part 1, where I cover what a heatmap is—and how using them can…

Level Up with Derived Columns: Two Neat Tricks That Will Improve Your Observability

When we released derived columns last year, we already knew they were a powerful way to manipulate and explore data in Honeycomb, but we didn’t realize just how many different ways folks could use them.