Honeycomb Blog

Notes on our first-ever o11ycon (2018 SF)

It wasn’t until the conference was ending that we learned that everybody thought we were kinda nuts. “Wow. You folks were really … brave? … to let everything hang on the success of open spaces.” “Can’t believe that went so well and was such fun. I never thought you guys would pull it off!” “What…
Read More...

o11ycon: A Conference For The Observability Community

Hey friends, We recently announced o11ycon, and the response has been terrific! Especially since we haven’t really even told you all what we’re planning. 🙂 First things first: o11y means observability. (Yes, abbreviations are annoying, but YOU try typing ‘observability’ twenty times a day with character limits. And o11y just sounds so cute!) o11ycon will…
Read More...

Observability: A Manifesto

Everybody and their freaking grandpa is now claiming to do observability, not stodgy old monitoring. Fine, great. Nice to be trendy I guess. But are they? What’s the difference? You all know how I define observability: the power to ask new questions of your system, without having to ship new code or gather new data…
Read More...

The New Best Engineer

If you make a habit of reading twitter or the writings of various thought lords and ladies of the internet, you’ve probably heard a lot of advice on what software engineers should do, like: Software engineers should deploy their own code! Software engineers must know how to operate their own services! EVERYBODY is in ops…
Read More...

Oncall and Sustainable Software Development

Yes, being on call typically and anecdotally sucks. I understand! If you’ve heard me speak, I often point out that I’ve been oncall since I was 17 years old—so I know how terrible it can be. But I believe strongly that it doesn’t have to be. Oncall can and should be different. It can be…
Read More...

Observations on the Enterprise of Hiring

Almost everybody hates interviewing. You aren’t wrong to hate it: interviewing is fucking broken, in ways that tear you down and rob you of your self-respect and your will to live.
Read More...

The Price is Right

Here at the hive, we’re working on something that isn’t code or new features(!), but is a big part of our business notwithstanding: figuring out the best way to help people understand how we price Honeycomb and the built-in assumptions we make about how they use Honeycomb. There are some issues (pricing is hard, film…
Read More...

Best Practices for Observability

Observability has been getting a lot of attention recently. What started out as a fairly obscure technical term, dragged from the dusty annals of control theory, has been generating attention for one simple reason: it describes a set of problems that more and more people are having, and that set of problems isn’t well-addressed by…
Read More...

Metrics: not the observability droids you’re looking for

I went to Monitorama last year for my first time. It was great; I had a terrific time. But I couldn’t help but notice how speaker after speaker in talk after talk spent time either complaining about the limitations of their solutions, or proudly/sadly showing off whatever terrible hacks they had done to get around…
Read More...

Bitten by a Kafka Bug – Postmortem

Dearest honeycombers, Yesterday, on Tuesday, Oct 17th, we experienced a partial service outage for some customers, and a small amount of data was dropped during ingestion (not any previously stored data). In terms of impact, 33% of users actively sending data during the incident window experienced partial loss of writes between 6:03 AM and 10:45…
Read More...