Incident Report: Running Dry on Memory Without Noticing
On November 6, 2019, we intermittently rejected 1-3% of customer telemetry data at ingest for four periods of 20 minutes each. The trigger of the incident was a slow memory leak that manifested over…
The Hive at Monitorama Baltimore 2019
As a human who entered this space via marketing, I often turn to my teammates for context. I recently asked the hive, why do we like Monitorama so much? Briefly: well curated community-driven event…
OpenTelemetry, OpenTracing, OpenCensus: An Introduction and Glossary
There’s been a fair bit of buzz lately about OpenTelemetry, which is the next major version of the OpenTracing and OpenCensus projects. The leadership of those two projects have come together to create OpenTelemetry,…
Becoming A Manager of Engineering Managers: What You Need To Know
In October Charity Majors joined Ian Nowland [VP | Eng | Datadog] and Rich Archibold [Sr Dir | Eng | Intercom] in a live panel discussion for Gitprime. Hosted by Marcus Blankenship, Becoming a…
Honeycomb at PagerDuty Summit
We had an excellent time at PagerDuty Summit 2019. The folks we met at the booth and in the hallway track felt particularly kindred to Team Honeybee: we’ve all been on-call, and we all…
A Culture of Observability Helps Engineers Hit the Spot (Instance)
At Honeycomb, we’re big fans of AWS Spot Instances. During a recent bill reduction exercise, we found significant savings in running our API service on Spot, and now look to use it wherever we…
Working Toward Service Level Objectives (SLOs), Part 1
In theory, Honeycomb is always up. Our servers run without hiccups, our user interface loads rapidly and is highly responsive, and our query engine is lightning fast. In practice, this isn’t always perfectly the…
Honeycomb Begins Another Chapter with a New Funding Round
It’s an exciting time here at Honeycomb HQ—today, we’re thrilled to announce our recent fundraise led by Scale Venture Partners, and to welcome Ariel Tseitlin to our board. On the first day of 2016,…
So You Want To Build An Observability Tool…
I’ve said this before, but I’m saying it again: observability is not a synonym for monitoring, and there are no three pillars. The pillars are bullshit. Briefly: monitoring is how you manage your known-unknowns,…
Working On Hitting a Release Cadence? CI/CD Observability Can Help You Get There
We recently sponsored our partner CloudBees’ conference DevOps World & JenkinsWorld in San Francisco and our message “Observe how Customers Experience Your Build” resonated well with the folks we met. Release engineers are critical…