Honeycomb Blog

Development at Honeycomb: Crossing the Observability Bridge to Production

For years, the “DevOps” community has felt focused on one main idea: what if we pushed our ops folks to do more development? To automate their work and write more code? That’s cool—and it’s clearly been working—but it’s time for the second wave of that movement: for developers (that’s us!) to own our code in production, and to be on point for operating / exploring our apps in the wild. Observability is that bridge: the bridge from developers understanding code running on local machines to understanding how it behaves in the wild. Observability is all about answering questions about your…

Read More...

Measure your Chef runs with Honeycomb

At Honeycomb, we love Chef! We wanted to know more about what was going on with our Chef runs, and what better tool to use to find out than Honeycomb?

Read More...

How Honeycomb Uses Honeycomb Part 8: A Bee’s Life

This post continues our dogfooding series from How Honeycomb Uses Honeycomb, Part 7: Measure twice, cut once: How we made our queries 50% faster…with data. To understand how Honeycomb uses Honeycomb at a high level, check out our dogfooding blog posts first — they do a better job of telling the story of problems we’ve solved with Honeycomb. This blog post peeks under the hood to go into greater detail around the mechanics of what we track, how we track it all, and how we think about the sorts of questions we want to answer. We’ve built up a culture…

Read More...

Honeycombers at LISA 2017

Did you go to LISA this year? I used to go back in the 1998-2003 timeframe (anyone remember playing the original Guitar Hero in that huge arcade in Seattle?) and I hope to make it back again someday soon. A lot of time has passed since those days, but the conference continues to offer attendees a wide range of useful and educational talks to choose from. In particular, the content on operating at scale has evolved upward much like the definition of “Large” since the conference’s inception 🙂 A couple of Honeycombers presented at LISA this year–here’s what they talked…

Read More...

How Honeycomb Uses Honeycomb, Part 3: End-to-End Failures

This post continues our dogfooding series from How Honeycomb Uses Honeycomb, Part 2: Migrating API Versions. At Honeycomb, one of our foremost concerns (in our product as well as our customers’) is reliability. To that end, we have end-to-end (e2e) checks that run each minute, write a single data point to Honeycomb, then expect that specific data point to be available for reads within a certain time threshold. If this fails, the check will retry up to 30 times before giving up. Monday morning, we were notified of some intermittent, then persistent, errors in these automated checks. We quickly verified…

Read More...

Honeycomb Triggers – Alert on your Data

We’re happy to announce the launch of Honeycomb Triggers—a method to get notifications when the data you send in to Honeycomb crosses configured thresholds. We’d like to show off how to use Triggers with a practical example. Check out the docs for more a conventional description of how to use them. We have a scheduled job that submits an event to Honeycomb and then attempts to read it back from the UI. This end to end (e2e) check catches overall problems that individual checks might miss. The job submits a report of its success (or failure) to our dogfooding Honeycomb…

Read More...

Nylas Guest Post: Ghosts in the WSGI Machine

This post is by Christine Spang, co-founder and CTO at Nylas, where they are building a better way to do email. They have a hybrid system of existing graphite and new Honeycomb analytics, and have been experimenting with using Honeycomb for debugging. Enjoy the story and screenshots … thanks Spang!!. A little while ago one of our engineers, Jackie, was on call at Nylas when she got an alert that said: Average API latency excluding draft_send, multi_send, & sync_deltas is over 1500ms. This is one of our core API alerts. It’s a really important alert, because it means the API…

Read More...

Part 4/5: Everyone is a DBA

DBAs may be the last remaining priesthood in our industry. But software engineers and operations engineers are increasingly finding themselves responsible for precious company data, and DBAs are increasingly adopting generalist skillsets and best practices. Not everyone is thrilled about this There is a surprising amount of fear and trembling among non-DBA engineers when it comes to managing data, because it has an alien feel and errors can be so permanent. Honeycomb can help. 🙂 Storage is one of our specialties. We’ve run massive, world-class clusters in MongoDB and MySQL and have extensive experience in plenty of other databases. This…

Read More...

Part 3/5: Dear Operations Engineers …

It’s time to shrug off the last vestiges of that martyr complex we’ve been trudging around with since the bad old days of the BOFH. We’ve got better things to do with our lives than being assholes to everyone. Stop trying to predict every possible failure — you can’t, anyhow – and stop toiling away half your life creating dashboards for people, dashboards of dashboards, and ways to auto-generate dashboards and metrics (that nobody can ever seem to find when they most need them). Honeycomb grew out of the best of the operations and data disciplines. We are grounded in…

Read More...

How Honeycomb Uses Honeycomb, Part 2: Migrating API Versions

Continued from How Honeycomb Uses Honeycomb, Part 1: The Long Tail. We recently released a new version of our API. As scarred veterans of building and supporting APIs, we made sure to retain backwards compatibility. Our code paths are versioned, but we’ve also versioned our docs to match. These things are good for developers even though they come at a cost — it takes time and effort to make sure that things continue to work. We have a strong interest in encouraging our customers to upgrade to the new API, especially at such an early stage, when we are shipping…

Read More...