Honeycomb Blog

How Honeycomb Uses Honeycomb, Part 9: Tracing The Query Path

This post continues our long-running dogfooding series from How Honeycomb Uses Honeycomb Part 8: A Bee’s Life. To understand how Honeycomb uses Honeycomb at a high level, check out our dogfooding blog posts first — they do a great job of telling the story of problems we’ve solved with Honeycomb. Last week we announced the general availability of tracing in Honeycomb. We’ve been dogfooding this feature extensively as part of an ongoing effort to keep Honeycomb fast and reliable. In this post, we’ll discuss how we use tracing internally, some of the ways it’s helped us improve our service, and some of the…

Read More...

Get deeper insights with Honeycomb Tracing

gif of looking at traces for api query timings

We’re excited to introduce Honeycomb Tracing! Now, you can both: Visualize individual traces to deeply understand request execution, and Break down, filter, and aggregate trace data to uncover patterns, find outliers, and understand historical trends. Tracing makes it easier to understand control flow within a distributed system. Waterfall diagrams concisely capture the execution history of individual requests, making it easy to answer questions such as: What does the high-level structure of the code look like? Which methods are especially slow? Are we making too many calls to a particular service? Are things happening serially when they could be parallelized? Is…

Read More...

Honeycomb goes Serverless: Send your app data without running agents

bee giving a "high five" to a person's finger

Serverless apps are growing in popularity, thanks to tools like AWS API Gateway and Lambda, and a growing number of powerful frameworks that simplify development and deployment. Complex applications are still complex, however, and regardless of your platform you’ll still need to think about observability. Today, we’re excited to announce our Agentless Integrations for AWS. There are a lot of ways to get data into honeycomb. Most of these – such as honeytail, our HoneyAWS suite, or the Kubernetes agent – require you to run an agent somewhere in your infrastructure. Our agentless integrations are different: they hook into the…

Read More...

Honeycombers at Influx Days (a late-ish report)

I finally got a chance to sit down and watch Emily and Christine’s talks from last November’s InfluxDays and right off the bat I have to apologize for not doing this sooner, because I certainly learned some stuff worth sharing! If you haven’t gotten to these yet, you should check them out: WHAT YOUR JAVASCRIPT DOES WHEN YOU’RE NOT AROUND As one might expect, Emily Nakashima’s talk was chock full of useful recommendations and her trademark understated humor. She points out (is that glee in her voice?) that in this era of ever-growing complexity, “your company is probably writing and…

Read More...

Troubleshooting in Honeycomb: Choose Your Own Adventure

When debugging, there’s often not a single right answer or path to follow. More often than not, two teammates will find themselves producing two different sets of Honeycomb queries, following different routes, yet still arriving at the same conclusion. With real, full-stack observability, you don’t have to walk the same path as everyone else to get to the truth.

Read More...

Bitten by a Kafka Bug – Postmortem

Dearest honeycombers, Yesterday, on Tuesday, Oct 17th, we experienced a partial service outage for some customers, and a small amount of data was dropped during ingestion (not any previously stored data). In terms of impact, 33% of users actively sending data during the incident window experienced partial loss of writes between 6:03 AM and 10:45 AM PDT. Of those, most lost less than half of their writes. In addition to this, a majority of users experienced a 30 minute period of partial read availability between 10:50 and 11:20 AM PDT. The type of outage we experienced was a new one…

Read More...

Nylas Makes Their API buzz with Honeycomb

At Honeycomb, we <3 Nylas, so we were super pleased to come across this short talk by Evan Morikawa (aka @e0m on Twitter): Evan does a great job of explaining the goals Nylas set for speeding up their API, and breaks the process they went through down into easy-to-understand steps, including a quick demo of our shiny new heatmap feature (which they got a little early, because we love them 😀 😀 :D) And we’re of course super pleased to hear that Nylas loves us right back! Evan goes on to talk about another cool approach to gaining observability into…

Read More...

Observability: What’s in a Name?

“Is observability just monitoring with another name?” “Observability: we changed the word because developers don’t like monitoring.” There’s been a lot of hilarious snark about this lately. Which is great, who doesn’t love A+ snark? Figured I’d take the time to answer, at least once. Yes, in practice, the tools and practices for monitoring vs observability will overlap a whole lot … for now. But philosophically there are some subtle distinctions, and these are only going to grow over time.* “Monitoring”, to anyone who’s been in the game a while, carries certain connotations that observability repudiates. It suggests that you…

Read More...

Our First Outage

Dear honeycomb users, On Saturday, Aug 19th, we experienced a service outage for all customers. This was our first-ever outage, even though we’ve had users in production for almost exactly one year, and paying customers for about 6 months. We’re pretty proud of that, but also overdue for an outage. We take production reliability very seriously for our customers. We know you rely on us to be available so you can debug your own systems, so we’ve always invested effort into defensive engineering and following best practices for a massive, multitenant system. We learned a lot from this outage, so…

Read More...

Event Foo: A Series of Unfortunate/Incredible Events

Good technical intuition is one of the things that defines a good senior engineer. And unpacking that intuition is the most valuable teaching tool. By making your implicit assumptions and experiences explicit, others can learn from them. Lately I’ve been having a lot of conversations with people about debugging with events, or structured log data, with a dawning realization that this is not a universal experience. What I thought was as normal as code reviews is an intuition fed by some highly specific experiences. Events seem ordinary and intuitive to many engineers—that’s how our brain works, after all—but not all…

Read More...