Try Honeycomb Intelligence, AI-native observability with MCP, Anomaly Detection, and CanvasLearn more

Explore Blog

Trace Propagation and Public API Endpoints in .NET: Part 1 (Disable All)
August 11, 2023
Martin Thwaites

Martin Thwaites

Trace Propagation and Public API Endpoints in .NET: Part 1 (Disable All)

One of the issues with the W3C trace context is that it doesn’t define any standards for how far a trace is to propagate. If a third party accidentally sends trace headers from their service, you’ll use their trace IDs and baggage data. This can have unwanted affects on your telemetry backend, such as the trace showing missing root spans, or including multiple API calls in a single trace at the top level. This makes understanding and debugging trace data hard. Worse though, the baggage data from the third party could contain PII data, which would therefore mean you’re processing PII without realizing it.

Tracing
Anything But Tech Debt
August 3, 2023
Emily Nakashima

Emily Nakashima

Anything But Tech Debt

Engineers often feel they aren’t allowed enough time to address tech debt. Product partners wonder why engineers spend so much time working on it—or at least talking about it. “The business” always seems to insinuate that engineers should do less of it, instead focusing on shipping value to customers. And despite all this, many engineering leaders worry their teams may actually be under-investing in tech debt, in ways that could negatively impact the business over the long term.

Software Engineering
Automatic Instrumentation for OpenTelemetry Go
August 1, 2023
Phillip Carter

Phillip Carter

Automatic Instrumentation for OpenTelemetry Go

The OpenTelemetry Go project now supports automatic instrumentation via eBPF! This is a big milestone for the project and makes it significantly easier to generate data from your Go apps.

OpenTelemetry
The Evolution of Sampling in Honeycomb: Introducing Refinery 2.0
July 26, 2023
Kent Quirk

Kent Quirk

The Evolution of Sampling in Honeycomb: Introducing Refinery 2.0

It’s rare to have too much telemetry—it’s not often that someone says “I wish I didn’t have all this information!” However, telemetry is data, and data is not necessarily information—particularly when you’re drowning in it. Honeycomb’s query engine is so fast and powerful that many customers can send us all their telemetry. As we say on our stickers, “The Backend Can Handle It.”

Sampling
Incident Management Steps and Best Practices
July 20, 2023
Valerie Silverthorne

Valerie Silverthorne

Incident Management Steps and Best Practices

Incident management is the way an organization reacts to any kind of outage (security, broken code, severe weather, or anything that’s disruptive to customer service). Incidents are inherently fraught, not just because they’re time consuming and costly, but because they can potentially poison the well with customers, investors, and even partners.

Incident Response
Evolving by Involving: Customer Success
July 17, 2023
Nick Travaglini

Nick Travaglini

Evolving by Involving

In this post, we’re going to lay out the guiding principle that unifies the diverse world of CS as we see it—and show how we put it into practice.

Software Engineering
Teams & Collaboration
On Becoming a VP of Engineering, Part 2: Doing the Job
July 14, 2023
Emily Nakashima

Emily Nakashima

On Becoming a VP of Engineering, Part 2: Doing the Job

Charity once said an off-hand sentence that became a mantra for my transition into the VP of Engineering role: “Directors run the company.” This was said in the context of thinking about how the various management roles around the company interact: line managers run teams and projects, directors run the day-to-day work of the company, and execs (including VPs) focus above all on strategy, external-facing matters, and longer-term planning for the company’s future.

Software Engineering
Observing Core Web Vitals with OpenTelemetry
July 13, 2023
Purvi Kanal

Purvi Kanal

Observing Core Web Vitals with OpenTelemetry

Each CWV measures a specific part of the end user experience. CWV scores can help identify gaps in web page performance. Additionally, Google uses CWV scores as one of the measures it uses to rank pages, which means they are important for SEO.

Frontend
Observability
OpenTelemetry
On Becoming a VP of Engineering, Part 1: The Path to VP
July 12, 2023
Emily Nakashima

Emily Nakashima

On Becoming a VP of Engineering, Part 1: The Path to VP

In February of 2020, I was promoted from Director of Engineering to Honeycomb’s first VP of Engineering. Although Charity wrote an extremely generous public announcement, I hesitated to talk about this new role for quite a while as I was figuring out the job. But since 2020, I’ve noticed how little candid writing there is about paths to the VPE role or what the job is really like.*

Software Engineering
We Did it Again: We’re a Leader in 2023 Gartner® Magic Quadrant™ for APM & Observability for the Second Year in a Row
July 10, 2023
Charity Majors

Charity Majors

We Did it Again: We’re a Leader in 2023 Gartner® Magic Quadrant™ for APM & Observability for the Second Year in a Row

When the Gartner Magic Quadrant Report came out in 2022, we did the professional equivalent of a spit take, then cheered wildly. NOT ONLY did they include observability for the first time ever in their newly revamped 2022 Magic Quadrant for APM & [now] Observability, but they also put us in the Leader Quadrant—our debut appearance!

Featured
News & Announcements
How Metrics Behave in Honeycomb
July 7, 2023
Mike Terhar

Mike Terhar

How Metrics Behave in Honeycomb

Honeycomb has the ability to receive events from application. These events can take the shape of Honeycomb wide events, OpenTelemetry trace spans, and OpenTelemetry metrics. Because Honeycomb’s backend is very flexible, these OpenTelemetry signals fit in just fine—but sometimes, they have a few quirks. Let’s dive into using metrics the Honeycomb way and cover a few optimizations.

Metrics
Driving Exceptional Support: Unleashing Support Power with Honeycomb
July 6, 2023
Roel Vista

Roel Vista

Driving Exceptional Support: Unleashing Support Power with Honeycomb

n technical support, ensuring customer satisfaction and quickly resolving issues are of utmost importance. At Honeycomb, we embrace a comprehensive approach by using our own platform—not only for engineering purposes, but to also empower our support team.

Dogfooding
Observability
How to Trial Honeycomb and OpenTelemetry
June 27, 2023
Kyle Moonwright

Kyle Moonwright

How to Trial Honeycomb and OpenTelemetry

Insightful proof-of-concepts with a tool can be difficult to undertake due to the demands on valuable resources: time, energy, and people. With a task as grand as observability, how could one truly test if Honeycomb and OpenTelemetry are right for their organization and meet their requirements? For this thought experiment, here’s a comprehensive description of the ideal product evaluation over the course of four weeks, given unlimited resources.

Observability
OpenTelemetry
There Are No Repeat Incidents
June 26, 2023
Fred Hebert

Fred Hebert

There Are No Repeat Incidents

People seem to struggle with the idea that there are no repeat incidents. It is very easy and natural to see two distinct outages, with nearly identical failure modes, impacting the same components, and with no significant action items as repeat incidents. However, when we look at the responses and their variations, we can find key distinctions that shows the incidents as related, but not identical.

Incident Response
Exotic Trace Shapes
June 23, 2023
Mike Terhar

Mike Terhar

Exotic Trace Shapes

OpenTelemetry and Beelines were designed with assumptions about the types of traffic that most users would trace. Based on these assumptions, web application and API calls fit very nicely into a trace waterfall view. This is also the set of assumptions for Refinery, our sampling proxy, to manage the traces it processes. These assumptions are true for most traffic.

Tracing
What Observability-Driven Development Is Not
June 21, 2023
Jessica Kerr

Jessica Kerr

What Observability-Driven Development Is Not

At Honeycomb, we are all about observability. In the past, we have proposed observability-driven development as a way to maximize your observability and supercharge your development process. But I have a problem with the terminology, and it is: I don’t want observability to drive your development.

Observability
Improving LLMs in Production With Observability
June 16, 2023
Phillip Carter

Phillip Carter

Improving LLMs in Production With Observability

In early May, we released the first version of our new natural language querying interface, Query Assistant. We also talked a lot about the hard stuff we encountered when building and releasing this feature to all Honeycomb customers. But what we didn’t talk about was how we know how our use of an LLM is doing in production! That’s what this post is all about.

LLMs
Software Engineering
How Our Love of Dogfooding Led to a Full-Scale Kubernetes Migration
June 14, 2023
Ian Smith

Ian Smith

How Our Love of Dogfooding Led to a Full-Scale Kubernetes Migration

When considering a migration to Kubernetes, as with any major tech upgrade or change, it’s imperative to understand the motivation for doing so. The engineering time and labor to execute a complex migration will take away from other priorities, making it crucial to have org-wide alignment on why the change makes sense.

Dogfooding
Don’t Let Observability Inflate Your Cloud Costs
June 13, 2023
Martin Thwaites

Martin Thwaites

Don’t Let Observability Inflate Your Cloud Costs

We saw a shift this year in how the technology sector honed in on sustainability from a cost perspective. In particular, looking at where they’re spending that revenue in the infrastructure and tooling space. Observability tooling comes under a lot of scrutiny as it’s perceived as a large cost center—and one that could be cut without affecting revenue. After all, if the business hasn’t had a problem in the last few months, we mustn’t need monitoring—right?

Observability
How Honeycomb Monitors Kubernetes
June 12, 2023
Nathan Lincoln

Nathan Lincoln

How Honeycomb Monitors Kubernetes

While Kubernetes comes with a number of benefits, it’s yet another piece of infrastructure that needs to be managed. Here, I’ll talk about three interesting ways that Honeycomb uses Honeycomb to get insight into our Kubernetes clusters. It’s worth calling out that we at Honeycomb use Amazon EKS to manage the control plane of our cluster, so this document will focus on monitoring Kubernetes as a consumer of a managed service.

Observability