Try Honeycomb Intelligence, AI-native observability with MCP, Anomaly Detection, and CanvasLearn more

Explore Blog

Getting Started With Log Management
September 6, 2024
Rox Williams

Rox Williams

Getting Started With Log Management

Whether you’re using logs to debug issues, keeping an eye on system performance, or protecting your infrastructure, good log management hygiene can make a huge difference. In this article, we’ll teach you the basics of log management, why it’s so important, and how you can contribute to important business goals.

Logging
The Evolution of Engineering and the Role of Observability 2.0 in Shaping the Future
September 5, 2024
Brian Chang

Brian Chang

The Evolution of Engineering and the Role of Observability 2.0 in Shaping the Future

Engineering has come a long way since the days of delivering discrete, point-in-time products that were often packaged on a CD and shipped to customers. The days of physical media and long development cycles are long gone. The advent of cloud computing and the rise of Software-as-a-Service (SaaS) transformed the landscape, creating a new model of continuous development and service delivery. This shift has not only revolutionized how software is developed, but has also redefined the engineer’s role.

Observability
Software Engineering
Getting Started with Refinery: Rules File Template
September 3, 2024
Max Aguirre

Max Aguirre

Getting Started With Refinery: Rules File Template

Sampling is a necessity for applications at scale. We at Honeycomb sample our data through the use of our Refinery tool, and we recommend that you do too. But how do you get started? Do you simply a set rate for all data and a handful of drop and keep rules, or is there more to it? What do these rules even mean, and how do you implement them?

Sampling
Always. Enable. Keepalives.
August 26, 2024
Lex Neva

Lex Neva

Always. Enable. Keepalives.

As part of our recent failure testing project, we ran into an interesting failure mode involving the OpenTelemetry SDK for Go. In this post, we’ll show you why our apps stopped sending telemetry for over 15 minutes and how we enabled keepalives to prevent this kind of failure from happening in the future.

Debugging
Dogfooding
Software Engineering
What Is Full-Stack Observability?
August 22, 2024
Rox Williams

Rox Williams

What Is Full-Stack Observability?

Simply put, full-stack observability is monitoring designed for modern, cloud-native architectures. It allows you to understand how your software system interacts at scale, across everything from traditional mainframes and legacy clients to modern serverless or Kubernetes-based services.

Observability
Aligning Business and Engineering Goals with Honeycomb SLOs
August 19, 2024
Priscilla Lam

Priscilla Lam

Aligning Business and Engineering Goals with Honeycomb SLOs

Setting clear, measurable goals is essential for any successful team. However, aligning those goals with the technical work can be challenging in the fast-paced world of software engineering. Engineers might focus on reducing latency or improving uptime, while business leaders look at revenue and customer satisfaction. It gets tricky to track the impact between the two to justify when specific engineering initiatives are important, why, and how they impact the bottom line. Everyone may feel the work is important, but it’s hard to see or remember why!

Product Updates
Service Level Objectives
A CoPE’s Guide to Alert Management
August 15, 2024
Nick Travaglini

Nick Travaglini

A CoPE’s Guide to Alert Management

Alerts are a perennial topic, and a CoPE will need to engage with them. The bounds of this problem space are formed by two types of alerts:  Reactive alerts (in Honeycomb, we call these Triggers): They are alerts that fire after some event, like crossing a pre-determined boundary.  Proactive alerts (Burn Alerts based on Honeycomb’s SLO feature): These give notice before crossing a threshold; in the case of SLOs, that means before failing to meet the stated objective.

Culture
Software Engineering
The CoPE and Other Teams, Part 2: Custom Instrumentation and Telemetry Pipelines
August 8, 2024
Nick Travaglini

Nick Travaglini

The CoPE and Other Teams, Part 2: Custom Instrumentation and Telemetry Pipelines

The previous post laid out the basic idea of instrumentation and how OpenTelemetry’s auto-instrumentation can get teams started. However, you can’t rely only on auto-instrumentation. This post will discuss the limitations in more detail and how a CoPE can help teams overcome them.

Culture
Software Engineering
Deploying the OpenTelemetry Collector to AKS
August 7, 2024
Martin Thwaites

Martin Thwaites

Deploying the OpenTelemetry Collector to AKS

The Collector is the focal point for telemetry inside your cluster. Instead of your containerized applications sending directly to your OpenTelemetry-capable backend (the place that allows you to ask questions of your telemetry), we send that data to an internal location first, then forward the data on.

OpenTelemetry
Tutorials
Apdex in Honeycomb
August 5, 2024
Max Aguirre

Max Aguirre

Apdex in Honeycomb

“How is my app performing?” is one of the most common, yet hardest questions to answer. There are myriad ways to measure this, like error rate, average response time, and so on. Enter the Application Performance Index (aka Apdex), a single metric that attempts to answer, “Are my application’s users happy?”

Observability
Software Engineering
Making Room for Some Lint
July 29, 2024
Fred Hebert

Fred Hebert

Making Room for Some Lint

It’s one of my strongly held beliefs that errors are constructed, not discovered. However we frame an incident’s causes, contributing factors, and context ends up influencing the shape of the corrective items (if any) that get created. I’ll cover these ideas by using our June 3rd incident where a database migration caused a large outage by locking up a shared database and making it run out of connections.

Dogfooding
Incident Response
Software Engineering
The CoPE and Other Teams, Part 1: Introduction & Auto-instrumentation
July 25, 2024
Nick Travaglini

Nick Travaglini

The CoPE and Other Teams, Part 1: Introduction & Auto-Instrumentation

The CoPE is made to affect, meaning change, how things work. The disruption it produces is a feature, not a bug. That disruption pushes things away from a locally optimal, comfortable state that generates diminishing returns. It sets things on a course of exploration to find new terrains which may benefit it more—and for longer.

Culture
Software Engineering
Destroy on Friday: The Big Day 🧨 A Chaos Engineering Experiment – Part 2 
July 23, 2024
Lex Neva

Lex Neva

Destroy on Friday: The Big Day 🧨 A Chaos Engineering Experiment – Part 2

In my last blog post, I explained why we decided to destroy one third of our infrastructure in production just to see what would happen. This is part two, where I go over the big day. How did our chaos engineering experiment go? Find out below!

Dogfooding
Software Engineering
What Makes for a ‘Good’ Pair Programming Session?
July 18, 2024
Ruthie Irvin

Ruthie Irvin

What Makes for a ‘Good’ Pair Programming Session?

Software changes so rapidly that developing on the cutting edge of it cannot fall to a single person. When it comes to asynchronously disseminating information about projects, code comments, PR conversations, Slack, RFCs, and other investigatory documents do a wonderful job, but no amount of async communication replaces the magic of two brains bouncing ideas off of each other.

Culture
Software Engineering
Teams & Collaboration
😫 Tired: Deploy on Friday / 🤯 Wired: Destroy on Friday! A Chaos Engineering Experiment – Part 1
July 16, 2024
Lex Neva

Lex Neva

Deploy on Friday? How About Destroy on Friday! A Chaos Engineering Experiment – Part 1

We recently took a daring step to test and improve the reliability of the Honeycomb service: we abruptly destroyed one third of the infrastructure in our production environment using AWS’s Fault Injection Service. You might be wondering why the heck we did something so drastic. In this post, we’ll go over why we did it and how we made sure that it wouldn’t impact our service.

Dogfooding
Software Engineering
Staffing Up Your CoPE
July 10, 2024
Nick Travaglini

Nick Travaglini

Staffing Up Your CoPE

Getting the right people working in the CoPE is crucial to success because these change agents must limber up the organization and promote the flexibility necessary to perform resilience.

Culture
Software Engineering
Why Every Engineering Team Should Embrace AWS Graviton4
July 9, 2024
Liz Fong-Jones

Liz Fong-Jones

Why Every Engineering Team Should Embrace AWS Graviton4

Two years ago, we shared our experiences with adopting AWS Graviton3 and our enthusiasm for the future of AWS Graviton and Arm. Once again, we’re privileged to share our experiences as a launch customer of the Amazon EC2 R8g instances powered by AWS Graviton4, the newest generation of AWS Graviton processors.

Dogfooding
Featured
The Hater’s Guide to Dealing with Generative AI
July 3, 2024
Tyler Wilson

Tyler Wilson

The Hater’s Guide to Dealing with Generative AI

Generative AI is having a bit of a moment—well, maybe more than just a bit. It’s an exciting time to be alive for a lot of people. But what if you see stories detailing a six month old AI firm with no revenue seeking a $2 billion valuation and feel something other than excitement in the pit of your stomach?

Conferences & Meetups
AI & LLMs
Navigating Software Engineering Complexity With Observability
June 27, 2024
Rox Williams

Rox Williams

Navigating Software Engineering Complexity With Observability

In the not-too-distant past, building software was relatively straightforward. The simplicity of LAMP stacks, Rails, and other well-defined web frameworks provided a stable foundation. Issues were isolated, systems failed in predictable ways, and engineers had time to innovate on new features for the business. And it was good.

Observability
Software Engineering
OpenTelemetry Best Practices #3: Data Prep and Cleansing
June 24, 2024
Martin Thwaites

Martin Thwaites

OpenTelemetry Best Practices #3: Data Prep and Cleansing

Having telemetry is all well and good—amazing, in fact. It’s easy to do: add some OpenTelemetry auto-instrumentation libraries to your stack and they’ll fill your disks with data pretty quickly. However, having good telemetry data—data that’s curated into being useful—is something that is both cost-effective and represents good value.

OpenTelemetry