Register now for O11yCon 2026. Get early bird rates until April 28th.Register now
Blog

Software Engineering

Establishing and Enabling a Center of Production Excellence
May 15, 2024
Nick Travaglini

Nick Travaglini

Establishing and Enabling a Center of Production Excellence

Software is in a crisis. This is nothing new. Complex distributed systems are perpetually in a state far from equilibrium, operating in what Richard Cook has called a “degraded mode.” It’s through a combination of technical artifacts, organizational practices and policies, and pure gumption that they manage to maintain themselves through time.

Culture
Software Engineering
Simulation Theory, Observability, and Modern Software Practices
April 29, 2024
Aiden Senner

Aiden Senner

Simulation Theory, Observability, and Modern Software Practices

The 1981 book Simulacra and Simulation by Jean Baudrillard is widely read and cited within academic circles but also permeates popular culture, influencing films, literature, and art. His theories notably influenced the Wachowski siblings’ The Matrix series, bringing some of his ideas into mainstream awareness.

Software Engineering
Where Does Honeycomb Fit in the Software Development Lifecycle?
February 2, 2024
Jessica Kerr (Jessitron)

Jessica Kerr (Jessitron)

Where Does Honeycomb Fit in the Software Development Lifecycle?

The software development lifecycle (SDLC) is always drawn as a circle. In many places I’ve worked, there’s no discernable connection between “5. Operate” and “1. Plan.” However, at Honeycomb, there is.

Observability
Software Engineering
Product Managing to Prevent Burnout
January 3, 2024
Winston Hearn

Winston Hearn

Product Managing to Prevent Burnout

I’ve been thinking about a risk that—if I’m not careful—could severely hinder my team’s ability to ship on time, celebrate success, and continue work after launch: burnout. I don’t see burnout mentioned often when the work of product management is discussed, but I believe it should be taken much more seriously.

Software Engineering
Teams & Collaboration
What Do Developers Need to Know About Kubernetes, Anyway?
November 10, 2023
Austin Parker

Austin Parker

What Do Developers Need to Know About Kubernetes, Anyway?

Stop me if you’ve heard this one before: you just pushed and deployed your latest change to production, and it’s rolling out to your Kubernetes cluster. You sip your coffee as you wrap up some documentation when a ping in the ops channel catches your eye—a sales engineer is complaining that the demo environment is slow. Probably nothing to worry about, not like your changes had anything to do with that… but, minutes later, more alerts start to fire off.

Software Engineering
What Happens to DevOps when the Kubernetes Adrenaline Rush Ends?
October 12, 2023
Martin Thwaites

Martin Thwaites

What Happens to DevOps when the Kubernetes Adrenaline Rush Ends?

Kubernetes has been around for nearly 10 years now. In the past five years, we’ve seen a drastic increase in adoption by engineering teams of all sizes. The promise of standardization of deployments and scaling across different types of applications, from static websites to full-blown microservice solutions, has fueled this sharp increase.

Software Engineering
What Is a Feature Flag? Best Practices and Use Cases
September 28, 2023
Emil Protalinski

Emil Protalinski

What Is a Feature Flag? Best Practices and Use Cases

Do you want to build software faster and release it more often without the risks of negatively impacting your user experience? Imagine a world where there is not only less fear around testing and releasing in production, but one where it becomes routine. That is the world of feature flags.

Software Engineering
Rescue Struggling Pods from Scratch
September 25, 2023
Mike Terhar

Mike Terhar

Rescue Struggling Pods from Scratch

Containers are an amazing technology. They provide huge benefits and create useful constraints for distributing software. Golang-based software doesn’t need a container in the same way Ruby or Python would bundle the runtime and dependencies. For a statically compiled Go application, the container doesn’t need much beyond the binary. Since the software is intended to run in a Kubernetes cluster, the container provides the release and distribution mechanism which the Helm chart uses to refer to these binaries. It also allows releasing multiple processor architectures to reference their own images. For general troubleshooting, some pretty good resources exist, like Refinery and the OpenTelemetry Collector.

OpenTelemetry
Software Engineering
Experiments in Daily Work
September 18, 2023
Jessica Kerr (Jessitron)

Jessica Kerr (Jessitron)

Experiments in Daily Work

In high school chemistry and then college physics labs, we learned a strong definition of “experiment.” Experiments are tied to the Scientific Method, responsible for advancement of human knowledge.

Software Engineering
Anything But Tech Debt
August 3, 2023
Emily Nakashima

Emily Nakashima

Anything But Tech Debt

Engineers often feel they aren’t allowed enough time to address tech debt. Product partners wonder why engineers spend so much time working on it—or at least talking about it. “The business” always seems to insinuate that engineers should do less of it, instead focusing on shipping value to customers. And despite all this, many engineering leaders worry their teams may actually be under-investing in tech debt, in ways that could negatively impact the business over the long term.

Software Engineering
Evolving by Involving: Customer Success
July 17, 2023
Nick Travaglini

Nick Travaglini

Evolving by Involving

In this post, we’re going to lay out the guiding principle that unifies the diverse world of CS as we see it—and show how we put it into practice.

Software Engineering
Teams & Collaboration
On Becoming a VP of Engineering, Part 2: Doing the Job
July 14, 2023
Emily Nakashima

Emily Nakashima

On Becoming a VP of Engineering, Part 2: Doing the Job

Charity once said an off-hand sentence that became a mantra for my transition into the VP of Engineering role: “Directors run the company.” This was said in the context of thinking about how the various management roles around the company interact: line managers run teams and projects, directors run the day-to-day work of the company, and execs (including VPs) focus above all on strategy, external-facing matters, and longer-term planning for the company’s future.

Software Engineering
On Becoming a VP of Engineering, Part 1: The Path to VP
July 12, 2023
Emily Nakashima

Emily Nakashima

On Becoming a VP of Engineering, Part 1: The Path to VP

In February of 2020, I was promoted from Director of Engineering to Honeycomb’s first VP of Engineering. Although Charity wrote an extremely generous public announcement, I hesitated to talk about this new role for quite a while as I was figuring out the job. But since 2020, I’ve noticed how little candid writing there is about paths to the VPE role or what the job is really like.*

Software Engineering
Improving LLMs in Production With Observability
June 16, 2023
Phillip Carter

Phillip Carter

Improving LLMs in Production With Observability

In early May, we released the first version of our new natural language querying interface, Query Assistant. We also talked a lot about the hard stuff we encountered when building and releasing this feature to all Honeycomb customers. But what we didn’t talk about was how we know how our use of an LLM is doing in production! That’s what this post is all about.

AI & LLMs
Software Engineering
How Traceloop Leverages Honeycomb and LLMs to Generate E2E Tests
May 30, 2023
Nir Gazit

Nir Gazit

How Traceloop Leverages Honeycomb and LLMs to Generate E2E Tests

At Traceloop, we’re solving the single thing engineers hate most: writing tests for their code. More specifically, writing tests for complex systems with lots of side effects, such as this imaginary one, which is still a lot simpler than most architectures I’ve seen.

Software Engineering
Tracing
All the Hard Stuff Nobody Talks About when Building Product with LLMs
May 26, 2023
Phillip Carter

Phillip Carter

All the Hard Stuff Nobody Talks About when Building Products with LLMs

There’s a lot of hype around AI, and in particular, Large Language Models (LLMs). To be blunt, a lot of that hype is just some demo bullshit that would fall over the instant anyone tried to use it for a real task that their job depends on. The reality is far less glamorous: it’s hard to build a real product backed by an LLM.

AI & LLMs
Software Engineering
5 Ways Honeycomb Saves Time, Money, and Sanity
May 18, 2023
Valerie Silverthorne

Valerie Silverthorne

5 Ways Honeycomb Saves Time, Money, and Sanity

If debugging has sucked the soul out of your engineers, we’ve got the answer: event-based observability. Instead of spending hours and resources trying to find out why an alert is sounding, event-based observability can quickly surface the correct cause of any issue. Here are five concrete ways teams can save time, money, and sanity using Honeycomb’s event-based approach to observability.

Observability
Software Engineering
How We Use Smoke Tests to Gain Confidence in Our Code
May 16, 2023
Jamie Danielson

Jamie Danielson

How We Use Smoke Tests to Gain Confidence in Our Code

Also known as confidence testing, smoke testing is intended to focus on some critical aspects of the software that are required as a baseline. The term originates in electronic hardware testing; as my colleague Robb Kidd stated, “We want to apply power and see if smoke comes out.” If a smoke test fails, there is almost definitely a problem to address. If it passes, it doesn’t necessarily mean there is no problem, but we can feel confident that the major functionality is okay.

Software Engineering
Generalists vs. Specialists: Figuring Out Your Path
April 4, 2023
MJ Jayaraman

MJ Jayaraman

Generalists vs. Specialists: Figuring Out Your Path

Last month, I had the opportunity to join Jen Dary on her podcast, Be Plucky. The conversation was great, and we covered a lot of topics. We unpacked the merits and nuances of growing as a generalist and/or a specialist. I also had the chance to reflect on and share my experience in changing roles and making a few specialization shifts in my career.

Software Engineering
How Do We Cultivate the End User Community Within Cloud-Native Projects?
March 20, 2023
Rynn Mancuso

Rynn Mancuso

How Do We Cultivate the End User Community Within Cloud-Native Projects?

The open source community talks a lot about the problem of aligning incentives. If you’re not familiar with the discourse, most of this conversation so far has centered around the most classic model of open source: the solo unpaid developer who maintains a tiny but essential library that’s holding up half the internet. For example, Denis Pushkarev, the solo maintainer of popular JavaScript library core-js, announced that he can’t continue if he’s not better compensated.

Software Engineering