A CoPE’s Guide to Alert Management
Alerts are a perennial topic, and a CoPE will need to engage with them. The bounds of this problem space are formed by two types...
The CoPE and Other Teams, Part 2: Custom Instrumentation and Telemetry Pipelines
The previous post laid out the basic idea of instrumentation and how OpenTelemetry’s auto-instrumentation can get teams started. However, you can’t rely only on auto-instrumentation....
Apdex in Honeycomb
“How is my app performing?” is one of the most common, yet hardest questions to answer. There are myriad ways to measure this, like error...
Making Room for Some Lint
It’s one of my strongly held beliefs that errors are constructed, not discovered. However we frame an incident’s causes, contributing factors, and context ends up...
The CoPE and Other Teams, Part 1: Introduction & Auto-Instrumentation
The CoPE is made to affect, meaning change, how things work. The disruption it produces is a feature, not a bug. That disruption pushes things...
Destroy on Friday: The Big Day 🧨 A Chaos Engineering Experiment - Part 2
In my last blog post, I explained why we decided to destroy one third of our infrastructure in production just to see what would happen....
What Makes for a 'Good' Pair Programming Session?
Software changes so rapidly that developing on the cutting edge of it cannot fall to a single person. When it comes to asynchronously disseminating information...
Deploy on Friday? How About Destroy on Friday! A Chaos Engineering Experiment - Part 1
We recently took a daring step to test and improve the reliability of the Honeycomb service: we abruptly destroyed one third of the infrastructure in...
Staffing Up Your CoPE
Getting the right people working in the CoPE is crucial to success because these change agents must limber up the organization and promote the flexibility...
Navigating Software Engineering Complexity With Observability
In the not-too-distant past, building software was relatively straightforward. The simplicity of LAMP stacks, Rails, and other well-defined web frameworks provided a stable foundation. Issues...
Framework for an Observability Maturity Model: Using Observability to Advance Your Engineering & Product
Everyone's talking about “observability,” but many don’t know what it is, what it’s for, or what benefits it offers. With this framing of observability in...
Investigating Mysterious Kafka Broker I/O When Using Confluent Tiered Storage
Earlier this year, we upgraded from Confluent Platform 7.0.10 to 7.6.0. While the upgrade went smoothly, there was one thing that was different from previous...
Independent, Involved, Informed, and Informative: The Characteristics of a CoPE
In part one of our CoPE series, we analogized the CoPE with safety departments. David Woods says that those safety departments must be: independent, involved,...
Establishing and Enabling a Center of Production Excellence
Software is in a crisis. This is nothing new. Complex distributed systems are perpetually in a state far from equilibrium, operating in what Richard Cook...
Simulation Theory, Observability, and Modern Software Practices
The 1981 book Simulacra and Simulation by Jean Baudrillard is widely read and cited within academic circles but also permeates popular culture, influencing films, literature,...