Ask Miss O11y: How Can I Convince My Organization to Invest in Instrumenting for Observability?

We recently hosted a Twitter Space, and a question came in regarding speaking to executives about instrumenting for observability. It’s a great topic we love expanding on. Here’s the answer we provided. 

If you’d like to join the next Twitter Space, follow @honeycombio. We’ll post information there about it.

“Dear Miss O11y,    

I’ve been following Honeycomb for a long time, and I understand where the insights from observability fit in. But larger orgs haven’t experienced this yet.

When you’re talking to a C-level executive or director, how do you speak to this? What success stories do you cite that have traction at this level?”

- Jason

Success Stories

Charity: Most of our customers care about the experience of their individual end users. Not every company, of course; but most companies, financial institutions, healthcare delivery, etc., they care. If all you have are monitoring tools, you can't answer questions about what the experience is like for any given route or request or error or payment. You need the high cardinality and high dimensionality that comes with observability.

Intercom, for example, tried out Honeycomb. They added just the customer ID to their instrumentation. All of a sudden, they were able to see that they had this queue building up. They had this vision of scaling up this database—it was going to cost tens of thousands of dollars. Then they realized that 60% of all execution time was going to one account that was paying them $20 per month.

So they realized: instead, they should limit the number of resources that one free account can consume. There's all of this very fine-grain stuff—so many problems in everyone's systems—that you generally have no idea about until you have actual telemetry.

Liz: Here's the other one: Vanguard is one of our clients. They were able to figure out that people who had many different accounts with them had a slow experience. They were able to optimize and fix it by using Honeycomb—by using observability. I think it’s really, really powerful that once you see what's going on, you know what you need to do to fix it and how to measure the impact of the fix.

Charity: And instead of waiting for people to complain, you're aware of these customer experiences.

CEOs love metrics

Martin: CTOs, CEOs—they love metrics. There are four basic metrics the 2021 State of DevOps report looks at: lead time for changes, deployment frequency, change failure rate, and mean time to recovery (MTTR). 

Liz: That comes from the book called Accelerate, by Dr. Nicole Forsgren and others. It's a great book. Basically, those are things that have impacted the rate at which people can change software in the enterprise. It turns out that observability positively impacts all those things.

Martin: So you can walk into those rooms and say, “These are the metrics we want to track.” They say, “Great! How are you going to improve them?” “Well, let me introduce you to observability.” It really plays into your hands when the thing you want to push for is something they already want.

Liz: Execs also look at observability as part of a cloud migration. With the migration, they want to increase development velocity. That's why you are doing this multi-year effort. The best way to measure it is DORA metrics. You cannot do a cloud migration at scale without also embracing observability.

Jess: Outcomes, outcomes, outcomes. Talking to a developer, I tell them what it's like to know what's going on in your system, but to someone in the business organization, I'm talking about outcomes. What effect does this have for customers? What effect does this have for onboarding? Recovery from incidents? Development velocity? My favorite quote from the State of DevOps report is: “Teams with good observability practices spend more time coding.”