Evolving by Involving 

This post was written by Nick Travaglini and Savannah Morgan and is the first in a series. We'll add links as they are released.

The customer success department at Honeycomb features a number of different roles dedicated to helping our customers succeed in every step of their observability journey. The work we do ranges from support engineers who provide timely assistance to customers, to customer architects who dive deep into the technical stuff, to product training who educate folks on features old and new. 

In our role as technical customer success managers, we work with our customers to develop scalable observability strategies. We do this by combining our department’s forces with the rest of Honeycomb’s resources to help businesses unlock the power of observability.

In this post, we’re going to lay out the guiding principle that unifies the diverse world of CS as we see it—and show how we put it into practice. That principle, as you might have already guessed, is that we make progress by working together. 

Teamwork makes the dream work

How does CS build this partnership with a customer? We start by asking questions (a lot of them!) so we can meet folks where they are. For example:

• How do you debug problems?
• What does an incident look like for your team today, and what would you like this to look like in the future?
• What’s the on-call experience like?
• What problems are you trying to solve through observability?
• What projects or goals are most important to your team this quarter, or this year?
• Are there any gaps in visibility?

From the CS perspective, there are a lot of puzzle pieces to put together before we can understand our customer’s current socio-technical landscape and contextualize the goals they’re trying to achieve. 

We need our customers to be curious about our own backgrounds and experiences, and the broader experience unlocked through observability. By asking questions, we challenge assumptions about each other and learn from one another. Assumptions that could be made by Honeycomb Customer Success (and would need to be verified or challenged) include:

• Expecting a distributed system. With the prevalence of microservices, it’s easy to work under the assumption that the systems we observe are distributed. This is not always the case—there are still a variety of monolithic structures that can benefit from tracing.
• Everyone is a developer. The developer experience is definitely a cornerstone of Honeycomb. However, there are folks in a variety of roles who we work with in Customer Success. Product, SRE, Ops, Support, Sales Engineering and others can all reap great insights from democratized data. 
• Instrumented services are fully visible. While auto-instrumentation is a great first step in bringing visibility to complex systems, adding custom context is an ongoing process. Customer Success needs to challenge the assumption that a service sending data is sending the right signals. This is something that can only be verified by talking to the people who use that instrumentation to solve problems. Does the available data help you debug? Are there any visibility gaps?
• Your organizational layout hasn’t changed. When CS works with a handful of people regularly, it can be easy to assume that the rest of your company’s structure hasn’t changed in a way that could impact your work. However, it’s important to reverify organizational layouts and goals regularly. Shifts in reports or cross-functional responsibilities can massively impact your OKRs, KPIs, and observability strategy. In order for us to best unlock visibility across the company, we need to know what priorities you have today and if anything has shifted since we last spoke. When changes happen, we can shift our support strategy to best align to your new structure and priorities. 

A customer can also make assumptions about Honeycomb or the Customer Success team, such as:

• Observability is only for software engineering and development. While it’s true that the power of distributed tracing and observability is often first unlocked in engineering, its effects can be felt everywhere. Proactively finding issues before customers leads to a more stable solution and platform, which benefits the entire company. Democratizing data allows field engineering teams such as Support or Sales Engineering to troubleshoot problems more independently, leading to fewer escalations and on-call tickets. This, in turn, can improve developer on-call experience. 
• Honeycomb will not help avoid vendor lock-in. Honeycomb is deeply invested in vendor-agnostic observability solutions. Our team wants your observability journey to succeed, and while we want to work with our customers as partners for years to come, this can only happen if they have a truly excellent experience with our product and people teams. No one should be stuck in a platform due to fear of data lock-in. This is why we have gone full force on OpenTelemetry, and are active contributors in this open source community. If you are not yet integrated into OpenTelemetry, our Customer Success team would love to talk through a path to migrate to this vendor-agnostic, thriving instrumentation community.
• The goals of the team are not relevant to Honeycomb/o11y. Our team strives to align our support to what matters most to you. Understanding the OKRs, KPIs, or other success metrics you and the team have heavily influences the type of support we provide. Our team has many levers to pull—bringing in DevRel to speak to your developer teams about topics like production excellence, including a customer architect to help work through tailored solutions, working sessions to help build SLOs and Boards, deep dives into OpenTelemetry, and more. Understanding what is most important to you allows us to tailor our sessions and the learning materials we send, and connect you with the right experts within our community. If we can’t personally help, we know someone who can!

Experiment, iterate, find answers

Much like observability, the power is in asking the right questions and having the visibility to experiment, iterate, and finally find answers. Only when we question these assumptions and gain clarity can we think differently and successfully work together. Through curiosity, visibility, and collaboration, we form a team.

But becoming a team doesn’t mean that we collapse all of our differences. The point of synchronizing is to find the ways to collaborate that amplify strengths and dampen weaknesses. It’s important to remember, though, that we’re heterogeneous—each person brings different strengths and weaknesses. 

We also need to keep in mind that we have to work together over time, diachronically, and so we can’t be in perfect but brittle lockstep. We need enough flexibility to adjust and trust that the other is looking out for us when Things Happen™. The specific degrees of difference, then, will modulate as our teams’ circumstances and goals change.

Who doesn't need a good forestry analogy?

To our mind, this situation looks a lot like a team of park rangers on a rescue mission. There are rangers on the ground—that’s our customer—making their way over the terrain towards the people they’ve set out to help. Flying above them in a helicopter is the CS team. A boots-on-the-ground perspective does the hard work of trekking through rough country and can inform the folks in the helicopter about nitty-gritty details. Meanwhile, up above, the helicopter has a bird’s eye view of the landscape and can forewarn those below of any opportunities or challenges ahead—or air-drop resources as necessary.

In actual Honeycomb terms, we’ve seen this play out in many ways. One example is a customer who had several of their developer teams in various stages of adopting distributed tracing and observability. The teams regularly turned to each other for assistance as teams’ skills waxed and waned over time. These local issues arose due to things like employee turnover, new policies and practices that impacted working habits, and changing expertise with Honeycomb’s feature set as the product itself changed. 

Eventually, though, teams’ mutual aid wasn’t enough. The whole organization decided they needed outside assistance to improve developer experience and address the on-call experience. So they brought the situation to our attention and we worked out a success plan. That plan guided our activities, like reworking alert paths and thresholds, thus empowering engineers—with data—to advocate for themselves and change priorities to do much-needed stability work. We also brought in Honeycomb DevRel speakers to share what they’ve learned from their many years working on the ground in development.

That organization is now well on its way to a healthier environment for its developers and a more reliable product for their users. And they’re getting there because we’re working together. Customer Success is a team sport.

If you’re a Honeycomb customer, or thinking about becoming one, know that Honeycomb CS has your back. Let us know about your goals and challenges, and check out some other examples of how we’ve helped folks like Heroku do great things.