Software Engineering   Culture  

Staffing Up Your CoPE

By Nick Travaglini  |   Last modified on July 8, 2024

Getting the right people working in the CoPE is crucial to success because these change agents must limber up the organization and promote the flexibility necessary to perform resilience.

We’ll look for teammates who share enough in common to work well together, but who don’t necessarily perfectly overlap so that they can play off each other’s strengths. We’ll also want them to come from a place of real commitment to ensuring great operations and a mindset that emphasizes the customer experience.

Finding people who match those characteristics doesn’t need to be hard. In this post, we’ll outline a strategy that’ll help you find and recruit these colleagues.

The challenge

The working members of your CoPE face a daunting challenge: By joining, they agree to the task of modifying a complex, adaptive system which is subject to financial, human, legal, and other constraints that have placed it in a locally optimal position. For the organization, things are going pretty well, all things considered. It’s normal—and expected—for things to turn out alright.

The CoPE members’ job is to convince the organization to change now, proactively, rather than wait until later (when things start going wrong) and thus the change will be a reactive one that takes more effort. That’s a tough ask because:

  • It’s not clear that the organization is anywhere close to the point where things aren’t going so well
  • When things normally go well, it’s easy to chalk up cases where things go poorly as one-offs
  • Until things do go poorly, it’s hard to determine what to do that will preempt a big qualitative change from good to bad

The solution

To meet that challenge, here’s what we recommend:

1) Enlist distinguished engineers (or adjacent positions) who already believe that changes in observability are needed.

Making a change requires people who will advocate for that change. It also requires that people listen to those advocates. The easiest way to get that is to find people who are already respected in their parts of the organization and who believe in the desired change.

If your organization is already using Honeycomb, then its History, Enhanced Reporting, and Activity Log features can provide good data to start your search. These three data sources allow you to see what people are querying (and how complex those queries are), how much they’re querying, and what else they’re doing in Honeycomb, respectively. That data can help you triangulate on relative “power users.” 

An important qualification is to look for people who will provide what Scott E. Page calls “diversity bonuses” once they are assembled. In The Diversity Bonus, Page lays out an argument that any group addressing complex challenges like those the CoPE will face requires a wide range of backgrounds, heuristics, information, and skills. Each of these added to a group provides a bonus because it changes how the group can engage with the problem space, and they can also be combined to form synthetic methods of working.

The details of the particular problem(s) that the group is addressing matter, so this diversity should be intentionally constructed. Thankfully, the book lays out guidelines about how to produce and accrue the greatest bonuses in various types of problems, like predicting or innovating. Consider making use of this theory when making staffing decisions.

2) Once the members are recruited, bring them together in a regular meeting to talk about the CoPE’s mandate, what they want to achieve, and why.

This initial group forms the basis of a social graph. That graph should represent social relations which can be enriched with information about normal engineering practices that each person and their team perform, such as periodic sprint cycles, regular post-incident meetings, or widely adopted standards and tools. It may also include informal relations like regular coffee meetings, carpool groups, and the like.

Let the group talk about their work and trust them to figure out, together, what needs to change and how. That conversation may benefit from a facilitator to ask open-ended questions and keep the discussion from stalling. As an output, the group should author a charter which includes a set of problems that it needs to scope and address, one or several stopping rules for its work, and rationale for these decisions.

Collecting this information up front is crucial because it will inform the development of a twinned strategy involving both passive and active tactics for making change. 

3) Get sign-off on the cooperatively-drafted charter.

As we discussed before, the CoPE may face headwinds from various parts of the organization; hence the need for committed resources and a location that permits it the leeway it needs. This means that clear expectations with its immediate neighbors, like a sponsoring department, are crucial. CoPE members can’t do their work if they don’t have the material, substantive resources necessary to make change.

Each member and the CoPE’s immediate ‘upstream’ supporters have to reach some clarity in order to coordinate. Therefore, a group review and commitment to follow through on the charter is paramount. Then, and only then, is the CoPE ready to get to work with the confidence and power to engage in creative destruction.


That concludes our discussion on finding the right people to place in a CoPE. Don’t miss the next post in the series, where we’ll go into where a CoPE should affect change, specifically in observability. 

If you missed the first post in the series, you can find it here: Establishing and Enabling a Center of Production Excellence.

For the second post, please go here: Independent, Involved, Informed, and Informative: The Characteristics of a CoPE.

Try Honeycomb for free while you wait for the next post!


Related Posts

Software Engineering   Observability  

Navigating Software Engineering Complexity With Observability

In the not-too-distant past, building software was relatively straightforward. The simplicity of LAMP stacks, Rails, and other well-defined web frameworks provided a stable foundation. Issues...

Software Engineering  

Investigating Mysterious Kafka Broker I/O When Using Confluent Tiered Storage

Earlier this year, we upgraded from Confluent Platform 7.0.10 to 7.6.0. While the upgrade went smoothly, there was one thing that was different from previous...

Software Engineering   Culture  

Independent, Involved, Informed, and Informative: The Characteristics of a CoPE

In part one of our CoPE series, we analogized the CoPE with safety departments. David Woods says that those safety departments must be: independent, involved,...