Building Observability Teams and Developing In-House Expertise

Developer productivity is intrinsically linked with the ability to safely build, deploy, and test changes in production. Observability plays a key role in the ability to design for and operate in production. Platform teams have become the de facto owners of developer tooling and reliability for companies operating at scale—from what we’ve seen, that increasingly includes observability.

Historically, observability has been the concern of site reliability engineering (SRE) or ops teams who are tasked with monitoring and understanding the health of the system. For modern teams building in the mid to late 2020s, we’d argue that the best way to roll out observability is through a platform team as opposed to using a standalone team or leaving individual product development teams to figure it out on their own. This flows as a natural consequence of platform teams owning more of the developer experience, and development teams owning their production services rather than relying upon ops or SRE teams.

Why does observability fall into developer experience rather than operations? Like testability, reliability, and accessibility, observability is a property of software systems. These practices require continuous investment rather than being a one-time addition, or having a one-size-fits-all solution. As they improve, benefits accrue for the developers that platform engineering teams serve, and for end users of your systems. Increasingly diverse organizations are adopting observability and rolling it out across their teams using a platform engineering mindset.

If you’re trying to move your company from the operational model to the upstream, faster feedback loops, unified storage model, this guide is about the organizational decisions you’ll need to make to set your teams up for success. And as a bonus, these changes also help you achieve your dual mandate of serving the business as well as your developers by unlocking the ability of non-developers to easily access data on key business metrics without having to guess what the “right way” is to look up the data.

An introduction to platform engineering

The need for platform engineering arose from a confluence of the cloud-native, SRE, DevOps, and DevEx movements. The Cloud Native Computing Foundation’s definition for cloud-native includes goals of scalability, development velocity, and operability. Similarly, DevOps and SRE both highlight a desire to shorten feedback loops and reduce operational toil in their definitions and practices. DevOps provides “Better Value, Sooner, Safer, and Happier” through culture and collaboration between development and operations groups. SRE joins together systems engineering and software skill sets to solve complex operational problems through developing software systems rather than manual toil.

As teams started shifting more ops effort left, they quickly discovered that while operations could be shifted left, not every development team could become an expert in all of the tooling underlying their ability to work in production. Teams adopting cloud-native technology such as Kubernetes would discover that they still needed someone to build and maintain the container runtime platform, even if development teams were empowered to build containers and schedule them on the cluster. Even the process of building such containers needed its own specialization for CI/CD platform systems. Accessibility, security, privacy, test engineering, and design all had this same problem of primary ownership shifting onto development teams, yet expertise and shared frameworks still being necessary. And larger organizations started realizing that a small investment in developer experience (abbreviated DevEx or DX) could deliver outsized gains by reducing friction and duplication across teams.

Increasingly, there are no longer ops teams, only engineers who write code and own that code in production. But you can’t expect everyone to learn and know everything. Platform engineering is one of the most promising models yet for how to separate concerns and allow people to specialize. By building a platform that encodes the right defaults, developers can focus on achieving outcomes rather than the nitty-gritty of creating and running a production-ready service.

Platform engineering teams typically write very little code themselves; instead, their output is choosing the right tools, followed by offering APIs to interface with production, documentation, and common standards. While you might think about a “platform-as-a-service” as the central offering of a platform team, the reality is that the cultivation of the platform is more important than the precise tools in use. A developer platform can be as simple as a template Git repository and some documentation, and could be as complex as a bespoke Backstage or Kratix deployment with built-in dashboards, one-click deploys, and more.

Regardless of how they are implemented, platform teams need to make it easy to do the right thing, and hard to do the wrong thing. They comprise software developers and product managers, user researchers, and even developer advocates in a sufficiently large organization. The one difference from an end-user-facing product? The customers are other engineering teams at the same company, not the general public.

Instead of engineers talking to other engineers, collaborating across or within team boundaries, the platform model calls for people talking to software, engineers talking to APIs. The point of collaboration no longer takes place in task tracking systems or meetings—it happens in APIs, SDKs and pull requests.

How observability benefits platform engineering as a whole

Embracing platform engineering requires high-quality observability. After all, observability and controllability are duals under control theory; having the ability to steer doesn’t matter if you can’t measure the results of the changes you are making. Ordering matters, and these tools are best used as a one-two punch: observability plus feature flags, observability plus progressive deployments. If you don’t have the ability to slice and dice and break down on high-cardinality dimensions, your ability to utilize feature flags will be sharply limited, because each flag and each build ID is a high-cardinality dimension. To get where you want to go, you need to both be able to move faster, see which way you’re going, and steer in the direction you want.

Yes, the compounding works bidirectionally. If you’re missing any ability to deploy whatsoever in a repeatable fashion, start with the fundamentals of CI/CD. Without reproducible deployments, you don’t know what you’re observing. Without fast CI/CD, you can’t iterate on instrumentation.

How do the principles of platform engineering apply to observability?

Many companies simply renamed their backend or SRE teams to “platform engineering.” This did not achieve the results they sought, because SRE itself has many diverse implementations (consultative, platform, embedded, etc.) and not every implementation was fit for the purpose leaders thought they were trying to achieve. Likewise, teams that rename their monitoring team to observability are doomed to struggle—because observability isn’t just about running the telemetry pipeline and storage stack as a data sinkhole, or having a team that firefights as the monitoring wizards. No one is out there spinning up new dev teams and new ops teams and asking them to collaborate with each other, because that doesn’t work. If half your people are writing code and half your people are operating it, that’s not a feedback loop.

Observability benefits from being offered as a value-add service instead of a managed minimum cost service. As we’ve discussed before, observability is a high-leverage investment. It is not purely an ops tool for operational outcomes, or just about errors, outages, and downtime. It is about understanding the lifecycle of your software, building better products, understanding user behavior—the feedback loop of feedback loops. Therefore, you need to use some of the core tenets of platform engineering when planning your observability effort, even if it does not initially fall under the platform organization:

Opinionated defaults: Every service template ships with OpenTelemetry auto-instrumentation, telemetry routing, and libraries for adding attributes that conform to semantic conventions.
Self-service APIs: Adjusting sampling, creating new datasets and attributes, configuring new boards and SLOs. All must be automatically accessible rather than requiring a ticket or talking to a human.
Make anti-patterns difficult: If telemetry routing works out of the box, there is not a temptation to add in additional vendor SDKs.
Own the abstractions: the observability team owns the libraries and the telemetry pipeline and the downstream vendor relationships, not the implementations (individual service instrumentation).

When every team has their own relationship with a different opinionated monitoring expert (or amateur on their team), you get fragmentation. Different teams instrument differently, store data in different systems, and can’t collaborate during cross-team incidents. Common instrumentation and semantic conventions through platform APIs create a shared language. When services from three different teams are involved in an incident, everyone’s looking at the same data, using the same tools and query language, following the same traces across boundaries. That’s only possible when observability flows through standardized systems rather than individual expertise.

How platform teams fulfill the observability team mandate

Platform teams contain abstractions and give single points of contact to teams and well-reasoned APIs rather than relying upon each team to have experts or individual human relationships with specialists for each area that the de facto platform would otherwise have to cover. Embedding SREs or platform engineers into individual teams once may have made sense, but as organizations have scaled, the general consensus is that platform engineering is best sustained through API endpoints rather than personal 1:1 engagement.

For observability, this API-first approach means standardized instrumentation becomes automatic. Instead of each team having to develop its own instrumentation standards and routing, picking their own tools, and designing their own dashboards, the platform provides libraries and templates that bake in the best practices for observability. Instead of tickets to request new dashboards, developers create them via infrastructure-as-code and API, which prevents accidental changes and drift and allows for revision control. The observability team’s job becomes encoding expertise into reusable abstractions, not being a bottleneck for every team’s monitoring needs.

Platform teams aid in observability as in-house experts that:

Evaluate observability tools objectively for organizational fit
Build integrations and “glue” connecting observability tools to workflows driven by data
Train teams on effective observability practices and patterns
Standardize instrumentation approaches across services
Make informed build vs. buy decisions for observability capabilities
Manage vendor relationships and hold vendors accountable
Continuously improve observability posture as the organization grows

Traps for the unwary

As with a container orchestration platform or a CI platform, no observability platform is actually useful unless developers use it. If you do not speak with your users and work with them to ensure you are meeting their needs, they will turn to other sources and cut you out. Shadow IT is the death of any observability platform effort. Product management, user research, and design thinking skills are a critical component of platform engineering, including observability.

Like with other categories in the domain of platform engineering, using a consistent set of tooling for observability benefits teams across your organization. That’s not to say that you need to use a single set of tools; instead, the important thing is to be opinionated and be able to outline to your organization what tool they should use for what purpose. This enables teams to have a consistent view of data when they need to debug incidents that cross multiple services; minimizes cognitive overhead, which is your teams’ scarcest resource; and most importantly, allows members of different engineering teams to enjoy internal mobility.

The failure mode of confusion and duplication is teams pointing fingers and blaming each other when there are incidents, contradictory data complicating analysis, and teams spending time recreating graphs in different tools as they progress through investigations. The observability team’s job is to sit between what vendors built and what your developers use daily, minimizing cognitive load and keeping everyone on the same page.

Ensuring that all of the systems used in a particular organization are cultivated, maintained, and adopted is a task for platform engineers. Integrating vendor systems isn’t just a matter for procurement, if those vendor systems are an integral part of how the organization works. Reducing cost and enforcing consolidation can be valuable where tools are cost centers, but where they are investments that unlock value, a different approach is necessary. Instead of asking “How little of this can we get away with?” the question becomes “How much can we invest in this before we start seeing diminishing returns?”

Before we dive into how to build these teams successfully, let’s name the common failure modes that sabotage observability transformations:

Reporting up via the IT/ops chain of command where observability gets managed like a cost center rather than strategic infrastructure
Staffing teams exclusively with infrastructure/SRE skill sets, missing the product perspective and business context needed for modern observability
Building everything yourself. Trying to compete with vendors who’ve invested hundreds of millions in columnar storage and query optimization
Building nothing yourself. Treating vendor tools as turnkey solutions without the glue code and abstractions your organization needs
Solving this at the team level and then stopping. Observability is an organizational problem; the nonlinear returns come from everyone using the same tools and contributing to the same dataset

Building and growing an observability team

Since observability typically accounts for 20% to 25% of cloud and platform spending, it naturally follows that approximately 20% to 25% of a platform team should have some degree of specialization in observability. While it might seem as if the natural way to fill those roles is from people who have previously worked as SREs or platform engineers, we would encourage you to look beyond those and draw from users of observability. Especially when working in the frontend domain, having empathy for the consumers of your observability stack goes a long way. And if you’re integrating into your software stack rather than building entirely from scratch, SRE and operational skills are not necessarily what you need. Otherwise, you will be reproducing the dysfunction of the previous “ops runs ops tools” dynamic.

Here’s one rule of thumb: by the time you have five platform engineers or SREs, you need one person to start specializing in observability. But that person doesn’t necessarily need to come from (and maybe shouldn’t come from) your bank of platform/SRE talent—sometimes the most effective specialists in observability come from the product side, closer to your business differentiators.

A good observability engineering team of three to five people can make an organization of 2,000 engineers dramatically more effective by providing the tools, training, and standards that enable everyone else to work faster and more confidently.

Conclusion

At the time of this writing (early 2026), AI is doubling the rate of pull requests. These foundations for understanding what you have pushed matter more than ever. AI amplifies whatever you already have. Strong platform practices make AI a force multiplier; weak fundamentals just get you to the crash faster. It’s like pouring rocket fuel on the ground and lighting a match rather than a controlled ignition.

Learn from the platform principles when designing your observability team and rollout strategy. Bake in instrumentation and telemetry routing by default. Build APIs for interacting with data that scale. Choose vendors strategically, whether commercial or open-source “vendor” (or even a mix). Hire the right people and give them the right mandate. And remember: your mission is to thoughtfully orchestrate the pieces so that your product engineers are spending time on unique business needs, not undifferentiated heavy lifting.

Plan, build, test, observe, then take off like a rocketship—with all of your telemetry nominal.