Observing the Future: The Power of Observability During Development

Just when you thought everything that could be shifted left has been shifted left, we’re sorry to say you’ve missed something: observability.

Modern software development—where code is shipped fast and fixed quickly—simply can’t happen without building observability in before deployments happen. Teams need to see inside the code and CI/CD pipelines before anything ships, because finding problems early makes them easier to fix. And don’t take “easier to fix” lightly: incidents found late not only impact the customer experience (and perhaps their loyalty), but they also hit developer productivity and job satisfaction hard.

Incorporating observability into your build cycles can make releases less stressful and alert-filled. They can be—dare we say—boring, meaning everyone can relax and get back to work that makes them happy: creating new features that result in innovative applications.

What does that look like in real life? HelloFresh’s platform team had been supporting a combination of several tools to help their teams troubleshoot production issues. However, they wanted to reduce the burden of maintaining and the cognitive load of using those solutions, while also encouraging adoption of modern practices. Honeycomb helped them first achieve faster incident response during deploys. It then enabled them to move towards a more proactive approach to preventing issues using SLOs.  

If you are looking to drive an observability culture in your org, here’s everything you need to know about pre-deployment observability and how teams can engage this development superpower.

Start with observability

When there’s a problem to find, it’s observability to the rescue. Observability gives teams a way to query data, explore leads on tricky bugs, and dive down endless rabbit holes—all while asking questions that hadn’t been thought of before. It’s a tall order, but observability is what makes the difference between a multi-day outage and finding problems before there is an outage.

The power of observability during development

The practice of observability during development is often referred to as “observability-driven development.” But what exactly does that mean? Think of it as a cultural practice or mindset that guides developers to think about uptime and availability as they code. Too often, observability is reserved for after code is deployed—but that’s not the most optimal strategy. Harnessing the question-asking power of observability to all the steps that happen before deployment empowers teams to make data-driven production choices. For example, how feature flags are developed and implemented. Now, the developers who wrote the code can really own the code because they can see in real time and in context what’s working, what’s not—and, eureka!, where it’s not working from the CI/CD pipeline to automated testing and more.

How to leverage pre-deployment observability

Teams can bake observability into the pre-deployment process using three main strategies: service-level objectives, tracing, and service maps.

SLOs

Service level objectives are a way to know, without a doubt, whether a system is functioning correctly according to what matters to you. They can be used to observe and monitor the state of your system and applications pre- and post-deployment to quickly identify how the deployment effected behavior. They’re straightforward to implement, and if done correctly, can let teams know something is wrong (we experienced this ourselves.) The secret to SLOs lies in the choice of objective: teams need to be certain they’re creating SLOs for performance issues that matter most, are measurable, and meaningful for their unique environments. SLOs are both an observability tool and a beneficiary of observability; teams can get the most bang for their buck out of tying observability data from a wide structured event (as on example) into the SLO. 

Tracing

Developers are uniquely situated to add tracing to their code as it’s being written, rather than bootstrapping it on later. Not only does the process of distributed tracing make observability easier at every stage, but it can be a pre-deployment game changer because teams can see what’s been added and changed with every pull request and within every CI/CD pipeline. Teams can jumpstart this process by teaming up with OpenTelemetry, which offers the option of auto-instrumentation for those getting started. 

Service Map

Modern application development is complicated and generally includes microservices, containers, multiple clouds, Kubernetes, and more. By implementing observability into the process, you can get a big picture view as you code. Honeycomb’s Service Map brings order to the chaos by providing a visualization of all moving parts so developers can code with downstream dependencies in mind, greatly reducing the potential for negatively impacting interconnected services with a deployment.

Build a culture to support observability during development

Observability can’t be seen as just “one more thing” developers have to do. It must become an intrinsic part of what they do, not a separate box to check. Research shows developers want to be creative and productive. However, research also shows that on average, they context-switch every 6.2 minutes. Anything that can give them back time in the day for better concentration and creativity is a win. Incorporating observability into their development cycles will do just that. 

Building with observability will help developers ensure that they are not introducing new code that will negatively impact performance, harm downstream applications, or disrupt customer usage. Additionally, and perhaps most importantly, it frees up developers from the time and stress that comes with having to debug post-deployment and keeps them doing what they like to do best: deliver innovation. A win win for both internal teams and external users. 

There are other benefits too: truly observable code can also be triaged by any member of the team, taking the burden off those few “problem solver” developers or legacy engineers and improving the quality of their work lives.

Change is hard. Modern software development is happening at a breakneck pace. But we’ve learned a lot about leveraging tools, process, and culture to create an environment where observability during development is your new best friend. Let us help you start your journey.