Bring Test Engineering into your DevOps practice

What do a test engineer and a DevOps or SRE team member have in common? The reality is that different teams need to proactively understand what is happening in production at critical milestones along the software engineering delivery cycle. In the words of Abby Bangser, senior test engineer at Moo, "Testing has so much in common with Ops and SRE teams. We need to ask interesting questions of production. We need no more debates whether a bug gets fixed. We all need to track production behavior and therefore we need continuous access to prod."

Flying blind no longer works. In a complex, cloud-native microservices environment, a test or delivery engineer absolutely must care about quality delivery. Everyone on the engineering team believes it’s so important to get ahead of issues and have a deep understanding of exactly how users interact with the system in order to resolve and improve over time.

How does a Platform Test Engineer Work with Other Teams?

“Testing is ultimately about quality delivery”, says Abby Bangser during o11ycast episode 16 with Liz Fong-Jones and Charity Majors. In order to understand how users interact with the service, you need do both exploratory tests as well as a series of automated tests. DevOps and testing teams care about building pipelines for quality delivery to get ahead of potential issues before they arise and impact users. Some organizations have moved towards a cross-pollination of different roles so that teams can focus and specialize on distinct aspects or primary jobs to be done.

At Moo, Abby and her team are on-call with engineers who also share on-call during business hours. Rather than using a first and second line support model, engineers at Moo directly handle primary issues in addition to escalations. Abby shares that Moo is definitely progressive by having platform test engineers work so closely with the dev team during on-call rotation, but also by creating exploratory tests for the engineers who are often focused on bigger picture aspects. Ultimately it’s about context, and switching from high level broad views to deep dive unique tests can be a lot to accomplish in a fast paced environment for any single team.

Test Engineers think Laterally, Yet Specialization is the new reality:

Specialization is very real. “So testing is making a come-back," said Charity Majors. It’s really difficult for any single team to be an expert across all the different processes and steps in the eng lifecycle and have a deep understanding of exactly what is occurring at any single point, such as when new code ships. Diversified roles and specialization have evolved that actually reduces pressure on any single team, relieving of additional workload. The ultimate outcome is everyone is accountable and responsibility is shared. Additionally everyone learns from each expert team.

Introducing DeliveryConf for bringing together Testing, Product, and DevOps teams

January 2020 was the first iteration of DeliveryConf, which attracted both individual engineers and managers across organizations large and small. This vendor-neutral conference sold out thanks to its focus on CI/CD, rather than on DevOps as a whole. The format of the conference uniquely focused on a reverse panel style with each session taking time to digest the topic and involve all participants, without giving the microphone to only session presenters. Liz Fong-Jones and Danyel Fisher also presented a talk titled Fast and Simple Deployments, and they thoroughly enjoyed the format and the interchange with a nice mix of audience members. Abby Bangser and Liz Fong-Jones shared some of their faves:

Jessica Kerr talked about the build versus buy conundrum and how we need to extract as much complexity from our day to day activities, and outsource, making it someone else's problem. With outsourcing, someone else carries all that complexity so that in-house teams only have to think about that clean API , and to a limited extent know how to debug, or at least who to talk to in the event that the API doesn't work.

One other talk that resonated was Steve Pereira’s talk on "Where's the map to your pipeline?" This boiled down to talking about value stream mapping your delivery process. Abby had worked with Thoughtworks for six and a half years learning through the client's delivery processes and mapping out all of the steps both online and offline that were needed to get any product into production. This was something that Abby used in the past, and evolved into turning to tracing at MOO.

We now trace our pipelines to understand the processes to get from development machine up to production, and it makes such a difference when things are visible.

Abby shared that the MOO team is able to go to continuous delivery for the oldest application, largest and rather than rely on a two week release cycle because the team took the manual steps and turned them into steps in our pipeline.

You don't have to fully automate everything as long as it's a checklist with an API. Maybe the API is fulfilled by a human. This is a talk given at SREcon by Max Luebbe, who is at Google describing the process of turning a Google cloud region from a matter of months to a matter of weeks. You don't get there by automating every single thing, but you figure out what's important to automate.

Testing is a critical element in the eng lifecycle. It’s the glue that brings devs and ops closer. In increasingly complex environments, you can’t be expert in all areas plus maintain excellence. Having a platform test engineer that proactively looks out for what’s coming from the dev team, providing others a heads up and applying rigor so that quality improves directly impacts the lives of all team-members. At Honeycomb we like to say Happy engineering teams make happy customers. Thanks to Abby Bangser for these invaluable insights and sharing.