Software Engineering  

How We Use Smoke Tests to Gain Confidence in Our Code

By Jamie Danielson  |   Last modified on May 12, 2023

Wikipedia defines smoke testing as “preliminary testing to reveal simple failures severe enough to, for example, reject a prospective software release.” Also known as confidence testing, smoke testing is intended to focus on some critical aspects of the software that are required as a baseline. The term originates in electronic hardware testing; as my colleague Robb Kidd stated, “We want to apply power and see if smoke comes out.” If a smoke test fails, there is almost definitely a problem to address. If it passes, it doesn’t necessarily mean there is no problem, but we can feel confident that the major functionality is okay.

How we do smoke tests at Honeycomb

Smoke Tests at Honeycomb

For smoke tests in our OpenTelemetry distributions, we run one or more example applications in a container. We then interrogate the telemetry exported to an OpenTelemetry Collector with tests written in Bash, using jq and the bats-core testing framework. Each application configures an SDK, utilizes some key components like instrumentation libraries and attributes, and exports telemetry to a Collector. The Collector then exports the telemetry to a file in json format. Tests are run against the json output and indicate whether the results are as expected, like the span names, attributes, and metrics.

This has multiple benefits: 

  • We have at least one example of how to use the library. 
  • Building and running the example in a container illustrates whether there are any build or runtime errors in a clean environment. 
  • Exporting telemetry to a Collector confirms exporting works, and can be received and parsed at an OTLP endpoint. 
  • Exporting from the Collector and saving to a json file lets us see the data itself, as we also upload this file as an artifact in our CI pipeline. 
  • By having a file with json output, we can use a language-agnostic testing framework that is repeatable across other projects and we can be precise about what fields we expect for each application.

 "traceId": "b29ba8bde2f87b487bf5c5385ded95c2",
 "spanId": "3e79a819f620a70e",
 "parentSpanId": "d0f406d4b8095df9",
 "name": "world",
 "startTimeUnixNano": "1682975000542426619",
 "endTimeUnixNano": "1682975000542472030",
 "attributes": [
  {"key":"SampleRate", "value": { "intValue": "1" }},
  {"key":"baggy", "value": { "stringValue": "important_value" }},
  {"key":"for_the_children", "value": { "stringValue": "another_important_value" }},
   {"key":"message", "value": { "stringValue": "hello world!" }}

See it in action

Check out our smoke tests in our Python SDK Distribution for OpenTelemetry, which run against our examples in the repository. There is a basic Hello World python app with a Dockerfile that is referenced in Docker Compose, along with a Collector. A test is run with a Makefile command make smoke, which builds the app and Collector, waits for telemetry to be exported, and then makes assertions: 

  1. That the span names include “hello” and “world” 
  2. That span attributes defined in the app are included on the span 
  3. That attributes added to baggage are propagated to child spans 
  4. And that metrics are produced 

The results of the test, including the actual json files containing the telemetry, are uploaded as artifacts during the CI workflow. These can also be run locally during development.

Smoke Test json

Smoke tests are a great start

Like any test suite, this is only one aspect of testing and works best as part of a workflow that includes other kinds of unit tests and automated checks for linting and formatting. But we have found that adding smoke tests early in the lifecycle of a new project—and continuing to iterate on those tests as the project grows—helps our team gain confidence in building features and upgrading dependencies.

Interested in trying Honeycomb? Sign up for our very useful and generous free tier and start exploring for yourself! 


Related Posts

Software Engineering   Culture  

Staffing Up Your CoPE

Getting the right people working in the CoPE is crucial to success because these change agents must limber up the organization and promote the flexibility...

Software Engineering   Observability  

Navigating Software Engineering Complexity With Observability

In the not-too-distant past, building software was relatively straightforward. The simplicity of LAMP stacks, Rails, and other well-defined web frameworks provided a stable foundation. Issues...

Software Engineering  

Investigating Mysterious Kafka Broker I/O When Using Confluent Tiered Storage

Earlier this year, we upgraded from Confluent Platform 7.0.10 to 7.6.0. While the upgrade went smoothly, there was one thing that was different from previous...