Ask Miss O11y: Is There a Beginner’s Guide On How to Add Observability to Your Applications?

"Dear Miss O11y,

I want to make my microservices more observable. Currently, I only have logs. I’ll add metrics soon, but I’m not really sure if there is a set path you follow. Is there a beginner's guide to observability of some sort, or best practice, like you have to have x kinds of metrics?

I just want to know what all possibilities are out there. I am very new to this space."

-Observability Newbie

The first thing that comes to mind would be to back away from metrics. I totally understand the drive towards them. However, if you're getting started, why not start with the best?

The best starting point would be OpenTelemetry. Implement the SDKs for tracing into your application. You can use Jaeger to get started. There are ‘Getting Started’ beginner's guides for each of the language SDKs. Once you graduate from something you can manage with Jaeger, a lot of observability vendors offer ‘free forever’ SaaS plans. Hint: we do, and it’s a very useful, generous plan.

This should give you low-level detail on everything your users are seeing, which you lose with metrics. Metrics can come later if you need them, as they are more focused on pod/infrastructure-based information, like CPU usage, etc.

Once you have that in place and you can see the trace data of requests flowing through your infrastructure, you can look at the specific areas of your application that could do with more visibility. Then, add more tracing information (spans and attributes) to get even better visibility.

From there, there are endless possibilities around service level objectives, service maps, high cardinality, and high dimensionality—the list goes on. Their usefulness will depend on the scale of the application and other things. Tracing is the first step, and it’s really easy to get started if you're on a modern version of your language.

If you're into reading, Charity Majors, George Miranda, and Liz from Honeycomb literally wrote the book on observability. It's more advanced than a beginner's guide, but it's definitely worth a read as you carry on in your journey.