O'Reilly Infrastructure & Ops Superstream Series

We’re always interested in improving the signal-to-noise ratio of our internal telemetry at Honeycomb. In an effort to reduce the amount of noise in our logs, we looked at reducing and deduplicating the logs emitted by our infrastructure and applications.

As the company experienced rapid growth, Duolingo remained steadfast in their commitment to delivering a high-quality user experience. This dedication led to the launch of a reliability initiative, which included the formation of a specialized team focused on observability. The engineering team recognized that comprehensive observability was critical to their mission. 

So, how do we get JSON logs into a backend analysis system like Honeycomb that primarily accepts OTLP data? In this post, we’ll cover how to use the filelog receiver component in the OpenTelemetry Collector to parse JSON log lines from logs files, as there are a few ways to achieve this.