NGINX Observability

NGINX is often the first point of contact for interaction with your service. With a very small amount of instrumentation, we can reliably find the source of any slowness in a massive distributed system in just a few clicks. For example: adding headers in app code with timing information for any calls out to data backends, microservices or third-party services.

By enriching NGINX logs with useful information about service behavior and parsing it into structured data, Honeycomb makes it possible to answer deep questions about systems instantly.

nginx observability
nginx observability - log configuration

Data Collection

The honeytail agent captures and streams log files to Honeycomb as they’re written. Honeytail converts the strings into structured json objects and reads the config file to add any extra fields defined, so we don’t have to change any regular expressions (or other hacks) to detect the new information.

You can also backfill old logs into Honeycomb to look at past data.

Getting Answers

When debugging a complex system, we usually want to start at the edge and systematically work our way down. With enriched NGINX logs in Honeycomb, you can ask questions like:

  • Which endpoints are slowest?
  • Is the service evenly slow across endpoints, or only high for ones that read from the database?
  • Is it equally slow handing off to all app instances, or are a few instances timing out at 30 seconds?
  • Are those running the same version, were some of them deployed recently, are the slow ones starved for memory?
  • Is it equally slow for all user IDs or app IDs (sent via upstream headers), or are some much slower?
  • Are they performing the same action or hitting the same endpoint?
  • Are all return codes the same, or is it only 500s?
nginx observability - query chart example