Apache Observability

Apache is often the first point of contact for interaction with your service. With a very small amount of instrumentation, we can reliably find the source of any slowness in a massive distributed system in just a few clicks. For example: adding headers in app code with timing information for any calls out to data backends, microservices or third-party services.

By enriching Apache logs with useful information about service behavior and parsing it into structured data, Honeycomb makes it possible to answer deep questions about systems instantly.

apache observability
apache observability - log configuration

Data Collection

The honeytail agent captures and streams log files to Honeycomb as they’re written. Honeytail converts the strings into structured json objects and reads the config file to add any extra fields defined, so we don’t have to change any regular expressions (or other hacks) to detect the new information.

You can also backfill old logs into Honeycomb to look at past data.

Getting Answers

When debugging a complex system, we usually want to start at the edge and systematically work our way down. With Apache logs in Honeycomb, you can ask questions like:

  • Which endpoints are slowest?
  • Is the increased latency distributed evenly across endpoints, or only some?
  • If only some endpoints are slower than usual, are there outliers inflating the average?
  • What do these outliers have in common? Are we being DDOSed?
  • Do the traffic patterns with elevated latency look like they’re timing out at 30 seconds or just slow?
  • Is there a single remote IP dominating my traffic? What about a single referrer?
  • What’s the distribution of HTTP status codes being returned? Overall? For a specific endpoint?
apache observability - query chart example