User Tips: Sending NGINX Logs to HoneycombBy Guest Blogger | Last modified on September 9, 2022
Written by Andrew Puch and Brian Langbecker
You use NGINX as a proxy for your application, and you want to leverage your favorite features in Honeycomb to help make sense of the traffic data. Have no fear: Honeycomb is more than capable and ready to help!
Things you will need:
- Node.js with NPM (to use a sample application)
- Honeycomb account
- Honeycomb API key (here’s a Free account)
Before you start with the instructions, let’s discuss a lightweight tool called Honeytail. This utility will tail log files, parse the various formats, and send the data to Honeycomb. This is the ✨ secret sauce ✨ to ingesting NGINX logs into Honeycomb.
For the sample today, we have a NodeJS Express application that uses Redis, MySQL, and NGINX to receive requests. Here’s the sample code.
Setting Honeytail Up
Begin by cloning this Honeytail repository.
Next, change into the
honeytail directory and edit the following environment variables and command in the
ENV HONEYCOMB_WRITE_KEY NULL ENV NGINX_LOG_FORMAT_NAME combined ENV NGINX_CONF /etc/nginx.conf ENV HONEYCOMB_SAMPLE_RATE 1 ENV NGINX_ACCESS_LOG_FILENAME access.log CMD [ "/bin/sh", "-c", "honeytail \ --parser nginx \ --writekey $HONEYCOMB_WRITE_KEY \ --file /var/log/nginx/$NGINX_ACCESS_LOG_FILENAME \ --dataset nginx \ --samplerate $HONEYCOMB_SAMPLE_RATE \ --nginx.conf $NGINX_CONF \ --nginx.format $NGINX_LOG_FORMAT_NAME \ --tail.read_from end" ]
HONEYCOMB_WRITE_KEY is the API key from your Honeycomb Account. Replace NULL with this value above. The key can be found here. You will use the NGINX parser to read your log file, though the regex parser can be useful for these types of log files.
NGINX_CONF to /tmp/nginx.conf since it is a shared Docker volume.
NGINX_ACCESS_LOG_FILENAME to /tmp/access.log since it is a shared Docker volume.
In the same file, you want to edit the
--file $NGINX_ACCESS_LOG_FILENAME \ --dataset YOUR_DATASET_HERE \
Pick a dataset file name, for instance,
nginx. Lastly, build your Honeytail Docker image:
docker build -t honeytail .
You now have a Docker container with Honeytail that you can use later on.
A bit more on NGINX
In your setup, by specifying
combined as your format and not specifying a specific
log_format, Honeytail will use the default one below for parsing the log lines.
// the combined log format is predefined by nginx const combinedLogFormat = `$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"`
If you have custom logging turned for NGINX, you will need to specify your log format. See here for more details.
Setting up our test application
Clone this repo.
Next, change into the
honeytail-tracing-logs-demo directory. Install the dependencies:
Next, you will set up your environment by copying your .env.sample to .env:
cp .env.sample .env
Update the details by using the API key and dataset name from above. See below:
OTEL_EXPORTER_OTLP_ENDPOINT="https://api.honeycomb.io" OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-key" OTEL_SERVICE_NAME="your-service-name"
Please note that you may need to add
x-honeycomb-dataset in your headers if you are running Honeycomb Classic. For example, you can set it to
test-app. This is where the Honeycomb traces will be sent. In this blog, we focus on NGINX.
Run the docker-compose via make:
Now you have a running set of Docker containers with NGINX, Honeytail, MySQL, Redis, and our application. Time to generate some load and see how your data looks in Honeycomb.
Start load testing and evaluating results
Run the load script and define how many requests per second you want:
Go to the Honeycomb UI and update the dataset to
nginx via the pull-down in the upper left.
Note: For the errors to show above, you will need to modify your schema to use
status. See below:
Next, run a few queries captured in the screenshots below. With this first one, you can see a breakdown of status codes by throughput containing 500 errors. Definitely something to dive into deeper.
In this screenshot, we actually create a heatmap based on the status codes, and use BubbleUp to surface the requests causing the 500 errors by drawing a box around that area. It looks like most of the 500s occur around
GET /?sleep=5 HTTP 1.1.
Lastly, you can look at the raw data below:
With these simple queries to start, you can explore your data and set up some dashboards.
What have you learned?
You have learned an easy way to parse NGINX logs using Honeytail. In today's example, you used an application with an NGINX proxy, parsed the logs, and sent that data to Honeycomb.
Have fun exploring your data more and creating your own NGINX setup, and don't forget to get your free account today!
Customer-Centric Observability: Experiences, Not Just Metrics
Frontend observability is a tricky problem. No website is free of errors or slowdowns; sites break down in weird ways for all kinds of reasons....
What Is a Telemetry Pipeline?
In a simple deployment, an application will emit spans, metrics, and logs which will be sent to api.honeycomb.io and show up in charts. This works...
5 Ways to Increase Release Velocity with Observability
The pressure on today’s development teams is real: innovate, release quickly, and then do it all again, only faster. Is it any surprise that studies...