Enriching Nginx Logs with request_time, Server Data, and Other Goodies

How can you make your nginx logs more awesome? Nginx has some fantastic data hiding in its log_format spec, but oddly enough, most of it is not enabled by default. Here are a couple of variables to add to your nginx logs to help you get the most from them.

Two suggestions

$request_time: Hands down, the most important measurement for tracking the performance of a dynamic service is the response time of each request that comes in … and even that measurement is not enabled by default. I can understand how it used to be that way — way back when all you served were static files, response time wasn’t very interesting. In the context of today’s modern web, however, with all its dynamic applications, everybody should add $request_time to their log_format spec. It will show the amount of time it took nginx to process the request as a floating point number, representing the time in seconds (to millisecond precision).

$server_name: It’s also understandable, in the world of services logging to local disk, why the name of the machine handling the request isn’t added to the log line - it’s the same for every request. However, when centralizing logs or handing them off to a 3rd party service, it’s vital to add $server_name to the log line so that you know where to dig deeper when you see an anomaly.

There are tons of additional variables available to augment your nginx access log. I encourage you to go dig around the nginx docs and see what else might be relevant to your application. (If you’re configuring nginx as a proxy, pay special attention to the upstream variables.)

The Honeycomb nginx installer

We’ve identified a short list of useful variables to add to your nginx config. The Honeycomb nginx installer script will (in addition to downloading honeytail, our agent, and configuring it for your nginx server) give you some suggestions about which ones are missing from your log_format.

Try it out! Download instructions are in our docs.

The installer script will:

• find your nginx config
• find your log files
• backfill a little bit of data (what’s in the current log file)
• maybe suggest some changes to the log format to log interesting fields
• start up the honeytail agent with the appropriate flags
• give you a link to the dataset in Honeycomb with some suggested queries

Enjoy!

p.s. If you’re curious what the script is going to do before running it, feel free to look at the source.