Product Updates   Observability   Logging  

New Custom Regex Log Ingestion

By Rachel Fong  |   Last modified on January 11, 2019

Unstructured text logs are so last decade, but sometimes you have to deal with them because they aren’t actually all that prehistoric in human years…or you just need us to ingest formats we don’t actually support yet.

We recently added custom regex support to honeytail, our agent for consuming existing log files, so you can now write your own custom formats for logs you want to analyze in Honeycomb.

Matching log lines will be parsed as events, with named regex capture groups corresponding to columns.

As an example, let’s ingest a Rails log file, one of a class of many objects that I would much rather never look through manually.

Rails log sample

Just construct a regex that pulls out the fields you want, and pass it to honeytail to get your awful looking log dumps parsed and ready to investigate in Honeycomb.

honeytail --writekey=MYWRITEKEY --dataset=MYDATASET `
    --parser=”regex” --regex.timefield="time" ``
    --regex.line-regex=”(?P<time>S+) (?P<source>.*): at=(?P<log_level>.*) method=(?P<http_method>.*) path="(?P<path>.*) ..."`

Rails logs in Honeycomb

Fascinating! Suspicious! Weird, even! Mostly, I’m excited that I can now click to sort all my requests by latency and filter to instantly see time-series graphs, instead of artisanally constructing an elaborate series of bash pipes.

Multiple line formats

We support passing multiple patterns, so you can parse logs with mixed output formats into the same dataset by adding additional —regex.line_regex flags to your honeytail call. (Order matters when we regex match your log lines, so put your most specific regexes first.)

Rails log sample with mixed output formats

honeytail --writekey=MYWRITEKEY --dataset=MYDATASET ``
    --parser=”regex” --regex.timefield="time" `
    --regex.line-regex=”(?P<time>S+) (?P<source>.*): at=(?P<log_level>.*) method=(?P<http_method>.*) path="(?P<path>.*) ..." `
    --regex.line-regex=”(?P<time>S+) (?P<source>.*): (?P<msg>.*)”`

FAQ

  • Nested capture groups are supported! (?P<outer>[^ ]* (?P<inner1>[^ ]*) (?P<inner2>[^ ]*))
    (Field names are WYSIWYG, e.g. inner1 instead of outer.inner1.)
  • Multiline regexes are not supported
  • Uses Google’s RE2 syntax

Check out the honeytail regex parser docs to get set up and try out some quickstart examples, and please contact us if you have questions. Happy string munging!

 

Related Posts

Observability   News & Announcements  

Announcing Honeycomb Support Business Hours in Europe

Earlier this year, Honeycomb announced the launch of data residency in Europe. To meet the growing needs of our customers in the region, we are...

Logging   Customer Stories  

Empowering Engineering Excellence: Achieving a 26% Reduction in On-call Pages at Amperity with Modern Observability for Logs

Amperity required an observability partner to facilitate their transition into the modern engineering era as their previous tooling struggled to support their growth strategy....

Observability   Metrics  

The Cost Crisis in Metrics Tooling

In my February 2024 piece The Cost Crisis in Observability Tooling, I explained why the cost of tools built atop the three pillars of metrics,...