New Custom Regex Log Ingestion
By Rachel Fong | Last modified on January 11, 2019Unstructured text logs are so last decade, but sometimes you have to deal with them because they aren’t actually all that prehistoric in human years…or you just need us to ingest formats we don’t actually support yet.
We recently added custom regex support to honeytail
, our agent for consuming existing log files, so you can now write your own custom formats for logs you want to analyze in Honeycomb.
Matching log lines will be parsed as events, with named regex capture groups corresponding to columns.
As an example, let’s ingest a Rails log file, one of a class of many objects that I would much rather never look through manually.
Just construct a regex that pulls out the fields you want, and pass it to honeytail
to get your awful looking log dumps parsed and ready to investigate in Honeycomb.
honeytail --writekey=MYWRITEKEY --dataset=MYDATASET `
--parser=”regex” --regex.timefield="time" ``
--regex.line-regex=”(?P<time>S+) (?P<source>.*): at=(?P<log_level>.*) method=(?P<http_method>.*) path="(?P<path>.*) ..."`
Fascinating! Suspicious! Weird, even! Mostly, I’m excited that I can now click to sort all my requests by latency and filter to instantly see time-series graphs, instead of artisanally constructing an elaborate series of bash pipes.
Multiple line formats
We support passing multiple patterns, so you can parse logs with mixed output formats into the same dataset by adding additional —regex.line_regex
flags to your honeytail
call. (Order matters when we regex match your log lines, so put your most specific regexes first.)
honeytail --writekey=MYWRITEKEY --dataset=MYDATASET ``
--parser=”regex” --regex.timefield="time" `
--regex.line-regex=”(?P<time>S+) (?P<source>.*): at=(?P<log_level>.*) method=(?P<http_method>.*) path="(?P<path>.*) ..." `
--regex.line-regex=”(?P<time>S+) (?P<source>.*): (?P<msg>.*)”`
FAQ
- Nested capture groups are supported!
(?P<outer>[^ ]* (?P<inner1>[^ ]*) (?P<inner2>[^ ]*))
(Field names are WYSIWYG, e.g.
inner1
instead ofouter.inner1
.) - Multiline regexes are not supported
- Uses Google’s RE2 syntax
Check out the honeytail regex parser docs to get set up and try out some quickstart examples, and please contact us if you have questions. Happy string munging!
Related Posts
Real User Monitoring With a Splash of OpenTelemetry
You're probably familiar with the concept of real user monitoring (RUM) and how it's used to monitor websites or mobile applications. If not, here's the...
Transforming to an Engineering Culture of Curiosity With a Modern Observability 2.0 Solution
Relying on their traditional observability 1.0 tool, Pax8 faced hurdles in fostering a culture of ownership and curiosity due to user-based pricing limitations and an...
Honeycomb + Google Gemini
Today at Google Next, Charity Majors demonstrated how to use Honeycomb to find unexpected problems in our generative AI integration. Software components that integrate with...