Honeycomb Blog

Honeycomb goes Serverless: Send your app data without running agents

bee giving a "high five" to a person's finger

Serverless apps are growing in popularity, thanks to tools like AWS API Gateway and Lambda, and a growing number of powerful frameworks that simplify development and deployment. Complex applications are still complex, however, and regardless of your platform you’ll still need to think about observability. Today, we’re excited to announce our Agentless Integrations for AWS. There are a lot of ways to get data into honeycomb. Most of these – such as honeytail, our HoneyAWS suite, or the Kubernetes agent – require you to run an agent somewhere in your infrastructure. Our agentless integrations are different: they hook into the…

Read More...

Structured Logging and Your Team

This guest blog post from Anton Drukh of snyk.io is part of our series on structured logging. From 1 service to over 50 today We run a distributed engineering team at Snyk. As we scale our team and infrastructure, we want to make sure that we don’t make a mess of the daily operation our system. We don’t want to fall into the trap of committing code that makes sense when it’s written, but not when it’s run in production. We started with a single service in our stack. Two years into adopting a micro-services architecture, we span over 50…

Read More...

Simple Structured Logging with NLog

We’re grateful for this guest post from Tim Wilde! You can find the source code for the examples he uses in his github repo. Strings are where data go to die There you go; I said it. How often have you found yourself contemplating some hair-brained regex scheme in order to extract an inkling of value from a string and wishing the data had just arrived in a well-structured package without all the textual fluff? So why do we insist on writing prose in our logs? Take “Exception while processing order 1234 for customer abc123” for example. There are at…

Read More...

New Custom Regex Log Ingestion

Unstructured text logs are so last decade, but sometimes you have to deal with them because they aren’ t actually all that prehistoric in human years…or you just need us to ingest formats we don’t actually support yet. We recently added custom regex support to honeytail, our agent for consuming existing log files, so you can now write your own custom formats for logs you want to analyze in Honeycomb. Matching log lines will be parsed as events, with named regex capture groups corresponding to columns. As an example, let’s ingest a Rails log file, one of a class of…

Read More...

You Could Have Invented Structured Logging

Sometimes we hear from folks who are a little bit intimidated by the notion of structured logging. Some common issues: There’s no approachable library for structured logging in my language. My logging today is messy, and changing it all is a daunting project. These are legitimate concerns! But I have some opinions: You might not need a fancy pants library You can make incremental changes to your logging setup. Structured logging is really all about giving yourself — and your team — a logging API to help you provide consistent context in events. An unstructured logger accepts strings. A structured…

Read More...

Build Observable Systems

What should you log? When your systems break, it’s great to be able to look at what they were doing just before they broke. A log is a common solution. But hands up if you’ve come across a log that looks like this: 11:32:33 Processing request for user 42 11:32:33 Request processed successfully 11:32:33 Processing request for user 43 11:32:33 WARNING: user 43 sprocket needs adjusting! 11:32:33 Processing request for user 44 11:32:33 Request processed successfully 11:32:34 NullPointerException in adjustSprocket() 11:32:34 Processing request for user 43 11:32:34 Request processed successfully 11:32:34 Processing request for user 44 What caused the exception?…

Read More...

Dynamic Sampling in Honeytail

A while ago I wrote a three part series on sampling, covering an introduction, some simple straight forward ways to do it, and some ideas for fancy implementations. I’m happy to say that that work has made its way in to Honeytail, our log tailing agent. Dynamic sampling in Honeytail works with a two phase algorithm – it measures the frequency of values in one or more columns for 30 seconds, computes appropriate sample rates for each value based on trying to fit a logarithmic curve to the traffic, then uses those values for the following 30 seconds. While it’s…

Read More...

Lies My Parents Told Me (About Logs)

Lots of us still believe some pretty silly things about logs. Most of these things used to be true! Some of them never really were. Sometimes they are “true enough” to get you a long ways, until you run into a wall and suddenly they no longer are. Any time there are changes in your scale or maturity or environment, you may need to reconsider your assumptions about logs, and these are good enough place to start. “Logs are cheap.” Not if you’re doing anything decently interesting with them, they’re not. Lots of people get intense sticker shock when they…

Read More...