Honeycomb Blog

o11ycon: A Conference For The Observability Community

Hey friends, We recently announced o11ycon, and the response has been terrific! Especially since we haven’t really even told you all what we’re planning. 🙂 First things first: o11y means observability. (Yes, abbreviations are annoying, but YOU try typing ‘observability’ twenty times a day with character limits. And o11y just sounds so cute!) o11ycon will be a one day, single-track, participatory event, bringing together forward-thinking engineers and leaders from every type and stage and size of company, to answer two significant questions: what is observability, and how do we, as a community, do it? What is observability? What does it…

Read More...

Simple Structured Logging with NLog

We’re grateful for this guest post from Tim Wilde! You can find the source code for the examples he uses in his github repo. Strings are where data go to die There you go; I said it. How often have you found yourself contemplating some hair-brained regex scheme in order to extract an inkling of value from a string and wishing the data had just arrived in a well-structured package without all the textual fluff? So why do we insist on writing prose in our logs? Take “Exception while processing order 1234 for customer abc123” for example. There are at…

Read More...

The Price is Right

Here at the hive, we’re working on something that isn’t code or new features(!), but is a big part of our business notwithstanding: figuring out the best way to help people understand how we price Honeycomb and the built-in assumptions we make about how they use Honeycomb. There are some issues (pricing is hard, film at 11!), but we’ve found the challenges to be more about words than about technology or sales process. Let me explain. We’ve heard from people who’ve been confused by the way we price and bounced without even trying Honeycomb because they think we are either…

Read More...

Best Practices for Observability

Observability has been getting a lot of attention recently. What started out as a fairly obscure technical term, dragged from the dusty annals of control theory, has been generating attention for one simple reason: it describes a set of problems that more and more people are having, and that set of problems isn’t well-addressed by our robust and mature ecosystem of monitoring tools and best practices. In a prime example of “this may be frustrating and irritating, but this is how language works” — observability, despite arriving on the computer architecture scene much later than monitoring, turns out to actually…

Read More...

You Could Have Invented Structured Logging

Sometimes we hear from folks who are a little bit intimidated by the notion of structured logging. Some common issues: There’s no approachable library for structured logging in my language. My logging today is messy, and changing it all is a daunting project. These are legitimate concerns! But I have some opinions: You might not need a fancy pants library You can make incremental changes to your logging setup. Structured logging is really all about giving yourself — and your team — a logging API to help you provide consistent context in events. An unstructured logger accepts strings. A structured…

Read More...

Event Foo: What Should I Add to an Event?

When we’re talking with people about how they should start using Honeycomb, many ask for guidance about what should go into an event. Though there are longer posts on this blog about what it means to be an event, this one is a “short” list of things to consider when you’re building events. What is actually useful is of course dependent on the details of your service, but most services can get something out of these suggestions. As a point of reference, the Honeycomb front end web server generates events with an average of 40 fields (±20), and our API…

Read More...

Event Foo: Building Better Events

This post from new Honeycomber Rachel Perkins is the seventh in our series on the how, why, and what of events. An event is a record of something that your system did. A line in a log file is typically thought of as an event, but events in Honeycomb can be a lot more than that—they can include data from different sources, fields calculated from values from within, or external to the event itself, and more. An event represents a unit of work. It can tell a story about a complete thing that happened–for example, how long a given request…

Read More...

Event Foo: Thou Shalt Not Aggregate at Source

This guest post from Alex Rasmussen of Freenome is the sixth in our series on the how, why, and what of events. When your application emits events, it’s usually emitting them for the benefit of a human operator – maybe you at 3am, if you’re unlucky. The operator wants as much information as possible, with as much context as possible. Keeping this in mind, here are a three things I always consider when creating events. Provide as much context as possible At minimum, an event should contain information about the process and host that emitted it, and the time at…

Read More...

Event Foo: Moar Context Better Events

This guest post from Mark McBride of Turbine Labs is the fifth in our series on the how, why, and what of events. As a systems engineer, an undervalued part of your job is storytelling. Every time you wire up an event to be dispatched to your logging system, it’s a note to your future self. Unfortunately these aren’t stories you read bundled up all cozy like with a cocktail. You read them as you’re frantically trying to put together a plan to get a broken system running again. Generating better events can dramatically improve your ability to respond to…

Read More...

Event Foo: Designing for Results

This guest post from Matt Klein of Lyft is the fourth in our series on the how, why, and what of events. Event based tracing, logging, and debugging are very powerful tools for distributed systems development and operations. However, without putting careful thought into the emitted events and the form they take, it’s easily possible (and all too frequent) that the output ends up producing an incomprehensible morass with little value. These few small tips, in my experience, go a long way towards preventing poor outcomes and realizing maximal value. Less Is More Emitting too many events is just as…

Read More...