Observability   Customer Stories  

Spread the Love: Appreciating Our Pollinators Community

By Jenni Boyer  |   Last modified on June 7, 2022

Have you heard the buzz about observability with Honeycomb 🐝? It’s the best tool on the market for observing your systems in real time to reduce toil and delight users. But don’t listen to us, listen to our buzzing community of “Pollinators”--this blog post is dedicated to them 💖

An invite is sent to our shared Slack workspace “Pollinators” to everyone who signs up for Honeycomb. As a customer advocate at Honeycomb, my goal is to help you find the right path at Honeycomb, and Pollinators is the welcome wagon that helps me help you 🌟

Whether you’re kicking the tires, you’re new to observability, or your team is actively evaluating Honeycomb, the first step is signing up for Honeycomb and jumping into what your new world with observability can look like. Sign up for free at honeycomb.io/signup or reach out to our sales team for an Enterprise trial, and check the email you signed up with for the Pollinators Slack invitation. (Or follow this direct link to request an invite 😉)

Who’s in Pollinators? Well, hopefully you soon 😊! The Hive’s rallying cry is Honeycomb Users and Honeycombers helping Honeycomb Users. All Honeycombers are in the Slack workspace. Our superstar engineers, developer advocates, and customer success team are in there, ready to answer your questions! Who else? Our customers, fellow trial-ers, and folks in our free tier. So if you have a question about how to get started, what to try with your stack, or questions about topics like SLOs (Service Level Objectives), tracing, events, and more, our community is here to help 🐝💖✨

Thank you to everyone who actively participates in Pollinators. Every day, we see unbounded kindness, curiosity, and helpfulness. And the Bees from Honeycomb are so thankful to have folks in our community discuss not only their technical questions, but their philosophical approaches to observability.

With love,

Jenni & the Bees

Some Recent Highlights

Getting Started with a Little Help from Our Friends

Nicholas Moorcroft (Zappi.io) 9:01 AM Hey again! I'm still on manual tracing and I'd now like to get my traces to propagate between my SPA and back-end Rails apps. I may have missed something but I don't see anything in the docs about cross-app propagation. Scenario: I'm wondering if there is a specific honeycomb header I can manually attach to my back-end requests? Example code: trace = { span_id: 123, trace_id: 444 } xhr.setRequestHeader('X-Honeycomb-Trace', trace) Where my Rails beeline will automatically check for the header, extract the trace and then use the span_id 123 as the parent for the next "unit of work" done by my Rails app.

Nicholas Moorcroft, Software/Observability Engineer at Zappi, is based in South Africa--thankfully, our community is timezone-agnostic with Pollinators all over the globe! Honeycomb’s product and community are deeply ingrained with the philosophy that learning from each other makes the journey that much better. With a little help from our friends, Nicholas got his tracing with the Ruby-Beeline, which is automatic instrumentation for Ruby applications, up and running at the start of his Honeycomb evaluation with our free-tier.

 In Nicholas’ own words: 

“What was really helpful at that point was when Pedro from carwow pointed out a part of the ruby-beeline code which seemed relevant. It provided some direction for me and I was able to find the actual part of the code I was interested in.

The thing I appreciate most about a Slack approach to community and engagement is that it can, and has, brought together a group of active users. Both technical and non-technical and allowed us to converse with one another. This is awesome in an age where companies can have varied tech stacks and face similar, but different, challenges when instrumenting new technologies. Additionally we get the benefits of having a direct line of communication with the business itself, be that for feedback or just general chatter.”

Zappi is an ad and innovation testing platform built by brands for brands, and is now one of Honeycomb’s newest Pro-tier customers 👋Welcome to the hive, Zappi!

Teamwork Makes the Dream Work - Discover Errors, Fix the Problem, Share the Knowledge, & Share the Love 💖

Ryan Ordway (orst.edu) 4:01 PM Honeycomb just helped me prove to our central IS group that the chain certificate for api.oregonstate.edu was expired. we get most of our SSL certs through Internet2/InCommon, and one of their intermediates expired over the weekend 100% of our oauth requests were failing. logins were still working, and deposits were still working. none of our triggers caught it. I just happened to see 100% failures on that endpoint as I was exploring the dataset to plan for building a dynamic sampler :facepalm: time to write more triggers I guess

Ryan Ordway, DevOps Engineer at Oregon State University Libraries & Press (OSULP), gives Honeycomb some sugar in the #love channel where Ryan was able to show his team that their chain certificate for their API was expired. 

In Ryan’s own words:

“It ended up not being a MAJOR issue because the API call that was failing can fail without preventing the rest of the app from loading, but it was causing a little bit of slowness. But having the API call instrumented (with the beeline, since it instruments external HTTP calls) allowed us to see the errors and alert the team on campus that manages that API so they could fix it.”

Ryan’s team is using Honeycomb Triggers for these particular issues--Triggers let you receive notification when your data in Honeycomb crosses the thresholds you configure, you can think of them to be like alerts. 

Ryan has also been a guest contributor to the Honeycomb Blog, see his piece here on how adopting observability at OSULP & Honeycomb: “The most important thing for us is saving money. I’ve finished migrating all of our tracing from our other vendor to Honeycomb, using the same little Honeycomb sandbox from when I started this adventure. Even better, I did it without having to pay more money. That’s huge!”

In the future, Ryan hopes his small but mighty team will be measuring their Service Level Indicators (SLIs) and SLOs with Honeycomb 🐝💪 Thank you Ryan for all of your contributions to our community, content, and product, we wouldn’t be the same without you!

Test in Production, Watch Deploys in Real-Time, Share Context Across Teams

matt (geckoboard) 6:58 AM The other week honeycomb helped us realise that one of our frequently used queries was doing a full table scan rather than using an index because it was filtering based on UPPER(id), but the index is on id. Can you see where we deployed the fix? :joy:

Honeycomb’s rallying cry has always been that we test in production, and much of our community does the same. Matt Button, Senior Infrastructure Engineer at Geckoboard, discovered something peculiar going on in the backend and the team was able to deploy the fix & see the results in real-time 👀 

In Matt’s own words:

“We were using some trace waterfalls to show a frontend engineer how the work the backend did differs when you change certain arguments in the request. While looking at one of the traces we noticed that one of the DB queries was actually taking up quite a large chunk of the request time, even though it should’ve been one of the fastest in that request. 

When we looked into it we realised that query wasn’t using the indexes we’d defined for it, which meant the time to execute that query could vary a fair bit, as seen in the “before” section of that pic. It turns out the id field was never going to have mixed-case characters, so the query didn’t need to filter on UPPER(id). That was the fix we rolled out in the screenshot (above).”

Having full-stack visibility is crucial for being able to ask any question of your system. 

Geckoboard built their own system health dashboards and had been using standard log search tools since the time when log search was new and shiny. But as their business has scaled up, so did their need to understand who is affected by a given issue, and Honeycomb helped them start connecting the dots.

Read more about Geckoboard’s experiences with Honeycomb:

Thank you Matt for your endless contributions to Honeycomb & the Pollinators community!

Sampling: One of Life’s Unavoidable Truths - Refine Your Data Without Losing Fidelity

Sampling is like eating your vegetables - you have to do it or else you’ll be in pain later 🥦💪 but also like vegetables, sampling is unique to your system, different depending on what you want, oddly shaped, and (may) give you gas 😂 depending on who you ask. Benjamin Franklin missed the third unavoidable truth in life, so I’ll revise it here: “in this world, nothing is certain but sampling, death, and taxes”. 

For teams running workloads at scale, the question becomes: are all of my events worth keeping? How can I reduce overall event volume while maintaining fidelity? Sampling is a complex science and intuitive art, just ask your local statistician, or any engineer trying to find a needle in a haystack. 

The best resource when trying to learn new techniques is to learn from each other:

Molly Struve 2:37 PM Little late to this conversation but @Ryan Ordway (orst.edu) we created a super handy redis and SQL sampler for dev.to that has been working marvelously. https://github.com/thepracticaldev/dev.to/blob/master/app/lib/honeycomb/noise_cancelling_sampler.rb You might not need all of it but feel free to use it as a template or get some ideas! app/lib/honeycomb/noise_cancelling_sampler.rb

Molly Struve, Lead Site Reliability Engineer at dev.to, shares her Redis & SQL Sampler with Pollinators as a template for people to learn from! Honeycomb has partnered with the Forem team to use their open-source production code to help pay it forward for the community. You can read about their Honeycomb journey in our HoneyBytes series, including the latest sampling post where we walk through the very implementation Molly shared above!

In Molly’s own words:

“The project I work on, Forem is a completely open source platform which makes it super easy to share our Honeycomb integration with others. Always happy to help where I can. Honeycomb is still pretty young so I think it’s important to have example projects that people can use as templates for their own setups”

Thank you for your dedication to educating others Molly!

polotek (Mode) 3:11 PM We have taken a first pass at dynamic sampling in our rails codebase. I thought I’d share a little bit. I’ve been following the guidance here https://docs.honeycomb.io/working-with-your-data/best-practices/sampling/#dynamic-sampling. And also listened to @lizf (Honeycomb)’s great talk here. https://www.youtube.com/watch?v=HFnBHA6NWUE The configuration is super simple. We were using static sampling and we put this behind a feature flag. config.sample_hook do |fields| if use_dynamic_sampling? Honeycomb::DynamicSampler.sample(fields) else # This essentially replicates the static sampling that honeycomb # does by default when the `sample_rate` config value is set. But # we need to do it here to support the dynamic feature flag Honeycomb::StaticSampler.sample(fields) end end

You’ll notice that we had to create a little static sampler class along with the dynamic sampler. This is because there doesn’t seem to be a good way to fall back to default behavior after you’ve invoked the sample_hook. module Honeycomb class StaticSampler extend Honeycomb::DeterministicSampler def self.sample(fields) sample_rate = fields['sample_rate'] || Config.sample_rate [should_sample(sample_rate, fields['trace.trace_id']), sample_rate] end end end

Additionally, Marco Rogers, Senior Software Engineer at Mode Analytics, shares his approach to dynamic sampling in the Open Source Ruby Beeline SDK Channel and what ensues after is an awesome 🤩 exchange between users and Honeycomb collaborating with tips, tricks, and sampling kickflips. 

In Marco’s own words:

“We needed to control the overall volume of events, but we wanted to keep every single thing that might indicate a problem. So dynamic sampling for us was about how to identify a successful (boring) event vs one that might have useful debugging data on it. Turns out that’s not as easy as throwing away all of the 200 status codes.”

Thank you for all of your contributions to Pollinators, Marco! The new-bees have lots to learn from you!

At Honeycomb, we recognize how important & tricky it is to sample correctly. Below are some resources (like the ones mentioned in Marco’s message) to help you understand and apply the  sampling method that works for you. 

System Optimization - Find Your Bottlenecks

pedro (carwow) 6:48 AM What a DB index can do. :smile: We noticed some bad performing endpoints on a user facing service and noticed a pattern similar to this one, after looking at some traces and looking at a few of the upstream services we noticed that same pattern on one of them. This pattern kind of pointed to a missing DB index depending on the query parameters passed to the API endpoint. We made a small change to test the hypothesis which affected just a small set of those requests (you can see a small drop on the left part of the chart), then we decided to do same to change to all of the same type of query parameters which we identified from Honeycomb BubbleUp and GroupBy and that is the drop you see on the right. This reduced the P99(duration) by an order of magnitude, from 1s to 100ms. We still have a few unoptimised combinations, but not too critical for the time being.

In distributed systems, small inefficiencies can create performance bottlenecks at scale. Pedro Chambino at carwow was able to observe some experimental optimizations with Honeycomb--again, testing in production!

Plus, seeing examples of how other people use Honeycomb can help you get more out of the tool. In Pedro’s own words:

“I think it has been more about being connected with the community and being able to learn about new features/changes to Honeycomb and how other people are using them. This was specially true with sampling, heatmap, and SLOs. Also, learning how we can use Honeycomb in more contexts, e.g. buildevents and airflow.”

Taking It to the Tweets

We love when the community takes to Twitter to share their Honeycomb experiences with the greater tech community. Here’s a fun one from Julian Simioni, Co-Founder of Geocode Earth:

Follow @honeycombio on Twitter, there’s plenty more where that came from. And tag us in your own stories!

Shout Out to Our Community of Contributors

While we’re showing our appreciation, check out our community of contributors. This is only going back to December 2019! When you upstream your improvements into our OSS codebases, it makes those changes available to the rest of the user community. Help us help you 💞

Go Beeline

Ruby Beeline

Python Beeline

Node Beeline

Java Beeline

libhoney-go

libhoney-rb

libhoney-js

OpenTelemetry Go Exporter

We also track third-party contributions! Have a cool Honeycomb project? Open a pull request and we’ll add it to the list 👩‍💻✨

Join the Hive

See what all the buzz is about! Request an invite to the Pollinators Slack today 🐝🍯💖

 

Related Posts

Observability   Customer Stories  

Transforming to an Engineering Culture of Curiosity With a Modern Observability 2.0 Solution

Relying on their traditional observability 1.0 tool, Pax8 faced hurdles in fostering a culture of ownership and curiosity due to user-based pricing limitations and an...

Observability   LLMs  

Honeycomb + Google Gemini

Today at Google Next, Charity Majors demonstrated how to use Honeycomb to find unexpected problems in our generative AI integration. Software components that integrate with...

OpenTelemetry   Observability  

Observing Core Web Vitals with OpenTelemetry: Part Two

In a previous blog post, we outlined how to set up our own auto-instrumentation to send Core Web Vitals data to Honeycomb. We recently released...