LoginOne Year of Graviton2 at Honeycomb
A year ago, we wrote about our experiences as early adopters of Graviton2, and how we were able to see 30% price-performance improvements on one dogfood workload from switching to the arm64 architecture. In…
Sweetening Your Honey
Are you looking for a better way to troubleshoot, debug, and really see and understand what weird behavior is happening in production? Service-level objectives (SLOs) and observability can help you do all that—but they…
A Recap of the KubeCon + CloudNativeCon Observability Track
OpenTelemetry has evolved so much since the 2019 KubeCon North America in San Diego, where I live-demoed OpenTelemetry on the keynote stage and highlighted our alpha to the world. We’re excited to be entering…
From 0 to Insight with OpenTelemetry in Go
Honeycomb allows you to send in data from a variety of sources. The steps to get there are a choose-your-own-adventure path and the choices to make may not always be clear to folks new…
Observations on ARM64 & AWS’s Amazon EC2 M6g Instances
At re:Invent in December, Amazon announced the AWS Graviton2 processor and its forthcoming availability powering Amazon EC2 M6g instances. While the first-generation Graviton processor that powered A1 instances was better suited to less compute-intensive…
Incident Report: Running Dry on Memory Without Noticing
On November 6, 2019, we intermittently rejected 1-3% of customer telemetry data at ingest for four periods of 20 minutes each. The trigger of the incident was a slow memory leak that manifested over…
Treading in Haunted Graveyards
Part 1: CI/CD for Infrastructure as Code At Honeycomb, we’ve often discussed the value of making software deployments early and often, and being able to understand your code as it runs in production. However,…
Making Instrumentation Extensible
Observability-driven development requires both rich query capabilities and sufficient instrumentation in order to capture the nuances of developers’ intention and useful dimensions of cardinality. When our systems are running in containers, we need an…
Dynamic Sampling by Example
Last week, Rachel published a guide describing the advantages of dynamic sampling. In it, we discussed varying sample rates to achieve a target collection rate overall, and having different sample rates for distinct kinds…
A New Bee’s First Oncall
I’m Honeycomb’s newest engineer, now on my eighth week at Honeycomb. Excitingly, I did my first week of oncall two weeks ago! Almost every engineer at Honeycomb participates in oncall, and I chose to…