Honeycomb Blog

How Honeycomb Uses Honeycomb, Part 9: Tracing The Query Path

This post continues our long-running dogfooding series from How Honeycomb Uses Honeycomb Part 8: A Bee’s Life. To understand how Honeycomb uses Honeycomb at a high level, check out our dogfooding blog posts first — they do a great job of telling the story of problems we’ve solved with Honeycomb. Last week we announced the general availability of tracing in Honeycomb. We’ve been dogfooding this feature extensively as part of an ongoing effort to keep Honeycomb fast and reliable. In this post, we’ll discuss how we use tracing internally, some of the ways it’s helped us improve our service, and some of the…

Read More...

Get deeper insights with Honeycomb Tracing

gif of looking at traces for api query timings

We’re excited to introduce Honeycomb Tracing! Now, you can both: Visualize individual traces to deeply understand request execution, and Break down, filter, and aggregate trace data to uncover patterns, find outliers, and understand historical trends. Tracing makes it easier to understand control flow within a distributed system. Waterfall diagrams concisely capture the execution history of individual requests, making it easy to answer questions such as: What does the high-level structure of the code look like? Which methods are especially slow? Are we making too many calls to a particular service? Are things happening serially when they could be parallelized? Is…

Read More...

New Honeycomb Integrations for PostgreSQL

We’re excited to announce that you can now use honeytail and rdslogs to send your Postgres query logs to Honeycomb. Honeycomb helps you answer the following questions (and more) about your database workload: “What does the overall distribution of query latency look like?” “Which queries are responsible for the spikes in query volume that we’re seeing?” “How did our query workload change after the latest deploy?” What real observability gets you Overall database statistics such as blks_read and blks_hit are useful for general tuning, but don’t tell you much about the actual queries your application is running, or how those…

Read More...

You Could Have Invented Structured Logging

Sometimes we hear from folks who are a little bit intimidated by the notion of structured logging. Some common issues: There’s no approachable library for structured logging in my language. My logging today is messy, and changing it all is a daunting project. These are legitimate concerns! But I have some opinions: You might not need a fancy pants library You can make incremental changes to your logging setup. Structured logging is really all about giving yourself — and your team — a logging API to help you provide consistent context in events. An unstructured logger accepts strings. A structured…

Read More...

Istio, Envoy and Honeycomb

Here at the hive, we’re exceedingly excited about the emerging future of the “service mesh”. Deploy a sidecar proxy such as Envoy in your infrastructure, and you get consistent support for advanced traffic control, fault injection, request-level observability, and other powerful features for every service. That’s a mighty useful tool to have when operating distributed systems. In particular, this technology makes canary deployments and incremental rollouts dramatically simpler — particularly when used with a control plane such as Istio. But the whole point of a canary is lost if you can’t usefully observe it! In order to deploy with confidence,…

Read More...

Honeycomb <3 Kubernetes Observability

Introducing the Honeycomb Kubernetes Agent and ksonnet integration We’re excited to release the Honeycomb Kubernetes Agent. The agent provides a flexible way to aggregate, structure, and enrich events from applications running on Kubernetes, before sending them to Honeycomb for you to explore. A huge thanks to the fine folks at Heptio who worked with us on developing this agent and making it available as a ksonnet mixin! Why You Might Care Kubernetes can be complicated, and identifying the root causes of problems within a cluster can be difficult. Is something wrong with a particular node? An application’s latest code? Did…

Read More...

Introducing Derived Columns

We’re excited to introduce derived columns! Derived columns let you run queries based on the value of an expression that’s computed from the columns in an event, making it easier to answer questions such as: What does service performance look like from the perspective of our ten highest-value customers? How many requests spent more than 50% of their request time talking to a database? What fraction of service requests complete in less than 100 milliseconds? How did performance compare between builds older than build ID 4300, and all newer builds? Let’s look at an example. Say you have events from…

Read More...

Event-Driven Instrumentation in Go is Easy and Fun

One of many things I like about Go is how easy it is to instrument code. The built-in expvar package and third-party libraries such as rcrowley/go-metrics are delightfully simple to use. But metrics aren’t quite enough! We’re here to encourage you to structure your instrumentation not just around metrics, but around events. Let’s make that idea concrete with an example. Imagine a frontend API for a queue service. It accepts user input, and writes it into a Kafka backing store. Something like this: func (a *App) handle(r *http.Request, w http.ResponseWriter) { userInfo := a.fetchUserInfo(r.Headers) unmarshalledBody := a.unmarshalInput(r.Body) kafkaWrite := a.buildKafkaWrite(userInfo,…

Read More...

Introducing Honeycomb’s TCP Agent for MongoDB

We’re excited to release honeycomb-tcpagent, an efficient way to get query-level visibility into your MongoDB deployment. honeycomb-tcpagent parses TCP traffic between MongoDB clients and servers, and reconstructs queries in a friendly JSON format. Honeycomb helps you explore this data to quickly uncover anomalies. Get started with Honeycomb and run the agent, or keep reading for more background and examples. Are you running a database that’s not MongoDB? Let us know! Support for MySQL is already in the works. Database Observability Means Lots of Questions About Lots of Data For any serious database performance work, the ability to fully capture a…

Read More...