Days to Minutes: How Freshpaint Pinpoints Customer Issues With Honeycomb

Freshpaint sits between healthcare organizations and the advertising and analytics platforms they rely on. The platform removes protected health information (PHI) from website events before forwarding sanitized data to Google Ads, Meta, and more than 100 other destinations. The result is hundreds of millions of events flowing through a multi-tenant platform every day.

Behind that volume is a complex infrastructure spanning Lambda, ECS, Kinesis, RDS, and S3. For Engineering Manager Mark Rogers and Staff Software Engineer M Bianchi, understanding how individual events moved through that system was becoming increasingly difficult. Freshpaint relied on AWS CloudWatch and X-Ray for observability, but the existing setup left gaps in the team's ability to connect the dots and explain what happened when customers reported discrepancies.

Those gaps showed up most when customers noticed discrepancies in their conversion numbers. Because Freshpaint intentionally removes certain data to protect patient privacy, some differences were expected. But the team lacked a reliable way to explain exactly why the numbers didn't match.

To investigate, Freshpaint built its own tracing tool. But because the homegrown solution only sampled events, the specific events behind a given discrepancy were often missing, making it hard to explain why a customer's numbers didn't match. Engineers had to pair raw data from Snowflake with individual traces and manually piece together evidence to explain each discrepancy. An investigation could take a day or two, and even then, the team couldn't always reach a definitive answer.

As Freshpaint scaled across customer volume, product complexity, and growing staff, the engineering team needed a means to move beyond individual knowledge and familiarity to manage troubleshooting.

There were several operational challenges they wanted to solve for:

• Opaque infrastructure alarms: CloudWatch and X-Ray provided infrastructure metrics and tracing capabilities, but the team lacked the end-to-end context needed to consistently connect a system-level symptom to the underlying cause across distributed components.
• Alerts pointed engineers in the wrong direction: Alerts fired on infrastructure symptoms, so a database load alarm might point to a problem eight steps away in a Lambda function.
• Alert volume obscured meaningful signals: Engineers had to spend time distinguishing actionable problems from routine noise.
• Customer-specific traffic patterns were hard to isolate: If one customer suddenly generated 10 or 100 times its normal traffic, the team couldn't readily identify that customer and understand the impact.

Freshpaint needed a way to investigate the system at the level where its operational questions arose: individual events, specific customers, and the relationships between components. That meant moving to an observability solution that could provide end-to-end visibility and clear signal through the noise.

Bianchi had used Honeycomb in a previous role and recognized its potential to address these challenges. Freshpaint put it to the test with a proof of concept, evaluating three requirements: whether it could support near-unsampled tracing at Freshpaint's event volume, fit within budget, and meet the company's requirements under a Business Associate Agreement (BAA).

Honeycomb cleared all three bars, and Freshpaint made it the foundation of its observability practice.

To achieve complete clarity across workflows, Freshpaint implemented unsampled event tracing on their core ingestion engine. By capturing high-cardinality metadata, including customer environment IDs and destination endpoints, the team gained precise visibility into every event moving through the platform.

“We operate a complex multi-tenant ingestion pipeline where customers need to know exactly what happened to their data. With Honeycomb, we capture high-cardinality attributes on every event, so we can filter straight down to a specific customer or destination without losing critical context,” shared Bianchi.

Two recent incidents show how that visibility changed the way the team diagnoses problems and assesses their impact.

A 2x impression surge

One customer's impression traffic doubled, and processing fell up to 20 minutes behind. Engineers drilled straight to the customer's environment ID in Honeycomb and identified the account in 10 minutes. Mark estimates CloudWatch alone would have taken an hour or more, and the team likely would have mistaken it for a general traffic increase.

A Google CAPI rate-limit cascade

In another incident, a customer sent orders of magnitude more traffic to the Google Ads Conversions API. Google rate-limited Freshpaint, retries piled up, and an ingest Lambda ran out of memory. Honeycomb traced the failure to a single environment ID. The team increased Lambda memory, contacted the customer, and used the same data to determine which other customers using that destination were affected.

“Most problems are obvious to diagnose now because we can group by customer in Honeycomb. It's usually one customer and one destination type. Scoping the blast radius, beyond detecting the cause, is just as useful,” noted Bianchi.

Event ingestion is Freshpaint's core promise to its customers, so the team needed a clear, shared definition of health. In Honeycomb, they established an ingest “successfully processed within 5 minutes” SLO of 99.95%.

At Freshpaint's scale, not every error signals a problem that needs intervention. A destination hiccups, a request retries, and the system recovers on its own. Before SLOs, even these transient errors could trigger alerts, leaving engineers to judge each one individually. Honeycomb SLOs gave the team a more consistent way to distinguish routine fluctuations from problems that threaten reliability. The error budget accounts for the level of failure the SLO allows, while burn rate alerts warn the team when errors are consuming that budget too quickly.

Honeycomb SLOs changed how the team works with alerts:

• Routine signals go to a lower-priority channel, so engineers can review them without being interrupted by every transient error.
• Fast error budget burn gets immediate attention whenever reliability is at risk of falling below the 99.95% target.
• SLOs set engineering priorities: when ingest meets its SLO, the team goes full throttle on the roadmap, and when it doesn't, reliability comes first.

“Before Honeycomb SLOs, we would have chased a lot of ghosts in the noise. Now, we can ignore the everyday blips and act when it's a real issue,” explained Mark.

Some of the data problems Freshpaint's customers encounter start outside Freshpaint itself. A website update can break data collection, or a change to a Google Ads setting can disrupt downstream event delivery. Without a way to spot these issues early, Freshpaint might not know anything was wrong until a customer noticed a discrepancy and contacted support.

Now, Freshpaint uses Honeycomb triggers to monitor each customer's event volume and success rate. When either deviates from expectations, the team can investigate and alert the customer before the issue becomes a support ticket.

That early visibility has changed the customer relationship. Freshpaint can often identify problems with a customer's setup before the customer notices them, turning troubleshooting into an opportunity to provide proactive support.

“We've had a couple of pretty significant customers thank us for telling them their own platform was broken before they knew it. One of our big goals with Honeycomb was to reduce customer-reported issues, and now we're finding problems before customers do,” said Bianchi.

Freshpaint's next step is to make observability part of how new features are designed. Every RFC and project plan has to answer one question: How will you know it's working?

Answering that question builds observability into the design from the start. Instrumentation and alerts ship with the feature, so the team knows whether it's working from day one and problems surface in Honeycomb before they reach a customer.

On the platform side, the team plans to extend tracing from the web UI through the backend services behind it, giving engineers one continuous view of how requests move through Freshpaint.

“We have a proven path with Honeycomb on our ingest flow, and it's inspiring our product teams to bring that same level of excellence to every feature. Because Honeycomb is Terraform-enabled, we want to codify observability into our software development lifecycle,” said Mark.

1. Prove it's a problem before you fix it
   Without an SLO, you can't tell whether something is broken or just not ideal. Let the SLO decide what deserves engineering time.
2. Instrument for the questions customers ask
   Capture high-cardinality attributes like customer and destination so you can slice straight to the answer for any customer.
3. Build observability into the design
   Ask every RFC the same question: How will you know it's working? Decide what you'll measure and what failure looks like before anyone writes production code.