Are Your Web Services Ready for a Reliability and Experience Upgrade?
You’ve been there before. The codebase started out sleek and efficient—every feature clicked into place like a puzzle piece. But as time passed and new features rolled out, the application began to groan under its own weight.
A once pristine, well-oiled machine became a bit… sluggish. Maybe it started with a few more 500 errors or a little latency here and there. Next, you’re drowning in performance issues and wondering, “Where did it go wrong?”
The truth is, we hear this story all too often from enterprise engineering teams. Complex systems evolve, and as they do, so does the potential for inefficiencies. What began as a nimble, high-performing app now faces bottlenecks and breakdowns, making every deploy a potential fire drill. It’s frustrating, exhausting, and worst of all—it slows down innovation. But here’s the thing: it doesn’t have to be this way.
That’s where observability comes in. With the right tools, you can not only detect where things went awry, but also fix them faster—so your team can focus on building what’s next instead of firefighting what’s broken. In this guide, we’ll share the stories of four Honeycomb customers who were in the thick of it—bogged down by sluggish systems and spiraling technical debt.
We’ll walk through how they used observability to pinpoint the pain, reclaim their performance, and get their development cycles humming again. Each story is a testament to how deep visibility, real-time insights, and a dash of curiosity can turn inefficiency into opportunity.
The growing pains of scale
In enterprise development teams, growing the codebase can be a double-edged sword. Sure, new features and enhancements are signs of progress, but with them come inefficiencies lurking beneath the surface. Over time, platform performance takes a hit, technical debt piles up, and those once-quick deploys start to drag. Sound familiar? As your application evolves, it’s not just about adding functionality—it’s about managing inefficiencies. These often manifest in slower performance, higher resource consumption, and a creeping sense that your platform is more fragile than ever. And the culprits? Here are some of the biggest offenders:
Redundant API calls
Each API call pulls on bandwidth, server power, and sometimes third-party costs. When redundant calls sneak into the system, they’re like little resource thieves, quietly driving up operational expenses and slowing down your app’s responsiveness. Cleaning up these calls isn’t just a cost-saving move—it’s about delivering a better, faster customer experience.
Database query inefficiencies
Inefficient database queries can wreak havoc on performance. Maybe it’s a single transaction running the same query over and over again, or similar queries fetching nearly identical data. Whatever the case, fixing these queries means faster load times and a more seamless experience for your users.
Inefficient cache layers
Efficient caching reduces the load on backend systems, speeds up data delivery, and prevents bottlenecks. But when your caching layers aren’t optimized, they can do more harm than good. Fine-tuning them is essential for keeping your platform humming along without hiccups.
Technical debt accumulation
Every system has its baggage. But when that technical debt gets too heavy, it slows down development and makes even simple tasks feel monumental. Reducing technical debt isn’t just a matter of tidying up—it’s a way to free your team to innovate faster, ship new features with confidence, and maintain a high bar for quality. These inefficiencies aren’t just nuisances; they’re performance killers that can cost you customers and dollars.
Modern observability in action at University of Oxford
Situation: The Bennett Institute for Applied Data Science at the University of Oxford is a leader in harnessing data for transformative research in healthcare, policy, and beyond. Their OpenSAFELY analytics platform is a key tool, used by 22 UK research and health data institutions for complex queries across over 160 research projects. These queries involve lengthy SQL operations on massive datasets, which can span several days. The challenge lies in efficiently monitoring these long-running jobs and ensuring system performance remains smooth, despite low user traffic but high processing demands.
Solution: Honeycomb stepped in to address the Institute's need for enhanced observability. By instrumenting long-running jobs, the engineers gained critical visibility into job scheduling and system operations.
This real-time insight allowed the team to effectively monitor job queues and manage data pipelines with greater precision. Honeycomb's capabilities helped demystify the previously opaque queueing times and reasons for delays, transforming how the team managed their complex tasks.
“Instrumenting long-running jobs provided visibility into our core operations—specifically job scheduling, which is crucial for us. It helped us understand queueing times and reasons for delays, which were previously a black box due to the volume of log activity. It’s important to us that our users have an experience with the system that is responsive, trustworthy, and reliable; Honeycomb helps us make that a reality.”
— Simon Davy, Senior Research Software Engineer, Bennett Institute for Applied Data Science, University of Oxford
Results: With Honeycomb’s observability, the Bennett Institute gained valuable insights into their system’s performance. They can now proactively identify and address issues before they escalate, significantly improving system service and reliability. Continuous optimizations have become more manageable, allowing the Institute to maintain high standards of performance and trustworthiness in their research operations.
Fender gets faster performance, happier customers
Situation: Synonymous with all things rock 'n' roll, Fender ventured into the digital space in 2016 with the launch of their apps to help users start and advance their guitar skills. Their 'Free to Play' campaign, which offered a million people free three-month access to Fender Play, was a huge success.
The result? A 10X increase in traffic. While Fender's serverless infrastructure scaled impressively, they encountered API latency issues that were beginning to impact performance.
Solution: With Honeycomb SLOs tracking Fender’s latency thresholds, the team was immediately alerted to the issue. Honeycomb makes debugging so fast and easy that Fender quickly identified the cause of the latency—a search cluster was slowing down, prompting the team to increase its size. After that, the latency SLO quickly started recovering.
“This was our first big win with Honeycomb, enjoying a night-and-day experience with getting ahead of issues and finding a root cause in just a minute. The best part is we only had slightly degraded API performance—with the Honeycomb SLO we were alerted to what was happening before it resulted in a poor user experience.”
— Michael Garski, Director of Platform Engineering, Fender
Results: By quickly identifying and resolving the latency problem, Fender not only improved API performance, but also ensured a seamless user experience. Honeycomb empowered the team to find the root cause in just a minute.
Intercom speeds up their busiest endpoint by 50%
Situation: Supporting over 20,000 businesses and connecting with a billion people worldwide, Intercom is all about revolutionizing sales, marketing, and support. Intercom’s customer-centric culture drives operations, but they faced challenges in detecting key error conditions and performance issues affecting individual users.
The snag? Their existing tools struggled with the high-cardinality nature of their data, making it a real challenge to pinpoint what was going wrong and exactly which customers were feeling the effects.
Solution: With Honeycomb’s high cardinality and advanced querying capabilities, the engineering team could dive deep into their data, slicing it by user types, usage patterns, and more to pinpoint exactly what was driving endpoint performance issues.
These insights empowered them to refactor their service, zero in on crucial code enhancements, and clearly document how these changes would boost performance. They rolled out the new code alongside the old, using Honeycomb to monitor and compare performance metrics until they were sure they’d hit the sweet spot with their improvements.
“Using Honeycomb enabled us to improve our customer experience across the board and decrease our resource spend. This in turn lets us turn down capacity as machines can finish work faster.”
— Jamie Osler, Product Engineer, Intercom
Results: Intercom gained a major speed boost in their busiest endpoint with a drop in median request times by 10-15% and an improvement in P99 request times by around 50%. This not only enhanced Intercom’s customer experience but also reduced resource consumption, allowing the company to decrease capacity and optimize their infrastructure.
CCP Games modernizes and migrates its codebase
Situation: CCP Games, the creators of EVE Online, operates a universe where hundreds of thousands of players interact across 7,800+ star systems with over a million daily market transactions. But the game's codebase was over 20 years old, and as the game evolved to include OAuth2, federated logins, and an extensive API ecosystem, debugging had become a massive challenge.
With layers of legacy code, debugging issues was like searching for a needle in a haystack, making it difficult for developers to quickly diagnose and resolve performance bottlenecks.
Solution: CCP Games turned to Honeycomb for its high-cardinality capabilities, which enabled them to achieve a granular understanding of their systems at scale. Honeycomb’s tracing provided visibility into how quickly messages were being processed and where delays occurred. As CCP Games began migrating from a monolithic structure to microservices, Honeycomb proved invaluable. Its ability to detect patterns across billions of requests in under three seconds allowed the team to quickly uncover hidden problems in their complex systems.
“When you can fix things quickly, discovering them quickly becomes vital to that feedback loop. People ask, ‘Where’s Honeycomb in the new architecture?’ The answer is it’s everywhere. It’s built in.”
— Nick Herring, Technical Director of Infrastructure, CCP Games
Results: The success of CCP Games' modernization wasn't just about technology; it also required a shift in developer culture. Honeycomb played a pivotal role in that transformation. For example, with Quasar, fixes can now be deployed in 30 minutes rather than 24 or more hours.
Honeycomb is now fully integrated into their architecture, providing constant visibility and helping teams address issues faster than ever.
Conclusion
At Honeycomb, we’ve seen firsthand how complex systems can become a tangled mess of inefficiencies and technical debt as they grow. But here’s the truth: just because your codebase has evolved into a labyrinth of features and functionality doesn’t mean you’re doomed to suffer slowdowns and performance hiccups.
With the right tools and a proactive mindset, you can transform those inefficiencies into opportunities for optimization and excellence. Our customer stories illustrate that tackling performance issues isn’t about quick fixes or band-aid solutions. It’s about diving deep into your data, understanding the root causes, and taking strategic action to refactor and refine. Whether it’s tackling those redundant API calls, database query inefficiencies, or that sneaky technical debt that accumulates over years of hotfixes and workarounds, Honeycomb helps you see beyond the noise and get to the heart of the problem.
We’ve walked this path with our customers, navigating the highs and lows of performance management. We know the frustration of unexplained slowdowns and the thrill of uncovering the insights that turn things around. And we’re here to tell you: you don’t have to go it alone. With Honeycomb, you gain a partner that’s as invested in your success as you are, armed with the expertise and tools to keep you ahead of the curve.