We just wrapped O11yDay London, and one theme ran through every session: writing code is no longer the hard part. Review queues, attention, operating agents in production, runaway token bills, and proving any of it is worth the money are where teams are getting stuck. The teams making progress are the ones who can see what's happening in production.

Here are some highlights. We'll have videos available in the near future.

Session recaps

Cobots, Dark Factories, and Architecting for AI (Sam Newman)

Sam, author of Building Microservices and Monolith to Microservices, opened the day with a talk about uncertainty, with a warning that anyone who tells you what the future looks like is selling something. Across the industry, measured gains from AI look closer to 10-15% than the 10x we were promised. Meanwhile, bigger PRs are overwhelming human review, and people are typing "LGTM" out of what he called cognitive surrender.

His reframe was to ask what human code review is for: correctness, shared learning, alignment with team practices, and auditability. Each of those can be handled in other ways (tests, fitness functions, automated checks). Borrowing from robotics, he suggested that not all code is equal. Critical modules might deserve a "cobot" approach, with human and AI working side by side, while less critical ones could be built in a "dark factory" with no one reading the code. Either way, trust comes from verifying output, not eyeballing it.

“Code's not truth. Never has been.”
Sam Newman

What's in people's heads and what's in production. That means observability has to extend into the delivery process itself. Agree as a team on why you do code review, find the modular boundaries where you can safely experiment, be intentional rather than rabid about AI, and use the time you gain to be more human.

Honeycomb: How We Built It and What's Coming Next (Liz Fong-Jones, Purvi Kanal, and Wolfgang Therrien, Honeycomb)

Honeycomb has roughly doubled its engineering output without adding engineers, and incidents nearly doubled with it. The team showed what it looks like to debug an agent with an agent, using Autobot, Honeycomb's internal code review agent.

Starting from a new Anomaly Detection alert, they followed an investigation through Canvas, which had already formed hypotheses, cited telemetry evidence, and surfaced candidate PRs before anyone opened the tab. From there, they moved to the Agent Timeline, which shows a single conversation's duration, LLM and tool calls, failures, tokens, and cost (about $1.70 and seven minutes for one review). Then, they moved to the AI Ecosystem page for the fleet view: seven agents, twelve tools, and a review agent responsible for 97% of spend. Zooming out to 14 days showed the failure spike lining up with the switch from reviewing once per PR to once per commit.

They also previewed what's next:

• Connectors that bring GitHub, Linear, and Braintrust context into Canvas
• Memory that lets agents retain the lore that normally lives in an engineer's head
• Team agents you can configure for specific services, SLOs, or anomalies

Finally, they pointed a coding agent at the finished canvas using Honeycomb's MCP server, and it opened a PR with the fix.

On-Demand Software: AI-Testing a Honeycomb Private Cloud Migration at Booking.com (Ben Hartshorne, Honeycomb, and Gural Vural, Booking.com)

Booking.com has used Honeycomb since 2016, at a scale of trillions of events a year. To get full value, they wanted to send far more data, which pointed them toward Honeycomb Private Cloud. Their hypothesis was that sending 5-10x more data would improve quality without a matching jump in cost, but they needed proof... and they only had two weeks.

Instead of a synthetic load test, Gural's team built AI agents that replayed thousands of real user queries (pulled from Honeycomb's activity log) against multiple environments, including their SaaS baseline. They did the same for Canvas and MCP usage. Private Cloud performance was comparable to SaaS, and the results were more representative. The test harness was built for this one evaluation, then thrown away.

Ben tied this to a concept he called falsework: temporary scaffolding, like the wooden frame under a bridge arch, that supports building the real thing. Because you plan to delete it, it doesn't have to be production quality, which makes it a great fit for AI-generated code.

Operational Excellence for Agents With Fin.ai (Brian Scanlan and Kesha Mykhailov, Fin)

Brian and Kesha packed a lot into one slot, but a lot of their talk centered around building good AI systems. "We're all AI engineers now," they said, which means two things: statistical rigor, and experimentation.

Brian covered how Fin rolled out Claude Code across the company with a plugin marketplace, enforced configuration, and a Honeycomb integration built on one very long shell script. Every behavior change now gets treated like a production experiment, with A/B arms and Honeycomb data. A few examples:

• Anthropic's brief mode cut output tokens by about 20%
• A hard nudge to delegate to cheaper models saved 7.4% in cost, versus 2.5% for a soft nudge
• A comment density hook produced 38.4% fewer comments
• A much-hyped token-saving tool turned out to save essentially nothing

Kesha then introduced Shrek, Fin's PR review agent, made up of subagents for problem statement, alignment, skills, correctness, and safety (for instance, PRs over 150 lines don't get approved, because small changes are safer to ship). Funnily enough, a cost-saving model swap caused Fin's first Shrek incident, because the cheaper model was 26% noisier and engineers got frustrated. Instead, they found a customer-centric signal: the percentage of PRs merged despite a rejection from the correctness agent. The engineers at Fin realized that agents really are sociotechnical systems, and every incentive you create has second-order effects.

How Is Agentic Product Engineering Affecting Platform Teams? (Abby Bangser, Syntasso/Kratix, and Adrian McCabe, Zopa Bank, moderated by Martin Thwaites, Honeycomb)

AI has multiplied what product engineers produce, but the people who review, run, and own that output haven't scaled with it. It's also no longer just engineers. Product managers and other roles are committing code, so capacity has to be measured against employees, not just engineering headcount. Platform teams are seeing PRs arrive with no prior conversation, sometimes three people building the same thing at once, and the burden lands on reviewers who have to operate code they didn't write.

Abby argued that internal platforms should cover what's unique to your business but common to your teams, and that agents, harnesses, and review tools now belong on that list. She prefers the word "coherency" to "standardization," since the goal is to reduce cognitive load so people want to use the platform.

Both agreed that "eating your vegetables" (good telemetry, error handling, tests) pays off more now that agents are constantly reading it.

You Can't Vibe Code Uptime... Or Can You? (Yana Chen, Baseten)

At Baseten, Yana works on reliability for an inference platform running over 100 Kubernetes clusters. In her talk at O11yDay London, she shared two stories about AI not quite doing the right thing.

The first: During a log shipper migration, Claude produced throwaway validation scripts quickly, but it told her 20 lagging nodes were "just a rollout" when the release had been suspended for seven months. Nothing was technically wrong, but the picture was incomplete, and she asked "are you sure?" 15 times across sessions, each one turning into a new check. It was not.

The second: An agent confidently named GOMAXPROCS as the root cause of dropped spans in Refinery, but it never checked the Go version Refinery actually used, which already had the solution. She compared it to the Woozle effect, where a claim repeated often enough becomes known to be "true."

“Somebody always pays.”
Yana Chen

Yana closed her talk with some advice: automate the routine work, and leave humans the rare and hard problems. AI made writing code cheap, but it made attention more expensive.

We Let AI Investigate Hundreds of Our Alerts (Johannes Weissensel and Mohammed Ragab, HelloFresh)

HelloFresh gave its Slack-based incident helper, TARS, an AI root cause agent, and saw service restored 20% faster on automatically created alerts, which—amazing—but there were issues along the way. Over 10% of investigations stalled, the agent sometimes finished after the on-call engineer had already fixed the issue, and less than 1% of investigations got user feedback.

To observe the agent, they built an OpenTelemetry pipeline to repair spans from the Claude Agent SDK (which had hidden subagents and swallowed errors) and sent them to Honeycomb's Agent Timeline. They layered on static rule checks plus an LLM judge scoring relevance, hallucination, and bias, and found stale instructions, unauthorized tool use, and a redundant step burning 34,000 tokens. They also found that messy telemetry, outdated runbooks, and noisy alerts now cost real tokens. Their conclusions:

• Fixing the old foundations helps humans and agents alike.
• A shared agent raises the floor for everyone, including the new joiner on their first shift.
• Building an agent is easy, but operating one is most of the work.

Your AI Problem Is Your 2016 Problems Dressed in a Trenchcoat (Rebecca Murphey, Honeycomb)

There is no one number that tells you whether engineering is worth it. What helps is a handful of old ideas from Amdahl's Law, Little's Law, and the theory of constraints: fix the slowest part, watch your queues, and expect a new bottleneck every time you remove one.

At Honeycomb, that bottleneck turned out to be code review. After an RFC listing what the team values in review, we found we could move much of it earlier, and automate more of it. We also found that about 80% of merged PRs had no human-driven changes. As a result, we updated our best practices to read that two developers must participate in a change, and one of them may be a machine. Our goal is to have machines review 25% of PRs on their own by year end, starting with low-risk changes.

“Review is the most expensive place to change your mind.”
Rebecca Murphey

Her parting advice: favor flow efficiency over cycle time, look at how much capacity goes to KTLO (robots are good at it), and ask where the next bottleneck will be, which might not be in software at all.

Everyone Is Looking at the CTO for the Answer on AI: 12 Months of Navigating AI (John Doran, Phorest)

The board wants to know how AI can drive revenue, customers want answers, and engineers are worried about slop and the safety of their jobs. As a CTO, John's approach was to get hands-on again and position himself as a learner. He shared how that played out at Phorest: the CEO shipped a PR, designers and PMs got hooked on building, and the board's question flipped from "where are we saving?" to "why aren't you spending more?"

Phorest launched Ivy, an AI booking agent that handles 95% of its chats, and set a goal to halve cycle time (first commit to production, measured at 44 hours). The hard part now is measurement, but John did share a good piece of advice: get a baseline, map your initiatives, note how long they took—and when the questions come (because they will), you can speak intelligently about what changed.

“Start the clock.”
John Doran

Nobody Knows What Anything Costs Anymore (Corey Quinn, The Duckbill Group)

Corey closed the day by opining that AI costs are, in addition to a finance problem, an observability problem. AI bills grow because an agent decides mid-request to do something else, retries, or resends a ton of context on every loop. Everything can return a 200 and still be a bad deal. He walked through his own examples: a crashloop that burned $50 before alerts caught it, a singing coffee pot that would have cost $1,400 at API prices, and a newsletter pipeline where traces showed half the spend went on work already done. He also cited an insurance claims agent averaging $0.73 per claim that occasionally hit $80.

“A bill that's itemized is very far from a bill that's explained.”
Corey Quinn

His recommendations are: include three fields on every span (cost, price at the time, and whether the result was accepted), share one request ID across retries, send telemetry using the OpenTelemetry GenAI semantic conventions instead of homegrown names, and apply spending limit enforcement outside the agent loop. And for the love of all that is holy, don't let an agent approve its own expenses.

See you next year, London!

We opened the day by asking everyone to find someone they didn't know and share stories, and to follow the Pac-Man principle: leave room in your circle so someone new can join in. Judging by the hallways and the hours after the closing keynote, you did just that.

Across ten sessions, the same questions kept coming up. Who reviews all this code? What does an agent cost, and is it worth it? How do we know what our agents are doing? None of the speakers claimed to have the answer, but all of them pointed to the same starting place: instrument, measure, and look at what's happening in production.

If any of this sparked something for you, or if you want the foundational theory behind it, pick up your free copy of Observability Engineering.