Ep. #91, Every Failure Becomes an Eval with Janaki Vivrekar

On episode 92 of o11ycast, Ray Myers joins Ken and Jess to explore how observability, reliability engineering, and formal software engineering practices are becoming even more important as AI coding agents take on larger roles in development. Rather than viewing AI as a replacement for established engineering disciplines, Ray argues that techniques like continuous delivery, testing, specifications, and formal verification provide the confidence needed to safely harness AI-generated code.

On episode 91 of o11ycast, Ken Rimple and Jess Kerr sit down with Janaki Vivrekar. Janaki shares how Amplitude is building AI-powered analytics agents, why evaluation frameworks are becoming essential to AI product development, and how teams can use observability techniques to improve agent performance over time. The conversation explores eval-driven development, production feedback loops, and the challenges of helping AI systems reason about complex business data.

On episode 90 of o11ycast, Ken Rimple and Jessica “Jess” Kerr speak with Cory Ondrejka. Together, they unpack the rise of agentic AI, the shifting identity of software engineers, and the growing importance of measuring real-world impact. Cory shares his concept of Outcome Engineering and how teams can adapt to a world where building is fast but validation is everything.Come find the Honeycomb team at O11yCon on May 21st!https://events.honeycomb.io/o11yConSF2026Use discount code SFCast2026 for 20% off

On episode 89 of o11ycast, Ken Rimple and Charity Majors are joined by Bryan Cantrill. They dive into the origins of observability, the realities behind AI productivity gains, and the tension between cloud convenience and infrastructure control. The discussion highlights how major tech shifts often look obvious only in hindsight.

On episode 88 of o11ycast, Ken Rimple and Jessica Kerr speak with Charity Majors about the shifting role of observability in modern software development. From AI-assisted instrumentation to reducing developer cognitive load, the episode examines how teams can move from reactive monitoring to continuous learning. It’s a deep dive into observability as a sense-making practice, not just a tooling problem.

On episode 87 of o11ycast, Ken Rimple and Jessica Kerr sit down with Lada Kesseler to explore how experienced engineers can work effectively with AI coding assistants. They discuss why AI feels like a fast, noisy black box, and how patterns like semantic zooming, feedback loops, testing, and observability can help developers stay in control. This episode is a deep dive into using AI without sacrificing clarity, quality, or trust.

In episode 86 of o11ycast, Ken Rimple and Jessica Kerr sit down with Amarilis Campos from Nubank. They explore how Brazil’s largest digital bank uses observability to build resilience and autonomy at scale. Amarilis shares how Nubank’s culture of “acting like owners, not renters” and its journey from metrics and logs to tracing with Honeycomb helped the company grow to over 100 million customers.

In episode 85 of o11ycast, Dr. Cat Hicks unpacks AI’s impact on software teams from a psychological and social-science perspective. Along with Ken, Jess, and Austin, she explores how AI magnifies long-standing tensions between solitary and collaborative models of development, and how fears about AI often reflect deeper issues like undervaluing collaboration or having unrealistic productivity expectations. The discussion also explores empathy, theory of mind, and pluralistic ignorance, highlighting why developers may prepare more for AI than for each other.

In episode 84 of o11ycast, Ken Rimple and Martin Thwaites welcome Maddy Montaquila, lead PM for .NET Aspire at Microsoft. This episode dives into the "magic" of Aspire, exploring how it streamlines the developer experience from frictionless app startup to telemetry-rich development. Discover how Aspire makes telemetry a first-class citizen, simplifying debugging and enhancing observability for modern applications.

In episode 83 of o11ycast, the Honeycomb team chats with Dan Ravenstone, the o11yneer. Dan unpacks the crucial, often underappreciated, role of the observability engineer. He discusses how this position champions the user, bridging the gap between technical performance and real-world customer experience. Learn about the challenges of mobile observability, the importance of clear terminology, and how building alliances across an organization drives successful observability practices.