Do we really need LLM observability?

In this conversation, Kyle Forster, Founder of RunWhen, discusses what AI agents actually need to work safely in production. He explains why logs and metrics make up only around 30% of the data reaching an LLM during an incident, and why the other 70%, the CLI calls, API queries and dead ends agents accumulate between incidents, behaves like the knowledge a senior SRE builds over time. Kyle shares why typical stacks wreck accuracy, what "safe for production" means when agents have write access, and how the SRE role is changing with advancements in AI.

In this episode of Operationally Intelligent, Ricardo Castro explores why so many platform teams fail by building tools nobody asked for. He breaks down the ivory tower mentality that plagues platform engineering, the importance of sitting with developers to understand their real pain points, and how operational data should drive every decision. Ricardo and Adam discuss why DevOps, SRE, and platform engineering aren't competing disciplines, how metrics like lead time for change and developer satisfaction can bridge the gap between engineering and leadership, and why treating internal tools as products is the key to adoption. The conversation makes a compelling case that observability is the thread connecting it all.

In this conversation, Karthik Kalyanaraman, Co-Founder and CTO of Langtrace, discusses the evolution and importance of observability in AI, particularly for large language models. He shares insights on the challenges faced by developers in implementing observability, the role of OpenTelemetry, and the significance of standards in preventing vendor lock-in. Karthik emphasizes the need for organizations to connect observability to business outcomes and the future potential of AI observability in building trust among engineering teams and executives.

In this episode of Operationally Intelligent, Nathen Harvey from Google Cloud discusses his journey to leading the DORA team and the significance of DORA in measuring software delivery performance. He explains the core metrics of DORA, the importance of capabilities and conditions for success, and how organizations can implement DORA effectively. The discussion also touches on the role of leadership, the impact of real-time data, and the future of DORA research, particularly in relation to AI.

In this episode of Operationally Intelligent, we chat to Hanson Ho from Embrace and discuss the intricacies of mobile observability, emphasizing the importance of user-centric monitoring and the challenges faced in collecting and analyzing data from mobile applications. We explore the role of OpenTelemetry in standardizing observability practices and provide insights into best practices for mobile teams. The discussion highlights the significance of understanding user experience and performance metrics in driving operational intelligence and business value.

In this episode of Operationally Intelligent, we're joined by Stephen Townshend , and we discuss the complexities of operational intelligence, site reliability engineering, and the role of data in making informed business decisions. We explore the challenges of navigating complex systems, the importance of effective communication within teams, and the illusion of control in large organizations. The conversation also touches on the potential of AI in enhancing operational intelligence and the need for trust in AI-driven decision-making.

In this episode of Operationally Intelligent, Adam Kinniburgh and Michael Hausenblas discuss the nuances of operational intelligence and observability, exploring the importance of learning from incidents, the challenges organizations face in transitioning to operational intelligence, and the role of culture in fostering a supportive environment. They also delve into cost considerations, data storage strategies, and the potential of AI in enhancing operational intelligence practices.