Ep. #71, Evaluating LLM-based Apps with Shir Chorev of Deepchecks

In episode 75 of o11ycast, Daniel Ravenstone and Adriana Villela dive into the challenges of adopting observability and OpenTelemetry in modern organizations. From breaking down vendor loyalty to getting developers to see the value of instrumentation, they explore how to foster a culture of curiosity and collaboration to make observability a core part of development.

In episode 74 of o11ycast, Liudmila Molkova unpacks the importance of semantic conventions in telemetry. The discussion highlights the challenge of agreeing on naming conventions, the role of working groups in maintaining these standards, and the potential for future integration with AI technologies. Liudmila underscores the importance of a shared, vendor-agnostic approach to telemetry, enabling smoother interoperability across platforms.

In episode 73 of o11ycast, Jessica Kerr, Martin Thwaites, and Austin Parker speak with Animesh Koratana, founder and CEO of PlayerZero, about the future of observability with AI. Discover how PlayerZero leverages LLMs to democratize institutional knowledge, improve software quality, and make problem-solving more accessible to all engineers.

Episode 72 of o11ycast explores the world of mobile observability with Hanson Ho, Android Architect at Embrace. Hanson unpacks how mobile's diverse device landscape impacts data collection and performance monitoring, the role of OpenTelemetry in simplifying these challenges, and the importance of tailoring observability strategies to real-world user experiences. Join hosts Charity, Jess, and Austin for this insightful examination of an often overlooked observability niche.

In Episode 71 of o11ycast, Jessica Kerr and Austin Parker sit down with Shir Chorev to delve into the nuances of incorporating Generative AI into applications and testing the outcomes. Shir shares her journey from a data scientist to founding Deepchecks, driven by her vision to enhance the quality and reliability of machine learning and AI in real-world scenarios. Gain valuable insights on how LLMs are reshaping the development landscape, making advanced projects more accessible, and strategies for mitigating adversarial use cases. Whether you're in DevOps, infrastructure, or just passionate about AI, this episode is packed with expert advice and cutting-edge information.

In episode 70 of o11ycast, Jess and Martin speak with Dan Gomez Blanco of Skyscanner. Dan shares his expertise on evangelizing better observability practices at Skyscanner and offers insights from his experience on the OpenTelemetry governance committee. Discover how observability can minimize organizational costs, the future of auto-instrumentation, and valuable advice on detecting and avoiding over-instrumentation. Plus, learn about the importance of tail sampling and OpenTelemetry’s semantic conventions.

In episode 69 of o11ycast, Jess and Martin speak with Charles Herring of WitFoo. Together they dive deep into the world of cybercrime investigations. Discover the intricate processes of data collection and analysis in the realm of cybersecurity, all while balancing the imperative of cost-effectiveness and citizen privacy.

In episode 68 of o11ycast, Jess and Martin speak with Michele Mancioppi of Dash0. This talk examines what it takes to make observability more accessible to non-experts in the space. Additional topics explored include the differences between monitoring and observability and Michele's shift from staff engineering to product management.

In episode 67 of o11ycast, Martin, Jess, and Liz dive deep on performance engineering. Pulling heavily from Liz’s extensive experience in the field, they share insights on utilizing data to balance resources like developers’ time and infrastructure costs. Listeners can expect to learn helpful lessons on infrastructure optimization, performance data accessibility, and why Honeycomb embraced Kubernetes.

In episode 66 of o11ycast, Jess and Martin speak with Iris Dyrmishi of Miro. They dive deep on what it takes to build an observability platform with open source tooling. Additionally, they explore the expense of outsourcing observability, the journey from logs to traces, tips for adopting new tools, and the spread of the FinOps movement.