Episode 15: Why Good Metrics Still Lead to Bad Decisions — and How to Fix It

Lis Costa, Chief of Innovation and Partnerships at the Behavioural Insights Team, joins High Signal to explore how behavioral science is reshaping public policy, digital platforms, and machine learning. She explains how defaults influence behavior at scale, why personalization and chatbots are unlocking new kinds of interventions, and what happens when AI systems meet real-world complexity. We also discuss the limits of nudging, the promise of boosting, and why building for human decision-making requires more than just good models. LINKS The Behavioral Insights Team (https://www.bi.team/) Lis Costa on LinkedIn (https://uk.linkedin.com/in/elisabeth-costa-6a5b35248) High Signal podcast (https://high-signal.delphina.ai/) Delphina's Newsletter (https://delphinaai.substack.com/)

Sudarshan Seshadri—VP of AI, Data Science, and Foundations Engineering at Alto Pharmacy—joins us to explore what it takes to build high-stakes AI systems that people can actually trust. He shares lessons from deploying machine learning and LLMs in healthcare, where speed, safety, and uncertainty must be carefully balanced. We talk about designing AI to support pharmacist judgment, the shift from bottlenecks to decision backbones, and why great data leaders are really architects of how irreversible decisions get made. LINKS Suddu on LinkedIn (https://www.linkedin.com/in/ss01/) Careers at Alto Pharmacy (https://www.alto.com/careers) High Signal podcast (https://high-signal.delphina.ai/) Delphina's Newsletter (https://delphinaai.substack.com/)

Roberto Medri, VP of Data Science at Instagram, explains why most experiments fail, how misaligned incentives warp product development, and what it takes to drive real impact with data science. He shares what teams get wrong about launches, why ego gets in the way of learning, and how Instagram turned Reels from a struggling product into a global success. A candid look at product, data, and decision-making inside one of the world’s most influential platforms. LINKS Roberto on LinkedIn (https://www.linkedin.com/in/robertomedri/) High Signal podcast (https://high-signal.delphina.ai/) Delphina's Newsletter (https://delphinaai.substack.com/)

Fei-Fei Li—co-director of Stanford’s Human-Centered AI Institute and one of the most respected voices in the field—reflects on AI’s evolution from the early days of ImageNet to the rise of foundation models. She explains why spatial intelligence may be the next major shift, how human-centered design applies in practice, and why AI should be understood as a civilizational technology—one that shapes individuals, communities, and society at large. LINKS Stanford HAI (https://hai.stanford.edu/) World Labs (https://www.worldlabs.ai/about) "The World I See", Fei-Fei's book (a must read!) (https://us.macmillan.com/books/9781250897930/theworldsisee/) Fei-Fei on X (https://x.com/drfeifei) Fei-Fei on LinkedIn (https://www.linkedin.com/in/fei-fei-li-4541247/) High Signal podcast (https://high-signal.delphina.ai/) Delphina's Newsletter (https://delphinaai.substack.com/)

Eoin O'Mahony—data science partner at Lightspeed, former Uber science lead, and one of the early architects of the system that kept NYC’s Citi Bikes available across the city—argues that positive metrics are meaningless if you don’t understand the mechanism behind them. At Uber, he was careful to make sure his launches both looked good on paper and made sense in practice. Now in venture, he’s applying that same rigor to unstructured data—using GenAI to scale a kind of work that’s long resisted systematization. LINKS Eoin's page at Lightspeed Ventures (https://lsvp.com/team-member/eoin-omahony/) Ramesh Johari on How to Build an Experimentation Machine and Where Most Go Wrong (https://high-signal.delphina.ai/episode/ramesh-johari-on-how-to-build-an-experimentation-machine-and-where-most-go-wrong) Chiara Farronato on Data Science Meets Management: Teamwork, Experimentation, and Decision-Making (https://high-signal.delphina.ai/episode/data-science-meets-management) Delphina's Newsletter (https://delphinaai.substack.com/)

Barr Moses—co-founder and CEO of Monte Carlo—thinks we’re headed for an AI reckoning. Companies are building fast, but most are still managing data like it’s 2015. In this episode, she shares high-stakes failure stories (like a $100M schema change), explains why full-stack observability is becoming essential, and breaks down how LLM agents are already transforming data debugging. From culture to tooling, this is a sharp look at what real AI readiness requires—and why so few teams have it. LINKS 2024 State of Reliable AI Survey – Monte Carlo (https://www.montecarlodata.com/blog-2024-state-of-reliable-ai-survey/) Delphina's Newsletter (https://delphinaai.substack.com/) Unity’s $100M Data Error – Schema Change Gone Wrong (https://www.theregister.com/2021/11/11/unity_stock_plunge/) Citibank’s $400M Fine for Risk Management Failures (https://www.reuters.com/article/us-citigroup-fine-idUSKBN26T0BK) Google’s AI Recommends Adding Glue to Pizza (https://www.theverge.com/2024/5/23/24162896/google-ai-overview-hallucinations-glue-in-pizza) Chevy Dealer’s AI Chatbot Agrees to Sell Tahoe for $1 (https://incidentdatabase.ai/cite/622/) The AI Hierarchy of Needs by Monica Rogati (HackerNoon) (https://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007) Data Quality Fundamentals by Barr Moses, Lior Gavish, and Molly Vorwerck (O’Reilly) (https://www.oreilly.com/library/view/data-quality-fundamentals/9781098112035/) Delphina's Newsletter (https://delphinaai.substack.com/)

Tim O’Reilly—founder of O’Reilly Media and one of the most influential voices in tech—argues we’re not witnessing the end of programming, but the beginning of something far bigger. He draws on past computing revolutions to explore how AI is reshaping what it means to build software, why real breakthroughs come from the edge—not incumbents—and what it takes to learn, teach, and build responsibly in the age of AI. LINKS The End of Programming as We Know It by Tim <--- Read this! (https://www.oreilly.com/radar/the-end-of-programming-as-we-know-it/) WTF? What’s the Future and Why It’s Up to Us (https://www.oreilly.com/tim/wtf-book.html) The fundamental problem with Silicon Valley’s favorite growth strategy (https://qz.com/1540608/the-problem-with-silicon-valleys-obsession-with-blitzscaling-growth) AI Engineering by Chip Huyen (https://www.oreilly.com/library/view/ai-engineering/9781098166298/) Delphina's Newsletter (https://delphinaai.substack.com/)

Stefan Wager—Professor at Stanford and expert on causal machine learning—has worked with leading tech companies including Dropbox, Facebook, Google, and Uber. He challenges the widespread assumption that better predictions mean better decisions. Traditional machine learning excels at prediction, but is prediction really what your business needs? Stefan explores why predictive models alone often fail to answer critical “what-if” questions, how causal machine learning bridges this gap, and provides practical advice for how you can start applying causal ML at work. LINKS Stefan's Stanford Website (https://www.gsb.stanford.edu/faculty-research/faculty/stefan-wager) Machine Learning and Economics, Stefan and Susan Athey's lectures for the Stanford Graduate School of Business (https://www.youtube.com/@stanfordgsb) Causal Inference: A Statistical Learning Approach (WIP!) (https://web.stanford.edu/~swager/causal_inf_book.pdf) Mastering ‘Metrics: The Path from Cause to Effect by Angrist & Pischke (https://www.masteringmetrics.com/) The Book of Why: The New Science of Cause and Effect by Judea Pearl and Dana Mackenzie (https://en.wikipedia.org/wiki/The_Book_of_Why) Causal Inference: The Mixtape by Scott Cunningham (https://mixtape.scunning.com/) A Technical Primer On Causality by Adam Kelleher (https://medium.com/@akelleh/a-technical-primer-on-causality-181db2575e41) What Is Causal Inference? An Introduction for Data Scientists by Hugo Bowne-Anderson and Mike Loukides (https://www.oreilly.com/radar/what-is-causal-inference/) The Episode on YouTube (https://www.youtube.com/watch?v=f9_Lt5p8avU&feature=youtu.be) Delphina's Newsletter (https://delphinaai.substack.com/)

Peter Wang—Chief AI Officer at Anaconda and a driving force behind PyData—challenges conventional thinking about AI’s role in software development. As AI reshapes engineering, are we moving beyond writing code to orchestrating intelligence? Peter explores why companies are fixated on models instead of integration, how AI is breaking traditional software workflows, and what this shift means for open source. He also shares insights on the evolving role of engineers, the commoditization of AI models, and the deeper questions we should be asking about the future of software. LINKS Peter Wang on LinkedIn (https://www.linkedin.com/in/pzwang/) Anaconda (https://www.anaconda.com/) Mistral Saba (https://mistral.ai/news/mistral-saba) Peter chatting with Hugo several years ago about the beginnings of PyData, NUMFOCUS, and Python for Data Science (https://vanishinggradients.fireside.fm/7) Delphina's Newsletter (https://delphinaai.substack.com/)

Ari Kaplan—Global Head of Evangelism at Databricks and a pioneer in sports analytics—explains why businesses fixated on AI often overlook the real advantage: making better decisions with their own data. He shares lessons from his work building analytics teams for Major League Baseball, advising McLaren’s F1 strategy, and helping companies apply AI where it actually works—without falling into hype-driven traps. SHOW NOTES Ari on LinkedIn (https://www.linkedin.com/in/arikaplan/) The Data Intelligence Platform For Dummies by Ari and Stephanie Diamond (https://www.databricks.com/resources/ebook/maximize-your-organizations-potential-data-and-ai) Databricks' AI/BI: Intelligent analytics for real-world data (https://www.databricks.com/product/ai-bi) That time Ari spoke with Travis Kelce  about how Travis and the Kansas City Chiefs use data and analytics! (https://www.linkedin.com/posts/arikaplan_wiley-databricks-genai-activity-7221214362575724545-RZwc/)