Episode 36: AI and the Judgment Problem in Data Science

Dawn Woodard (Distinguished Engineer, LinkedIn), Andrés Bucchi (LATAM Airlines), and Jeremy Hermann (CEO & Co-Founder, Delphina) join High Signal for a deep dive into the shifting architecture of data science & analytics in the era of AI. As the industry moves from static dashboards to vibe coding and conversational querying, this panel of industry veterans explores why traditional data fundamentals—strict cataloging, verifiable outputs, and a single source of truth—are suddenly the most critical bottlenecks in the AI era. We dig into the sobering reality of the "source of truth" problem, where the speed of AI-generated code far outpaces our ability to define what "correct" actually looks like in a complex enterprise. The conversation reveals how AI is breaking legacy experimentation platforms, the transition of the data analyst into a "verifier" of AI-generated workflows, and why "headless" security architectures are essential for the next generation of autonomous agents. From the limitations of LLMs in causal reasoning to the challenges of integrating AI into "brownfield" enterprise codebases, this discussion provides a grounded framework for leaders navigating the gap between AI hype and operational reality. LINKS Dawn on LinkedIn Andrés on LinkedIn Jeremy on LinkedIn Build Your Bracket with Data in the Delphina Sandbox High Signal podcast Watch the podcast episode on YouTube Delphina's Newsletter

Martin Tingley, Head of Windows Experimentation at Microsoft and former Head of the Experimentation Platform Analysis Team at Netflix, talks about why humans are the bottleneck in experimentation, and how a five-level maturity framework points the way toward self-optimizing software. Our conversation traces the path from basic hypothesis testing to a frontier where Generative AI creates, evaluates, and refines product variants in a closed loop. We explore the architectural shift required to move from testing single variants to optimizing entire parameter spaces, and how startups are already using AI to generate production-ready landing pages for Fortune 500 companies in hours rather than weeks. Tingley also shares a strategic lens on "experimentation programs," explaining how plotting the distribution of treatment effects across different product areas can serve as a powerful tool for capital allocation and high-level strategy. LINKS Martin on LinkedIn Want Your Company to Get Better at Experimentation? by Iavor Bojinov, David Holtz, Ramesh Johari, Sven Schmit and Martin Tingley (Harvard Business Review) Avoid the Pitfalls of A/B Testing by Iavor Bojinov, Guillaume Saint-Jacques and Martin Tingley (Harvard Business Review) Martin & Co.'s Seven Part Blog Series on Experimentation at Netflix Roberto Medri (Meta) on High Signal: The Incentive Problem in Shipping AI Products — and How to Change It Tim O’Reilly on High Signal: The End of Programming As We Know It Watch the podcast episode on YouTube Delphina's Newsletter

Bozena Pajak, VP of Learning at Duolingo, joins High Signal to discuss the evolution of AI at Duolingo: from personalized difficulty models to the current generative frontier where AI characters provide low-stakes and high impact conversational practice. We discuss the role of AI in overcoming one of the biggest hurdles in language acquisition, speaking anxiety. We also talk about how Bozena's team leverages agentic workflows to scale content and why the next wave of personalization involves shifting from difficulty levels to "thematic lenses" tailored to specific user interests. LINKS Bozena on LinkedIn The original AI: how your brain tracks language patterns, a Duolingo blog post How Duolingo uses AI to create lessons faster, a Duolingo blog post Duolingo is hiring a Learning Scientist (Efficacy Research), a Director of Learning Design (Language Learning), and a Director of Learning Design (Immersive Language Learning) High Signal podcast Watch the podcast episode on YouTube Delphina's Newsletter

Benn Stancil, writer and co-founder of Mode, joins High Signal to ask some uncomfortable questions about the current AI moment. Is now actually a terrible time to start a company? If the tools you build on today are obsolete in six months, at what point does the head start stop mattering? Is all that context engineering you're doing a waste of time, destined to go the way of Boolean search syntax in the 90s? Benn argues that AI is turning us all into Steve Jobs, not the visionary who delegated, but the one who berated people over pixel placement. As AI takes over the doing, our job becomes obsessing over the polish. He makes the case that technical debt may be self-healing: if future models can untangle the mess today's models made, then messy code isn't debt…it's a spec for a clean rewrite. We also dig into why Claude Cowork can't work. AI has these uncanny ticks you can't beat out, so anything it writes "as you" will smell like AI. The solution isn't better AI writing—it's to stop pretending we write to each other at all. Benn envisions a future where communication is radically intermediated: I dump facts into a shared repository, your AI reads them, and nobody bothers with the social decoration in between. LINKS Benn’s blog on Substack Benn.website, with links to all everything else Benn related Will there ever be a worse time to start a startup? Today's frontier is tomorrow's tech debt. Why Cowork can’t work: The future isn’t collaborative. Producer theory: Platforms are overrated. Tim O’Reilly on High Signal: The End of Programming As We Know It Watch the podcast episode on YouTube Delphina's Newsletter

Nicholas Moy, former Head of Research at Windsurf & now at Google DeepMind, joins High Signal to discuss the shift from "co-driving" to a truly "agentic" era of development. We discuss Windsurf's journey from early prototypes that struggled with compounding errors to the successful launch of their agentic coding product. Nick explains that building a startup in the current climate requires a strategy of "disrupting yourself" to avoid the innovator’s dilemma; companies must be ready to pivot as soon as a new frontier model makes previously impossible features viable. He argues that traditional technical moats are increasingly fragile, and true defensibility now comes from real-world usage data, brand reputation, and a deep intuition for what users need at the frontier of these capabilities. LINKS Nicholas Moy on LinkedIn Introducing Google Antigravity, a New Era in AI-Assisted Software Development “A Flash of Deflation - Gemini 3 Flash represents a step function increase in model deflation : a gauntlet thrown” by Thomas Tunguz Tomasz Tunguz on Why a Trillion Dollars of Market Cap Is Up for Grabs (and How AI Teams Will Win It) High Signal podcast Watch the podcast episode on YouTube Delphina's Newsletter

Cara Dailey, VP and Head of Data Strategy at Early Warning (the parent company of Zelle), joins High Signal to discuss the evolution of high-stakes data leadership and governance. From her early work in online advertising at DoubleClick to shaping data strategy at Nike and holding Chief Data Officer roles at Bank of the West and T. Rowe Price, Cara has seen every iteration of the data leader’s role. Now, she’s navigating her 'product era'—shaping the data strategy for Early Warning's Decisions Intelligence business, where she leverages rich financial data and data science to drive fraud monitoring and modeling. In this episode, Cara shares her pragmatic 'progress over perfection' approach to governance, why she’s abandoning monolithic platforms in favor of incremental data products, and her 80/20 rule for balancing operational rigor with innovation. We also discuss why 'loving' data isn't enough—you have to actually 'take care' of it—and why AI is finally shining a spotlight on the often-neglected fundamentals of data stewardship and conversational BI. LINKS Cara Dailey on LinkedIn Why AI Adoption Fails: A Behavioral Framework for AI Implementation, A High Signal Conversation with Lis Costa (Chief of Innovation, Behavioural Insights Team) Watch the podcast episode on YouTube High Signal podcast Delphina's Newsletter

Chris Child, VP of Product, Data Engineering at Snowflake, joins High Signal to deliver a new playbook for data leaders based on his recent MIT report, revealing why AI is paradoxically creating more work for data teams, not less. He explains how the function is undergoing a forced evolution from back-office “plumbing” to the strategic core of the enterprise, determining whether AI initiatives succeed or fail. The conversation maps the new skills and organizational structures required to navigate this shift. We dig into why off-the-shelf LLMs consistently fail to generate useful SQL without a semantic layer to provide business context, and how the most effective data engineers must now operate like product managers to solve business problems. Chris provides a clear framework on the shift from writing code to managing a portfolio of AI agents, why solving for AI risk is an extension of existing data governance, and the counterintuitive strategy of moving slowly on foundations to unlock rapid, production-grade deployment. LINKS MIT Technology Review Report: Redefining Data Engineering in the Age of AI The Evolution of the Modern Data Engineer: From Coders to Architects Why Most AI Agents Fail (and What It Takes to Reach Production) with Anu Brahadwaj (Atlassian) The End of Programming As We Know It with Tim O'Reilly The Incentive Problem in Shipping AI Products — and How to Change It with Roberto Medri (Meta) Andrej Karpathy — AGI is still a decade away Chris Child on LinkedIn High Signal podcast Watch the podcast episode on YouTube Delphina's Newsletter

Liz Costa of the Behavioral Insights Team returns to High Signal to deliver a critical behavioral science playbook for the AI era focused on human and business impact. We discuss why the potential of AI can only be fulfilled by understanding a single bottleneck: human behavior. The conversation reveals why leaders must intervene now to prevent temporary adoption patterns from calcifying into permanent organizational norms, the QWERTY Effect, and how to move organizations past simply automating drudgery to achieving deep integration. We dig into why AI adoption is fundamentally a behavioral challenge, providing a diagnostic framework for leaders to identify stalled progress using the Motivation-Capability-Trust triad. Liz explains how to reframe AI deployment by leveraging Loss Aversion to bypass employee skepticism, and how to design workflows that improve human reasoning rather than replace it. The conversation provides clear guidance on intentional task offloading, the power of using AI to stress-test decisions, and why sanctioning employee experimentation is essential to discovering high-value use cases. LINKS AI & Human Behaviour: Augment, Adopt, Align, Adapt Thinking Fast and Slow in AI How does LLM use affect decision-making? Defaults, Decisions, and Dynamic Systems: Behavioral Science Meets AI with Lis Costa (High Signal) The Behavioral Insights Team Lis Costa on LinkedIn High Signal podcast Watch the podcast episode on YouTube Delphina's Newsletter

Lance Martin of LangChain joins High Signal to outline a new playbook for engineering in the AI era, where the ground is constantly shifting under the feet of builders. He explains how the exponential improvement of foundation models is forcing a complete rethink of how software is built, revealing why top products from Claude Code to Manus are in a constant state of re-architecture simply to keep up. We dig into why the old rules of ML engineering no longer apply, and how Rich Sutton's "bitter lesson" dictates that simple, adaptable systems are the only ones that will survive. The conversation provides a clear framework for leaders on the critical new disciplines of context engineering to manage cost and reliability, the architectural power of the "agent harness" to expand capabilities without adding complexity, and why the most effective evaluation of these new systems is shifting away from static benchmarks and towards a dynamic model of in-app user feedback. LINKS Lance on LinkedIn Context Engineering for Agents by Lance Martin Learning the Bitter Lesson by Lance Martin Context Engineering in Manus by Lance Martin Context Rot: How Increasing Input Tokens Impacts LLM Performance by Chroma Building effective agents by Erik Schluntz and Barry Zhang at Anthropic Effective context engineering for AI agents by Anthropic How we built our multi-agent research system by Anthropic Measuring AI Ability to Complete Long Tasks by METR Your AI Product Needs Evals by Hamel Husain Introducing Roast: Structured AI workflows made easy by Shopify Watch the podcast episode on YouTube Delphina's Newsletter

Paras Doshi (Head of Data, Opendoor; former data leader at Amazon) joins High Signal to unpack the playbook for building an indispensable data function. He shares his experience tackling the classic scaling challenge of fragmented data at Opendoor, where rapid growth led to inconsistent metrics across the business, and turning the data function into a centralized strategic asset. We dive deep into how to earn a true seat at the table, why he believes AI is creating the "100x individual contributor," and how the principles of agency, autonomy, and adaptability are the new essentials for data careers. The conversation also explores the pragmatic divide between batch and real-time ML, how to identify a truly data-led company, and why leaders must shield their top talent to unlock disproportionate impact. LINKS Paras Doshi on LinkedIn Insight Extractor, Paras' blog on analytics, data science, and business intelligence Watch the conversation on YouTube Delphina's Newsletter