Episode 4: How to Build an Experimentation Machine and Where Most Go Wrong

In this episode of High Signal, Chris Wiggins—Chief Data Scientist at The New York Times, Professor at Columbia University, and co-author of How Data Happened—shares how organizations can move beyond prediction to actionable decision systems. Drawing on his work at The New York Times and in academia, Chris explains how to scale data teams, optimize systems, and align data science with organizational impact. Key topics from the conversation include: • From Prediction to Prescription: Why organizations need to focus on interventions that drive outcomes, illustrated with insights like, “Imagine a hospital prescribing treatments instead of just diagnosing conditions.” • The AI Hierarchy of Needs: Foundational practices, such as data logging and engineering, that enable advanced machine learning and AI. • Personalization and Optimization: How reinforcement learning and exploration-exploitation methods help optimize KPIs and adapt to user context. • Scaling Data Teams: Strategies for attracting and retaining talent by emphasizing autonomy, mastery, and purpose. • Empathy as a Data Science Skill: The importance of collaborating with other teams and understanding their goals to drive adoption and success. 🎧 Tune in to learn how to build decision systems, integrate causality into workflows, and develop scalable data science teams for real-world impact. You can find more on our website: https://high-signal.delphina.ai/ LINKS - Chris Wiggins' Website (https://datascience.columbia.edu/people/chris-h-wiggins/) - Chris Wiggins on LinkedIn (https://www.linkedin.com/in/wiggins/) - How Data Happened: A History from the Age of Reason to the Age of Algorithms (https://en.wikipedia.org/wiki/How_Data_Happened) - The AI Hierarchy of Needs by Monica Rogati (https://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007) - The Book of Why by Judea Pearl (https://en.wikipedia.org/wiki/The_Book_of_Why)

In this episode of High Signal, Hilary Mason—renowned data scientist, entrepreneur, and co-founder of Hidden Door—shares her unique insights into the evolving world of data science and generative AI. Drawing from her pioneering work at Fast Forward Labs, Bitly, and Hidden Door, Hilary explores how creativity, judgment, and empathy are reshaping the data landscape. Highlights from the discussion include: - Judgment as a Competitive Edge: Hilary emphasizes the enduring importance of human judgment in framing problems and evaluating AI outputs. - The Future of Generative AI: She discusses its transformative potential while cautioning against over-reliance on prompts, advocating for systems rooted in rich context. - Building for Creativity with Hidden Door: Hilary shares how her company turns generative AI’s liabilities into assets, creating immersive, bias-aware storytelling experiences. - The Shifting Role of Data Science Careers: With automation redefining entry-level roles, Hilary outlines how data professionals can focus on transferable skills to stay ahead. - Navigating AI Strategy in Leadership: She offers pragmatic advice on balancing the hype of AI with practical business impact, aligning leadership expectations with achievable goals. The conversation concludes with Hilary’s optimistic take on how the data science community can continue to thrive by embracing creativity, empathy, and interdisciplinary collaboration. 🎧 Tune in to gain practical insights into building robust AI systems, navigating career shifts, and leveraging generative AI for meaningful innovation. You can find more on our website: https://high-signal.delphina.ai/ LINKS Hilary Mason on LinkedIn (https://www.linkedin.com/in/hilarymason/) Hidden Door (https://www.hiddendoor.co/) Fast Forward Labs Reports (https://blog.fastforwardlabs.com/reports) Of Oaths and Checklists By DJ Patil, Hilary Mason and Mike Loukides (https://www.oreilly.com/radar/of-oaths-and-checklists/)

In this episode of High Signal, Gabriel Weintraub (the Amman Professor of Operations, Information, and Technology at Stanford Graduate School of Business), brings his expertise in market design, data science, and operations, enriched by his experience with global platforms like Uber and Mercado Libre, to a conversation that spans practical strategies, cultural insights, and global perspectives on data and AI. Highlights from the discussion include: - Bridging the C-Level and Technical Divide: Gabriel emphasizes the importance of aligning leadership with on-the-ground teams to build effective, data-driven organizations. - Starting with the Basics: From building pipelines to identifying high-ROI projects, Gabriel outlines foundational steps for companies adopting data science and AI. - Cultural Transformation for Experimentation: He explains why fostering an experimentation culture, where negative results are valued for learning, is essential for success. - Opportunities in Latin America: Gabriel shares insights on the unique challenges and immense potential of the Latin American tech ecosystem, including the critical role of startups and the need for local innovation systems. - Generative AI’s Role in Driving Impact: Discussing generative AI’s transformative potential, Gabriel highlights its capacity to lower barriers for smaller teams while emphasizing the importance of problem-first approaches. The conversation concludes with a forward-looking exploration of opportunities in government, education, and healthcare, and Gabriel’s optimism about building ecosystems where startups and local talent thrive. 🎧 Tune in to learn from Gabriel’s thoughtful perspectives on navigating the complexities of building data-driven cultures, the global AI landscape, and how to leverage data for impactful change. You can find more on our website: https://high-signal.delphina.ai/

Ramesh Johari (Stanford, Uber, Airbnb, and more) explores the art and science of online experimentation, especially in the context of marketplaces and tech companies. Ramesh shares insights on how organizations evolve from basic experimentation practices to becoming fast, adaptive, and self learning organizations. We dive into challenges like the risk aversion trap, the importance of learning from negative results, and how generative AI is reshaping the experimentation landscape. We also talk about common failure modes and the types of things you're probably doing wrong, along with strategies to avoid these pitfalls. Plus, we discussed the role of incentives, the necessity of data driven decision making, and what it means to experiment in high stakes environments.

Chiara Farronato (Harvard Business School) discusses how digital platforms like Airbnb and Uber have transformed industries. She explores the challenges of fostering collaboration between managers and data scientists, bridging communication gaps, and building data-driven cultures. Chiara also delves into the complexities of managing peer-to-peer marketplaces and the evolving role of data in decision-making. This episode offers key insights for business leaders working with technical teams and navigating platform-based innovation.

Hugo Bowne-Anderson welcomes Andrew Gelman, professor at Columbia University, to discuss the practical side of statistics and data science. They explore the importance of high-quality data, computational skills, and using simulation to avoid misleading results. Andrew dives into real-world applications like election predictions and highlights causal inference’s critical role in decision-making. This episode offers insights into balancing statistical theory with applied data analysis, making it a must-listen for both data practitioners and those interested in how statistics shapes our world.

Michael Jordan (UC Berkeley) on the future of machine learning as it extends to a planetary scale in "The Next Evolution of AI: Markets, Uncertainty, and Engineering Intelligence at Scale." In this episode, Mike speaks with Hugo about the evolution of AI, the importance of integrating machine learning, computer science, and economics, and how AI can scale to address planetary-level challenges.