The Prediction Crisis: Your Model Fits, but Does It Predict?

A model that fits your data isn't the same as a model that predicts accurately. In this episode, Dr. Philip Ramsey and Dr. Tiffany Rau make the case that science and engineering have a prediction crisis, and talk through what it takes to build models that hold up at conditions you haven't run yet.In this episodeWhat "prediction" actually means: how a model performs at settings that weren't used to train itWhy complex systems break simple models, and what a nanoscale map of human brain tissue says about complexityLessons from hurricane forecasting, and why Tiffany took a programmer into the labPredictive models as learning tools, not just deliverablesProcess characterization, scale-down models, and thought experiments in drug developmentWhy the "optimum" in your design space may be somewhere you'd never want to operateA step-by-step workflow: quality data, space-filling designs, measurement system checks, machine learning, validation, and studying the modelWhat to do tomorrow: scope your first model small enough to finish, and learn from people who do this work every dayFurther readingShapson-Coe A, et al. "A petavoxel fragment of human cerebral cortex reconstructed at nanoscale resolution." Science 384, eadk4858 (2024). https://doi.org/10.1126/science.adk4858 — the Harvard–Google study Phil mentions: about 57,000 cells and 150 million synapses in roughly one cubic millimeter of cortex, 1.4 petabytes of data.HostsNathaniel Leies (host)Philip Ramsey, Ph.D. — Emeritus Professor of Statistics, University of New Hampshire; VP of Data Science and Analytics, Predictum Inc.Tiffany Rau, Ph.D. — Owner, Rau Consulting, global biotechnology consultant specializing in R&D, process development, and the commercialization of biopharmaceutical manufacturing processes--Got a question? Want to hear your voice on a future episode? 📬 Write to us at [email protected] We read everything, and we'd love to feature listener questions on the show.⭐ If this episode was useful, please subscribe, rate the show, and share it with someone who'd get value from it.🌐 Visit us on https://beyondthebatch.io to learn more.--Produced by Predictum Inc., in partnership with Rau Consulting.Questions, feedback, or a topic you'd like us to cover? Write to [email protected]. All platforms: beyondthebatch.io

Many projects fail or hit roadblocks. But the moment a project "fails"  usually comes too late, rather than much earlier when someone on the team knew failure to be a likely outcome.Phil Ramsey and Tiffany Rau dig into why struggling R&D and development projects keep getting funded, how to tell when to stop or regroup, and why gap analyses save years of work.In this episode:Why a program that delivered zero commercial successes still got its leader promoted, and what that says about cultureDevelopment reasons to stop or regroup: the wrong expression system, a CDMO platform that doesn't fit the molecule, or a pipeline that has moved onWhy stopping development isn't the same as failureWhat success means at each stage, from the target product profile and QTPP to process variability you can't controlThree kinds of development projects (variant, derivative, breakthrough), and why breakthroughs need stopping criteria on day oneTreating data as inventory: findable, consistently named, and documented with its contextWhat tighter venture funding forces small teams to do, and when an aggressive timeline becomes a silly oneThe gap analysis that would have caught a product the plant could never makeChapters:00:00 Intro00:23 Projects that should have been stopped03:39 Development reasons to stop or regroup05:17 Regrouping isn't failure06:44 What success means from discovery to scale-up09:49 Three kinds of development projects12:31 Keeping data you can find and trust15:52 Designed experiments and prediction20:20 Funding pressure on small teams23:29 What to do this week if your project is in trouble26:09 Gap analysis and the chemistry nobody could makeHosts:Nathaniel Leies, hostPhilip Ramsey, Ph.D., Emeritus Professor of Statistics, University of New Hampshire; Vice President of Data Science and Analytics, Predictum Inc.Tiffany Rau, Ph.D., Owner, Rau Consulting; global biotech and pharmaceutical consultant across the drug development lifecycleMentioned: Beyond the Batch Ep. 2 on uncertaintyCoBaseKRM, an application for capturing knowledge from scientific and engineering modeling, produced by Predictum: https://cobasekrm.comContact: Send questions or a voice recording to [email protected]. More at beyondthebatch.io.Produced by Predictum Inc., in partnership with Rau Consulting.

Every generation of scientists has faced complexity, but are today's kinetic systems (cell and gene therapies, advanced materials, complex manufacturing processes) are structurally harder to understand than what came before? In this episode, hosts, Dr. Tiffany Rau, Dr. Philip Ramsey, and Nathaniel Leies explain why classical statistical methods often aren't built for the job, why "boundary designs" miss where the real action happens, and why uncertainty (not just a prediction itself) has to be part of every decision.They also get into designing for manufacturability, how regulators actually respond to quantified risk, and where machine learning methods designed for applied science and engineering problem solving actually helps.This is Part 2 of our 2-part series on the most dangerous mistakes in CMC.--Got a question? Want to hear your voice on a future episode? 📬 Write to us at [email protected]. 📬 We read everything, and we'd love to feature listener questions on the show.If this episode was useful, please subscribe, rate the show, and share it with someone who'd get value from it.🌐 Visit us on beyondthebatch.io to learn more.--Beyond the Batch is Predictum's podcast on CMC, data science, and the real-world problems facing R&D, quality, and manufacturing teams. Hosted by Nathaniel Leies (Predictum), with Dr. Tiffany Rau (CMC consultant) and Dr. Philip Ramsey (Professor Emeritus, University of New Hampshire; VP of Data & Analytics, Predictum). Stock Media provided by Musinova / Pond5#CMC #Biotech #Podcast #PharmaManufacturing #LifeSciences

The most expensive sentence in biotech might be "I'll worry about that in Phase 3."In the first episode of Beyond the Batch, host Nathaniel Leies sits down with cohosts, Dr. Tiffany Rau and Dr. Philip Ramsey, to dig into the biggest failure modes in CMC (Chemistry, Manufacturing, and Controls) and why so many of them trace back to teams underestimating the journey ahead of them.Tiffany, a CMC consultant who's built out facilities and led regulatory strategy across vaccines, monoclonal antibodies, and cell and gene therapy, explains why teams so often misjudge the sheer number of interlocking decisions between preclinical work and commercialization. Phil, a statistician and longtime R&D consultant connects that same failure to a lack of technical continuity: fragmented teams, knowledge that walks out the door when someone changes roles, and no single person accountable for the project from day one.In this episode:What CMC actually means, and why it's the hub everything runs throughThe most dangerous mistake CMC practitioners make, and why it compounds for yearsPractical concepts and tools covered:Starting with the end in mind makes projects faster and more successfulSystems engineers ("integrators") as critical (and often-times forgotten) contributors for cross-functional successPareto frontier decision-makingWhy respecting what your cross-functional colleagues know is more than good mannersGot a question, or want to hear your voice on a future episode?Write to us at [email protected]. We read everything, and we'd love to feature listener questions on the show.If this episode was useful, the best way to help it find more people like you is to subscribe, rate the show, and share it with someone who'd get value from it.Visit us at https://beyondthebatch.io to learn more.—Beyond the Batch is Predictum's podcast on CMC, data science, and the real-world problems facing R&D, quality, and manufacturing teams. Hosted by Nathaniel Leies (Predictum), with Dr. Tiffany Rau (CMC consultant) and Dr. Philip Ramsey (Professor Emeritus, University of New Hampshire; VP of Data & Analytics, Predictum).#CMC #Biotech #Podcast #PharmaManufacturing #LifeSciences