Statistical Sleuthing: Can you spot a problem?

Does cheating make you more creative? A provocative study from Harvard Business School suggested that it does. But data detectives uncovered suspicious patterns in the results, and a Harvard investigation found even more troubling changes in the data. In Part 2 of our investigation into the Francesca Gino research misconduct case, we dig into the now-retracted “Evil Genius” study and the suspicious changes that made its results look better, then turn to Harvard law professor Lawrence Lessig's defense of Gino: Could innocent mistakes by “an army of research assistants” explain what happened? Along the way, we explore the trap of circular reasoning in statistics, why good data management matters, and how much responsibility researchers bear for their data even when it's out of their hands.Statistical topicsStatistical sleuthingData forensicsData integrityData cleaningData managementReproducibilityStatistical significanceSpreadsheetsPredictor variablesOutcome variablesCircular reasoningResearch misconductMethodologic Morals“You cannot define your predictor variable using your outcome variable.”“If it takes an army of research assistants to clean your data, you're probably doing something wrong and maybe you need a statistician.”“Being too smart to commit stupid fraud is not a defense. Smart people do stupid things all the time.”“The more hoops your explanation has to jump through, the less convincing it becomes.”ReferencesGino F, Wiltermuth SS. Evil genius? How dishonesty can lead to greater creativity. Psychol Sci. 2014;25(4):973-981. Retracted. doi:10.1177/0956797614520714Retraction notice: Evil genius? How dishonesty can lead to greater creativity. Psychol Sci. 2023;34(8):947. doi:10.1177/09567976231187595Simonsohn U, Simmons JP, Nelson LD. Data falsificada (part 3): “The cheaters are out of order.” Data Colada. June 23, 2023. https://datacolada.org/111Simonsohn U, Simmons JP, Nelson LD. Exhibits 3, 4, and 5. Data Colada. September 16, 2023. https://datacolada.org/114Simmons JP. Harvard's Gino report reveals how a dataset was altered. Data Colada. July 9, 2024. https://datacolada.org/118Lessig L. The Gino Case. Website and podcast series presenting arguments challenging Harvard's research misconduct findings. https://theginocase.info/Simonsohn U, Simmons JP, Nelson LD. Data Falsificada. ResearchBox. Research data, analysis code, and supporting materials accompanying the Data Colada investigations. https://researchbox.org/1630.34Harvard Business School. Investigative report concerning allegations of research misconduct involving Francesca Gino. Harvard Business School Research Integrity Investigation Report. 2023. Accessed October 18, 2026. https://datacolada.org/wp-content/uploads/Harvard-Report-on-Gino.pdfKristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding  Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program  Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Program that Kristin teaches in::Epidemiology and Clinical Research Graduate Certificate Program Kristin’s manuscript writing course:https://instats.org/seminar/from-data-to-discussion-writing-a-scientFind us on:Kristin -  LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - Intro (06:50) - The Claim: Superstars and Fraud (08:09) - Lessig’s Army of Research Assistants (22:15) - Sorted Data and 13 Suspicious Values (27:32) - Harvard’s Recovered Files from Gino’s Laptop (30:30) - Twelve People Reclassified as Cheaters (34:01) - Four RAT Totals That Don’t Add Up (45:25) - Lessig’s Research Assistant Theory (49:03) - Circular Logic and the Googling Defense (01:04:39) - Where the Evidence Leaves Us (01:07:18) - Rating the Claim (01:09:38) - Methodological Morals

A famous Harvard researcher. A blockbuster finding about honesty. Suspicious data, failed replications, and a $25 million lawsuit. Welcome to one of the strangest research misconduct scandals in recent memory. In the first of two episodes about the Francesca Gino case, we investigate a famous experiment claiming that signing an honesty pledge at the top of a form makes people less likely to cheat. We follow the trail from a failed replication to some extraordinary detective work involving an Excel spreadsheet—and conduct a little statistical sleuthing of our own. Along the way, we discover how suspicious changes to a handful of observations can alter the story a scientific study tells.Statistical topicsStatistical sleuthingData integrityData managementReplicationSensitivity analysisEffect sizesStatistical significanceReproducibilitySpreadsheetsConceptual replicationDirect replicationCalcChainMethodologic Morals“There's cleaning your data, and then there's laundering your data.” “Good data record keeping is still sexy. Version control, baby.”ReferencesShu LL, Mazar N, Gino F, Ariely D, Bazerman MH. Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end. Proc Natl Acad Sci U S A. 2012;109(38):15197-15200. Retracted. doi:https://doi.org/10.1073/pnas.1209746109Kristal AS, Whillans AV, Bazerman MH, Gino F, Shu LL, Mazar N, Ariely D. Signing at the beginning versus at the end does not decrease dishonesty. Proc Natl Acad Sci U S A. 2020;117(13):7103-7107. doi:https://doi.org/10.1073/pnas.1911695117Retraction for Shu et al., Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end. Proc Natl Acad Sci U S A. 2021;118(38):e2115397118. doi:https://doi.org/10.1073/pnas.2115397118Data Colada investigationsSimonsohn U, Simmons JP, Nelson LD. Evidence of fraud in an influential field experiment about dishonesty. Data Colada. August 17, 2021. https://datacolada.org/98Simonsohn U, Simmons JP, Nelson LD. Data falsificada (part 1): “Clusterfake.” Data Colada. June 17, 2023. https://datacolada.org/109Simonsohn U, Simmons JP, Nelson LD. Data falsificada (part 2): “My class year is Harvard.” Data Colada. June 20, 2023. https://datacolada.org/110Simonsohn U, Simmons JP, Nelson LD. Data falsificada (part 3): “The cheaters are out of order.” Data Colada. June 23, 2023. https://datacolada.org/111Simonsohn U, Simmons JP, Nelson LD. Data falsificada (part 4): “Forgetting the words.” Data Colada. June 30, 2023. https://datacolada.org/112Simonsohn U, Simmons JP, Nelson LD. Exhibits 3, 4, and 5. Data Colada. September 16, 2023. https://datacolada.org/114Simmons JP. Harvard's Gino report reveals how a dataset was altered. Data Colada. July 9, 2024. https://datacolada.org/118Additional resourcesLessig L. The Gino Case. Website and podcast series presenting arguments challenging Harvard's research misconduct findings. https://theginocase.info/Gino v Harvard University. CourtListener. Federal court docket and publicly available filings. https://www.courtlistener.com/docket/67659904/gino-v-harvard-university/Simonsohn U, Simmons JP, Nelson LD. Data Falsificada. ResearchBox. Research data, analysis code, and supporting materials accompanying the Data Colada investigations. https://researchbox.org/1630.34Kristal AS, et al. Research materials and data accompanying Signing at the beginning versus at the end does not decrease dishonesty. Open Science Framework. https://osf.io/4b7mu/overviewCorrection for Kristal et al., Signing at the beginning versus at the end does not decrease dishonesty. Proc Natl Acad Sci U S A. 2026;123(27):e2610441123. doi:https://doi.org/10.1073/pnas.2610441123Harvard Business School. Investigative report concerning allegations of research misconduct involving Francesca Gino. Harvard Business School Research Integrity Investigation Report. 2023. Accessed October 18, 2026. https://datacolada.org/wp-content/uploads/Harvard-Report-on-Gino.pdfKristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding  Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program  Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Program that Kristin teaches in::Epidemiology and Clinical Research Graduate Certificate Program Kristin’s manuscript writing course:https://instats.org/seminar/from-data-to-discussion-writing-a-scientFind us on:Kristin -  LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - Cold Open: Two Frauds in a Paper About Honesty (02:41) - The Claim: Spreadsheets Are the Villain (05:43) - The 2012 Honesty Pledge Study (09:17) - Signing at the Top Wins (13:31) - Failed Replications: Five Conceptual Tries, Then a Direct One (17:39) - How Data Colada Got the Data and Found Car Insurance Fraud (24:52) - Out-of-Order...

Can you spot problems in a scientific study just by reading the published paper? We put our statistical sleuthing skills to the test with a famous study on procrastination and deadlines that was retracted more than 20 years after it was published. But were there clues hidden in plain sight in the published results—and would we have noticed anything suspicious at the time? We put on our data-detective hats and find a few statistical clues of our own before turning to Data Colada’s investigation of the original data, where the story gets even stranger. Along the way, we hunt for missing participants, question suspiciously enormous effects, meet some unlikely data twins, and discover why sometimes the best statistical tool is simply asking: Do these data behave like real people?Statistical topicsStatistical sleuthingEffect sizesDegrees of freedomReplicationMissing dataMethodologic Morals“A data detective's toolbox is surprisingly simple. Check the degrees of freedom, look for twins, question giant effects, and ask whether the data actually make human sense.”“You don't have to be a statistician to catch a statistical criminal.”ReferencesData Colada blog posts:https://datacolada.org/138https://datacolada.org/139Data and code: https://researchbox.org/7135Retraction Watch article: https://retractionwatch.com/2026/09/03/procrastination-study-duke-dan-ariely-psychological-science-data-colada-tampering-retraction/ Original 2002 paper: https://journals.sagepub.com/doi/10.1111/1467-9280.00441 Retraction notice: https://journals.sagepub.com/doi/10.1177/09567976261488042 Hyndman K, Bisin A. Replication of "Procrastination, Deadlines, and Performance: Self-Control by Precommitment". Psychol Sci. 2026;37(8):557-571. doi:10.1177/09567976261460772Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding  Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program  Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Program that Kristin teaches in::Epidemiology and Clinical Research Graduate Certificate Program Kristin’s manuscript writing course:https://instats.org/seminar/from-data-to-discussion-writing-a-scientFind us on:Kristin -  LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com (00:00) - Introduction (02:15) - The Procrastination Paper and Study Two (08:51) - Suspiciously Tidy Results (16:18) - Study One: Real Classroom Stakes (20:06) - Missing Data and First Red Flags (28:17) - How Data Colada Got the Data (33:43) - When the Replication Failed (35:36) - Red Flags: Data Twins and Sanity Checks (47:21) - Study One Gets the Spreadsheet Treatment (51:45) - The Grading Scheme Unravels (57:51) - Manuscript Versions Reveal the Truth (01:08:11) - Rating the Claim and Methodological Morals

What happens when you discover that your published paper is wrong? Does retracting it ruin your reputation—or could owning the mistake actually make things better? In this episode, we meet the inaugural winners of the Ctrl-Z Awards, which recognize scientists for correcting or retracting their own work. Behavioral ecologist Kate Laskowski takes us inside a scientific detective story involving suspicious spider data, three retracted papers, and the fear that going public might tank her career. Then computational social scientist Alyssa Smith tells us how one small error in a figure led her to retract the paper at the center of her PhD and job talk. Along the way, we talk about data forensics, the stigma around retractions, and why self-correcting scientists are the heart of science.Statistical topicsData management and documentationOpen scienceReproducible workflowsResearch integrityRetractionsScientific misconductStatistical forensicsMethodologic Morals“Science is self-correcting, which means scientists need to be self-correcting too.”“Good documentation is boring—right up until it saves you.”“If you want scientists to correct their mistakes, reward them for correcting their mistakes.”“I'd rather be considered an honest idiot than for people to wonder if I'm untrustworthy.”ReferencesRetraction Watch: https://retractionwatch.com/Ctrl-Z Awards: https://retractionwatch.com/ctrl-z-award/Announcement of inaugural Ctrl-Z winners: https://centerforscientificintegrity.org/2026/08/04/2026-winners-ctrl-z-award-courage-retracting-own-work/ Kate Laskowski’s blog post: https://laskowskilab.faculty.ucdavis.edu/2020/01/29/retractions/Kate Laskowski’s Data Detective Course syllabusAlyssa Smith’s site: https://asmithh.github.io/ Laskowski KL, Montiglio PO, Pruitt JN. Individual and Group Performance Suffers from Social Niche Disruption [retracted in: Am Nat. 2020 Feb;195(2):393. doi: 10.1086/708066.]. Am Nat. 2016;187(6):776-785. doi:10.1086/686220Laskowski KL, Pruitt JN. Evidence of social niche construction: persistent and repeated social interactions generate stronger personalities in a social spider [retracted in: Proc Biol Sci. 2020 Jan 29;287(1919):20200077. doi: 10.1098/rspb.2020.0077.]. Proc Biol Sci. 2014;281(1783):20133166. Published 2014 Mar 26. doi:10.1098/rspb.2013.3166Modlmeier AP, Laskowski KL, DeMarco AE, et al. Persistent social interactions beget more pronounced personalities in a desert-dwelling social spider [retracted in: Biol Lett. 2020 Feb;16(2):20200062. doi: 10.1098/rsbl.2020.0062.]. Biol Lett. 2014;10(8):20140419. doi:10.1098/rsbl.2014.0419Smith AH, Green J, F Welles B, Lazer D. Emergent structures of attention on social media are driven by amplification and triad transitivity [retracted in: PNAS Nexus. 2026 May 07;5(5):pgag137. doi: 10.1093/pnasnexus/pgag137.]. PNAS Nexus. 2025;4(4):pgaf106. Published 2025 Apr 1. doi:10.1093/pnasnexus/pgaf106Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding  Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program  Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin -  LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com

Statistical power is one of the most important ideas in statistics—and one of the most misunderstood. In this episode, we explain what statistical power really means, why researchers need to think about it before they collect their data, and what goes wrong when they calculate “post hoc power” after the study is over. Along the way, we hunt for UFOs, test Kristin’s psychic abilities, explore the ethics of underpowered studies, and discover why post hoc power is really just a p-value wearing a fake mustache. Then statistician and powerlifter Andrew Althouse joins us for the inside story of a years-long scientific feud over post hoc power, where we discuss what happens when being wrong about statistics turns into doubling down, angry letters to the editor, and debates about how science corrects itself.Statistical topicsEffect sizeFalse negativesP-valuesPost hoc powerPower calculationsResearch ethicsSample sizeScientific controversySensitivity analysisStatistical powerStatistical significanceStudy designType II errorMethodologic Morals“Big questions need studies powerful enough to answer them.”“Post hoc power is circular reasoning masquerading as new information.”“Know when to cut your losses. A mistake gets more expensive every time you double down.”ReferencesAlthouse AD, Chow ZR. Comment on “Post-hoc power: If you must, at least try to understand.” Ann Surg. 2019;270:e78-e79.Althouse AD. Post hoc power: Not empowering, just misleading. J Surg Res. 2021;259:A3-A6.Altman DG. Statistics and ethics in medical research: III. How large a sample? BMJ. 1980;281:1336-1338.Bababekov YJ, Hung YC, Hsu YT, et al. Is the power threshold of 0.8 applicable to surgical science?—empowering the underpowered study. J Surg Res. 2019;241:235-239.Bababekov YJ, Stapleton SM, Mueller JL, et al. A proposal to mitigate the consequences of type 2 error in surgical science. Ann Surg. 2018;267:621-622.Chang DC, Stapleton SM. Response: The proliferation and misinterpretation of “as safe as” statements in surgical science: A call for professional discourse to search for a solution. J Surg Res. 2021;259:A12-A15.Freiman JA, Chalmers TC, Smith H Jr, et al. The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial: Survey of 71 negative trials. N Engl J Med. 1978;299:690-694. Hoenig JM, Heisey DM. The abuse of power: The pervasive fallacy of power calculations for data analysis. Am Stat. 2001;55:19-24.Nuzzo RL. Statistical Power. PM&R. 2016;8(9):907-912. doi:10.1016/j.pmrj.2016.08.004Nuzzo RL. Post hoc Power. PM&R. 2021;13(4):422-424. doi:10.1002/pmrj.12476Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding  Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program  Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin -  LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - Intro (00:43) - What is Statistical Power? (03:22) - The UFO Telescope Analogy (06:19) - The Psychic Ability Experiment (10:36) - Planning Power Before You Study (12:46) - History and the Landmark 1978 Paper (17:42) - Enter Post-Hoc Power: The Problem (19:26) - Three Problems with Post-Hoc Power (24:33) - The Surgery Journal Disaster (29:59) - Guest Interview: Andrew Althouse (32:30) - The 2018 Paper and the Controversy Begins (39:54) - Doubling Down and the Cost of Being Right (53:03) - Rating the Claim

Can weed protect your brain as you get older? We unpack a study linking lifetime cannabis use to larger brain volumes and better cognitive test scores in older adults—and investigate how those findings became headlines urging readers to “pass the pot to grandma.” Along the way, we explore collapsing categorical variables, confounding, clinically versus statistically significant results, multiple testing, and how a press release subtly changed a story. We also meet permanently stoned mice, learn about Carl Sagan’s surprising connection to cannabis research, and discover how a single missing word can change the entire meaning of a study.Statistical topicsCategorical variablesConfoundingEffect sizeFalse discovery rateHealthy volunteer biasMeasurement errorMultiple testingObservational studiesRecoding variablesScience communicationStatistical vs practical significanceMethodologic Morals“Always follow the chain from data to result to conclusions and pay attention to the words along the way.”“Huge samples can give tiny differences delusions of grandeur.”ReferencesGuha A, Fu Z, Calhoun V, Hutchison KE. Lifetime Cannabis Use Is Associated With Brain Volume and Cognitive Function in Middle-Aged and Older Adults. J Stud Alcohol Drugs. 2026;87(4):739-751. doi:10.15288/jsad.25-00346Press Release: https://news.cuanschutz.edu/news-stories/study-finds-cannabis-usage-in-middle-aged-and-older-adults-associated-with-larger-brain-volume-better-cognitive-functionBilkei-Gorzo A, Albayram O, Draffehn A, et al. A chronic low dose of Δ9-tetrahydrocannabinol (THC) restores cognitive function in old mice. Nat Med. 2017;23(6):782-787. doi:10.1038/nm.4311NY Post: https://nypost.com/2026/02/07/health/cannabis-may-benefit-aging-brains-study-finds/David Grinspoon: https://www.nytimes.com/2020/07/02/science/lester-grinspoon-dead.html?unlocked_article_code=1.7lA.CyS1.neeqGGNgx8g1&smid=url-shareKristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding  Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program  Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin -  LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - Opening: The Brain on Drugs (00:37) - The Cannabis Headline (03:38) - The Claim: Brain Benefits for Older Adults (06:53) - The Mouse Study: Pothead Mice vs. Control Mice (11:25) - From Mice to Humans: UK Biobank (15:01) - Measurement Problems and Category Collapse (21:39) - Results: Statistically Significant, Clinically Meaningless (27:13) - Individual Brain Regions and Cherry-Picking (32:32) - Cognitive Tests: What Actually Matters (40:09) - How It Gets Lost in Translation (45:32) - Rating the Claim

Can the meningitis vaccine do double duty against gonorrhea? We examine a fascinating randomized trial inspired by years of encouraging observational studies—and what happened when those earlier results were finally put to the test. Along the way, we discuss ecological studies, case-control studies, why statisticians distrust phrases like “trend toward significance,” how intention-to-treat differs from per-protocol analysis, and why no amount of statistical adjustment can fully substitute for randomization. Along the way we celebrate outrageously named clinical trials, ask whether “flirting with statistical significance” belongs in a romance novel or a medical journal, and discover that “practice safe statistics” may be the best advice of the episode.Statistical topicsCase-control studiesConfoundingEcological studiesGeneralizabilityIntention-to-treatMeta-analysisNull resultsObservational studiesP-valuesPer-protocol analysisRandomized clinical trialsSurvival analysisMethodologic Morals“Don't mistake a p-value of 0.06 for a p-value of 0.04 with bad luck.” “Practice safe statistics; randomize whenever possible.”ReferencesSeib KL, Donovan B, Jin F, et al. Meningococcal B Vaccine to Prevent Neisseria gonorrhoeae Infection. N Engl J Med. 2026; 395: 349-61. Petousis-Harris H, Paynter J, Morgan J, et al. Effectiveness of a group B outer membrane vesicle meningococcal vaccine against gonorrhoea in New Zealand: a retrospective case-control study. The Lancet. 2017; 390: 1603–10.Wang, B., Mohammed, H., Andraweera, P., McMillan, M., Marshall, H., 2024. Vaccine effectiveness and impact of meningococcal vaccines against gonococcal infections: A systematic review and meta-analysis. Journal of Infection. 2024; 89: 106225. Molina JM, Bercot B, Assoumou, L, et al. Doxycycline prophylaxis and meningococcal group B vaccine to prevent bacterial sexually transmitted infections in France (ANRS 174 DOXYVAC): a multicentre, open-label, randomised trial with a 2 × 2 factorial design. The Lancet Infectious Diseases. 2024; 24: 1093–1104.Thng C, Eskandari S, Jin F, et al. Efficacy of the meningococcal vaccine against Neisseria gonorrhoeae: a randomised clinical trial (MenGO). npj Vaccines. 2026.Still Not Significant | Probable Error: List of funny disguises scientists use to describe p>.05Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding  Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program  Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin -  LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:15) - - Introduction (00:51) - - The Claim: Can a Meningitis Vaccine Prevent Gonorrhea? (04:10) - - The Biology Behind the Idea (10:37) - - How Observational Studies Made the Link (14:44) - - Earlier Clinical Trials: MENGO and DOXYVAC (17:30) - - Flirting with Significance: A Statistical Rant (22:40) - - GoGoVax: Study Design and Analysis (33:31) - - The Shocking Null Result (35:38) - - Why These Results Don't Generalize (37:42) - - Rating the Claim

Can you improve your endurance by sitting in a hot tub instead of doing another hard workout? We dig into a study that put elite runners through five weeks of what we affectionately call “endurance hot tubbing” to see whether heat alone could trigger the same cardiovascular adaptations as altitude training. Along the way, we explore crossover study designs, multiple testing, why significant physiological changes don’t necessarily translate into better athletic performance, and how an exploratory regression analysis ended up answering the wrong scientific question. We also discover that serious hot tub research involves surprisingly little relaxation, debate whether bubbles should count as an experimental condition, and learn why distance runners pay good money to sleep in tents with some of the oxygen sucked out.Statistical topicsbest subsets regressioncausal inferencecrossover studiesexploratory analysesmediation / mechanismsmodel specificationmultiple testingpeer reviewsample size(00:15) - Introduction (00:44) - Can Hot Tubs Replace Exercise? (01:27) - Crazy Runners and Crazy Training (05:44) - The Amazon Delivery System Metaphor (12:12) - From Heat Training to Hot Tubs (13:33) - The Study: 10 Runners in Hot Water (22:31) - Results and the VO2 Max Question (29:26) - Statistical Sleuthing: Best Subsets Regression (36:50) - The Peer Review Problem (41:39) - Rating the Claim Methodologic Morals“A model can be very good at answering the wrong question.” “The label ‘peer reviewed’ does not guarantee ‘carefully reviewed’.”ReferencesJenkins EJ, Killick JA, Zerilli O, et al. Long-term passive heat acclimation enhances maximal oxygen consumption via haematological and cardiac adaptation in endurance runners. J Physiol. Published online November 20, 2025. doi:10.1113/JP289874Stembridge M, Jenkins E. Marathon training: Why hot baths might help you run faster. The Conversation. Published March 16, 2026. doi:10.64628/AB.6vknm9eshKristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding  Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program  Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin -  LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:15) - Introduction (00:44) - Can Hot Tubs Replace Exercise? (01:27) - Crazy Runners and Crazy Training (05:44) - The Amazon Delivery System Metaphor (12:12) - From Heat Training to Hot Tubs (13:33) - The Study: 10 Runners in Hot Water (22:31) - Results and the VO2 Max Question (29:26) - Statistical Sleuthing: Best Subsets Regression (36:50) - The Peer Review Problem (41:39) - Rating the Claim

Could four one-minute bursts of exercise really improve blood sugar? We try “exercise snacks” ourselves before taking a close look at the clinical trial that inspired headlines. We explain why the study’s main result wasn’t statistically significant, how 34 secondary outcomes complicated the story, and what pre-registration can reveal about a study after it’s published. Along the way, we compare notes on our own exercise-snacking adventures, debate continuous glucose monitors, and ask how much evidence a single study should generate before it becomes health news.Statistical topicsCrossover designMultiple testingPre-registrationPrimary vs secondary outcomesRandomized controlled trialResearch transparencyMethodologic Morals“It's good relationship advice to be transparent. It's also good research advice.”“If the primary outcome is not significant, say it up top.”ReferencesBabir FJ, Marcotte-Chénard A, Sandilands RE, et al. Exercise snacks performed in real-world settings reduce postprandial hyperglycaemia and glycaemic variability in individuals living with type 2 diabetes: a randomised crossover study. Diabetologia. 2026;69(8):2200-2211. doi:10.1007/s00125-026-06741-2https://www.washingtonpost.com/wellness/2026/05/28/4-minutes-exercise-day-could-help-control-blood-sugar/clinicaltrials.gov pre-registration with changes: https://clinicaltrials.gov/study/NCT06382246?term=NCT06382246&rank=1&tab=history&a=1&b=2#version-content-panelKristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding  Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program  Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin -  LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - Intro (04:06) - The claim: four minutes a day (07:05) - Our own N of 1 experiments (13:24) - The study (21:54) - Primary outcome: complete miss (25:16) - Secondary outcomes to the rescue? (35:02) - Statistical sleuthing and transparency (44:23) - Rating the claim

How do you decide whether a clinical trial “worked”? In Part 2 of our Galleri series, we examine the landmark randomized trial of a blood test designed to detect more than 50 cancers. We explore why different outcome measures led to dramatically different headlines, discuss primary versus secondary outcomes, pre-registration, hierarchical testing, and post hoc analyses, and explain why mortality remains the outcome everyone is waiting for. Along the way, we uncover a statistical mystery involving dozens of missing cancers and discover how a little arithmetic can sometimes reveal more than a press release.Statistical topicscancer screeningexploratory analyseshierarchical testingmissing datamultiple testingoutcome measurespost hoc analysespre-registrationprimary and secondary outcomesrandomized clinical trialsscreening testsMethodologic Morals“When the simple numbers don't add up, pay attention. The arithmetic may be trying to tell you something.”“The first question should not be, did it work? It should be, what counts as success?”ReferencesGiridhar KV, et al. Safety and performance results from PATHFINDER 2, a registrational study of a multi-cancer early detection test in an intended-use population. Presented at the 2026 American Society of Clinical Oncology (ASCO) Annual Meeting. May 2026.Hubbell E, Clarke CA, Aravanis AM, Berg CD. Modeled Reductions in Late-stage Cancer with a Multi-Cancer Early Detection Test. Cancer Epidemiol Biomarkers Prev. 2021;30(3):460-468. doi:10.1158/1055-9965.EPI-20-1134Neal RD, Johnson P, Clarke CA, et al. Cell-Free DNA-Based Multi-Cancer Early Detection Test in an Asymptomatic Screening Population (NHS-Galleri): Design of a Pragmatic, Prospective Randomised Controlled Trial. Cancers (Basel). 2022;14(19):4818. Published 2022 Oct 1. doi:10.3390/cancers14194818ASCO slides: https://grail.com/wp-content/uploads/2026/05/Swanton_ASCO-2026_NHS-Galleri_FINAL-Slides-05.26.2026.pdfUK registry protocol:  https://www.isrctn.com/ISRCTN91431511 Clinicaltrials.gov protocol: https://clinicaltrials.gov/study/NCT05611632 Common biases in cancer screening studiesCancer screening studies are subject to several well-known biases that can make a screening test appear more effective than it actually is. Three of the most important are:Lead-time bias: Screening advances the time of diagnosis, making survival from diagnosis appear longer even if the patient's lifespan is unchanged. For example, if a screening test detects a Stage II cancer at age 60 that otherwise would have been diagnosed because of symptoms at age 62, but the patient dies at age 68 regardless, survival from diagnosis appears to increase from 6 years to 8 years even though the patient did not live any longer. Length bias: Screening preferentially detects slower-growing, less aggressive cancers because they remain detectable for longer than fast-growing cancers. For example, a slow-growing cancer that remains in Stage I for 5 years is much more likely to be found by screening than an aggressive cancer that progresses to symptoms within months. This can make screened patients appear to have better survival simply because screening preferentially found the less aggressive cancers. Overdiagnosis: Screening detects cancers that would never have caused symptoms or death during a person's lifetime, leading to unnecessary diagnosis and treatment. For example, a screening test may detect a very slow-growing prostate or thyroid cancer in an older adult that would never have become clinically important if it had remained undiscovered. Kristin and Regina’s online courses: Demystifying Data: A Modern Approach to Statistical Understanding  Clinical Trials: Design, Strategy, and Analysis Medical Statistics Certificate Program  Writing in the Sciences Epidemiology and Clinical Research Graduate Certificate Program Programs that we teach in:Epidemiology and Clinical Research Graduate Certificate Program Find us on:Kristin -  LinkedIn & Twitter/XRegina - LinkedIn & ReginaNuzzo.com(00:00) - Intro (03:39) - The Claim: Not Ready for Primetime (03:58) - Trial Design: 142,000 Participants (07:50) - The Primary Outcome Problem (20:29) - The Primary Endpoint: Complete Miss (22:14) - Three Arguments for the Defense (28:29) - - Statistical Sleuthing: Missing Cancers (41:14) - - The Stage Shift Argument (50:30) - - Rating the Claim