Metascience Matters

Samuel Westwood is Senior Lecturer in Psychology (Education) at the Institute of Psychiatry, Psychology & Neuroscience at King's College London.Samuel joined King’s College London in 2023. He specialises in neuromodulation, neurodevelopmental disorders, and open and reproducible research practices. His work focuses on providing best practice guidance in translational cognitive neuroscience and psychiatric research. Samuel earned a degree in BSc Psychology at the University of Birmingham, then went on to achieve a PhD in Cognitive Neuroscience at Aston University, after which he was a post-doc at King’s in the Department of Child & Adolescent Psychiatry. Before joining King’s, Samuel was a lecturer at the University of Westminster and the University of Wolverhampton.OUTLINE:0:00 - Highlights2:06 - Intro3:01 - How Sam got interested in brain stimulation5:25 - Publishing null results from transcranial direct current stimulation10:49 - Conducting a systematic review of the field13:31 - Methodological issues across types of brain stimulation16:11 - Which brain stimulation/clinical indication combination are most promising?24:05 - Clinical trial of transcranial direct current stimulation and cognitive training for ADHD in adolescent boys31:51 - Personalized approaches to brain stimulation and cognitive training37:23 - Pursuing metascience more broadly42:34 - Founding the RIOT Science Club48:07 - Most desired structural changes to science over the next 5-10 years51:15 - Dealing with scientific fraud, incentives57:11 - Advice for young scientists59:25 - Forthcoming developments and where Sam's work can be found online1:00:58 - Outro

Chris Chambers is a Professor of Cognitive Neuroscience at Cardiff University who focuses on the promotion and effectiveness of open research practices. He co-founded Registered Reports, Exploratory Reports, Verification Reports, the Transparency and Openness Promotion guidelines, the Royal Society Replications initiative, the UK network of open research working groups, the Peer Reviewers' Openness Initiative, the UK Reproducibility Network (UKRN), the GW4 Undergraduate Psychology Consortium programme for promoting reproducible science, the Royal Society Rapid Review Network for COVID-19 Registered Reports, and the Peer Community in Registered Reports. He currently chairs the Registered Reports committee supported by the Center for Open Science, serve on the UKRN steering committee and sits on the Advisory Board of Nature Human Behaviour.He is the author of the Seven Deadly Sins of Psychology: A Manifesto for Reforming the Culture of Scientific Practice, which won the 2018 British Psychological Society Book Award (Best Academic Monograph) and the 2018 PROSE Award from the Association of American Publishers. OUTLINE:0:00 - Highlights2:19 - Intro3:06 - Starting registered reports5:44 - Blogosphere conversations that spawned registered reports7:26 - Pre-registration vs. Registered reports10:39 - Precursors to registered reports17:10 - Reviewing the first registered reports at Cortex21:25 - Initial reactions from neuroscience and psychology to RRs25:26 - The growth of RRs across journals27:54 - Peer Community In Registered Reports (Registered Reports 2.0)33:30 - Stage 1 RR triage/snapshot solution to the wait time problem36:57 - New challenges with Registered Reports 2.039:53 - Programmatic research at Peer Community In Registered Reports43:26 - Clinical trials and registered reports 47:47 - Current conversations around registered reports50:39 - Researcher evaluation in a hypothetical world where null data are embraced56:01 - Forthcoming developments related to registered reports59:49 - Advice for young scientists1:03:32 - Outro

Beth Clarke is an incoming Junior Professor of Psychological Metascience at the Leibniz Institute for Psychology (ZPID). Her work examines how research methods and practices have been changing in psychology (especially social and personality psychology) over the replication crisis.Google Scholar: https://scholar.google.com/citations?user=KTF_Du0AAAAJ&hl=enLooking our limitations in the eye: A call for more thorough and honest reporting of study limitations:https://compass.onlinelibrary.wiley.com/doi/10.1111/spc3.12979The prevalence of direct replication articles in top-ranking psychology journals:https://pubmed.ncbi.nlm.nih.gov/39418470/P-curving social and personality psychology experiments: Changes from 2006 to 2019:https://osf.io/preprints/psyarxiv/txzeu_v1RegCheck:https://regcheck.app/OUTLINE:0:00 - Highlights2:34 - Intro3:29 - What Beth's professorship in metascience says about the maturity of the field7:10 - Initial interest in metascience10:54 - Hopefulness for the future of psychology, metascience, and science in general13:16 - Study of how researchers ignore the limitations of their work21:20 - Thoughts on registered reports 22:56 - Are certain subfields of psychology better at acknowledging limitations? 24:52 - Should limitations be acknowledged in abstracts, titles, and elsewhere in papers?25:55 - Study of the prevalence of direct replication articles in top psychology journals34:42 - How do you choose what studies to replicate?36:06 - P-curving social and personality psychology experiments45:34 - Developing RegCheck, an AI tool to compare preregistrations and their associated papers50:30 - What systemic interventions would you most like to see in the next 5-10 years that would most benefit the scientific community?54:37 - Directions for Beth's future lab55:51 - Advice for undergraduates59:16 - Outro

Lukas Röseler is the Managing Director at the Center for Open Science at the University of Münster. He is the Editor-in-Chief at the journal Replication Research, part of the Replication Journal Federation, is an active member of the Framework for Open and Reproducible Research Training (FORRT) community, and developed a video game called Guess the Replication, where players guess whether studies successfully replicated. Replication Research:http://replicationresearch.org/Replication Journal Federation:https://forrt.org/rjf/Framework for Open and Reproducible Research Training (FORRT) community:https://forrt.org/Guess The Replication:https://lukasroeseler.github.io/GuessTheReplication/Dominance and Prestige: Meta-Analytic Review of Experimentally Induced Body Position Effects on Behavioral, Self-Report, and Physiological Dependent Variables:https://www.researchgate.net/profile/Robert-Koerner/publication/360577810_Dominance_and_prestige_Meta-analytic_review_of_experimentally_induced_body_position_effects_on_behavioral_self-report_and_physiological_dependent_variables/links/62823b133a23744a728598c2/Dominance-and-Prestige-Meta-Analytic-Review-of-Experimentally-Induced-Body-Position-Effects-on-Behavioral-Self-Report-and-Physiological-Dependent-Variables.pdfOUTLINE:0:00 - Intro2:10 - Leading the Münster Center for Open Science6:17 - How Lukas became interested in metascience10:53 - Cultural change in science13:43 - Being Editor-in-Chief of Replication Research20:00 - Funding the journal, diamond open access24:55 - Prestigious journals' money extraction scheme28:17 - Distributed institutional support for diamond OA journals31:50 - The Replication Journal Federation39:24 - Journals for replication, or replications for journals?44:34 - Guess The Replication50:57 - Prediction markets for replication54:32 - Meta-analysis of body positioning effects/power posing1:02:46 - If science were aligned to truth, how would researchers be evaluated?1:14:36 - Advice for young people

Alexander Gibson is a PhD student at the Queensland University of Technology and the Australian Centre for Health Services Innovation studying the intersection of metascience and clinical machine learning. One of his focus areas is data provenance, the Who, What, Where, When, Why, and How of datasets, and how neglecting this can lead to bad outcomes in medical machine learning not only in research, but also for clinical practice and medical device approval. CONTACT RANDY:Feedback: [email protected] LINKS:Alex's preprint on unreliable diabetes and stroke datasets:https://www.medrxiv.org/content/10.64898/2026.02.24.26347028v2OUTLINE:0:00 - Introduction3:14 - The beginning of Alex's interest in clinical predictive modeling5:05 - Alex's interest in metascience6:42 - Choosing a dissertation topic/metrics hacking in machine learning9:49 - Preprint on data provenance in medical datasets12:33 - The diabetes and stroke datasets Alex investigated16:46 - Major irregularities in the data23:29 - TRIPOD+AI guidelines for auditing machine learning studies25:26 - How unreliable studies can impact clinical practice and medical device patents26:42 - Citation networks27:37 - AI-generated formulaic medical machine learning studies31:50 - Strategies for high-quality data provenance33:53 - Patents citing unreliable studies, and how to integrate data provenance into peer review35:23 - The biggest problems for clinical predictive modeling studies37:02 - Resources and tools for improving rigor in machine learning38:45 - Metrics reporting40:45 - Choosing decision thresholds in predictive models42:59 - The importance of clinical context in metrics reporting45:21 - The unreasonable effectiveness of age and sex as predictors47:53 - The roles of academia and industry in improving clinical machine learning studies50:07 - Explanation versus prediction52:51 - Advice and resources for students54:27 - Outro

Charles Piller, an Investigative Correspondent for Science Magazine, is the author of the 2025 book Doctored: Fraud, Arrogance, and Tragedy in the Quest to Cure Alzheimer's. He has received more than 40 honors or awards from American Association for the Advancement of Science/Kavli Foundation, National Institute of Health Care Management, Society of Professional Journalists, National Academies/Eric and Wendy Schmidt, Los Angeles Press Club, American Society of Tropical Medicine and Hygiene, Gerald Loeb Foundation and UCLA, Western Publications Association, First Amendment Coalition, California State University, Computer Press Association, American Society of Business Press Editors, California Newspaper Publishers Association, National Council on Crime and Delinquency, and other organizations. His work has been nominated for the Pulitzer Prize nine times by STAT, the Los Angeles Times and The Sacramento Bee.CONTACT RANDY:Feedback: [email protected] LINK:Blots on a field? https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-diseasePicture imperfecthttps://www.science.org/content/article/research-misconduct-finding-neuroscientist-eliezer-masliah-papers-under-suspicionDoctored: Fraud, Arrogance, and Tragedy in the Quest to Cure Alzheimer'shttps://www.amazon.com/Doctored-Fraud-Arrogance-Tragedy-Alzheimers/dp/1668031248OUTLINE:0:00 - Introduction6:05 - How Charles became a science journalist7:58 - Charles' 2022 piece, "Blots on a field?"14:39 - Responses from scientists to this piece22:22 - Charles' 2024 piece, "Picture imperfect"30:29 - How Charles began working on his book "Doctored: Fraud, Arrogance, and Tragedy in the Quest to Cure Alzheimer's"33:15 - Charles' experience working with scientific sleuths36:45 - The incentive for positive data in science46:09 - Charles' NYT op-ed about how to combat fraud in Alzheimer's research; attacks from the Trump administration on science53:22 - Charles' thoughts on registered reports and other ways to improve scientific incentives58:34 - Upcoming work59:22 - Advice for listeners1:02:20 - Outro

Tim Errington is the Senior Director of Research at the Center for Open Science. He led the Reproducibility Project: Cancer Biology, Systematizing Confidence in Open Research and Evidence (SCORE), as well as the implementation and evaluation of initiatives such as Registered Reports, Registered Revisions, responsible conduct of research trainings, and open science badges.CONTACT RANDY:Feedback: [email protected] LINKS:Investigating the replicability of preclinical cancer biology (Reproducibility Project: Cancer Biology led by Tim Errington):https://elifesciences.org/articles/71601Bayer replication study:https://www.nature.com/articles/nrd3439-c1Amgen replication study:https://www.nature.com/articles/483531aReproducibility in Cancer Biology: Challenges for assessing replicability in preclinical cancer biology (Companion paper to Reproducibility Project: Cancer Biology):https://elifesciences.org/articles/67995What is replication?:https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000691Study comparing standard reports and registered reports in psychology:https://journals.sagepub.com/doi/10.1177/25152459211007467Blog post on the seemingly magical success of revision experiments:https://rajlaboratory.blogspot.com/2013/06/the-magical-results-of-reviewer.htmlGoogle's AI co-scientist paper:https://arxiv.org/abs/2502.18864Machine-readable documents:https://journals.sagepub.com/doi/10.1177/2515245920970949How open science helps researchers succeed:https://elifesciences.org/articles/16800ZBW's Expedition to Open Science Land:https://expedition-open-science.org/OUTLINE:0:00 - Introduction4:47 - Tim's origin story as a cancer biologist6:38 - Initial interest in metascience 9:24 - Starting the Reproducibility Project: Cancer Biology12:07 - How were the studies that were replicated chosen?14:41 - Publishing the Reproducibility Project: Cancer Biology as a registered report17:26 - Results from the Reproducibility Project: Cancer Biology20:28 - Tim's experience throughout the years running the Reproducibility Project: Cancer Biology25:21 - The difficulty of running cancer biology studies27:54 - Judging whether a replication is successful31:23 - What has the response to the Reproducibility Project: Cancer Biology been?37:52 - Why aren't replication rates higher?40:26 - Challenges of running cancer biology replication studies45:43 - Caveats of preclinical disease models49:13 - The incentive for positive data in science57:05 - Systemic intervention vs. Individual policing1:01:04 - The value of registered reports1:07:38 - Registered revisions1:10:48 - Falsifying theories early at the preclinical stage1:15:21 - Different institutions (e.g., academic, industry) conducting different studies (e.g., preclinical, clinical)1:17:34 - New initiatives at the Center for Open Science (Replication project of social/behavioral sciences, automated tools for predicting replication success, LifeCycle journal)1:23:02 - AI scientists are trained on biased literature; distrust of academic literature in drug discovery1:28:46 - Peer review1:32:51 - Narrative in science1:35:06 - 100-200 years into the future1:40:29 - Advice for high school/undergraduate listeners1:42:51 - Metascience manifests in every field1:44:46 - Philosophy of science1:47:49 - Outro

Brian Nosek is the cofounder and Executive Director of the Center for Open Science. He co-developed the Implicit Association Test, a method that advanced the study of implicit bias. He then co-founded three non-profit organizations: Project Implicit to advance research about implicit bias, the Society for the Improvement of Psychological Science to improve the research culture in his home discipline, and the Center for Open Science (COS) to improve rigor, transparency, integrity, and reproducibility across research disciplines. He led the Reproducibility Project in Psychology, a replication of 100 studies from psychology, as well as multiple other replication projects, along with policy reforms such as open science badges, and the investigation of prediction markets for study replication.CONTACT RANDY:Feedback: [email protected] LINKS:Estimating the reproducibility of psychological science (Reproducibility Project in Psychology led by Brian Nosek):https://www.science.org/doi/10.1126/science.aac4716False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant:https://journals.sagepub.com/doi/10.1177/0956797611417632What is replication?:https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000691Metascience conference:https://metascience.info/International Conference on the Science of Science and Innovation:https://icssi.org/International Congress on Peer Review and Scientific Publication:https://peerreviewcongress.org/MetaROR:https://metaror.org/OUTLINE:0:00 - Introduction2:56 - Brian's origin story as a psychologist6:35 - Initial interest in metascience 10:57 - "Scientific Utopia," starting the Reproducibility Project in Psychology, and major fraud cases17:57 - How did the Reproducibility Project in Psychology come together?23:40 - Results of the Reproducibility Project in Psychology27:03 - Response from the field, institutional changes30:39 - Advice for high school/undergrad listeners34:03 - Outro

Eugenie Reich is an attorney committed to taking on scientific fraud, understanding the incentives that drive it, and recovering misdirected research funding. She is also a former investigative science journalist committed to correcting the scientific record. Her 2009 book Plastic Fantastic, details a major fraud cause in physics at Bell Labs. Two of the cases she has litigated cases were against Biogen ($900 million settlement), and the Dana-Farber Cancer Institute ($15 million settlement).CONTACT RANDY:[email protected] LINKS:Cargo Cult Science:https://calteches.library.caltech.edu...The PubPeer Conundrum:https://www.tandfonline.com/doi/epdf/...New Scientist:https://www.newscientist.com/Skeptical Inquirer:https://skepticalinquirer.org/Special thanks to Dylan BouscherOUTLINE:0:00 - Introduction4:20 - How Eugenie became interested in science6:25 - Interest in scientific fraud7:38 - Deciding to become a science journalist8:58 - Fraud in physics at Bell Labs (the subject of her 2009 book Plastic Fantastic)20:15 - Bell Labs coming under private ownership22:53 - Interviewing scientists for her book26:50 - The response from the physics community31:10 - Deciding to become a lawyer33:28 - The False Claims Act36:29 - The qui tam provision of the False Claims Act37:46 - The $900M case against Biogen44:02 - The day-to-day of working on this case46:30 - Impact of the case on biotech and pharma48:40 - The $15M case against Dana-Farber50:42 - Do universities have an incentive to protect accused researchers?54:15 - Is the scale of fraud too large to be fixed?57:56 - Does fraud damage public trust in science?1:00:44 - Tools/solutions to combat these problems1:04:32 - Advice to practicing scientists1:05:55 - Advice and resources for listeners1:08:28 - Outro

James Heathers is the Founder and Director of the Medical Evidence Project, a venture of The Center for Scientific Integrity. He aims to reduce medical harm and improve patient outcomes by identifying and publicizing errors and miscondcut in the medical literature. He uses forensic meta-analytical techniques to detect and deconstruct errors arising from low-quality science and fraudulent work in areas that involve large numbers of patients.CONTACT RANDY:[email protected] LINKS:The original GRIM test paper: http://www1.psych.purdue.edu/~gfranci...Machine-readable documents: https://journals.sagepub.com/doi/10.1...Chaos in the Brickyard: https://www.science.org/doi/10.1126/s...Books:The Man Who Only Loved Numbers: https://en.wikipedia.org/wiki/The_Man...The Emperor of All Maladies:https://en.wikipedia.org/wiki/The_Emp...OUTLINE:0:00 - Introduction5:27 - How James has been the last couple days, and his radio voice6:40 - Writing the first book on Forensic Metascience10:47 - James' PhD work on heart rate variability14:12 - Starting to work on metascience18:40 - The GRIM test26:40 - Programmatically scaling statistical checks33:25 - Centering quantitative results and peripheralizing narrative in scientific papers39:36 - AI for metascience, unearthing data underneath narrative48:11 - Pursuing cases of misconduct52:45 - Founding and Directing the Medical Evidence Project56:10 - Incentives for positive data, post-publication review, challenging the binary of positive and negative data1:05:41 - Advice and resources for listeners1:12:40 - Optimism over pessimism1:16:58 - Outro