186: Evaluating journal quality

We discuss whether preprint servers and journals should require author identity verification for submitting manuscripts. This would probably speed up the submission process, but is this worth the potential downsides? We also discuss the similarities and differences between academia and professional sports and a weird case of author identity theft. Other links * The BJKS podcast https://bjks.buzzsprout.com Social media links - Dan on Bluesky (https://bsky.app/profile/dsquintana.bsky.social) - James on Bluesky (https://bsky.app/profile/jamesheathers.bsky.social) - Everything Hertz on Bluesky (https://bsky.app/profile/hertzpodcast.bsky.social)

Dan and James chat about a a new 'pop-up journal' concept for addressing specific research questions. They also answer a listener question from a journal grammar editor and discuss a new PNAS article on paper mills Links * The pop-up journal (https://popupjournal.com) * The episode (https://everythinghertz.com/58) where Dan's wife went into labor * The PNAS paper mill paper (https://www.pnas.org/doi/10.1073/pnas.2420092122) * A blog post (https://reeserichardson.blog/2025/08/04/a-do-or-die-moment-for-the-scientific-enterprise/) from the PNAS paper lead author, Reese Richardson. * The Nature piece (https://www.nature.com/articles/d41586-025-02446-5) on the paper Social media links - Dan on Bluesky (https://bsky.app/profile/dsquintana.bsky.social) - James on Bluesky (https://bsky.app/profile/jamesheathers.bsky.social) - Everything Hertz on Bluesky (https://bsky.app/profile/hertzpodcast.bsky.social) Citation Quintana, D. S., & Heathers, J. (2025, August 7). 193: The pop-up journal, Everything Hertz [Audio podcast], DOI: 10.17605/OSF.IO/2ZMQ7

Dan and James answer listener questions on outsourcing in academia and differences in research culture between academic institutions and commercial institutions. Social media links - Dan on Bluesky (https://bsky.app/profile/dsquintana.bsky.social) - James on Bluesky (https://bsky.app/profile/jamesheathers.bsky.social) - Everything Hertz on Bluesky (https://bsky.app/profile/hertzpodcast.bsky.social) Citation Quintana, D. S., & Heathers, J. (2025, July 1). 192: Outsourcing in academia, Everything Hertz [Audio podcast], DOI: 10.17605/OSF.IO/3MC2R

James and Dan discuss James' newly funded 'Medical Evidence Project', whose goal is to find questionable medical evidence that is contaminating treatment guidelines. Links * James' blog post (https://jamesclaims.substack.com/p/how-should-we-fund-scientific-error) from last year * The carthorse child blog post (https://medium.com/hackernoon/introducing-sprite-and-the-case-of-the-carthorse-child-58683c2bfeb) * The blog post (https://jamesclaims.substack.com/p/introducing-the-medical-evidence) announcing the project * A write up in Nature (https://www.nature.com/articles/d41586-025-01739-z?utm_source=bluesky&utm_medium=social&utm_campaign=nature&linkId=14776408) about the project Other links - Dan on Bluesky (https://bsky.app/profile/dsquintana.bsky.social) - James on Bluesky (https://bsky.app/profile/jamesheathers.bsky.social) - Everything Hertz on Bluesky (https://bsky.app/profile/hertzpodcast.bsky.social) Citation Quintana, D. S., & Heathers, J. (2025, June 4). 191: Cleaning up contaminated medical treatment guidelines Everything Hertz [Audio podcast], DOI: 10.17605/OSF.IO/537BN

We chat about two new studies that took different approaches for evaluating the impact of paying reviewers on peer review speed and quality. Links * James' 450 movement proposal (https://jamesheathers.medium.com/the-450-movement-1f86132a29bd) * The paper (https://journals.lww.com/ccmjournal/fulltext/9900/effect_of_monetary_incentives_on_peer_review.488.aspx) from Critical Care Medicine * The preprint (https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1) from Biology Open Other links - Dan on Bluesky (https://bsky.app/profile/dsquintana.bsky.social) - James on Bluesky (https://bsky.app/profile/jamesheathers.bsky.social) - Everything Hertz on Bluesky (https://bsky.app/profile/hertzpodcast.bsky.social) Citation Quintana, D. S., & Heathers, J. (2025, April 2). 190: What happens when you pay reviewers?, Everything Hertz [Audio podcast], DOI: 10.17605/OSF.IO/PHQ2K

Dan and James discuss a recent piece that proposes a post-publication review process, which is triggered by citation counts. They also cover how an almetrics trigger could be alternatively used for a more immediate post-publication critique. Links * The Chonicle piece (https://www.chronicle.com/article/social-science-is-broken-heres-how-to-fix-it?sra=true) by Andrew Gelman and Andrew King [Free to read with email registration] * The paper (https://psycnet.apa.org/fulltext/2022-14587-001.html) by Peder Isager and collegues on how to decide what papers we should replicate. Here is the preprint (https://files.de-1.osf.io/v1/resources/2gurz/providers/osfstorage/5f4f4314a392b9002f1d9576?action=download&direct&version=2). * The ERROR project (https://error.reviews/about/) Other links Everything Hertz on Bluesky - Dan on Bluesky (https://bsky.app/profile/dsquintana.bsky.social) - James on Bluesky (https://bsky.app/profile/jamesheathers.bsky.social) - Everything Hertz on Bluesky (https://bsky.app/profile/hertzpodcast.bsky.social) Citation Quintana, D. S., & Heathers, J. (2025, Mar 2). 189: Crit me baby, one more time, Everything Hertz [Audio podcast], DOI: 10.17605/OSF.IO/3X5UR

Dan and James discuss a recent editorial which argues that double-blind peer review is detrimental to scientific integrity. Links * The editorial from Christopher Mebane: https://doi.org/10.1093/etojnl/vgae046 Other links Everything Hertz on Bluesky - Dan on Bluesky (https://bsky.app/profile/dsquintana.bsky.social) - James on Bluesky (https://bsky.app/profile/jamesheathers.bsky.social) - Everything Hertz on Bluesky (https://bsky.app/profile/hertzpodcast.bsky.social) Support us on Patreon (https://www.patreon.com/hertzpodcast) and get bonus stuff! $1 per month: A 20% discount on Everything Hertz merchandise, access to the occasional bonus episode, and the the warm feeling you're supporting the show $5 per month or more: All the stuff you get in the one dollar tier PLUS a bonus episode every month Citation Quintana, D. S., & Heathers, J. (2025, Jan 30). Double-blind peer review vs. scientific integrity, Everything Hertz [Audio podcast], DOI: 10.17605/OSF.IO/6XS29

We chat about the events that started the replication crisis in psychology and Dorothy Bishop's recent resignation from the Royal Society Links * The resignation blogpost (http://deevybee.blogspot.com/2024/11/why-i-have-resigned-from-royal-society.html) from Dorothy Bishop * The bluesky post (https://bsky.app/profile/sarahwieten.bsky.social/post/3lbtsqc6jcs2z) from Sarah Weiten that asked the question, "If you had to cite an event that opened the "replication crisis" era, what would you point to?" * The "Year of Horrors" paper (https://www.ejwagenmakers.com/2012/Wagenmakers2012Horrors.pdf) from Eric-Jan Wagenmakers Other links Everything Hertz on Bluesky - Dan on Bluesky (https://bsky.app/profile/dsquintana.bsky.social) - James on Bluesky (https://bsky.app/profile/jamesheathers.bsky.social) - Everything Hertz on Bluesky (https://bsky.app/profile/hertzpodcast.bsky.social) Support us on Patreon (https://www.patreon.com/hertzpodcast) and get bonus stuff! $1 per month: A 20% discount on Everything Hertz merchandise, access to the occasional bonus episode, and the the warm feeling you're supporting the show $5 per month or more: All the stuff you get in the one dollar tier PLUS a bonus episode every month Citation Quintana, D. S., & Heathers, J. (2024, Dec 3). 187: What started the replication crisis era?, Everything Hertz [Audio podcast], DOI: 10.17605/OSF.IO/EC7QH

In this episode we chat about a Nordic approach for evaluating the journal quality and how we should be teaching undergraduates to evaluate journal and article quality Links * The Norwegian journal register (https://kanalregister.hkdir.no/en/informasjonsartikler/about-the-norwegian-register) * The Finnish journal register (https://julkaisufoorumi.fi/en/publication-forum) * Episode 22 (https://everythinghertz.com/22), where we played "Pokemon or Cholesterol medication?" Other links Everything Hertz on social media - Dan on twitter (https://www.twitter.com/dsquintana) - James on twitter (https://www.twitter.com/jamesheathers) - Everything Hertz on twitter (https://www.twitter.com/hertzpodcast) - Everything Hertz on Facebook (https://www.facebook.com/everythinghertzpodcast/) Support us on Patreon (https://www.patreon.com/hertzpodcast) and get bonus stuff! $1 per month: A 20% discount on Everything Hertz merchandise, access to the occasional bonus episode, and the the warm feeling you're supporting the show $5 per month or more: All the stuff you get in the one dollar tier PLUS a bonus episode every month Citation Quintana, D. S., & Heathers, J. (2024, Nov 13). 186: Evaluating journal quality, Everything Hertz [Audio podcast], https://doi.org/10.17605/OSF.IO/KB37U

We discuss the recent retraction of a paper that reported the effects of rigour-enhancing practices on replicability. We also cover James' new estimate that 1 out of 7 scientific papers are fake. Links * The story (https://www.nature.com/articles/d41586-024-02907-3) about data integrity concerns in 130 women’s health papers * James' new preprint (https://osf.io/23zcr) with the estimate that 1 out of 7 scientific papers are fake * The retracted paper in Nature Human Behavior (https://www.nature.com/articles/s41562-023-01749-9) by Protzko and coworkers * The Matters Arising article (https://rdcu.be/dVXN8) from Bak-Coleman and Devezer, who initially raised concerns about the paper from Protzko and coworkers. * The Everything Hertz merch store (https://everything-hertz-podcast.creator-spring.com) * The paper (https://journals.sagepub.com/doi/10.1177/0165551507086261) about puns/jokes in paper titles * The "Everything Hertz" paper (https://www.frontiersin.org/journals/physiology/articles/10.3389/fphys.2014.00177/full) from James * Dan's only paper (https://www.biologicalpsychiatryjournal.com/article/S0006-3223(15)00528-4/abstract) with a pun in the title Other links Everything Hertz on social media - Dan on twitter (https://www.twitter.com/dsquintana) - James on twitter (https://www.twitter.com/jamesheathers) - Everything Hertz on twitter (https://www.twitter.com/hertzpodcast) - Everything Hertz on Facebook (https://www.facebook.com/everythinghertzpodcast/) Support us on Patreon (https://www.patreon.com/hertzpodcast) and get bonus stuff! $1 per month: A 20% discount on Everything Hertz merchandise, access to the occasional bonus episode, and the the warm feeling you're supporting the show $5 per month or more: All the stuff you get in the one dollar tier PLUS a bonus episode every month Citation Quintana, D. S., & Heathers, J. (2024, Oct 4). 185: The Retraction, Everything Hertz [Audio podcast], https://doi.org/10.17605/OSF.IO/528SF