Metascience 101 - EP5: "How and Why to Run an Experiment"

IN THIS EPISODE: Professor Heidi Williams, Professor Paul Niehaus, and Matt Clancy walk through academic, non-profit and private sector paths to research, the importance of your surroundings, and how you can find good use-inspired questions.“Metascience 101” is a nine-episode set of interviews that doubles as a crash course in the debates, issues, and ideas driving the modern metascience movement. We investigate why building a genuine “science of science” matters, and how research in metascience is translating into real-world policy changes. Chapters(00:00:00) Introduction(00:01:36) Public service in government(00:09:23) Academia-adjacent jobs(00:14:47) Time in the private sector(00:17:23) Thinking about PhD route(00:23:59) Guiding and advising students in academia(00:25:36) What to do during your PhD(00:28:50) Institutions, surroundings, and peers(00:35:57) Finding use-based questions(00:39:53) Sales pitch to do research This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.macroscience.org

IN THIS EPISODE: Journalist Derek Thompson and economist Eli Dourado investigate the bottlenecks standing in the way of the invention vs. the diffusion of ideas. They discuss whether new ideas are getting harder to find, how to get these new ideas to scale, and how a crisis can spur effective implementation.“Metascience 101” is a nine-episode set of interviews that doubles as a crash course in the debates, issues, and ideas driving the modern metascience movement. We investigate why building a genuine “science of science” matters, and how research in metascience is translating into real-world policy changes. Chapters(00:00:00) Introduction(00:01:44) Are ideas getting harder to find?(00:07:25) Thinking through implementation(00:18:26) Crises spurring effective implementation(00:25:53) Drug research This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.macroscience.org

IN THIS EPISODE: Journalist Dylan Matthews leads a conversation with Open Philanthropy CEO Alexander Berger, Professor Tyler Cowen, and IFP Co-CEO Caleb Watney. Together, they explore the relationship between effective, robust scientific institutions and notions of political legitimacy.“Metascience 101” is a nine-episode set of interviews that doubles as a crash course in the debates, issues, and ideas driving the modern metascience movement. We investigate why building a genuine “science of science” matters, and how research in metascience is translating into real-world policy changes. Chapters(00:00:00) Introduction(00:01:24) Current science research funding(00:10:13) Reason for bureaucratic restrictions(00:14:50) Legibility of science policy to the public(00:25:06) Legible science and curing disease(00:29:07) NIH as vital part of funding infrastructure(00:31:57) Immigration(00:42:55) Scientific clusters and place-based policy(00:46:08) Science powers and immigration(00:49:33) Making science utopian and fun(00:53:12) Science role models(00:55:49) Science policy elsewhere(00:58:17) Overrated / Underrated This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.macroscience.org

IN THIS EPISODE: Journalist Dylan Matthews sits down with Professor Tyler Cowen, Matt Clancy, and Jacob Trefethen to discuss whether there are tensions between accelerating science and safety. With case studies where society has faced this tradeoff between progress in science and safety, they work through strategies we can use to accelerate science safely.“Metascience 101” is a nine-episode set of interviews that doubles as a crash course in the debates, issues, and ideas driving the modern metascience movement. We investigate why building a genuine “science of science” matters, and how research in metascience is translating into real-world policy changes. Chapters(00:00:00) Introduction(00:00:37) Being pro-science while understanding downsides of science(00:08:39) Why does science seem so safe currently?(00:10:21) Indicators of increasing risk(00:13:26) Major changes and “wild times” in history(00:17:21) Inevitability of the path of science?(00:23:54) Feedback loop on safety(00:26:46) Reliance on a hegemon for safety(00:29:33) Nuclear catastrophe chain example(00:32:54) Information security and reputation incentives(00:36:17) Balancing safety and progress(00:39:26) Best regulated area?(00:43:58) Differential technology development(00:46:25) Underrated / Overrated This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.macroscience.org

IN THIS EPISODE: Professor Heidi Williams, Professor Paul Niehaus, Emily Oehlsen, and Jim Savage dive in on a practical “how-to” for experimentation and evaluation in metascience. They discuss how to keep metascience experimentation and evaluation relevant to policymakers.“Metascience 101” is a nine-episode set of interviews that doubles as a crash course in the debates, issues, and ideas driving the modern metascience movement. We investigate why building a genuine “science of science” matters, and how research in metascience is translating into real-world policy changes. Chapters(00:00:00) Introduction(00:00:37) Research on research(00:03:34) Identifying talent, RISE program(00:14:44) Impact evaluation, GiveDirectly(00:17:57) Decision making with impact evaluation(00:31:45) Replications(00:35:11) Existential vs. operational experiments and evaluation(00:42:21) Outcome measurability: benchmarks, surrogates and new evidence(00:52:01) GiveDirectly as example of evidence-based approach(00:55:01) Government evidence-based as a way This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.macroscience.org

IN THIS EPISODE: Stripe Press’s Tamara Winter talks through the broad range of scientific funding institutions with guests Professor Tyler Cowen, Arc Institute Co-Founder Professor Patrick Hsu, and Convergent Research CEO Adam Marblestone. They pay special attention to the renaissance in new, exploratory scientific funding models.“Metascience 101” is a nine-episode set of interviews that doubles as a crash course in the debates, issues, and ideas driving the modern metascience movement. We investigate why building a genuine “science of science” matters, and how research in metascience is translating into real-world policy changes. Chapters(00:00:00) Introduction(00:01:53) Why now?(00:06:16) Focused Research Organizations(00:12:15) Arc Institute(00:16:15) Fast Grants(00:22:26) Convergent project selection(00:27:31) Counterfactuals and when they matter(00:35:43) Type of person for FROs(00:41:29) Is OpenAI an FRO?(00:45:30) Dealing with risk(00:51:55) Underinvested weird science(00:56:13) Government’s role(01:00:20) Reasons for excitement(01:07:12) What’s next?(01:12:32) Get involved This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.macroscience.org

IN THIS EPISODE: Journalist Kelsey Piper interviews Convergent Research CEO Adam Marblestone and Professor Paul Niehaus on the inputs to scientific production. They talk through the funding ecosystem, labor force, the culture of scientific labs, and the search for important questions.“Metascience 101” is a nine-episode set of interviews that doubles as a crash course in the debates, issues, and ideas driving the modern metascience movement. We investigate why building a genuine “science of science” matters, and how research in metascience is translating into real-world policy changes. Chapters(00:00:00) Introduction(00:01:49) Picking scientific questions with a long view for impact(00:06:15) GiveDirectly example(00:08:54) Scientific labor force(00:13:21) The field architect(00:20:03) Indicators on the value of scientific questions(00:23:15) Ideal scientific architecture(00:29:22) Bettering the funding ecosystem(00:42:29) Culture in science(00:47:54) Tradeoff between impact and academic convention(00:53:48) From a ‘doing’ career to a research career(00:57:00) Benefit of a roadmap for communicating broadly This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.macroscience.org

IN THIS EPISODE: OpenPhil CEO Alexander Berger interviews economist Matt Clancy and Stripe co-founder Patrick Collison to talk about whether science itself is slowing down, one of the key motivating concerns in metascience. They look at the challenges of measuring scientific progress, the reasons why progress might be slowing down, and what we might be able to do about it. “Metascience 101” is a nine-episode set of interviews that doubles as a crash course in the debates, issues, and ideas driving the modern metascience movement. We investigate why building a genuine “science of science” matters, and how research in metascience is translating into real-world policy changes. Chapters (00:00:00) Introductions(00:01:25) Per-capita slowing in science(00:05:03) Measuring breakthroughs via inputs(00:09:20) Measuring breakthroughs via outputs(00:12:53) Books and sources(00:14:57) Predictions from these models(00:21:35) After the low hanging fruit era(00:31:30) How much is institutional?(00:45:17) Solutions to these problems(01:01:03) Over- and underrated(01:07:37) Other creative endeavors(01:11:05) Audience comments This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.macroscience.org

IN THIS EPISODE: Journalist Dylan Matthews sits down with economist Heidi Williams and IFP co-founder Caleb Watney to set the scene. They talk about the current state of science in America, what metascience aims to achieve, and what empirical experimentation in metascience is revealing. “Metascience 101” is a nine-episode set of interviews that doubles as a crash course in the debates, issues, and ideas driving the modern metascience movement. We investigate why building a genuine “science of science” matters, and how research in metascience is translating into real-world policy changes. Chapters(00:00:00) Introduction(00:01:36) Science is important(00:06:59) Pharmaceuticals and patents(00:10:31) From PhD to independent scientist(00:14:05) The changing landscape of team science(00:19:52) Immigration of scientists(00:24:00) Solutions to improve science funding(00:36:51) Doing metascience This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.macroscience.org

This week on the Macroscience podcast: an interview with Jon Askonas, a senior fellow at the Foundation for American Innovation and an assistant professor of politics at the Catholic University of America. Inspired in part by “Technological Stagnation is a Choice” — Jon’s terrific review of Vaclav Smil’s Invention and Innovation — I wanted to have Jon on the show to think through the ideological underpinnings of the movement around progress and abundance. We discuss metascience as a technocratic endeavor, the aesthetics of progress, the fundamental irrationality of breakthrough innovations, and why progress needs to bring greater moral force to its visions for the future. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.macroscience.org