Are AI benchmarks doomed?

AI benchmarks saturate quickly, struggle to capture what we care about, and cost more than ever to build. But are they doomed? Greg Burnham, who leads Epoch's benchmarking team, and Tom Adamczewski, who developed MirrorCode, push back on the pessimism and dig into what the next generation of AI benchmarks could look like.

Daniel Litt is a professor of mathematics at the University of Toronto. He has been a careful observer of AI’s progress toward accelerating mathematical discovery, sometimes skeptical and sometimes enthusiastic. Topics we cover: the hardest problems models can solve today, whether there is convincing evidence that AI is speeding up math research, and what’s missing before AI might have a shot at solving Millennium Prize problems.We also discuss how to measure progress in math, including Epoch AI’s new FrontierMath: Open Problems benchmark which evaluates models on meaningful unsolved math research problems.

Professor Luis Garicano isn’t your usual academic economist. Academically, his theories have heavily influenced how modern economists understand the structure of firms and the labor market. But his influence hasn’t been confined to the ivory towers of academia — Luis spent three years in the EU parliament, seeing first-hand how EU policy gets made. This has given him an unusually grounded view of how institutions actually work.Through this institutional lens, Luis has been keeping an eye on how organizations like the EU have been responding to rapid AI developments — and he’s deeply concerned. In this episode, Luis chats with our co-hosts Andrei Potlogea and Anson Ho about: Why he disagrees with Daron Acemoglu about the macroeconomics of AI and how policy should orient to thisHow AI could disrupt the training ladder for entry-level workers, such that they can’t learn economically valuable skills—with major consequences.Why he thinks the EU AI Act has many major issues, and what he would like the EU to do instead-- Episode links --Youtube: https://youtu.be/L8IRbTab2FkTranscript: https://epoch.ai/epoch-after-hours/luis-garicano-not-so-simple-macroeconomics-of-ai-- Timestamps --0:00:00 – Will AI trigger explosive growth?0:06:26 – Short-run macroeconomic effects0:11:29 – The decline of junior jobs0:20:21 – The missing training ladder0:39:31 – Europe’s AI regulation problem0:52:46 – Who captures AI value?01:08:17 – AI, interest rates & fiscal future

Stanford economist Phil Trammell has been rigorously thinking about the intersection of economic theory and AI (incl. AGI) for over five years, long before the recent surge of interest in large language models. In this episode of Epoch After Hours, Phil Trammell and Epoch AI researcher Anson Ho discuss what economic theory really has to say about the development and impacts of AGI: what current economic models get wrong, the odds of explosive economic growth, what “GDP” actually measures, and much more!-- Episode links --Transcript: https://epoch.ai/epoch-after-hours/economics-of-ai-- Timestamps --00:00 Problems with existing work on the economics of AI10:18 Declining returns to R&D18:28 What real GDP misses26:57 Task-based models & AI automation49:32 The limits of economic theory01:09:11 How to detect an economic singularity01:23:32 Increasing returns to scale-- Credits --Design: Robert SandlerPodcast Production & Editing: Caroline Falkman Olsson & Anson Ho Special thanks to The Producer’s Loft for their support with recording and editing this episode — https://theproducersloft.com/

What will AI progress look like over the next 15 years? Informed by current trends, Epoch AI researchers Jaime Sevilla and Yafah Edelman argue that the default expectation should be wild. They discuss whether AI will solve the Riemann Hypothesis in 5 years, what AI agents will be able to do in 2030, and what happens if we have 100,000 self-improving robots. They also explore what might make progress much faster or slower than they expect. 0:00:00 - Preview0:00:41 - Intro: Does 5× compute scaling continue?0:08:15 - Largest training run in 2030 & what does it imply?0:12:44 - Impact on Software Engineering & other cognitive tasks0:23:27 - Economic impacts near the end of the decade0:31:34 - 2030 bifurcation: Slow down or take off?0:35:49 - Physical vs cognitive automation0:44:37 - Timelines and impact of full cognitive automation1:02:37 - Returns to intelligence1:08:51 - Three cruxes after 2035 (Robots, technology & intelligence)1:16:28 - What happens in 2040?  1:23:16 - Recap: Three eras of forecasting1:37:42 - Closing remarksFor full transcripts of all Epoch After Hours episodes, visit: https://epoch.ai/epoch-after-hours-- Credits --Participants: Jaime Sevilla & Yafah EdelmanDesign: Robert SandlerPodcast Production & Editing: Caroline Falkman Olsson & Anson Ho Special thanks to The Producer’s Loft for their support with recording and editing this episode — https://theproducersloft.com/

When might AI truly transform our world, and what will that transformation look like? Even within our own research team, timelines for AGI differ substantially. In this episode, the two Epoch AI researchers with the longest and the shortest AGI timelines candidly examine the roots of their disagreements. Ege and Matthew dissect each other’s views, and discuss the evidence, intuitions and assumptions that lead to their timelines diverging by factors of two or three for key transformative milestones.The hosts discuss:Their median timelines for specific milestones (like sustained 5%+ GDP growth) that highlight differences between optimistic and cautious AI forecasts.Whether AI-driven transformation will primarily result from superhuman researchers (a "country of geniuses") or widespread automation of everyday cognitive and physical tasks.Moravec's Paradox today: Why practical skills like agency and common sense remain challenging for AI despite advancements in reasoning, and how this affects economic impact.The interplay of hardware scaling, algorithmic breakthroughs, data availability (especially for agentic tasks), and the persistent challenge of transfer learning.Prediction pitfalls and why conventional academic AI forecasting might miss the mark.A world with AGI: moving from totalizing "single AGI" or "utopia vs. doom" narratives to consider economic forces, decentralized agents, and the co-evolution of AI and society.

In our first episode of Epoch After Hours, Ege, Tamay and Jaime dig into what they expect AI to look like by 2030; why economists are underestimating the likelihood of explosive growth; the startling regularity in technological trends like Moore's Law; Moravec’s paradox, and how we might overcome it; and much more!