Justified Posteriors: The Economics of AI Podcast podcast

Is AI Replacing Programmers or Boosting Them?

0:00
1:04:20
Spol 15 sekunder tilbage
Spol 15 sekunder frem

Justified Posteriors reads “Writing Code vs. Shipping Code” by Mert Demirer, Leon Musolff, and Liyuan Yang

In this week’s episode of Justified Posteriors, we update our beliefs with evidence from an ambitious new paper estimating the impact of AI on software production productivity.

Demirer (friend of the show), Musolff, and Yang combine public GitHub records for over 100,000 developers with confidential Microsoft data to trace the effect of distinct generations of AI coding tools — autocomplete, sync agents, and async agents — on the code production hierarchy: lines of code, files, commits, pull requests, projects, and releases. The main empirical finding is attenuation of the effect of AI at each step. Enormous gains of 1000% productivity increases or more at the top of the chain translate into about a 30% increase in shipped releases. The model they use to explain this result is closely connected to Kremer’s O-ring logic, which regular listeners will recognize as from a few episodes back. Seth likes the spirit of the model, but feels it is overcomplicated for this context, for reasons he explains.

In addition to discussing the data analysis, Andrey and Seth have a good back-and-forth about what we can conclude from it and extrapolate to the economy more generally. The implication that grabbed Seth’s attention is a sentence in this paper’s abstract. Nested in the summary of the careful empirical exercise is an estimated elasticity of substitution of 0.25 between AI and human effort. Big if true! Seth points out the enormous long-run implications of humans being complements to AI in what seems to be the most AI-friendly of tasks: as AI gets cheaper, the human share of income goes up, wages skyrocket, and ultimately AI boosts jobs instead of taking them. That’s a huge real-world hook. Seth and Andrey discuss whether, and if so how much, we update our beliefs in this direction, with Andrey being careful to point out the difficulties of extrapolating from a partial equilibrium elasticity to long-run macro consequences.

This episode is sponsored by Revelio Labs — a great source of labor economics data for academics and firms. Now available on WRDS.

Priors → Posteriors

Prior 1: Does access to AI coding tools boost lines of code written by more than 100%?

* Seth: 95% → 99%. Seth came in confident and left more so. Great to see giant numbers.

* Andrey: 80% → 95%. A high yes, hedged because “which developers” and “which tools” do a lot of work in that sentence.

Prior 2: Does AI boost economic value by 50% or less of the factor by which it boosts lines of code?

* Seth: 95% → 97.5%. I have personally produced a great deal of economically worthless code lately.

* Andrey: 85% → 95%.

Prior 3: Are AI coding tools a gross complement to human labor?

Seth’s answer depends on the level of aggregation:The average normie programmer 20%→20% (unchanged)A human engineering department 33%→40%A software company / open source project 60%→85–90%The economy as a whole 33%→33% (unchanged)

Andrey: 75% complement at the sectoral level, and he’d put it as low as the programming department — because right now the code that comes out is not shippable without substantial human input. Posterior: still a complement, mildly supported. He declines, on the record and repeatedly, to extrapolate to the macroeconomy. No fun!

References

The paper under review

* Mert Demirer, Leon Musolff & Liyuan Yang, “Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools,” NBER Working Paper 35275 (May 2026).

* The authors’ own summary: “Writing code versus shipping code”, VoxEU, June 2026.

Prior work by the same team

* Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng & Tobias Salz, “The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers,” Management Science (2026). 4,867 developers, roughly a 26% increase in completed tasks, larger gains for the less experienced.

Related Research and Prior Episodes

* METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” — the RCT Andrey refers to, in which 16 experienced developers took 19% longer with AI while believing they were 20% faster. arXiv version.

* METR’s own update, “We are Changing our Developer Productivity Experiment Design” (Feb 2026) — developers increasingly refuse to be randomized into working without AI, which biases the estimated speedup downward.

* Michael Kremer, “The O-Ring Theory of Economic Development,” Quarterly Journal of Economics 108(3), 1993. Our episode on it: Weak Links, Strong Predictions: Kremer’s O-Ring at 30.

* Josh Gans & Avi Goldfarb, “O-Ring Automation,” NBER Working Paper 34639 — and our conversation with Avi: Avi Goldfarb on Prediction Machines, O-Ring Tasks, and How AI is Reshaping Economics.

* Aaron Chatterji, Tom Cunningham, David Deming, Zoë Hitzig, Christopher Ong, Carl Shan & Kevin Wadman, “How People Use ChatGPT,” NBER Working Paper 34255 — a reference for Andrey’s point about substitution toward home production. Non-work usage grew to over 70% of messages; computer programming is a small share.

* Our episode on the same phenomenon in a different market: The Economics of Book Slop — more books, not obviously more valuable books. More apps, not obviously more downloads. Rhymes.

* The universal token budget: Alex Imas — Demand Collapse, Bargaining with Machines, and Behavioral AI Economics and Seb Krier on AGI, the Coasean Singularity, and EDM.

Table of Contents

* Introduction and Today’s Paper — [00:00]

* Priors: [02:58]

* The Evidence: Ideal and Possible Experiments— [18:34]

* The Evidence: Adoption Events and the Attenuating Waterfall — [24:56]

* The Evidence: Model and Simulation Results — [42:56]

* Aggregates, App Stores, and Posteriors — [56:00]

Transcript

Introduction and Today’s Paper [00:00]

Seth: The abstract of the paper has the following sentence: blah, blah, blah, based on the results in the model, there’s an estimated elasticity of substitution of 0.25, so high complementarity between AI and human effort, which indicates strong complementarities. Wow. As the people say, big if true. Humans and AI, 0.25 complements. Everybody worried about AI taking all our jobs — wrong. All labor share to 100%.

Welcome to the Justified Posteriors podcast, the podcast that updates beliefs about the economics of AI and technology. I’m Seth Benzell, with a 0% productivity impact on my code writing, as measured by my podcast release schedule, coming to you from the Pocono Mountains of eastern Pennsylvania.

Andrey: And I’m Andrey Fradkin, coming to you from San Francisco, California. Justified Posteriors is sponsored by the fine folks at Revelio Labs, and please do sign up to our podcast and our Substack whenever you get the chance.

Seth: Today we’re talking about a really interesting empirical study investigating the impact of AI tool use — autocomplete, synchronous agents, asynchronous agents — on people’s productivity in writing code. This is that kind of hard empirical data that maybe has the potential to move our beliefs. So I’m cautiously optimistic that I’m going to learn a lot from this one.

Andrey: It’s the big question, in many ways. We have these tools. We’re using them. What do we get out of them? Are we really that much more productive? That is the question on everyone’s mind, especially since so many of these tools are very costly. There are people token maxing under the belief that the more tokens that are used, the more valuable the output will be.

Seth: People are going broke over the tokens. We discussed a universal token budget when Alex Imas was on — or maybe that was with Seb. What are people getting when they’re actually paying for them? The paper we read to look into this is called “Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools,” from Demirer, Leon Musolff, and Liyuan Yang. So, butchered every name, as is common for us.

Andrey: Mert is a co-author of the pod, so very excited to be…



Get full access to Justified Posteriors at empiricrafting.substack.com/subscribe

Flere episoder fra "Justified Posteriors: The Economics of AI Podcast"