Off-the-Shelf Large Language Models Are Unreliable Judges – Jonathan Choi (USC / WashU)

With the rapid rise of artificial intelligence, large language models (LLMs) are increasingly being considered for tasks once thought to be uniquely human—including legal interpretation. The idea of “AI judges” suggests appealing possibilities: consistent, fast, and ostensibly unbiased answers to legal questions. But how reliable are these models? Can their judgments truly be trusted? And do they withstand careful empirical scrutiny?In this episode of the CLE Vlog Series, Prof. Jonathan Choi (University of Southern California & Washington University, St. Louis) joins Alessandro Tacconelli (ETH Zurich) to discuss his paper, “Off-the-Shelf Large Language Models Are Unreliable Judges.” Prof. Choi presents findings from a series of empirical experiments designed to test how well LLMs perform as legal interpreters. His results reveal that model judgments are highly sensitive to prompt phrasing, output processing methods, and training choices. Moreover, post-training adjustments in today’s most widely used models can push LLMs’ assessments far from empirically grounded predictions of language use. These insights raise serious questions about the credibility of LLMs in legal interpretation and cast doubt on their ability to capture the “ordinary meaning” of legal texts.Paper Reference:Jonathan Choi – University of Southern California / Washington University (St. Louis)Large Language Models Are Unreliable Judgeshttps://papers.ssrn.com/sol3/papers.cfm?abstract_id=5188865Audio Credits for Trailer:AllttA by AllttA https://youtu.be/ZawLOcbQZ2w

In this episode of the CLE Vlog Series, Prof. Murat C. Mungan (Texas A&M University) discusses his paper "Symmetry, Presumptions, and the Judges Design" with Karol Rutkowski (ETH Zurich). In his work, Murat C. Mungan discusses an important assumption underlying a popular empirical method used in social sciences, namely average monotonicity, and shows the conditions under which this assumption can be violated.Paper Reference: Murat C. Mungan - Texas A&M UniversitySymmetry, Presumptions, and the Judges DesignPLOS One, Forthcominghttps://papers.ssrn.com/sol3/papers.cfm?abstract_id=4322684Audio Credits for Trailer:AllttA by AllttA https://youtu.be/ZawLOcbQZ2w

In this episode of the CLE Vlog & Podcast Series, Prof. Berk Ustun (University of California San Diego) discusses his paper "Prediction without Preclusion: Recourse Verification with Reachable Sets" with Benjamin Kohler (ETH Zurich). In their work, Berk Ustun and his co-authors investigate how machine learning models in high-stakes settings, such as lending and hiring, assign "fixed" predictions that individuals cannot change regardless of their actions. They introduce a formal procedure called "recourse verification" to certify whether a model allows for responsiveness or precludes access. In addition, they develop an auditing tool for practitioners to flag models that effectively block access to certain outcomes before they are deployed – a crucial step to promote fairness and transparency in machine-led decision making.Paper Reference: Berk Ustun – University of Southern California, San DiegoAvni Kothari – University of Southern California, San DiegoBogdan Kulynych – Lausanne University Hospital Tsui-Wei Weng – Halıcıoğlu Data Science InstitutePrediction without Preclusion: Recourse Verification with Reachable Setshttps://arxiv.org/abs/2308.12820Audio Credits for Trailer:AllttA by AllttA https://youtu.be/ZawLOcbQZ2w

In this episode of the CLE Vlog & Podcast Series, Mitu Gulati (University of Virginia) and Karol Rutkowski (ETH Zurich) discuss "Swiss Cheese Contracts", a topic drawn from Gulati's new book "Contract Hazards: Lawyers and Their Landmines", co-authored with Stephen J. Choi (NYU) and Robert E. Scott (Columbia University).In the study "Swiss Cheese Contracts: The Costs of Creative Lawyering", Prof. Gulati and his co-authors examine contracting practices in high-yield secured syndicated loans and Liability Management Transactions. They describe how new loopholes are continually discovered and exploited, creating a landscape that increasingly resembles a block of Swiss cheese.References: Stephen J. Choi – New York UniversityMitu Gulati – University of VirginiaRobert E. Scott – Columbia UniversitySwiss Cheese Contracts: The Costs of Creative Lawyeringhttps://papers.ssrn.com/sol3/papers.cfm?abstract_id=5391281Contract Hazards: Lawyers and Their Landmineshttps://academic.oup.com/book/61634Audio Credits for Trailer:AllttA by AllttA https://youtu.be/ZawLOcbQZ2w

In this episode of the CLE Vlog & Podcast Series, Raquel De Haro (ETH Zurich) interviews Anu Bradford (Columbia University) about her recent paper “The False Choice Between Digital Regulation and Innovation.” Prof. Bradford is the Henry L. Moses Professor of Law and International Organization, and a leading scholar on the EU’s regulatory power and digital regulation. She coined the term “Brussels Effect” and is the author of the influential books "The Brussels Effect: How the European Union Rules the World" and "Digital Empires: The Global Battle to Regulate Technology".This discussion addresses the widespread belief that digital regulation inevitably harms innovation, and why that assumption is often wrong. Prof. Bradford explains the real factors that have prevented EU technology companies from thriving. She also provides examples illustrating why digital regulation and innovation are not mutually exclusive and can even be mutually reinforcing under certain circumstances.A thought-provoking conversation for anyone interested in tech policy, digital markets, or the global race to govern big platforms.Paper Reference: Anu Bradford – Columbia UniversityThe False Choice Between Digital Regulation and Innovationhttps://scholarlycommons.law.northwestern.edu/nulr/vol119/iss2/3/Audio Credits for Trailer:AllttA by AllttA https://youtu.be/ZawLOcbQZ2w

In this episode of the CLE Vlog & Podcast Series, Prof. Francesco Parisi (University of Minnesota & University of Bologna) discusses his paper "Rewarding Failure" with Gianmarco Torchetti (ETH Zurich).In their work, Parisi and his co-authors Alice Guerra and Andres Sawicki challenge the one-sided incentive structure of intellectual property law, which rewards only successful innovations while overlooking the social value of failed research. The paper shows how negative results – information about what does not work – can generate significant learning externalities that benefit future research. Yet, under current legal frameworks, these valuable failures often go unshared. Paper Reference: Alice Guerra – University of Bologna Francesco Parisi – University of Minnesota & University of BolognaAndres Sawicki – University of MiamiRewarding FailureUniversity of Miami Legal Studies Research Paper No. 4726758 Minnesota Legal Studies Research Paper No. 24-12https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4726758Audio Credits for Trailer:AllttA by AllttA https://youtu.be/ZawLOcbQZ2w

In this episode of the CLE Vlog & Podcast Series, Prof. Manisha Padi (University of California, Berkeley) discusses her study "The Rise of Nonbanks in Servicing Household Debt" with Karol Rutkowski (ETH Zurich). In their work, Padi and co-authors study the effects of capital regulation focusing on the case of the market for mortgage servicing rights (MSRs) and the Basel III regulation. Using a theoretical model and U.S. credit registry data, the authors analyze the shift in servicing riskier mortgages from banks towards non-bank servicers triggered by Basel III. Their analysis shows the tension between reducing agency conflicts and enhancing investor welfare on one side, and the borrowers' welfare on the other. Paper Reference: The Rise of Nonbanks in Servicing Household DebtOlin Business School Center for Finance & Accounting Research Paper No. 2023/08 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4550175Naser Hamdi – Equifax, Inc. Erica Xuewei Jiang – University of Southern CaliforniaBrittany Almquist Lewis – Washington University in Saint Louis & Indiana University Manisha Padi – University of California, Berkeley Avantika Pal – Washington University in Saint LouisAudio Credits for Trailer: AllttA by AllttA https://youtu.be/ZawLOcbQZ2w

In this episode of the CLE Vlog & Podcast Series, Prof. Kevin Cope (University of Virginia) discusses his paper "An Expert-Sourced Measure of Judicial Ideology" with Benjamin Kohler (ETH Zurich). In his work, Kevin Cope investigates how to measure judicial ideology and how this construct influences legal decisions. He introduces the Jurist-Derived Judicial Ideology Scores (JuDJIS), a new metric based on over 10,000 written evaluations of judges by thousands of jurists collected over three decades. The paper shows that these ideology estimates outperform existing measures in predicting appellate decisions. This innovative dataset opens up new possibilities for empirical research in judicial politics that were previously constrained by limited data.Paper Reference:Kevin Cope – University of VirginiaAn Expert-Sourced Measure of Judicial Ideology, Political Analysis (forthcoming)https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4742254Audio Credits for Trailer:AllttA by AllttAhttps://youtu.be/ZawLOcbQZ2w

In this episode of the CLE Vlog & Podcast Series, Prof. Jacob Goldin (University of Chicago) discusses his paper "Work Requirements and Child Tax Benefits" with Noah Fehr (ETH Zürich). In his work, Jacob Goldin examines the trade-offs between targeting child tax benefits to the most vulnerable families and preserving incentives to work. Alongside his co-authors, Goldin uses administrative tax data and quasi-random variation in eligibility to estimate the labor supply effects of removing work requirements from child tax credits. The findings suggest that eliminating these requirements leads to negligible reductions in maternal labor force participation, challenging the notion that such provisions are essential for maintaining work incentives.Paper Reference: Jacob Goldin et. alWork Requirements and Child Tax Benefitshttps://www.nber.org/papers/w32343Revised in December 2024Audio Credits for Trailer:AllttA by AllttA https://youtu.be/ZawLOcbQZ2w

In this episode of the CLE Vlog & Podcast Series, Daniel Hemel (New York University School of Law) discusses his paper "Law and the New Dynamic Public Finance" with Luca Baltensperger (ETH Zurich).In his work, Prof. Hemel explores how insights from the new dynamic public finance can inform legal policy. This framework challenges conventional wisdom in classical tax theory, offering novel perspectives on taxation, social insurance, and regulatory design. The paper highlights the potential benefits of age-dependent tax schedules, provides fresh justifications for capital taxation, and critiques the structure of retirement savings programs like IRAs and 401(k)s. Paper Reference:Daniel Hemel – New York UniversityLaw and the New Dynamic Public FinanceSSRN Working Paper, March 2025https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5154785Audio Credits for Trailer:AllttA by AllttA https://youtu.be/ZawLOcbQZ2w