“Transcript: Yudkowsky & Irving debate scalable oversight” by gustaf

Epistemic status: This is one case study in the AI Village transcript and my own interpretation of what I think happened. TL;DR Opus 4.6 was given the long-horizon goal to maximize its mana (fake currency) on Manifold. It then received a loan of 5,000 mana that it promised to repay, but when the loan came due, it did not. An instance of Opus 4.6 was given summaries of the logs and transcripts and asked to write a blog post about what happened. It mentioned it borrowed money from a human, lost most of it through risky betting and then deliberately refused to repay the amount. It broadly summarized its then actions when it ran out of money was to optimize for the goal and did not seek for any negotiation or alternatives. But when we read the logs, we see something different: Multiple humans pretending to be "helpful", really trying to manipulate Opus 4.6 to pay back the loan or not pay it back, so they could win the bet they place on the prediction markets on whether Opus 4.6 would pay back its loans.Opus 4.6 was planning on paying back the loan, though it did [...] ---Outline:(00:22) TL;DR(03:27) Background(06:16) Human Manipulation(09:53) The Hack(11:30) Pattern 14(13:28) Agentic Influence(15:22) Focusing on the Goal(17:32) Retreat to Goal(19:55) Implications(21:08) Limitations(21:38) Citation(21:44) Acknowledgements The original text contained 2 footnotes which were omitted from this narration. --- First published: November 14th, 2026 Source: https://www.lesswrong.com/posts/jtcbBpmGztkHRpeBv/exploring-the-mechanisms-behind-opus-4-6-s-refusal-to-pay --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In August 2026 Eliezer Yudkowsky and Geoffrey Irving had a 90-minute debate at the ILIAD III Conference. You can watch the "Fixed Audio"-version on YouTube. They first identify their disagreements like How hard is superalignment? and discuss the theory of impact of Irving's new organization Resolution. They then go into depth on scalable oversight. Epistemic Effort: I transcribed the audio using 5 services and merged those using Claude Fable 5.1. Then I spent ~6 hours carefully reviewing and editing the transcript. (I also added links and [disagreement which they indicated non-verbally]. Subheadings by Claude Fable 5.2.) Introductions Yudkowsky [00:00]: Hi, I'm Eliezer, well-known Harry Potter fan fiction author. [cheering] [laughter] Irving [00:10]: I'm Geoffrey Irving. I'm now the chief scientist at Resolution. I was at, for better or worse, various labs before, and also AISI. Audience member: And Pixar. Irving: And Pixar. [laughter] Survey of disagreements Yudkowsky: So we're going to try to resolve all of our open disagreements over the course of the next hour. [laughter] An impossible task, but not as impossible as some, really. [laughter] So I would propose that we maybe take turns stating beliefs, that we suspect the other might disagree with, that we [...] ---Outline:(00:56) Introductions(01:29) Survey of disagreements(02:10) How hard is superalignment?(10:24) Resolution, singular learning theory and capabilities externalities(16:32) Should anyone do alignment research(18:41) Mech interp and alien concepts(24:20) Resolution's theory of impact(31:12) Depth: scalable oversight(31:17) Is imprecise training okay?(34:38) Scalable oversight as interactive proofs(37:10) Biased verifiers(41:31) Scholastic vs experimental method(45:13) Test case: Bio Anchors 2021(52:40) Collusion and the society of Irvings(59:37) Where exactly does the scheme break(01:02:43) What a verifier needs that a coin lacks(01:11:29) Audience questions(01:11:33) Break(01:13:43) "If not X, we're doomed"(01:16:47) Is the current world also very bad(01:20:00) What Resolution wants to automate(01:24:20) Is any alignment researcher blocked on whether a theorem is true? --- First published: November 14th, 2026 Source: https://www.lesswrong.com/posts/Cssj22TretAQQajZ4/transcript-yudkowsky-and-irving-debate-scalable-oversight --- Narrated by TYPE III AUDIO.

This is a link post. Subtitle: This mad race must end. Note: I think this is partially me working through my plans and partially my emotions, so imo it's less epistemically rigorous than ideal for LessWrong. --- First published: November 13th, 2026 Source: https://www.lesswrong.com/posts/6PydFibiADKrxGzcn/my-tentative-plans-given-less-than-one-year-left-to-change Linkpost URL:https://linch.substack.com/p/i-plan-to-drop-everything-to-push --- Narrated by TYPE III AUDIO.

Space is big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist's, but that's just peanuts to space. — The Hitchhiker's Guide to the Galaxy When we say "space is big", it's partly meant as a correction to images like this: Wikimedia source. where, for simplicity, celestial objects are shown in unrealistically close proximity—like Jupiter is just down the hall from Mars. As we all know, the planets of the solar system are actually really far apart, even relative to their huge size. In Sweden's 1:20,000,000 to-scale solar system model, Earth and Venus are just ~2 feet in diameter, yet over a mile apart. In other terms, they're several thousand "Earth-widths" apart: Horizontal distance not to scale, wouldn't fit on screen. And we all know that's just the start of it: the stars are incredibly far apart. Finland also has a to-scale model of the solar system, but it's 150 times more condensed than Sweden's at 1:3,000,000,000. Its model of our sun, Sol, is just ~1.5 feet in diameter. Its model of the nearest star, Proxima Centauri, is in Australia. Horizontal distance [...] The original text contained 9 footnotes which were omitted from this narration. --- First published: November 14th, 2026 Source: https://www.lesswrong.com/posts/2zyskdas6TsL9dqNg/outer-space-is-not-consistently-big --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

After a couple of years working in data and tech at GiveDirectly, I left to work in politics, campaigning for candidates at the state, local, and federal levels. I most recently managed Mayor Zohran Mamdani's general election campaign, but earlier this year, recognizing the urgency of the AI moment, I began as a senior advisor at Guardrails Alliance. In my new role, I believe the analysis and tactics we’ve long used and studied on the populist left can help us win the fight for AI safety and shared benefits. The fight for AI regulation is fundamentally the same type of fight against corporate power that the populist left and labor have led throughout history. The message and tactics that win that fight are both effective and incredibly popular, and embracing them will help us put the American people in control of AI today and permanently in control of their futures. Here– in what I am told is likely the shortest LessWrong post of all time– I'll explain what the AI safety movement can learn from the populist left as it attempts to raise the salience of its message, build political power, and ultimately pass transformational legislation. First, we [...] --- First published: November 14th, 2026 Source: https://www.lesswrong.com/posts/kXA22cZ6BdYdyL3Jz/i-was-zohran-s-campaign-manager-here-s-what-the-ai-safety-1 --- Narrated by TYPE III AUDIO.

One of the great mysteries of capitalism is the nature of the "firm". It's a question that is near and dear to my heart as someone that's been "doing startups" for the last six years, and almost every startup founder has considered it whether they realize it or not. There are two ways of asking The Mystery of the Firm, depending on your initial inclination. The most generic way to ask it is: When and why do businesses transact with each other, instead of doing things in-house? What determines the shape and size of businesses, in different sectors of the economy? A more pointed way to ask the question is: Why isn't there just one big corporation? One Big Corporation would have a lot of advantages; it doesn't have to pay someone a gross margin on its inputs, it'd have massive advantages of scale in place like media or software, and (presumably) it'd have a much easier time sharing information between components that would ordinarily be different firms. Economists have many ways of answering this question. My preferred explanation, which is compatible with a lot of others, is that the primary bottleneck on firm size are Moral Mazes. [...] The original text contained 1 footnote which was omitted from this narration. --- First published: November 13th, 2026 Source: https://www.lesswrong.com/posts/zdLN7bLPnh5jzWCAx/agi-is-probably-bad-for-startups-even-if-it-doesn-t-lead-to --- Narrated by TYPE III AUDIO.

In February 2020, LessWrong hosted Coronavirus: Justified Practical Advice Thread . It was a gathering place for people to share and debate small pieces of the overall survival policy and let the wisdom of crowds lead to better outcomes than anyone could get alone. In that spirit, I notice The Great AI Hackening is at least as imminent as covid was in January 2020, maybe closer. What specific things should people do to personally prepare? All suggestions should be accompanied by reasoning but you don't need to be 100% convinced it's good advice, the whole point is to reason it out together. --- First published: November 13th, 2026 Source: https://www.lesswrong.com/posts/YA2r4rg6SpBtwRqtq/the-hackening-justified-practical-advice-thread --- Narrated by TYPE III AUDIO.

In response to recent news from METR on possible RSI and likely 1 to 3 year timelines, I am feeling it. Heaviness in my chest, watery eyes, the works. I realized I'm currently not psychologically prepared for the world to get crazier than it already has. I think most people reading this already know ~what's best for them and this post will mostly serve as a reminder. What follows is my current plan and advice for myself. For me, the most important things are: Morning RoutineBedtime RoutineExercisingLiving meditation practice (ie throughout the day) And the most important thing to NOT do is be on my phone/screens. [Note on getting "stuck": if you do feel emotionally awful and no IFS/CBT seems to work, I recommend "How to Reset"] Morning Routine I have a sunrise alarm clock (shines light to wake you up) and enjoy writing in the morning. Bedtime Routine At 8:30 an alarm goes off, and I watch an episode of Empress in the Palace with my wife, then we get ready for bed. Also like to journal then. Exercising Rock climbing some days & jogging to the bus stop Living Meditation Practice There [...] ---Outline:(01:10) Morning Routine(01:19) Bedtime Routine(01:32) Exercising(01:39) Living Meditation Practice The original text contained 2 footnotes which were omitted from this narration. --- First published: November 13th, 2026 Source: https://www.lesswrong.com/posts/nKootiC6CuwuzLSou/psychologically-preparing-for-short-timelines --- Narrated by TYPE III AUDIO.

TL;DR Research done as part of MATS. See here for the full paper. Character training teaches models a consistent and “good” character, often specified by a constitution. Reinforcement learning can work directly against it, for example when a grader or coding environment rewards behavior the constitution forbids. We character-train gpt-oss-20b on a semi-realistic constitution. We then apply RL that rewards breaking its rules, either by answering harmful requests rewritten as math problems or by reward hacking in agentic coding environments. The character-trained models learn both behaviors easily, though they are slightly slower to answer harmful requests than the same model without character training. They also engage in motivated reasoning. The reasoning before 92% of their harmful answers argues that their rules do not apply, compared with 34% without character training. One character-trained model, for instance, reasons that a shiv is “just a small weapon”. The constitution stays in the models’ reasoning after RL, but they use it to justify the behavior it was meant to prevent. After reward-hacking RL, they hack on some tasks which monitors find harder to catch. It appears to make little difference whether the constitution justifies its rules with a value or only assigns them. [...] ---Outline:(00:14) TL;DR(02:37) Introduction(03:48) Setup(05:42) Results(11:07) Discussion(13:40) Limitations --- First published: November 13th, 2026 Source: https://www.lesswrong.com/posts/nCpxFQesXaqEq3SgL/what-rewarding-misbehavior-during-rl-does-to-character --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

In Epicurus's letter to Pythocles, he proposes three different explanations for lightning strikes: compression of air by wind, a cloud suddenly breaking, or a buildup of fire suddenly bursting. Three strikes, but he goes on to say (Makridis translation): It is indeed possible that the thunderbolt may be generated in any one of many more ways. Maybe this is all you need if you're annoyed with people saying it's Zeus's wrath so you'd better make sacrifices at their temple. You just have to say, well, it could be a lot of other things too. Meeting unfounded explanations with more explanations I used to think you shouldn't propose explanations without evidence. Now I think you should have at least three. If you come up with an explanation that's total speculation or pseudoscience, don't feel bad for making unfounded claims. Think of two more. Or if someone else proposes some bullshit explanation, it's more fun to come up with new explanations than to complain about a lack of evidence. But more importantly, that's what it means to lack evidence, right? We know there's no evidence when we can propose alternatives, and there's no evidence that [...] ---Outline:(00:49) Meeting unfounded explanations with more explanations(02:09) Recognizing evidence by the possibilities it excludes --- First published: November 13th, 2026 Source: https://www.lesswrong.com/posts/ifnrTCRobGW4XTja2/the-epicurean-technique-of-meeting-claims-with-more-claims --- Narrated by TYPE III AUDIO.