#18 Nathan Labenz on reinforcement learning, reasoning models, emergent misalignment & more

For episode 22 of Consistently Candid, I chatted with Andy Masley. Andy is the former director of Effective Altruism DC, and now writes full-time on his Substack, Twitter, and website about EA, AI, policy, and many other topics! (Note: There were some lighting issues on my end for part of this episode, which I have somewhat clumsily attempted to correct with Claude. Sorry about that!).Topics covered:How Andy went  viral discussing AI’s water useTalking about AI risk with climate-minded peopleAndy’s p(doom) and uncertaintyEA’s bad press and the Trump administration’s attacksDating culture and professionalism in EACan AI really think?Why people resist anthropomorphising AIAnimal welfare and moral circlesAI welfare vs. AI safetyIs current AI conscious?Doing better EA communication 

After a long hiatus, I am resurrecting my podcast! For this episode, I spoke with Steven Adler. Steven worked on policy and safety at OpenAI between 2020 and 2024. He has since left to pursue independent writing to raise public awareness about AI risks and co-founded Guidelight AI, an organisation focused on scrutinising and improving safety practices at major AI companies. Guidelight was one of two organisations instrumental in spearheading the Pacing the Frontier open letter, which has garnered 1300+ signatories from employees at frontier labs. It has also published a scorecard of AI control practices across OpenAI, Anthropic, Meta, Google, and xAI.Topics covered in the episode:Why Steven left OpenAI in 2024 — o1, NDAs, safety staff turnoverCounterarguments to AI pessimism, and where Steven's cruxes lieThe OpenAI–Hugging Face incident and OpenAI's responseAI control: monitoring, prevention, and whether it scales to superintelligenceWhy incident reporting is inadequate, and communicating risk when nothing visibly bad has happenedWhere AI policy stands: SB 53, RAISE, Illinois, the EU AI Act, and how weak enforcement isCompanies quietly diluting their own safety commitmentsThe Pacing the Frontier letter, what "pacing" means and how it's landingGuidelight's control scorecard: method, findings, and theory of changeLinks:Follow Steven on Twitter and subscribe to his SubstackThe Pacing the Frontier open letterRead Guidelight's AI control scorecard

In this episode, I chatted with Frances Lorenz, events associate at the Centre for Effective Altruism. We covered our respective paths into AI safety, the emotional impact of learning about x-risk, what it's like to be female in a male-dominated community and more!Follow Frances on TwitterSubscribe to her SubstackApply for EAG London!

Gabe Alfour is a co-founder of Conjecture and an advisor to Control AI, both organisations working to reduce risks from advanced AI. We discussed why AI poses an existential risk to humanity, what makes this problem very hard to solve, why Gabe believes we need to prevent the development of superintelligence for at least the next two decades, and more. Follow Gabe on TwitterRead The Compendium and A Narrow Path

A lot has happened in AI since the last time I spoke to Nathan Labenz of The Cognitive Revolution, so I invited him back on for a whistlestop tour of the most important developments we've seen over the last year!We covered reasoning models, DeepSeek, the many spooky alignment failures we've observed in the last few months & much more!Follow Nathan on TwitterListen to The Cognitive Revolution My Twitter & Substack 

The Fun Theory Sequence is one of Eliezer Yudkowsky's cheerier works, and considers questions such as 'how much fun is there in the universe?', 'are we having fun yet' and 'could we be having more fun?'. It tries to answer some of the philosophical quandries we might encounter when envisioning a post-AGI utopia. In this episode, I discussed Fun Theory with Noah Topper, who loyal listeners will remember from episode 7, in which we tackled EY's equally interesting but less fun essay, A List of Lethalities. Follow Noah on Twitter and check out his Substack!

John Sherman is the host of the For Humanity Podcast, which (much like this one!) aims to explain AI safety to a non-expert audience. In this episode, we compared our experiences of encountering AI safety arguments for the first time and the psychological experience of being aware of x-risk, as well as what messaging strategies the AI safety community should be using to engage more people. Listen & subscribe to the For Humanity Podcast on YouTube and follow John on Twitter!

Buck Shlegeris is the CEO of Redwood Research, a non-profit working to reduce risks from powerful AI. We discussed Redwood's research into AI control, why we shouldn't feel confident that witnessing an AI escape attempt would persuade labs to undeploy dangerous models, lessons from the vetoing of SB1047, the importance of lab security and more. Posts discussed:The case for ensuring that powerful AIs are controlledWould catching your AIs trying to escape convince AI developers to slow down or undeploy?You can, in fact, bamboozle an unaligned AI into sparing your lifeFollow Buck on Twitter and subscribe to his Substack!

In this episode, Aaron Bergman and Max Alexander are back to battle it out for the philosophy crown, while I (attempt to) moderate. They discuss the Very Repugnant Conclusion, which, in the words of Claude, "posits that a world with a vast population living lives barely worth living could be considered ethically inferior to a world with an even larger population, where most people have extremely high quality lives, but a significant minority endure extreme suffering." Listen to the end to hear my uninformed opinion on who's right.Read Aaron's blog post on suffering-focused utilitarianism Follow Aaron on Twitter Follow Max on TwitterMy Twitter

Deger Turan is the CEO of forecasting platform Metaculus and president of the AI Objectives Institute. In this episode, we discuss how forecasting can be used to help humanity coordinate around reducing existential risks, Deger's advice for aspiring forecasters, the future of using AI for forecasting and more!Enter Metaculus's Q3 AI Forecasting Benchmark TournamentGet in touch with Deger: [email protected]