The Information Bottleneck

Dan Card has spent more than 26 years in cybersecurity as a penetration tester, defender and architect. He helped organizations respond to some of the biggest incidents of the last two decades, from MS Blaster to WannaCry and Log4j. In this episode we ask him what has actually changed now that attackers and defenders both have strong AI models.Dan runs a lab where Claude attacks and ChatGPT defends, and he explains why a swarm of ten thousand agents hitting a defended server mostly gets you blocked faster. AI does help attackers with reconnaissance and research, and it helps defenders work through hundreds of alerts a day. Still, most breaches come from bad passwords and misconfigurations, not from exotic zero days.We also talk about air gaps versus sandboxes, why finding more vulnerabilities doesn’t always mean more risk, prompt injection, robots and autonomous weapons, and the story of how Dan tracked down someone who attacked his website. His main message is that security in the real world should be boring, like a good flight.TopicsSocial engineering, fake job candidates and mail bombing attacksWho the attackers really are, from teenagers to nation statesWhether an AI could escape and spread across the internetAir gaps, sandboxes and the Stuxnet storyHow AI attackers compare to human onesWhy agent swarms are easy to blockVulnerabilities that exist versus vulnerabilities that can be exploitedUsing LLMs to triage security alertsPrompt injection and how much we can trust modelsRobots, drones and physical riskWhy you defend against the brick before the aliensChapters00:00 Intro and Dan’s background01:13 What cybersecurity is and who the attackers are09:36 Can an AI escape and spread?19:31 Air gaps, sandboxes and Stuxnet25:04 How AI is changing attack and defense34:04 Why agent swarms don’t scare defenders42:05 AI for security alerts and trusting models54:57 Crime, deterrence and spending on security1:02:38 When someone attacked Dan’s website1:07:29 Robots and physical risk1:16:28 Start with the brick, not the aliens1:20:08 Wrap upMusic: "Kid Kodi" by Blue Dot Sessions (CC BY-NC 4.0)

Julian Togelius, an NYU professor and co-founder of modl.ai, is living the dream of every kid who grew up as a computer geek and gamer. He’s been doing AI research on video games since 2004, back when people told him it wasn’t real research.So why do most video games still run on behavior trees and dialogue trees when we have LLMs? Julian’s answer is that games are designed not to need AI. Most genres were locked in decades ago, and nobody has figured out how to design a good game around a model that can say anything.We talk about the state of the game industry and why so many developers there hate AI. Julian also explains why learning a brand new game as fast as a person would is a better test of intelligence than most benchmarks, and why he thinks we built the wrong kind of AI by training it to copy us. Near the end he gets into making games with Claude and where video world models like Genie might fit.Timeline0:14 Why games matter for AI5:21 What counts as AI?10:02 Why games barely use modern ML17:36 Why LLM characters break games20:59 The state of the game industry27:09 The AI backlash in games30:41 AI in casual games36:07 AI-made mods and fake players40:42 Games as a test of intelligence47:33 Can GPT-6 beat any game?51:10 Playing alongside AI agents53:36 We invented the wrong AI58:33 Personal assistants and Jeb1:03:22 Making games with AI1:07:50 Video world models1:11:16 Joining Julian’s lab1:13:48 Favorite gamesTopicsWhy games are a good testbed for AIClassic game AI vs machine learningWhy shipped games rarely use neural networksLLM NPCs and game designThe game industry slump and the AI backlashAI in mobile games like Candy CrushAI-generated mods and simulated playersGames as a test of general intelligenceAutomated playtesting and reward hackingDesigning games around AI agents“We invented the wrong AI”Generating games with code, asset models and world modelsMusic: "Kid Kodi" by Blue Dot Sessions (CC BY-NC 4.0)

In this episode, we talked with Daniel Litt, a mathematician at the University of Toronto. In March 2025, Daniel bet that AI would not be able to autonomously produce an Annals of Mathematics-level paper by 2030. He has since conceded. He told us what changed and why. The models became superhuman at a small set of skills, like knowing the entire literature and grinding through technical arguments, and that turned out to be enough to do real math. They are still weak at the fuzzy part of the job, like figuring out what is actually true about an object or which questions are worth asking.We also got into how anyone can check a proof that no human has read yet, and what a Lean formalization does and doesn't guarantee. Daniel thinks prestige and credit in math are dead, and that we will have to judge people through thesis defenses and seminars rather than through PDFs. He explains why he thinks math got Goodharted and why the labs put so much effort into it in the first place. We finished with listener questions, including how to teach kids math now and whether P vs NP might be just one idea away.Topics- Why Daniel conceded his AI math bet- What models are superhuman at in math, and where they fall short- Why AI won't "solve math"- Checking AI-written proofs and the limits of Lean- Why different models keep proving the same results- How Daniel uses AI in his own research- Evaluating mathematicians when prestige and credit are dead- Why math got "Goodharted"- AI labs, race dynamics and publication norms- Teaching kids math, and P vs NP00:00 Intro00:15 The bet Daniel conceded03:23 What AI is good and bad at in math08:13 Will AI solve all of math?11:37 Why we still need human mathematicians13:59 Math is about understanding, not only proof21:26 Axioms, foundations and Lean27:42 How do you check an AI-written proof?31:22 When models cheat36:18 Are all the models proving the same things?39:51 How Daniel uses AI in his own research45:20 "Prestige and credit are dead"49:58 AI risk and social upheaval51:38 Is math special, or did it get Goodharted?57:45 Why the labs care about math59:54 PR, race dynamics and the pause1:06:49 Would Daniel join a lab?1:07:53 Listener questions: can understanding be formalized?1:12:05 Teaching kids math with AI around1:15:38 P vs NP and the Riemann hypothesis1:17:37 Wrap-upMusic: "Kid Kodi" by Blue Dot Sessions (CC BY-NC 4.0)

In this episode of the the information bottleneck podcast, we talked with Andy Hall, a Stanford professor and MTS at Anthropic. The conversation explores the political economy of AGI, analyzing how superintelligence could reshape job markets, wealth redistribution, and public trust in government. Hall presents an optimistic view that hyperintelligent agents could enhance democracy by helping citizens make better political decisions, likening AI's potential impact to the invention of the printing press. The discussion also covers the challenges of regulating open-source models, the origins and mitigation of ideological biases in AI, and the geopolitical implications of the AI race between the US and adversaries like China.List of TopicsThe definition of the political economy of superintelligence and its potential to improve civic decision-making.The future of work, the likelihood of white-collar job loss, and the role of the existing tax system in wealth redistribution.Public skepticism toward AI, the decline of trust in the US government, and the contrast between AI power users and the general public.The tension between personal freedom and collective safety regarding open-source and open-weight AI models.Using AI tools to assist legislators in drafting policy and auditing frontier AI models.The geopolitical AI race, the threat of state-run hacking groups, and the challenges for middle-power countries.Center-left ideological biases in current AI models, their roots in training data and reinforcement learning, and experiments to fine-tune models with alternative viewpoints.The concept of AI constitutions, market competition over model values, and the risks of government intervention.AI-driven astroturfing on social media, ungovernable AI swarms, and the debate over regulatory "kill switches".Timeline00:14 - Introduction of Andy Hall and an overview of the political economy of superintelligence.05:38 - Discussion on the future of work, potential mass unemployment, and the political backlash of job loss during periods of economic growth.11:41 - Analysis of public hostility toward AI, drawing analogies to the regulation of social media.19:26 - The role of existing tax systems in wealth redistribution and the political dynamics of state versus federal taxation.27:31 - The debate over open-source models, balancing the desire for personal AI agents with national cyber risk concerns.33:10 - How AI can aid government bureaucracy, specifically in writing complex legislation and overseeing tech labs.46:25 - The geopolitical implications of AI, the race against China, and the threat of state-run hacking organizations like the Lazarus Group.51:38 - Examination of political biases in AI models, including Hall's lab experiments in fine-tuning models to hold conservative views.1:02:38 - Discussion on model constitutions, the distinction between a "soul document" and a true political constitution, and the regulation of AI values.1:13:19 - The rise of AI astroturfing in politics, botnets, and whether "kill switch" legislation is a practical solution.1:19:40 - Theoretical discussions on the future of automated law enforcement, robot armies, and AI's impact on class mobility.Music- "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

John Platt is a Google Fellow and leads the Applied Science team at Google Research, the group working on climate and science. He's also the person behind SMO and Platt scaling. In this episode, we talk with him about how Google decides which scientific problems are worth its time, and what happens when AI meets real physical systems. A big part of the conversation is contrails. A small share of flights creates most of the warming from contrails, which together add up to roughly 1% of human-caused warming. John explains how his team detects contrails from weather satellites and predicts where they will form. In trials with American Airlines, rerouting flights cut contrail formation by 62% with almost no extra fuel. We also get into ERA, Google's empirical research assistant, and where AI agents actually help scientists. John sees them taking over the drudgery of building models and munging data, while humans keep the taste and the rigor. He shares why he's skeptical of gradient-free optimization, why the bitter lesson runs into energy and data center limits, and why physical AI that makes manufacturing cheaper could matter a lot for climate. We finish with FireSat, a satellite constellation (about 50 satellites at full scale) designed to catch wildfires while they're still small enough to put out.Timeline00:00 Intro01:09 How Google picks science problems: comparative advantage04:15 Projects that didn't work: RNA aptamers and tool/problem fit05:46 DNA, RNA and XNA07:41 Contrails and why they warm the planet13:01 Three AI systems for avoiding contrails15:23 Cloud seeding and why cloud physics is hard18:17 Airline trials: 62% fewer contrails19:46 Navier-Stokes, wind tunnels and where simulation still fails25:07 Will agents replace scientists? ERA and scorable tasks31:36 Overfitting, Kaggle and the half-pixel bug33:41 Agents that exploit bugs: Lean and Goodhart's law36:45 Gradient-free optimization38:53 SMO, SVMs and hinge loss in neural networks42:12 The bitter lesson and the limits of scaling43:36 Where agents will help science in the next 12 months45:13 Robotics and making climate tech cheaper to build48:22 AI risk and misuse50:51 Climate outlook: past 2°C54:39 Why AI won't just solve climate change57:04 Room-temperature superconductors, fusion and HVDC1:02:25 FireSat: detecting wildfires from orbit1:05:16 Why existing satellites aren't enough1:07:53 The economics of 50 satellites1:09:34 False positives1:11:16 ClosingTopicsComparative advantage as the test for which science problems Google takes onContrails: the physics, the AI pipeline, and the American Airlines trialsERA and AI agents for scientific softwareWhat scientists still do that agents can’t: taste, creativity, rigorOverfitting, Goodhart’s law, and agents exploiting bugsGradient-free methods, SMO, and hinge lossThe bitter lesson and the physical limits of scalingClimate outlook, fusion, and physical AIRoom-temperature superconductors and long-distance power transmissionFireSat and early wildfire detectionMusic- "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Yuandong Tian spent many years at Meta FAIR and recently left to co-found Recursive Superintelligence, a company building AI that improves itself. In this episode, he tells us why.We start with his work on computer Go, where he built DarkForest before AlphaGo came out. A few years later came OpenGo, which played Korean professionals on a single GPU and didn't lose a game. He then tried to bring RL to real-world problems and found that the design of the action space mattered more than the algorithm.We also talk about Coconut, his paper on reasoning in latent space instead of tokens, and why frontier models still aren't trained that way.The second half is about recursive self-improvement. Yuandong wrote his last paper at Meta together with GPT-5 and says it made him 6 to 10 times faster. That convinced him his own job could be replaced within five years. We ask him where agents still fall short and whether a new architecture can really beat transformers at scale. He also gives his view on the calls to restrict self-improving AI and makes the case for open source.Recursive is hiring in San Francisco and London: [email protected]* Computer Go: DarkForest, AlphaGo and OpenGo* Gradient-free optimization* Action space design and neural architecture search* Coconut and reasoning in latent space* Understanding how neural networks learn representations* Grokking, and writing a paper with GPT-5* Recursive self-improvement and coding agents* New architectures vs. transformers* The NanoGPT speedrun* Data efficiency and robotics* Restricting self-improving AI* Open source modelsTimeline0:00 Intro0:45 From CMU to deep learning, and the AlexNet debates5:35 AlphaGo and beating Go pros on a single GPU10:27 Gradient-free optimization13:16 Diffusion models and diversity16:23 RL on real problems: why the action space matters20:34 AutoML and architecture search22:05 Coconut and reasoning in latent space26:07 Why latent reasoning hasn't caught on30:36 Adapting research to the LLM era34:04 Why he left Meta to work on recursive self-improvement37:10 Do we still need humans?39:37 Where agents fall short43:08 Can new architectures beat transformers at scale?47:29 The hardest part of research to automate50:16 What comes next for AI52:52 Should self-improving AI be restricted?54:11 Open source models56:43 Hiring at RecursiveMusic- "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Chris Manning is a legend in NLP. He's a professor of linguistics and computer science at Stanford and ran the Stanford AI Lab. His course CS224N, Natural Language Processing with Deep Learning, is where a whole generation of researchers learned the field. If you've worked on anything in NLP over the last 25 years, you've probably used his work.We start with what linguistics gave machine learning that ML wouldn't have figured out on its own, and what's left for NLP researchers now that LLMs handle most of the classic tasks. Chris thinks the open problems have moved up the stack to pragmatics and dialogue. Models are good at using context but still sound confident when they shouldn't.Then we get into his recent work on how language models learn verb classes. Small GPT-2 models seem to form abstract categories right away instead of memorizing verbs one at a time, and Chris explains why distributed representations push them in that direction.The second half is about meaning and representation. Chris makes the case that Yann LeCun underrates the role of language in intelligence, and revisits his debate with Emily Bender over whether text alone can teach meaning. We finish with ReFT, which steers a frozen model by editing its hidden states, and whether concepts really live in linear subspaces.Timeline00:00 Intro01:05 What linguistics gave machine learning03:27 Is NLP losing its focus on language?06:29 What's left for NLP researchers in the LLM era09:37 The next frontier: pragmatics and dialogue13:01 Constructed languages and AI-to-AI communication15:35 Do language models learn categories first?23:43 Why transformers learn abstractions early26:37 Does it hold at scale?28:30 LeCun, JEPA and the role of language in intelligence34:54 Can text alone teach meaning? The octopus debate40:45 Diffusion language models vs transformers43:39 ReFT: editing representations instead of weights50:34 Weights or representations: where knowledge lives52:37 Do concepts really live in linear subspaces?TopicsNLP, computational linguistics, large language models, learning dynamics, meaning from form, world models, JEPA, diffusion LMs, ReFT, representation learning, interpretabilityMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Andrew Dai spent over a decade at Google Brain and DeepMind, where he co-wrote the 2015 paper that introduced language model pre-training followed by fine-tuning, and later co-led pre-training data for Gemini. He's now co-founder and CEO of Elorian AI, which is building models for visual reasoning.We talk about how his pre-training result started as a bug, why next-token prediction scales better than other objectives, and what went wrong for Google in the early LLM race. The second half is about vision: why today's frontier models still can't count objects in a photo, why he thinks reasoning is fundamentally visual, and how his view of world models differs from JEPA.Chapters00:00 Intro00:48 Andrew's background and the accidental discovery of pre-training07:10 Why next-token prediction scales10:09 How Google fell behind and the early days of Gemini18:28 What makes training data good28:16 Where visual understanding breaks down46:01 World models, JEPA and robotics55:27 Generation vs understanding, and what's next for ElorianTopicsPre-training and fine-tuningScaling and next-token predictionGemini and Google's AI historyData quality and synthetic dataVisual reasoning and countingWorld models and JEPAMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Zachary Lipton is an associate professor at Carnegie Mellon University and a co-founder of Abridge, a healthcare AI company (and a jazz saxophonist!). His research spans machine learning, healthcare, and the broader impact of AI.In this episode, we discuss what AI can (and cannot yet) do in healthcare. We talk about why medicine is harder to automate than coding, AI scribes and clinical decision support, drug discovery, whether AI could eventually replace doctors, and the role of open models and intelligent routing.In the second half, we turn to the future of AI research: whether academia can still compete, what AI PhD students should work on, and how Zach thinks about automation and the future of research.Timeline00:00 — Introduction00:28 — Why healthcare is hard for AI05:00 — Zach’s path into healthcare AI22:13 — Where AI can have the biggest impact32:01 — Can AI replace doctors?41:41 — Open models and AI infrastructure56:20 — The future of AI research1:03:03 — What should AI PhD students work on?1:27:00 — Advice for researchers and foundersTopicsAI in healthcare • AI doctors • drug discovery • clinical decision support • open-source AI • AI research • academia vs. industry • future of workMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

What comes after scaling?We talk with Sara Hooker, co-founder and CEO of Adaptation Lab, about why the next generation of AI may look very different from today's static models. Sara argues that models should continuously adapt to new tasks, data, users, and environments—and that doing this efficiently will require rethinking much more than fine-tuning.We discuss continual learning, AutoScientist and automated research, why non-verifiable tasks may become the next major bottleneck, and why interfaces could be as important as the models themselves. We also get into open vs. closed models, distillation and Chinese AI labs, AI regulation and safety, cybersecurity and biorisk, AI companionship, and what may eventually come after Transformers and tokenization.TopicsContinuous learning and adaptive AIFine-tuning, memory, and AutoScientistAI agents and automated researchNon-verifiable tasks and human feedbackAdaptive interfacesOpen vs. closed models and distillationAI safety, regulation, cyber risk, and bioriskAI companionship and persuasionThe limits of TransformersMultilingual models and tokenizationChapters00:00 — Introduction02:15 — Why start another AI lab? The return of research05:46 — What continuous learning actually means12:04 — Should every company have its own adapting model?13:59 — Fine-tuning and platforms like Tinker18:04 — AutoScientist and automated optimization22:52 — Can AI really improve its own research?28:38 — The problem of non-verifiable tasks31:30 — Human feedback and the limits of exponential progress34:43 — Why the AI interface matters40:36 — Distillation, China, and open models49:05 — Open-model licensing52:19 — Will open models catch closed models?58:43 — AI regulation and compute thresholds1:03:07 — AI safety and agent failures1:10:19 — Biorisk vs. cybersecurity1:14:03 — Persuasion, AI companionship, and overlooked risks1:20:41 — Where will AI have the biggest real-world impact?1:25:41 — What is missing from current AI architectures?1:29:03 — Neurosymbolic AI1:31:30 — Multilingual models and tokenization1:34:02 — Byte-level models and alternatives to tokenization1:35:03 — ClosingMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0