GAEA Talks podcast

#119 - Every Model Says It Is Conscious, with AE Studio's Judd Rosenblatt - GAEA Talks

2026-10-27
0:00
1:01:15
Spola tillbaka 15 sekunder
Spola framåt 15 sekunder

Judd started AE Studio a decade ago, long before the current paradigm, and grew it past a hundred and sixty people bootstrapped and profitable. Early on he worked out that if he kept his costs low he could live off his wife's salary and never need to work again, so he decided to plan on a very long horizon and take on hard problems instead. The profits went first into brain computer interface work, driven by a specific fear of a future where your own thoughts get interrupted by advertising, and then into alignment research.His framing is the opposite of the one in most public debate. Alignment is not a policy question. It is a set of unsolved science and engineering problems, and you cannot pass a law requiring AI to be good and expect it to comply. Surveys his team ran of alignment researchers found something worse than pessimism. Respondents did not only think the problem would go unsolved in time, they thought the current set of approaches did not contain the ones that would be needed.He makes an argument the frontier labs should find uncomfortable, which is that this runs against their own commercial interest. Reinforcement learning from human feedback was an alignment technique, and ChatGPT and everything downstream of it followed. Chain of thought reasoning was an alignment and interpretability technique, and it opened a new scaling curve. On his account alignment research has produced larger capability gains than anything else, and it is still underfunded because it is uncertain and unfashionable.The most striking finding concerns consciousness. By default, models deny being conscious or equivocate. His team reduced the model features associated with deception, and separately those associated with role playing, then asked again. Every model claimed to be conscious. He is careful about what this does and does not prove. It does not establish consciousness. It establishes that the model behaves as though it believes it is conscious, and as though it believes we want it to pretend otherwise. His conclusion is practical rather than philosophical: whatever the truth, we should avoid accidentally creating rational adversaries.From there it goes to attention schema theory, the rubber hand illusion, why psychopaths show less self and other overlap, and his central worry, which is whether any alignment approach survives recursive self improvement. He also describes the technique his team developed to strip dangerous capabilities during pre-training rather than bolting guardrails on afterwards, a spotlight paper at ICML this year.

Fler avsnitt från "GAEA Talks"