Simons Foundation Lectures podcast

ICM 2026 Invited Section Lecture – Philippe Rigollet

0:00
52:47
Rewind 15 seconds
Fast Forward 15 seconds
The Mean-Field Dynamics of TransformersWe develop a mathematical framework that interprets Transformer attention as an interacting particle system and studies its continuum (mean-field) limits. By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows, synchronization models (Kuramoto), and mean-shift clustering. Central to our results is a global clustering phenomenon whereby tokens cluster asymptotically after long metastable states where they are arranged into multiple clusters. We further analyze a tractable equiangular reduction to obtain exact clustering rates, show how commonly used normalization schemes alter contraction speeds, and identify a phase transition for long-context attention. The results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.Uploaded 06-10-26 Learn more about your ad choices. Visit megaphone.fm/adchoices

More episodes from "Simons Foundation Lectures"