Best AI papers explained podcast

Generalization Dynamics of Language Model Pretraining

0:00
22:32
Reculer de 15 secondes
Avancer de 15 secondes

Large language models often exhibit unexpected behavioral fluctuations during pre-training, repeatedly shifting between memorizing shallow patterns and demonstrating true generalization, a phenomenon termed mode-hopping. This instability occurs independently of standard optimization metrics and persists even after training on trillions of tokens across various model scales. Consequently, tracking these hidden generalization dynamics using targeted evaluation suites enables researchers to identify superior intermediate checkpoints and strategically curate pre-training data to improve reasoning and alignment robustness.

D'autres épisodes de "Best AI papers explained"