Best AI papers explained podcast

Generalization Dynamics of Language Model Pretraining

0:00
22:32
Recuar 15 segundos
Avançar 15 segundos

Large language models often exhibit unexpected behavioral fluctuations during pre-training, repeatedly shifting between memorizing shallow patterns and demonstrating true generalization, a phenomenon termed mode-hopping. This instability occurs independently of standard optimization metrics and persists even after training on trillions of tokens across various model scales. Consequently, tracking these hidden generalization dynamics using targeted evaluation suites enables researchers to identify superior intermediate checkpoints and strategically curate pre-training data to improve reasoning and alignment robustness.

Mais episódios de "Best AI papers explained"