Best AI papers explained podcast

A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets

0:00
24:18
Reculer de 15 secondes
Avancer de 15 secondes

This paper analyzes policy optimization in reinforcement learning post-training, focusing on how base models, regularization, and clipping affect optimization and sample complexity. They demonstrate that standard methods like REINFORCE can suffer from severe coverage collapse, failing to reach optimality even with high-quality base models. To resolve this, the text introduces Approximate Policy Mirror Descent (APMD), a meta-algorithm that periodically resets the reference policy to maintain response coverage throughout training. Furthermore, the sources examine specific implementations such as PPO-NoClip, PPO-Lite, and squared-loss objectives like KimiPO and Rebel-OnPolicy. Ultimately, the research provides theoretical guarantees showing that applying strong regularization alongside reference resets yields stable, efficient, and coverage-preserving global convergence.

D'autres épisodes de "Best AI papers explained"