Best AI papers explained podcast

A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets

0:00
24:18
Spol 15 sekunder tilbage
Spol 15 sekunder frem

This paper analyzes policy optimization in reinforcement learning post-training, focusing on how base models, regularization, and clipping affect optimization and sample complexity. They demonstrate that standard methods like REINFORCE can suffer from severe coverage collapse, failing to reach optimality even with high-quality base models. To resolve this, the text introduces Approximate Policy Mirror Descent (APMD), a meta-algorithm that periodically resets the reference policy to maintain response coverage throughout training. Furthermore, the sources examine specific implementations such as PPO-NoClip, PPO-Lite, and squared-loss objectives like KimiPO and Rebel-OnPolicy. Ultimately, the research provides theoretical guarantees showing that applying strong regularization alongside reference resets yields stable, efficient, and coverage-preserving global convergence.

Flere episoder fra "Best AI papers explained"