AI Post Transformers podcast

Stochastic Weight Averaging: Wider Optima and Better Generalization

0:00
NaN:NaN:NaN
Retroceder 15 segundos
Avanzar 15 segundos
This episode examines "Averaging Weights Leads to Wider Optima and Better Generalization," which proposes Stochastic Weight Averaging (SWA): instead of decaying the learning rate to near zero, keep it high or cyclical and take an equal-weight average of the weights SGD visits. The discussion builds the background step by step. It covers mode connectivity, which suggests good solutions form a connected region rather than isolated pits. It covers Fast Geometric Ensembling, whose snapshots sit on the periphery of that region. And it covers the link between wide minima and generalization, including caveats from Entropy-SGD and from the rescaling argument for ReLU/BatchNorm networks. The episode also contrasts SWA with exponential moving averages and classical Polyak–Juditsky averaging. It walks through the practical recipe: train normally for about three quarters of the budget, continue at a high or cyclical rate, fold weights into a running average once per epoch, then recompute BatchNorm statistics. Listeners get a skeptical framing around three claims (accuracy gains, the width story, and approximating an ensemble at single-model serving cost) along with the question of whether the baseline is fair. Sources: 1. Averaging Weights Leads to Wider Optima and Better Generalization — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018 http://arxiv.org/abs/1803.05407 2. Averaging Weights Leads to Wider Optima and Better Generalization — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (UAI) https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization 3. Acceleration of Stochastic Approximation by Averaging — Boris Polyak, Anatoli Juditsky, 1992 https://scholar.google.com/scholar?q=Acceleration+of+Stochastic+Approximation+by+Averaging 4. Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy without Increasing Inference Time — Mitchell Wortsman, Gabriel Ilharco, et al., 2022 https://scholar.google.com/scholar?q=Model+Soups%3A+Averaging+Weights+of+Multiple+Fine-Tuned+Models+Improves+Accuracy+without+Increasing+Inference+Time 5. Fast Geometric Ensembling / Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs — Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (NeurIPS) https://scholar.google.com/scholar?q=Fast+Geometric+Ensembling+%2F+Loss+Surfaces%2C+Mode+Connectivity%2C+and+Fast+Ensembling+of+DNNs 6. Flat Minima — Sepp Hochreiter, Jürgen Schmidhuber, 1997 https://scholar.google.com/scholar?q=Flat+Minima 7. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang, 2017 (ICLR) https://scholar.google.com/scholar?q=On+Large-Batch+Training+for+Deep+Learning%3A+Generalization+Gap+and+Sharp+Minima 8. Sharp Minima Can Generalize For Deep Nets — Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio, 2017 (ICML) https://scholar.google.com/scholar?q=Sharp+Minima+Can+Generalize+For+Deep+Nets 9. Sharpness-Aware Minimization for Efficiently Improving Generalization — Pierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam Neyshabur, 2021 (ICLR) https://scholar.google.com/scholar?q=Sharpness-Aware+Minimization+for+Efficiently+Improving+Generalization 10. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs — Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson, 2018 https://scholar.google.com/scholar?q=Loss+Surfaces%2C+Mode+Connectivity%2C+and+Fast+Ensembling+of+DNNs 11. Essentially No Barriers in Neural Network Energy Landscape — Felix Draxler, Kambis Veschgin, Manfred Salmhofer, Fred Hamprecht, 2018 https://scholar.google.com/scholar?q=Essentially+No+Barriers+in+Neural+Network+Energy+Landscape 12. Linear Mode Connectivity and the Lottery Ticket Hypothesis — Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, Michael Carbin, 2020 (ICML) https://scholar.google.com/scholar?q=Linear+Mode+Connectivity+and+the+Lottery+Ticket+Hypothesis 13. Git Re-Basin: Merging Models modulo Permutation Symmetries — Samuel Ainsworth, Jonathan Hayase, Siddhartha Srinivasa, 2022 https://scholar.google.com/scholar?q=Git+Re-Basin%3A+Merging+Models+modulo+Permutation+Symmetries 14. SGDR: Stochastic Gradient Descent with Warm Restarts — Ilya Loshchilov, Frank Hutter, 2017 (ICLR) https://scholar.google.com/scholar?q=SGDR%3A+Stochastic+Gradient+Descent+with+Warm+Restarts 15. Snapshot Ensembles: Train 1, Get M for Free — Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John Hopcroft, Kilian Weinberger, 2017 (ICLR) https://scholar.google.com/scholar?q=Snapshot+Ensembles%3A+Train+1%2C+Get+M+for+Free 16. Cyclical Learning Rates for Training Neural Networks — Leslie N. Smith, 2017 (WACV) https://scholar.google.com/scholar?q=Cyclical+Learning+Rates+for+Training+Neural+Networks 17. Don't Decay the Learning Rate, Increase the Batch Size — Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le, 2018 (ICLR) https://scholar.google.com/scholar?q=Don%27t+Decay+the+Learning+Rate%2C+Increase+the+Batch+Size 18. A Simple Baseline for Bayesian Uncertainty in Deep Learning (SWAG) — Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, Andrew Gordon Wilson, 2019 https://scholar.google.com/scholar?q=A+Simple+Baseline+for+Bayesian+Uncertainty+in+Deep+Learning+%28SWAG%29 19. Stochastic Gradient Descent as Approximate Bayesian Inference — Stephan Mandt, Matthew D. Hoffman, David M. Blei, 2017 https://scholar.google.com/scholar?q=Stochastic+Gradient+Descent+as+Approximate+Bayesian+Inference 20. Averaging Weights of Multiple Runs: Lookahead Optimizer and LAWA / Latest Weight Averaging — Kaddour; Zhang et al. (Lookahead); related work on LAWA, 2022-2019 https://scholar.google.com/scholar?q=Averaging+Weights+of+Multiple+Runs%3A+Lookahead+Optimizer+and+LAWA+%2F+Latest+Weight+Averaging 21. Understanding Edge-of-Stability Training Dynamics (Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability) — Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, Ameet Talwalkar, 2021 https://scholar.google.com/scholar?q=Understanding+Edge-of-Stability+Training+Dynamics+%28Gradient+Descent+on+Neural+Networks+Typically+Occurs+at+the+Edge+of+Stability%29 22. Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging — Jean Kaddour, 2022 https://scholar.google.com/scholar?q=Stop+Wasting+My+Time%21+Saving+Days+of+ImageNet+and+BERT+Training+with+Latest+Weight+Averaging 23. Scaling Law with Learning Rate Annealing / WSD schedules (MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies) — Shengding Hu et al., 2024 https://scholar.google.com/scholar?q=Scaling+Law+with+Learning+Rate+Annealing+%2F+WSD+schedules+%28MiniCPM%3A+Unveiling+the+Potential+of+Small+Language+Models+with+Scalable+Training+Strategies%29 Interactive Visualization: Stochastic Weight Averaging: Wider Optima and Better Generalization

Otros episodios de "AI Post Transformers"