
Stochastic Weight Averaging: Wider Optima and Better Generalization
19/11/2026
0:00
NaN:NaN:NaN
This episode examines "Averaging Weights Leads to Wider Optima and Better Generalization," which proposes Stochastic Weight Averaging (SWA): instead of decaying the learning rate to near zero, keep it high or cyclical and take an equal-weight average of the weights SGD visits. The discussion builds the background step by step. It covers mode connectivity, which suggests good solutions form a connected region rather than isolated pits. It covers Fast Geometric Ensembling, whose snapshots sit on the periphery of that region. And it covers the link between wide minima and generalization, including caveats from Entropy-SGD and from the rescaling argument for ReLU/BatchNorm networks. The episode also contrasts SWA with exponential moving averages and classical Polyak–Juditsky averaging. It walks through the practical recipe: train normally for about three quarters of the budget, continue at a high or cyclical rate, fold weights into a running average once per epoch, then recompute BatchNorm statistics. Listeners get a skeptical framing around three claims (accuracy gains, the width story, and approximating an ensemble at single-model serving cost) along with the question of whether the baseline is fair.
Sources:
1. Averaging Weights Leads to Wider Optima and Better Generalization — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018
http://arxiv.org/abs/1803.05407
2. Averaging Weights Leads to Wider Optima and Better Generalization — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (UAI)
https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization
3. Acceleration of Stochastic Approximation by Averaging — Boris Polyak, Anatoli Juditsky, 1992
https://scholar.google.com/scholar?q=Acceleration+of+Stochastic+Approximation+by+Averaging
4. Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy without Increasing Inference Time — Mitchell Wortsman, Gabriel Ilharco, et al., 2022
https://scholar.google.com/scholar?q=Model+Soups%3A+Averaging+Weights+of+Multiple+Fine-Tuned+Models+Improves+Accuracy+without+Increasing+Inference+Time
5. Fast Geometric Ensembling / Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs — Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (NeurIPS)
https://scholar.google.com/scholar?q=Fast+Geometric+Ensembling+%2F+Loss+Surfaces%2C+Mode+Connectivity%2C+and+Fast+Ensembling+of+DNNs
6. Flat Minima — Sepp Hochreiter, Jürgen Schmidhuber, 1997
https://scholar.google.com/scholar?q=Flat+Minima
7. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang, 2017 (ICLR)
https://scholar.google.com/scholar?q=On+Large-Batch+Training+for+Deep+Learning%3A+Generalization+Gap+and+Sharp+Minima
8. Sharp Minima Can Generalize For Deep Nets — Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio, 2017 (ICML)
https://scholar.google.com/scholar?q=Sharp+Minima+Can+Generalize+For+Deep+Nets
9. Sharpness-Aware Minimization for Efficiently Improving Generalization — Pierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam Neyshabur, 2021 (ICLR)
https://scholar.google.com/scholar?q=Sharpness-Aware+Minimization+for+Efficiently+Improving+Generalization
10. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs — Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson, 2018
https://scholar.google.com/scholar?q=Loss+Surfaces%2C+Mode+Connectivity%2C+and+Fast+Ensembling+of+DNNs
11. Essentially No Barriers in Neural Network Energy Landscape — Felix Draxler, Kambis Veschgin, Manfred Salmhofer, Fred Hamprecht, 2018
https://scholar.google.com/scholar?q=Essentially+No+Barriers+in+Neural+Network+Energy+Landscape
12. Linear Mode Connectivity and the Lottery Ticket Hypothesis — Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, Michael Carbin, 2020 (ICML)
https://scholar.google.com/scholar?q=Linear+Mode+Connectivity+and+the+Lottery+Ticket+Hypothesis
13. Git Re-Basin: Merging Models modulo Permutation Symmetries — Samuel Ainsworth, Jonathan Hayase, Siddhartha Srinivasa, 2022
https://scholar.google.com/scholar?q=Git+Re-Basin%3A+Merging+Models+modulo+Permutation+Symmetries
14. SGDR: Stochastic Gradient Descent with Warm Restarts — Ilya Loshchilov, Frank Hutter, 2017 (ICLR)
https://scholar.google.com/scholar?q=SGDR%3A+Stochastic+Gradient+Descent+with+Warm+Restarts
15. Snapshot Ensembles: Train 1, Get M for Free — Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John Hopcroft, Kilian Weinberger, 2017 (ICLR)
https://scholar.google.com/scholar?q=Snapshot+Ensembles%3A+Train+1%2C+Get+M+for+Free
16. Cyclical Learning Rates for Training Neural Networks — Leslie N. Smith, 2017 (WACV)
https://scholar.google.com/scholar?q=Cyclical+Learning+Rates+for+Training+Neural+Networks
17. Don't Decay the Learning Rate, Increase the Batch Size — Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, Quoc V. Le, 2018 (ICLR)
https://scholar.google.com/scholar?q=Don%27t+Decay+the+Learning+Rate%2C+Increase+the+Batch+Size
18. A Simple Baseline for Bayesian Uncertainty in Deep Learning (SWAG) — Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, Andrew Gordon Wilson, 2019
https://scholar.google.com/scholar?q=A+Simple+Baseline+for+Bayesian+Uncertainty+in+Deep+Learning+%28SWAG%29
19. Stochastic Gradient Descent as Approximate Bayesian Inference — Stephan Mandt, Matthew D. Hoffman, David M. Blei, 2017
https://scholar.google.com/scholar?q=Stochastic+Gradient+Descent+as+Approximate+Bayesian+Inference
20. Averaging Weights of Multiple Runs: Lookahead Optimizer and LAWA / Latest Weight Averaging — Kaddour; Zhang et al. (Lookahead); related work on LAWA, 2022-2019
https://scholar.google.com/scholar?q=Averaging+Weights+of+Multiple+Runs%3A+Lookahead+Optimizer+and+LAWA+%2F+Latest+Weight+Averaging
21. Understanding Edge-of-Stability Training Dynamics (Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability) — Jeremy Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, Ameet Talwalkar, 2021
https://scholar.google.com/scholar?q=Understanding+Edge-of-Stability+Training+Dynamics+%28Gradient+Descent+on+Neural+Networks+Typically+Occurs+at+the+Edge+of+Stability%29
22. Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging — Jean Kaddour, 2022
https://scholar.google.com/scholar?q=Stop+Wasting+My+Time%21+Saving+Days+of+ImageNet+and+BERT+Training+with+Latest+Weight+Averaging
23. Scaling Law with Learning Rate Annealing / WSD schedules (MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies) — Shengding Hu et al., 2024
https://scholar.google.com/scholar?q=Scaling+Law+with+Learning+Rate+Annealing+%2F+WSD+schedules+%28MiniCPM%3A+Unveiling+the+Potential+of+Small+Language+Models+with+Scalable+Training+Strategies%29
Interactive Visualization: Stochastic Weight Averaging: Wider Optima and Better Generalization
Otros episodios de "AI Post Transformers"



No te pierdas ningún episodio de “AI Post Transformers”. Síguelo en la aplicación gratuita de GetPodcast.








