AI Post Transformers podcast

On-Policy Self-Distillation Using Majority Vote Instead of Ground Truth

0:00
NaN:NaN:NaN
Retroceder 15 segundos
Avanzar 15 segundos
This episode examines "On-Policy Self-Distillation without Any Supervision," a 2026 preprint that asks whether a model can build its own teacher from agreement among its sampled answers, with no ground-truth solutions. It first lays out the ladder of training methods: supervised fine-tuning, RLVR and GRPO, classic on-policy distillation, and on-policy self-distillation, which still needs gold solutions. It also covers majority voting, as in self-consistency and TTRL. The method samples eight answers per prompt and takes a majority vote. It skips prompts where the vote is weak or unanimous, uses the longest agreeing rollout as the teacher's reference, and trains the student only on rollouts that disagreed with the vote. The discussion stresses caveats: the pseudo-labels matched gold only 86.7 percent of the time on a 64-prompt probe, and small benchmarks like AIME are noisy. It also notes that non-thinking Qwen3 has far more headroom than thinking mode. Listeners get a clear explanation of why a noisy self-generated answer key can still help, and of how far that claim should be trusted. Sources: 1. On-Policy Self-Distillation without Any Supervision — Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian, Di Fu, Nuno Vasconcelos, 2026 http://arxiv.org/abs/2608.06296 2. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD) — Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, Olivier Bachem, 2024 (ICLR) https://scholar.google.com/scholar?q=On-Policy+Distillation+of+Language+Models%3A+Learning+from+Self-Generated+Mistakes+%28GKD%29 3. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Denny Zhou et al., 2022 (ICLR 2023) https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models 4. TTRL: Test-Time Reinforcement Learning — Yuxin Zuo, Kaiyan Zhang, et al. (Tsinghua / Shanghai AI Lab), 2025 https://scholar.google.com/scholar?q=TTRL%3A+Test-Time+Reinforcement+Learning 5. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (OPSD) — Zhao et al. (cited in the U-OPSD paper as Zhao et al., 2026a), 2026 https://scholar.google.com/scholar?q=Self-Distilled+Reasoner%3A+On-Policy+Self-Distillation+for+Large+Language+Models+%28OPSD%29 6. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO) — Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, et al. (DeepSeek-AI), 2024 https://scholar.google.com/scholar?q=DeepSeekMath%3A+Pushing+the+Limits+of+Mathematical+Reasoning+in+Open+Language+Models+%28introduces+GRPO%29 7. Self-Distillation Enables Continual Learning (SDFT) — Idan Shenfeld et al., 2026 https://scholar.google.com/scholar?q=Self-Distillation+Enables+Continual+Learning+%28SDFT%29 8. Reinforcement Learning via Self-Distillation (SDPO) — Jonas Hübotter, Frederike Lübeck, Lejs Deen Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, Andreas Krause, 2026 https://scholar.google.com/scholar?q=Reinforcement+Learning+via+Self-Distillation+%28SDPO%29 9. MiniLLM: Knowledge Distillation of Large Language Models — Yuxian Gu, Li Dong, Furu Wei, Minlie Huang, 2024 https://scholar.google.com/scholar?q=MiniLLM%3A+Knowledge+Distillation+of+Large+Language+Models 10. Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning / Spurious Rewards: Rethinking Training Signals in RLVR — Rulin Shao et al. (Spurious Rewards); related work from Qwen-math RLVR analyses, 2025 https://scholar.google.com/scholar?q=Reinforcement+Learning+with+Verifiable+Rewards+Implicitly+Incentivizes+Correct+Reasoning+%2F+Spurious+Rewards%3A+Rethinking+Training+Signals+in+RLVR 11. Large Language Models Can Self-Improve — Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, Jiawei Han, 2023 https://scholar.google.com/scholar?q=Large+Language+Models+Can+Self-Improve 12. Intuitor / Learning to Reason without External Rewards; RENT: Maximizing entropy of the output as reward — Xuandong Zhao et al.; Mihir Prabhudesai et al., 2025 https://scholar.google.com/scholar?q=Intuitor+%2F+Learning+to+Reason+without+External+Rewards%3B+RENT%3A+Maximizing+entropy+of+the+output+as+reward 13. R-Zero: Self-Evolving Reasoning LLM from Zero Data — Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, et al., 2025 https://scholar.google.com/scholar?q=R-Zero%3A+Self-Evolving+Reasoning+LLM+from+Zero+Data 14. Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning — Junlong Ke, Zichen Wen, Weijia Li, Conghui He, Linfeng Zhang, 2026 https://scholar.google.com/scholar?q=Respecting+Self-Uncertainty+in+On-Policy+Self-Distillation+for+Efficient+LLM+Reasoning 15. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes — Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Yuanheng Zhu, Dongbin Zhao, 2026 https://scholar.google.com/scholar?q=Revisiting+On-Policy+Distillation%3A+Empirical+Failure+Modes+and+Simple+Fixes Interactive Visualization: On-Policy Self-Distillation Using Majority Vote Instead of Ground Truth

Otros episodios de "AI Post Transformers"