AI Post Transformers podcast

Scaling Laws for Precision: Bits, Parameters, and Training Tokens

0:00
NaN:NaN:NaN
Retroceder 15 segundos
Avanzar 15 segundos
This episode examines "Scaling Laws for Precision," which argues that numeric format belongs in the planning equation alongside model size and training data, rather than being treated as a deployment detail. It reviews the Chinchilla baseline and why overtraining is now common. It then explains the paper's central idea: lower precision acts like shrinking the model, captured by an "effective parameter count" whose gains flatten around six or seven bits. The discussion separates post-training quantization, quantization-aware training and full low-precision training. It also covers the experimental setup of 465 small-scale pretraining runs and the caveat that the fitted constants are brittle, so the functional forms matter more than the values. The first finding it reaches is that post-training quantization hurts more the longer a model has trained, which challenges the push toward 4-bit and the default of 16-bit training. Listeners get a framework for judging precision-versus-size bets in an era of FP8 and FP4 hardware. Sources: 1. Scaling Laws for Precision — Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, Aditi Raghunathan, 2024 http://arxiv.org/abs/2411.04330 2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020 https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models 3. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, and others (DeepMind), 2022 https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29 4. FP8 Formats for Deep Learning — Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, and others (NVIDIA, Arm, Intel), 2022 https://scholar.google.com/scholar?q=FP8+Formats+for+Deep+Learning 5. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58) — Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei, 2024 https://scholar.google.com/scholar?q=The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits+%28BitNet+b1.58%29 6. The case for 4-bit precision: k-bit Inference Scaling Laws — Tim Dettmers, Luke Zettlemoyer, 2023 https://scholar.google.com/scholar?q=The+case+for+4-bit+precision%3A+k-bit+Inference+Scaling+Laws 7. Language models scale reliably with over-training and on downstream tasks — Samir Yitzhak Gadre et al., 2024 https://scholar.google.com/scholar?q=Language+models+scale+reliably+with+over-training+and+on+downstream+tasks 8. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — Nikhil Sardana, Jonathan Frankle, 2023 https://scholar.google.com/scholar?q=Beyond+Chinchilla-Optimal%3A+Accounting+for+Inference+in+Language+Model+Scaling+Laws 9. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits — Shuming Ma et al., 2024 https://scholar.google.com/scholar?q=The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits 10. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022 https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers 11. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — Zeyuan Allen-Zhu, Yuanzhi Li, 2024 https://scholar.google.com/scholar?q=Physics+of+Language+Models%3A+Part+3.3%2C+Knowledge+Capacity+Scaling+Laws 12. Pretraining Large Language Models with NVFP4 — NVIDIA, 2025 https://scholar.google.com/scholar?q=Pretraining+Large+Language+Models+with+NVFP4 13. Scaling Laws for Floating Point Quantization Training — Xingwu Sun et al., 2025 https://scholar.google.com/scholar?q=Scaling+Laws+for+Floating+Point+Quantization+Training 14. Scaling Law for Quantization-Aware Training — Mengzhao Chen et al., 2025 https://scholar.google.com/scholar?q=Scaling+Law+for+Quantization-Aware+Training 15. QuaRot / SpinQuant: Outlier-free 4-bit inference via rotations — Saleh Ashkboos et al.; Zechun Liu et al., 2024 https://scholar.google.com/scholar?q=QuaRot+%2F+SpinQuant%3A+Outlier-free+4-bit+inference+via+rotations Interactive Visualization: Scaling Laws for Precision: Bits, Parameters, and Training Tokens

Otros episodios de "AI Post Transformers"