
Scaling Laws for Precision: Bits, Parameters, and Training Tokens
18/11/2026
0:00
NaN:NaN:NaN
This episode examines "Scaling Laws for Precision," which argues that numeric format belongs in the planning equation alongside model size and training data, rather than being treated as a deployment detail. It reviews the Chinchilla baseline and why overtraining is now common. It then explains the paper's central idea: lower precision acts like shrinking the model, captured by an "effective parameter count" whose gains flatten around six or seven bits. The discussion separates post-training quantization, quantization-aware training and full low-precision training. It also covers the experimental setup of 465 small-scale pretraining runs and the caveat that the fitted constants are brittle, so the functional forms matter more than the values. The first finding it reaches is that post-training quantization hurts more the longer a model has trained, which challenges the push toward 4-bit and the default of 16-bit training. Listeners get a framework for judging precision-versus-size bets in an era of FP8 and FP4 hardware.
Sources:
1. Scaling Laws for Precision — Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, Aditi Raghunathan, 2024
http://arxiv.org/abs/2411.04330
2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020
https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models
3. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, and others (DeepMind), 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29
4. FP8 Formats for Deep Learning — Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, and others (NVIDIA, Arm, Intel), 2022
https://scholar.google.com/scholar?q=FP8+Formats+for+Deep+Learning
5. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58) — Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei, 2024
https://scholar.google.com/scholar?q=The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits+%28BitNet+b1.58%29
6. The case for 4-bit precision: k-bit Inference Scaling Laws — Tim Dettmers, Luke Zettlemoyer, 2023
https://scholar.google.com/scholar?q=The+case+for+4-bit+precision%3A+k-bit+Inference+Scaling+Laws
7. Language models scale reliably with over-training and on downstream tasks — Samir Yitzhak Gadre et al., 2024
https://scholar.google.com/scholar?q=Language+models+scale+reliably+with+over-training+and+on+downstream+tasks
8. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — Nikhil Sardana, Jonathan Frankle, 2023
https://scholar.google.com/scholar?q=Beyond+Chinchilla-Optimal%3A+Accounting+for+Inference+in+Language+Model+Scaling+Laws
9. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits — Shuming Ma et al., 2024
https://scholar.google.com/scholar?q=The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits
10. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh, 2022
https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers
11. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws — Zeyuan Allen-Zhu, Yuanzhi Li, 2024
https://scholar.google.com/scholar?q=Physics+of+Language+Models%3A+Part+3.3%2C+Knowledge+Capacity+Scaling+Laws
12. Pretraining Large Language Models with NVFP4 — NVIDIA, 2025
https://scholar.google.com/scholar?q=Pretraining+Large+Language+Models+with+NVFP4
13. Scaling Laws for Floating Point Quantization Training — Xingwu Sun et al., 2025
https://scholar.google.com/scholar?q=Scaling+Laws+for+Floating+Point+Quantization+Training
14. Scaling Law for Quantization-Aware Training — Mengzhao Chen et al., 2025
https://scholar.google.com/scholar?q=Scaling+Law+for+Quantization-Aware+Training
15. QuaRot / SpinQuant: Outlier-free 4-bit inference via rotations — Saleh Ashkboos et al.; Zechun Liu et al., 2024
https://scholar.google.com/scholar?q=QuaRot+%2F+SpinQuant%3A+Outlier-free+4-bit+inference+via+rotations
Interactive Visualization: Scaling Laws for Precision: Bits, Parameters, and Training Tokens
Otros episodios de "AI Post Transformers"



No te pierdas ningún episodio de “AI Post Transformers”. Síguelo en la aplicación gratuita de GetPodcast.








