AI Post Transformers podcast

4-bit Precision Beats 8-bit: k-bit Inference Scaling Laws

0:00
NaN:NaN:NaN
Retroceder 15 segundos
Avanzar 15 segundos
This episode examines the paper "The case for 4-bit precision: k-bit Inference Scaling Laws," which asks, for a fixed total number of model bits, which weight precision gives the best zero-shot accuracy. It opens with a puzzle: a 30-billion-parameter model stored in 8 bits versus a 60-billion-parameter model stored in 4 bits. The paper finds the 4-bit model wins across almost every family tested. The discussion builds the background step by step. It covers why bits are the right unit when small-batch generation is memory-bound, and how inference scaling laws differ from training laws. It also separates "zero-shot" evaluation from "zero-shot" quantization. The episode then walks through the knobs studied across more than 35,000 experiments on OPT, BLOOM, Pythia and GPT-2 models from 19 million to 176 billion parameters. Those knobs are block-wise normalization, data types such as integer, float, quantile and dynamic exponent, and a proxy-based method that keeps outlier dimensions in 16-bit. Listeners get a clear explanation of the trade-offs, along with a skeptical look at how far "4-bit is optimal" reaches given the specific methods and bit accounting used. Sources: 1. 4-bit Precision Beats 8-bit: k-bit Inference Scaling Laws https://arxiv.org/pdf/2212.09720 2. Language Models are Unsupervised Multitask Learners — Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019 https://scholar.google.com/scholar?q=Language+Models+are+Unsupervised+Multitask+Learners 3. Language Models are Few-Shot Learners — Tom B. Brown et al. (OpenAI), 2020 https://scholar.google.com/scholar?q=Language+Models+are+Few-Shot+Learners 4. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, et al., 2020 https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models 5. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale — Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, 2022 https://scholar.google.com/scholar?q=LLM.int8%28%29%3A+8-bit+Matrix+Multiplication+for+Transformers+at+Scale 6. Zero-Shot Learning with Semantic Output Codes / Learning to Detect Unseen Object Classes by Between-Class Attribute Transfer — Mark Palatucci et al. (2009); Christoph Lampert, Hannes Nickisch, Stefan Harmeling (2009), 2009 https://scholar.google.com/scholar?q=Zero-Shot+Learning+with+Semantic+Output+Codes+%2F+Learning+to+Detect+Unseen+Object+Classes+by+Between-Class+Attribute+Transfer 7. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — Frantar, Ashkboos, Hoefler, Alistarh, 2022 https://scholar.google.com/scholar?q=GPTQ%3A+Accurate+Post-Training+Quantization+for+Generative+Pre-trained+Transformers 8. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — Lin, Tang, Tang, Yang, Dang, Han, 2023 https://scholar.google.com/scholar?q=AWQ%3A+Activation-aware+Weight+Quantization+for+LLM+Compression+and+Acceleration 9. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models — Xiao, Lin, Seznec, Demouth, Han, 2022 https://scholar.google.com/scholar?q=SmoothQuant%3A+Accurate+and+Efficient+Post-Training+Quantization+for+Large+Language+Models 10. QLoRA: Efficient Finetuning of Quantized LLMs (NF4 data type) — Dettmers, Pagnoni, Holtzman, Zettlemoyer, 2023 https://scholar.google.com/scholar?q=QLoRA%3A+Efficient+Finetuning+of+Quantized+LLMs+%28NF4+data+type%29 11. QuIP / QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks — Chee, Cai, Kuleshov, De Sa; Tseng et al., 2023-2024 https://scholar.google.com/scholar?q=QuIP+%2F+QuIP%23%3A+Even+Better+LLM+Quantization+with+Hadamard+Incoherence+and+Lattice+Codebooks 12. AQLM: Extreme Compression of Large Language Models via Additive Quantization — Egiazarian et al., 2024 https://scholar.google.com/scholar?q=AQLM%3A+Extreme+Compression+of+Large+Language+Models+via+Additive+Quantization 13. The case for 4-bit precision follow-ups: 'How Good Are Low-bit Quantized LLaMA3 Models?' / 'A Comprehensive Evaluation of Quantization Strategies for LLMs' — Huang et al.; Li et al., 2024 https://scholar.google.com/scholar?q=The+case+for+4-bit+precision+follow-ups%3A+%27How+Good+Are+Low-bit+Quantized+LLaMA3+Models%3F%27+%2F+%27A+Comprehensive+Evaluation+of+Quantization+Strategies+for+LLMs%27 14. Scaling Laws for Precision — Kumar, Ankner, Spector, Bordelon, Muennighoff, Paul, Pehlevan, Ré, Raghunathan, 2024 https://scholar.google.com/scholar?q=Scaling+Laws+for+Precision 15. Efficiently Scaling Transformer Inference — Pope et al., 2022 https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference 16. Training Compute-Optimal Large Language Models (Chinchilla) — Hoffmann et al., 2022 https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29 17. FP8 Formats for Deep Learning — Micikevicius et al., 2022 https://scholar.google.com/scholar?q=FP8+Formats+for+Deep+Learning 18. KIVI / KVQuant: KV-cache quantization — Liu et al.; Hooper et al., 2024 https://scholar.google.com/scholar?q=KIVI+%2F+KVQuant%3A+KV-cache+quantization Interactive Visualization: 4-bit Precision Beats 8-bit: k-bit Inference Scaling Laws

Otros episodios de "AI Post Transformers"