AI Post Transformers podcast

Scaling Inference Demand into Chinchilla: Training Smaller Models Longer

0:00
NaN:NaN:NaN
Retroceder 15 segundos
Avanzar 15 segundos
This episode examines "Beyond Chinchilla-Optimal," a paper from Databricks MosaicML that adds inference demand to LLM scaling laws. Chinchilla sizing counts only training compute, which is why it lands on about 20 tokens per parameter. The paper instead fixes a target loss and minimizes training plus lifetime inference cost. Heavy expected demand then favors a smaller model trained far longer, as in Llama 3 8B at roughly 1,900 tokens per parameter. The discussion covers the numerical solution and the FLOPs-only savings, such as about 16% for a 30B-quality model serving five trillion tokens. It then moves to dollar costs, where inference hardware utilization is far lower than training utilization, so the savings are larger. It also covers the paper's sweep of 47 trained models, which tests whether small models keep improving at extreme data ratios, and its refit of the Chinchilla coefficients. The hosts flag a weakness: the extreme end of the sweep rests mostly on one small model, while the recommendations are meant for much larger ones. Listeners get a clear explanation of why over-training can be economically rational, and a skeptical look at how well the evidence supports it. Sources: 1. Scaling Inference Demand into Chinchilla: Training Smaller Models Longer https://raw.githubusercontent.com/mlresearch/v235/main/assets/sardana24a/sardana24a.pdf 2. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. (DeepMind), 2022 https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models 3. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, et al. (OpenAI), 2020 https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models 4. LLaMA: Open and Efficient Foundation Language Models — Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. (Meta AI), 2023 https://scholar.google.com/scholar?q=LLaMA%3A+Open+and+Efficient+Foundation+Language+Models 5. Scaling Data-Constrained Language Models — Niklas Muennighoff, Alexander Rush, Boaz Barak, et al., 2023 https://scholar.google.com/scholar?q=Scaling+Data-Constrained+Language+Models 6. Llama 2: Open Foundation and Fine-Tuned Chat Models / The Llama 3 Herd of Models — Hugo Touvron et al.; Llama Team, AI@Meta, 2023 / 2024 https://scholar.google.com/scholar?q=Llama+2%3A+Open+Foundation+and+Fine-Tuned+Chat+Models+%2F+The+Llama+3+Herd+of+Models 7. Scaling Laws for Precision — Tanishq Kumar, Zachary Ankner, Benjamin Spector, et al., 2024 https://scholar.google.com/scholar?q=Scaling+Laws+for+Precision 8. Resolving Discrepancies in Compute-Optimal Scaling of Language Models — Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, Yair Carmon, 2024 https://scholar.google.com/scholar?q=Resolving+Discrepancies+in+Compute-Optimal+Scaling+of+Language+Models 9. Efficiently Scaling Transformer Inference — Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, et al. (Google), 2022 https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference 10. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023 https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention 11. Llama 2 / Chinchilla-style compute accounting: 'Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws' — Nikhil Sardana, Jacob Portes, Sasha Doubov, Jonathan Frankle, 2024 https://scholar.google.com/scholar?q=Llama+2+%2F+Chinchilla-style+compute+accounting%3A+%27Beyond+Chinchilla-Optimal%3A+Accounting+for+Inference+in+Language+Model+Scaling+Laws%27 12. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, et al. (Microsoft), 2024 https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting 13. Language models scale reliably with over-training and on downstream tasks — Samir Yitzhak Gadre et al., 2024 https://scholar.google.com/scholar?q=Language+models+scale+reliably+with+over-training+and+on+downstream+tasks 14. Chinchilla Scaling: A replication attempt — Tamay Besiroglu, Ege Erdil, Matthew Barnett, Josh You, 2024 https://scholar.google.com/scholar?q=Chinchilla+Scaling%3A+A+replication+attempt 15. Go smol or go home — Harm de Vries, 2023 https://scholar.google.com/scholar?q=Go+smol+or+go+home 16. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations / Predictable scale: Part I (hyperparameter scaling) — Hägele et al.; Yao et al., 2024 https://scholar.google.com/scholar?q=Scaling+Laws+and+Compute-Optimal+Training+Beyond+Fixed+Training+Durations+%2F+Predictable+scale%3A+Part+I+%28hyperparameter+scaling%29 17. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann et al., 2022 https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29 18. Scaling Laws for Fine-Grained Mixture of Experts — Jakub Krajewski et al., 2024 https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts 19. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar, 2024 https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+can+be+More+Effective+than+Scaling+Model+Parameters 20. Mixture-of-Experts scaling and inference cost analyses, including DeepSeek-V2/V3 reports — DeepSeek-AI, 2024 https://scholar.google.com/scholar?q=Mixture-of-Experts+scaling+and+inference+cost+analyses%2C+including+DeepSeek-V2%2FV3+reports Interactive Visualization: Scaling Inference Demand into Chinchilla: Training Smaller Models Longer

Otros episodios de "AI Post Transformers"