
Scaling Inference Demand into Chinchilla: Training Smaller Models Longer
18/11/2026
0:00
NaN:NaN:NaN
This episode examines "Beyond Chinchilla-Optimal," a paper from Databricks MosaicML that adds inference demand to LLM scaling laws. Chinchilla sizing counts only training compute, which is why it lands on about 20 tokens per parameter. The paper instead fixes a target loss and minimizes training plus lifetime inference cost. Heavy expected demand then favors a smaller model trained far longer, as in Llama 3 8B at roughly 1,900 tokens per parameter. The discussion covers the numerical solution and the FLOPs-only savings, such as about 16% for a 30B-quality model serving five trillion tokens. It then moves to dollar costs, where inference hardware utilization is far lower than training utilization, so the savings are larger. It also covers the paper's sweep of 47 trained models, which tests whether small models keep improving at extreme data ratios, and its refit of the Chinchilla coefficients. The hosts flag a weakness: the extreme end of the sweep rests mostly on one small model, while the recommendations are meant for much larger ones. Listeners get a clear explanation of why over-training can be economically rational, and a skeptical look at how well the evidence supports it.
Sources:
1. Scaling Inference Demand into Chinchilla: Training Smaller Models Longer
https://raw.githubusercontent.com/mlresearch/v235/main/assets/sardana24a/sardana24a.pdf
2. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. (DeepMind), 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models
3. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, et al. (OpenAI), 2020
https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models
4. LLaMA: Open and Efficient Foundation Language Models — Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. (Meta AI), 2023
https://scholar.google.com/scholar?q=LLaMA%3A+Open+and+Efficient+Foundation+Language+Models
5. Scaling Data-Constrained Language Models — Niklas Muennighoff, Alexander Rush, Boaz Barak, et al., 2023
https://scholar.google.com/scholar?q=Scaling+Data-Constrained+Language+Models
6. Llama 2: Open Foundation and Fine-Tuned Chat Models / The Llama 3 Herd of Models — Hugo Touvron et al.; Llama Team, AI@Meta, 2023 / 2024
https://scholar.google.com/scholar?q=Llama+2%3A+Open+Foundation+and+Fine-Tuned+Chat+Models+%2F+The+Llama+3+Herd+of+Models
7. Scaling Laws for Precision — Tanishq Kumar, Zachary Ankner, Benjamin Spector, et al., 2024
https://scholar.google.com/scholar?q=Scaling+Laws+for+Precision
8. Resolving Discrepancies in Compute-Optimal Scaling of Language Models — Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, Yair Carmon, 2024
https://scholar.google.com/scholar?q=Resolving+Discrepancies+in+Compute-Optimal+Scaling+of+Language+Models
9. Efficiently Scaling Transformer Inference — Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, et al. (Google), 2022
https://scholar.google.com/scholar?q=Efficiently+Scaling+Transformer+Inference
10. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al., 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
11. Llama 2 / Chinchilla-style compute accounting: 'Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws' — Nikhil Sardana, Jacob Portes, Sasha Doubov, Jonathan Frankle, 2024
https://scholar.google.com/scholar?q=Llama+2+%2F+Chinchilla-style+compute+accounting%3A+%27Beyond+Chinchilla-Optimal%3A+Accounting+for+Inference+in+Language+Model+Scaling+Laws%27
12. Splitwise: Efficient Generative LLM Inference Using Phase Splitting — Pratyush Patel, Esha Choukse, Chaojie Zhang, et al. (Microsoft), 2024
https://scholar.google.com/scholar?q=Splitwise%3A+Efficient+Generative+LLM+Inference+Using+Phase+Splitting
13. Language models scale reliably with over-training and on downstream tasks — Samir Yitzhak Gadre et al., 2024
https://scholar.google.com/scholar?q=Language+models+scale+reliably+with+over-training+and+on+downstream+tasks
14. Chinchilla Scaling: A replication attempt — Tamay Besiroglu, Ege Erdil, Matthew Barnett, Josh You, 2024
https://scholar.google.com/scholar?q=Chinchilla+Scaling%3A+A+replication+attempt
15. Go smol or go home — Harm de Vries, 2023
https://scholar.google.com/scholar?q=Go+smol+or+go+home
16. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations / Predictable scale: Part I (hyperparameter scaling) — Hägele et al.; Yao et al., 2024
https://scholar.google.com/scholar?q=Scaling+Laws+and+Compute-Optimal+Training+Beyond+Fixed+Training+Durations+%2F+Predictable+scale%3A+Part+I+%28hyperparameter+scaling%29
17. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann et al., 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29
18. Scaling Laws for Fine-Grained Mixture of Experts — Jakub Krajewski et al., 2024
https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts
19. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters — Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar, 2024
https://scholar.google.com/scholar?q=Scaling+LLM+Test-Time+Compute+Optimally+can+be+More+Effective+than+Scaling+Model+Parameters
20. Mixture-of-Experts scaling and inference cost analyses, including DeepSeek-V2/V3 reports — DeepSeek-AI, 2024
https://scholar.google.com/scholar?q=Mixture-of-Experts+scaling+and+inference+cost+analyses%2C+including+DeepSeek-V2%2FV3+reports
Interactive Visualization: Scaling Inference Demand into Chinchilla: Training Smaller Models Longer
Otros episodios de "AI Post Transformers"



No te pierdas ningún episodio de “AI Post Transformers”. Síguelo en la aplicación gratuita de GetPodcast.








