Glossary Entry

Compute-Optimal Scaling

The Chinchilla result that for a fixed training compute budget, loss is minimized by scaling parameters and data together, at roughly 20 training tokens per parameter.

Training LLMs

Also called: Chinchilla scaling, Chinchilla-optimal, compute-optimal training

Seed source: Hoffmann et al. 2022

Given a compute budget of roughly 6ND FLOPs for N parameters and D tokens, the Chinchilla paper showed the loss-minimizing split scales both about equally, landing near 20 tokens per parameter; its 70B model trained on 1.4T tokens outperformed the much larger Gopher trained on the same compute.

Production models now deliberately train far past this point (Llama 3 405B at 38 tokens per parameter, recent frontier models by much larger multiples), because a model that will serve billions of requests is worth overtraining to buy more quality per inference FLOP. Compute-optimal is optimal for training loss, not for lifetime economics.