Glossary Entry

Model FLOPs Utilization

The fraction of a hardware fleet's peak compute that ends up in a model's actual forward and backward passes; large training runs typically achieve around 40%.

Training Hardware

Also called: MFU

Seed source: PaLM paper (Chowdhery et al. 2022)

MFU divides the FLOPs a training run theoretically needs (roughly 6 times parameters times tokens) by the FLOPs the hardware could have delivered in the same wall-clock time at peak rate. Everything lost to communication stalls, pipeline bubbles, kernel overheads, and restarts shows up as the gap to 100%.

It is the standard scorecard for large-scale training efficiency: the PaLM paper that popularized it reported 46.2% on 6,144 TPU v4 chips, Meta reports 38-43% for Llama 3 405B on 16k H100s, and 40% is a common planning number. Even the best engineering keeps only about half the purchased silicon doing useful model math.