Glossary Entry

Arithmetic Intensity

The number of floating-point operations a workload performs per byte of data it moves, which determines whether it is compute-bound or memory-bound.

Hardware Optimization

Also called: operational intensity, FLOPs per byte

Seed source: How to Scale Your Model (roofline chapter)

Arithmetic intensity is total FLOPs divided by total bytes communicated. Compare it to a chip’s own ratio of peak compute to memory bandwidth (roughly 295 FLOPs per byte for an H100): workloads above that ridge are compute-bound, workloads below it leave the arithmetic units idle waiting for memory.

For transformer weight matmuls the intensity is approximately the number of tokens flowing through the layer together, which is why batch size is the central lever in LLM serving: prefill and training arrive with thousands of tokens per pass and run compute-bound, while single-request decoding has an intensity of about 1 and is starved for bytes.