Glossary Entry

High-Bandwidth Memory

Stacked DRAM packaged next to an accelerator's compute die, holding model weights and activations and feeding them to the chip at terabytes per second.

Hardware

Also called: HBM

Seed source: NVIDIA H100 datasheet

HBM is the fast on-package memory that accelerators read model weights and KV caches from during every forward pass: 80 GB at 3.35 TB/s on an H100, 192 GB at 7.37 TB/s on Google’s Ironwood TPU. Its two specs, capacity and bandwidth, set two of the hardest limits in LLM systems.

Capacity bounds what fits on a chip (weights, optimizer state, and every concurrent request’s KV cache), and bandwidth bounds how fast memory-bound work like token-by-token decoding can possibly run, regardless of how many FLOPs the chip advertises.