Glossary Entry

Data Parallelism

Training parallelism where every chip holds a copy of the model and processes a different slice of the batch, synchronizing gradients once per step.

Training Hardware

Also called: data parallel training, FSDP, fully sharded data parallel, ZeRO

Seed source: How to Scale Your Model (training chapter)

Data parallelism is the workhorse of distributed training: replicate the model, split the data, and average gradients across replicas with an AllReduce at each step. Because communication happens only once per step and can overlap with compute, it scales well as long as each chip keeps a large enough per-chip batch to stay compute-bound.

Its sharded variant (FSDP, also known as ZeRO) splits the weights and optimizer state across chips and gathers them just-in-time, removing the memory cost of full replicas for the same total communication volume. That makes it the default first cut in most large training recipes, composed with tensor, pipeline, or expert parallelism when the model itself must be split.