Glossary Entry

Pipeline Parallelism

Training parallelism where consecutive layers live on different chips and activations flow between stages, cheap in bandwidth but prone to idle "bubbles".

Training Hardware

Also called: pipeline parallel, pipeline bubbles

Seed source: GPipe (Huang et al. 2019)

Pipeline parallelism assigns contiguous blocks of layers to different chips and passes activations down the line, so the only communication is one activation tensor per stage boundary. That makes it the cheap way to span slow links between racks and pods.

The cost is pipeline bubbles: chips idle while the pipeline fills and drains each step. Micro-batching and interleaved forward/backward schedules shrink the bubbles, and frontier recipes like DeepSeek-V3’s 16-way pipeline stack it with expert and data parallelism.