Glossary Entry

Continuous Batching

A serving scheduler that admits and retires requests at every decode step, keeping the running batch full instead of waiting for a whole batch to finish.

Deployment Optimization

Also called: in-flight batching, iteration-level scheduling, dynamic batching

Seed source: Orca (Yu et al., OSDI 2022)

Static batching groups requests and waits for the slowest one, so a single long response holds the whole batch hostage while the GPU idles. Continuous batching reschedules at iteration granularity: when a sequence finishes, a queued request takes its slot on the very next decode step.

Because decode throughput scales almost linearly with batch size until the hardware’s roofline, keeping the batch full is worth a lot: Anyscale measured roughly 8x throughput from continuous batching alone and 23x combined with paged KV-cache memory (vLLM) versus naive batching. It is now standard in every serious serving engine (vLLM, TGI, TensorRT-LLM, SGLang).