Glossary Entry

Speculative Decoding

An inference trick where a small draft model proposes several tokens and the large model verifies them in one batched pass, spending spare compute to cut latency.

Optimization LLMs

Also called: speculative sampling

Seed source: Leviathan et al. 2023

Token-by-token decoding leaves an accelerator memory-bound, with most of its arithmetic idle. Speculative decoding exploits that slack: a cheap draft model guesses the next few tokens, and the big model checks all of them in a single forward pass, accepting the longest correct prefix. A rejection sampling scheme keeps the output distribution exactly identical to the big model’s own.

Verification of several tokens costs little more than generating one (the weights are being streamed anyway), so accepted guesses are nearly free speedup. Draft lengths of a handful of tokens are typical, with bigger wins on predictable text like code.