Glossary Entry

Inter-Token Latency

The time between successive output tokens once a response has started streaming, set by the memory-bound decode phase.

Deployment Metrics

Also called: TPOT, time per output token

Seed source: LLM Inference at Scale with TGI

Inter-token latency (also called time per output token, TPOT) is the pace of the visible stream: each decode step must read all the model’s weights plus the request’s KV cache from memory to mint one token, so the floor on this metric is bytes divided by memory bandwidth, not FLOPs.

It degrades as batch size and context length grow (more KV cache bytes per step), which is exactly the latency side of the latency-throughput trade that batching decides. End-to-end response time is roughly TTFT plus inter-token latency times the number of output tokens.