Glossary Entry

Time to First Token

The delay between sending a request and receiving the first token of the response, dominated by the compute-bound prefill of the prompt.

Deployment Metrics

Also called: TTFT

Seed source: LLM Inference at Scale with TGI

TTFT is the pause before an LLM’s answer starts streaming. It is set by how long the model takes to process the entire prompt in one parallel pass (prefill) and build the KV cache, so it grows with prompt length and is limited by the chip’s compute rate rather than its memory bandwidth.

Serving systems treat it as one of the two core latency objectives, alongside inter-token latency. Prompt caching improves it directly by skipping prefill for prompt prefixes the system has already processed.