Glossary Entry

Logit Lens

An interpretability technique that applies a model's unembedding matrix to intermediate-layer activations, revealing which output tokens the model is leaning toward at each layer.

Interpretability LLMs

Also called: tuned lens

Seed source: Interpreting GPT, the logit lens (nostalgebraist, 2020)

The unembedding matrix normally translates only the final layer’s residual stream into vocabulary logits. The logit lens applies it to every intermediate layer as well, letting you watch a model refine its next-token guess layer by layer. In GPT-style models the eventual answer often becomes visible in the rankings well before the final layer.

Its known weakness is basis drift: early layers write in coordinate systems the output unembedding no longer matches, so direct decoding degrades toward noise exactly where the interesting computation begins. The tuned lens (Belrose et al., 2023) fixes this by regressing a per-layer correction, and Anthropic’s Jacobian lens (2026) fixes it by differentiating the network itself, transporting each activation through the remaining layers before unembedding.