Thanks to skip connections, each transformer layer adds its attention and MLP outputs to a running vector rather than replacing it: the update has the form h(l+1) = h(l) + attention + MLP. Interpretability researchers call this running sum the residual stream and treat it as the model’s central bus: every component reads its input from the stream and writes its result back in, and the unembedding at the top turns the final stream state into token logits.
The framing matters because it gives interpretability one shared object to study. Anything a model “knows” mid-computation must exist as a pattern in some layer’s residual stream, so techniques like probes, the logit lens, sparse autoencoders, and the Jacobian lens are all different ways of asking that same vector questions.