Glossary Entry

Induction Head

An attention head that finds an earlier occurrence of the current token and copies what came after it, letting a transformer continue patterns it has already seen in its context.

Interpretability LLMs

Also called: induction heads, copying head, copying heads

Seed source: A Mathematical Framework for Transformer Circuits (Elhage et al., 2021)

An induction head completes repeated sequences: having seen “total” earlier, it helps the model turn “tot” into “total” the next time. It works together with an earlier head that records which token preceded each position, so the pair forms a small two-layer circuit.

Induction heads were one of the first mechanisms identified by reading a transformer’s weight matrices directly, and they remain a standard test case for interpretability methods.