A model that tracks millions of concepts cannot give each one its own orthogonal axis in a few-thousand-dimensional activation space. Superposition is the workaround networks learn spontaneously: represent features as directions that are only almost orthogonal. High-dimensional geometry makes this cheap, since the number of unit vectors with pairwise overlaps below a small tolerance grows exponentially with dimension, and sparsity makes it safe, since features that rarely co-occur rarely interfere.
Superposition explains why single neurons are usually polysemantic (each neuron reads many overlapping directions) and why interpretability needs direction-finding tools like sparse autoencoders rather than neuron-by-neuron inspection. Elhage et al. (2022) demonstrated the phenomenon in tiny toy models, where sparse features arrange themselves into regular geometric configurations such as pentagons to share limited dimensions.