Expert parallelism parks different experts of a mixture-of-experts layer on different chips, then ships each token to the chips holding the experts its router selected and gathers the results back. The routing exchange is an AllToAll: every chip sends a different slice of its tokens to every other chip, a pattern that is affordable on switched datacenter fabrics but sensitive to load imbalance, since an overloaded expert turns one chip into a hot spot while its neighbors idle.
It is the reason sparse frontier models can store trillions of parameters: no single chip ever needs the whole expert set, only its shard, and it composes with data, tensor, and pipeline parallelism in real training and serving recipes.