Tensor parallelism slices individual weight matrices across chips, so every chip holds a slab of every layer and the layer’s output must be reassembled from partial results. That puts communication on the critical path of every single layer, which is why this cut is reserved for the fastest interconnect (within an NVLink node or a few torus hops) and typically tops out around 8- to 16-way before communication swamps compute.
It is the standard way to fit a model whose layers are too big for one chip, both in training and in latency-sensitive serving, and is usually composed with data and pipeline parallelism in frontier-scale recipes.