A 50/50 experiment that comes back 50.2/49.8 over a million users is not “close enough”: a chi-squared test on the counts shows odds below one in five hundred thousand of that gap arising by chance. Something upstream (bots, redirects, telemetry loss, identity bugs, asymmetric triggering) is deleting users from one arm non-randomly, which means the surviving populations are no longer comparable and the effect estimate is meaningless.
The check costs one line of code, which is why mature platforms run it automatically and withhold results until it passes. Microsoft reported roughly 6% of experiments affected. The standard metaphor is a fever: one symptom, many possible diseases, all serious.