An A/A test can find no real effect because none exists, which is exactly what makes it useful: every “significant” result is a false positive by construction, so the observed rejection rate audits the whole pipeline. Sustained rejection above 5% points at biased assignment, broken logging, or variance formulas that ignore clustering (the classic culprit being ratio metrics computed at a finer grain than the randomization unit).
The second job is measurement: A/A data estimates each metric’s real variability, feeding honest power calculations. The published guidance from Microsoft’s experimentation group is to run A/A tests continuously alongside real experiments, as a canary that never stops singing.