The p-value answers a precise, narrow question: if the treatment truly did nothing, how often would chance alone produce a difference this large? It is not the probability the effect is real, and the gap between the two matters enormously in practice: how much a significant result should move your beliefs depends on the base rate of true effects among the things you test and on the test’s power, via Bayes’ rule.
Its guarantees are also conditional on discipline. A p-value threshold holds for one pre-committed analysis of a pre-committed sample; repeatedly testing accumulating data, choosing among many metrics after the fact, or selecting flattering segments all inflate the real false-positive rate far above the nominal 5%.