Glossary Entry

A/B Testing

Comparing two versions of a system by randomly assigning users to each and measuring which performs better on a chosen metric, so changes are judged on evidence rather than opinion.

Metrics Decision Making Statistics

Also called: A/B test, A/B tests, split testing

Seed source: Trustworthy Online Controlled Experiments (Kohavi, Tang & Xu)

An A/B test randomly splits traffic between the current system (the control) and a changed version (the treatment), then compares an agreed metric between the two groups. Because assignment is random, any reliable difference in the metric can be attributed to the change itself rather than to seasonality, marketing pushes, or differences in who happened to see each version.

In machine learning work, A/B tests are the standard way to prove a model actually beats the process it replaced: the old heuristic or workflow keeps serving part of the traffic as a live baseline, and the model must outperform it on the business metric, not just on offline accuracy. Running them properly needs deliberate engineering (randomisation, logging, and enough traffic for statistical power), which is why measurement belongs in the project plan from the start.