Glossary Entry

Sequential Testing

Statistical designs that permit analyzing an experiment repeatedly while it runs, by spending the error budget deliberately across looks instead of pretending each look is the only one.

Statistics A/B test Decision Making

Also called: group sequential test, always-valid inference, mSPRT

Checking an experiment daily and stopping at the first significant result inflates the false-positive rate several-fold, but the desire to stop early is legitimate: large effects deserve fast decisions and harmful ones fast rollbacks. Sequential designs make the monitoring legal. Group sequential tests, imported from clinical trials, pre-plan a few interim analyses with strict early thresholds that relax toward the final look (O’Brien-Fleming boundaries, generalized by alpha-spending functions). Always-valid methods like the mixture sequential probability ratio test go further, producing p-values that hold simultaneously at every moment, so continuous dashboards are simply safe.

The practical selection rule, from Spotify’s published comparison: group sequential tests are more powerful when the sample size can be planned in advance, while always-valid inference suits open-ended monitoring for regressions.