Glossary Entry

Minimum Detectable Effect

The smallest true effect an experiment is designed to detect reliably at its chosen power and significance level; the honesty clause of a power calculation.

Statistics Metrics A/B test

Also called: MDE

The MDE converts an experiment’s sample size into a plain-language capability statement: “this test will notice a 2% relative lift; it will usually miss a 1% one.” Fixing any two of MDE, sample size, and power determines the third, so quoting an experiment’s duration without its MDE hides the most important design fact.

Choosing the MDE is a business judgment dressed as statistics: it should be the smallest effect that would still justify shipping and maintaining the change. Setting it optimistically small makes experiments impossibly long; setting it conveniently large produces fast experiments that can only detect miracles.