Glossary Entry

Novelty Effect

A short-lived lift from users exploring anything new, which fades as curiosity does; its mirror image, the primacy effect, is a temporary penalty while users relearn a changed interface.

Statistics A/B test Metrics

Also called: primacy effect

Both effects make an experiment’s first days a biased forecast of its steady state, in opposite directions: novelty flatters the treatment, primacy handicaps it. The standard checks are to compare new users (who have no habits to be novel against or disrupted from) with existing users, and to distrust launch decisions that depend on the earliest data.

The caveat from the Microsoft team that named the pattern deserves equal billing: most apparent novelty and primacy trends in short experiments are statistical artifacts of cumulative plots, whose early segments are simply noisy. Diagnose with new-versus-existing splits and longer runs, not by eyeballing a wobbly first week.