Glossary Entry

Mutation Testing

A way to measure whether a test suite would actually notice a bug, by seeding small deliberate faults (mutants) into the code and checking that at least one test fails for each.

Evaluation Metrics

Also called: mutation tests, mutation test, mutant, mutants, mutation score

Seed source: Petrović and Ivanković, State of Mutation Testing at Google (ICSE 2018)

Code coverage tells you which lines a test suite executes. It does not tell you whether the suite asserts anything useful about them, so a suite can cover a function completely while passing no matter what the function returns. Mutation testing closes that gap. A tool rewrites the code in small ways, flipping a comparison, deleting a statement, replacing a constant, and reruns the tests against each mutant. A mutant that makes a test fail is killed; one that survives points at behaviour the tests never check.

The technique matters more once coding agents write both the code and its tests, because an agent can produce green tests that ratify whatever it implemented rather than the behaviour that was asked for. Surviving mutants are a mechanical, reviewer-free signal that the tests are weaker than they look. Google runs a diff-based version inside its mandatory code review, showing only the surviving mutants on changed lines, and open-source tools such as Stryker and PIT do the same for JavaScript, .NET, and Java projects.