Glossary Entry

Elastic Weight Consolidation

A continual-learning method that fights catastrophic forgetting by adding a quadratic penalty anchoring each weight near its old-task value, scaled by the Fisher information measuring how much the old task depends on that weight.

Training Optimization

Also called: EWC

Seed source: Kirkpatrick et al. 2017

EWC comes from a Bayesian argument: the posterior over weights after task A is Laplace-approximated as a Gaussian centered at the trained solution, with the diagonal of the Fisher information as its precision. Training on task B while staying probable under that Gaussian turns into minimizing the task B loss plus a per-weight spring penalty; weights that mattered to task A are held stiffly, weights task A never used remain free.

In practice EWC trades plasticity for stability along a frontier controlled by its penalty strength, and simple experience replay is often competitive with or better than it. Its lasting contribution is the framing: forgetting is about which weights move, not how many, an idea that echoes through synaptic intelligence, sparse memory finetuning, and the rest of the modern continual-learning toolbox.