Glossary Entry

Expected Calibration Error

A summary of how far a model's stated probabilities are from observed frequencies, computed by binning predictions by confidence and averaging the gap between mean confidence and hit rate in each bin, weighted by bin size.

Metrics Evaluation Decision Making

Also called: ECE, expected calibration error, reliability diagram

Seed source: Guo et al., On Calibration of Modern Neural Networks (2017)

Expected calibration error (ECE) asks a simple question of a probabilistic classifier: when it says 0.8, is it right about 80% of the time? Predictions are sorted into bins by stated probability, usually ten equal-width bins, and for each bin the absolute gap between the average stated probability and the observed accuracy is taken. ECE is the average of those gaps weighted by the share of predictions in each bin, so a value of zero means perfectly calibrated and a value of 0.1 means the stated probabilities are off by about ten points on average.

The same bins plotted as accuracy against confidence give a reliability diagram, which shows which confidence bands are overconfident or underconfident rather than just how badly. ECE is sensitive to the number of bins and to the sample size in each, and it says nothing about ranking quality: a model can have low ECE and poor accuracy, or high accuracy and terrible ECE. It matters most when a probability, not a label, is what downstream code acts on.