Cohen’s kappa measures agreement between two labelings beyond what chance would produce, ranging from 1 (perfect) through 0 (chance-level) to negative values (systematic disagreement). The quadratic weighted variant adds a penalty matrix that grows with the squared distance between categories, so predicting “3” for a true “4” costs far less than predicting “0”, which is exactly the right shape for ordinal targets like severity grades or ratings.
Because classification losses ignore category order, a common competition-winning recipe treats the ordinal target as a regression problem and then optimises the rounding thresholds on validation data to maximise kappa directly. One quirk to respect: the chance-correction term depends on the marginal distribution of your predictions, so the score can move when you merely reshape how often you predict each category.