Glossary Entry

Linear Regression

A model predicting the mean of a continuous outcome as a weighted sum of features, classically fitted by least squares.

Models Statistics

Also called: ordinary least squares, OLS, least squares regression

Seed source: Penn State STAT 501, Regression Methods

Linear regression models the expected value of a continuous outcome as a weighted sum of the features, with observations scattering around that mean. Fitting by least squares is equivalent to maximum likelihood under Gaussian noise, which is the probabilistic reading that generalizes to the rest of the GLM family.

Its coefficients have the cleanest interpretation in machine learning: one unit of a feature shifts the expected outcome by that feature’s weight, holding the rest fixed. The price is a set of assumptions (linearity, independent errors, equal variance) that deserve checking with residual plots before the coefficients are trusted.