Loss function

What a loss function is

A loss function maps an event, or the values of one or more variables, onto a real number that represents some cost associated with that event.^[1] It does not say what the right answer is; it puts a price on being wrong. In machine learning the same idea is usually stated more concretely: Google's Machine Learning course defines loss as a numerical value indicating the difference between a model's predictions and the actual values.^[2]

How it is used

Once a loss function has been chosen, the learning problem becomes an optimisation problem that is solved by minimising the loss.^[1] The stated goal of model training is to minimise loss, bringing it as close to zero as possible.^[2] Much of the rest of training — which parameters to adjust, in which direction, by how much — follows from that single number.

A small example makes the arithmetic concrete. Suppose a regression model predicts a price of 105 when the actual value is 100. The error is 5. An absolute-error loss records 5; a squared-error loss records 25. Squaring punishes large mistakes much more heavily than small ones, so choosing that loss is already a decision about what kind of error matters.

For regression, two common ways of calculating loss are mean absolute error and mean squared error, and the two differ in how sensitive they are to outliers.^[2]

Common forms

Squared error loss penalises the square of the error and is widely used in least squares, t-tests and regression.^[1]

Zero-one loss assigns one when a classification is wrong and zero otherwise; in information theory the same quantity is known as Hamming distortion.^[1]

Absolute error loss is not differentiable at zero.^[1] For training methods that rely on gradients, that kink in the curve is something the implementation has to handle.

Where it comes from

The idea can be traced to Laplace, and Harald Cramér used loss functions in actuarial science in the 1920s.^[1] Abraham Wald reintroduced the concept into statistics in 1950, in his book Statistical Decision Functions.^[1] Leonard J. Savage later proposed losses based on regret.^[1] Seen as a sequence, the history shows the same object — a number attached to an outcome — proving useful well beyond fitting curves to data, as a way of judging decisions whose consequences only become known afterwards.

Limits

Squared error loss can be dominated by outliers, so alternatives such as Huber loss and log-cosh loss are used instead.^[1] Deming and Taleb pointed out that real-world losses are often discontinuous, asymmetric and not differentiable.^[1] A delay that doubles in length, for instance, may more than double in cost, and some failures carry a cost that jumps rather than rises smoothly. Choosing a loss function is therefore not a technicality: it encodes what the model should treat as an expensive mistake, and a loss that is convenient to optimise may not be the one that matches the real cost.