Overfitting

What overfitting is

Overfitting is the production of an analysis that corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably ^[1]. Google's machine learning glossary states the same idea in plainer terms: a model performs well on training data but poorly on new, unseen data ^[2]. The opposite of overfitting is generalisation. A model generalises well if it accurately predicts on new data, which is a sign that it has not overfit ^[2].

A useful way to picture the difference is a student preparing for an exam. A student who memorises the exact answers to a practice paper can reproduce them perfectly, but a slightly different question leaves them stuck. A student who has learned the underlying method answers both the practice paper and the new questions reasonably well. The first student has fitted the specific examples rather than the pattern behind them.

How it happens

Overfitting typically happens when a model has more parameters than the data can reasonably support ^[1]. With enough capacity, a model can bend itself around the peculiarities of whatever examples it was shown instead of the regularities that would carry over to new cases. Google lists unrepresentative training data and overly complex models among the common causes ^[2]. These two causes pull in the same direction: if the examples are a poor sample of the world, or if the model is complicated enough to memorise them, the fit is to the sample rather than to the world.

Bias, variance and the two failure modes

An overfitted model has low bias and high variance, while an underfitted model has high bias and low variance ^[1]. Low bias means the model is flexible enough to follow the training data closely; high variance means that closeness is unstable, so a different sample would produce a noticeably different model. Underfitting is the mirror image: too rigid to capture the structure, and consistently wrong in the same way. The bias–variance trade-off is the standard way to describe the two failure modes together ^[1], and it is the reason overfitting is normally discussed alongside underfitting rather than on its own.

How it is detected

Overfitting is detected when performance on a validation set is much worse than on the training set ^[1]. Since the validation examples played no part in fitting the model, a large gap between the two scores is evidence that the fit does not travel. The same gap can be watched over the course of training: on a generalization curve, overfitting shows up as diverging loss curves for the training and validation sets, with validation loss turning upward while training loss keeps falling ^[2].

What is done about it

Common countermeasures include model comparison, cross-validation, regularisation, early stopping, pruning, Bayesian priors, dropout, ensemble methods and feature engineering ^[1]. The list is long because the problem has several entry points: some techniques limit how complicated the model is allowed to become, others change how the data is presented or how many examples are effectively used, and others check the model's behaviour against data it has not seen. Combating overfitting often requires more data than would otherwise be necessary ^[1], which makes data collection a practical part of the remedy and not just a preliminary step.

Limits and risks

Overfitting is not only a modelling inconvenience. Overfitted models can be less portable, and the discussion notes that memorisation of training data may leak sensitive information, citing litigation over Stable Diffusion and GitHub Copilot ^[1]. When a model has absorbed details of individual training examples rather than general structure, those details can travel with it and surface in its outputs. That turns a statistical failure into a problem with legal and privacy dimensions, and it is one reason why the gap between training performance and new-data performance is watched so closely.