Cross-validation

What it is

Cross-validation is a model validation technique. It uses resampling and data partitioning to assess how well a statistical result will generalise to an independent data set^[1]. What it produces is not one fitted model but an estimate of how a modelling procedure would behave on data it has not seen.

How it works

The best-known form is k-fold cross-validation. The sample is randomly split into k equally sized subsets called folds^[1]. Each fold serves once as the validation set while the remaining k-1 folds are used for training, and the results are averaged over the k runs^[1]. With ten folds — a common choice — the model is fitted ten times, each time on nine tenths of the data and checked on the remaining tenth^[1]. The averaging matters because a single split can be easy or hard by accident; letting every part of the data take its turn as the held-out part makes the estimate less dependent on which points happened to be set aside.

The split itself is not optional. scikit-learn's documentation states that learning the parameters of a prediction function and testing it on the same data is a methodological mistake, because a model that simply repeats the labels it has just seen would score perfectly but predict nothing useful on unseen data^[2]. The library calls that situation overfitting and prescribes holding out part of the available data as a test set^[2].

Other variants trade computational cost against the way the data are partitioned. Leave-p-out cross-validation holds out p points at a time and requires the binomial number of combinations; leave-one-out is the case p = 1, which needs n runs^[1]. Simpler still are the holdout method, which works from a single split, and repeated random sub-sampling, also called Monte Carlo cross-validation^[1].

Where it comes from

The technique is credited in the literature to M. Stone, whose paper "Cross-Validatory Choice and Assessment of Statistical Predictions" appeared in 1974^[1]. Geisser published related work in 1975, the same year the alternative name "rotation estimation" appeared^[1].

Where it is used

Cross-validation belongs to the standard toolchain of applied statistics and machine learning: it is the routine way to compare candidate models, choose settings, and report an expected error before a model meets new data. scikit-learn devotes a documentation chapter to cross-validation as the means of evaluating estimator performance^[2].

Limitations

Cost comes first: the model has to be trained repeatedly, which is expensive when the data are large or the fitting procedure is slow^[1]. Results can also show high variance, and the procedure carries selection bias^[1]. A subtler problem is leakage: carrying out preprocessing or feature selection on the whole data set before splitting introduces bias, because the held-out part has already influenced the pipeline by the time validation begins^[1]. The bias enters at the moment the split stops being the first step.