Supervised learning

What supervised learning is

Supervised learning is a machine-learning paradigm in which an algorithm learns a mapping from inputs to outputs from paired examples: inputs together with their correct outputs. Data in this paired form is called labelled data.^[1] Google's machine learning course describes the same idea as using labelled data to train models that predict outcomes for new, unseen data.^[2]

How it works

The training set has to be representative of the wider data the model will meet, and every item in it is paired with the correct output, called the label or target.^[1] Inputs are converted into feature vectors, and the learning algorithm searches for a function g that maps the input space X to the output space Y.^[1] In Google's description, the model is fed labelled examples so that it can learn the relationship between features and labels.^[2]

Training minimises a loss over the training data. One route is empirical risk minimisation, which minimises the error measured on the training set itself; the other is structural risk minimisation, which adds a penalty term to control the bias-variance trade-off.^[1]

A small example makes the shape of the problem clearer. Suppose each input is an image, turned into a feature vector of pixel values, and the label is a category such as "cat" or "dog". Training adjusts the function g so that its outputs match the labels on as many training items as possible, while the penalty term discourages a function that simply memorises the training items and has nothing useful to say about a new image. The same pattern applies whenever inputs can be described by features and outputs can be written down as labels.

Measuring success

What matters is generalisation: the model's accuracy on data it has not seen, usually estimated with a held-out test set or with cross-validation.^[1] In the Google course, trained models are evaluated by comparing their predictions on unseen data with the actual values, and are then used for inference.^[2]

Algorithm families

Common families include support vector machines, linear and logistic regression, naive Bayes, linear discriminant analysis, decision trees, k-nearest neighbours, and neural networks such as multilayer perceptrons.^[1] None of them is best for every problem: the no free lunch theorem holds that no single algorithm wins across all problems.^[1]

Limits

Supervised models are prone to overfitting, and the bias-variance trade-off is difficult to tune.^[1] High-dimensional inputs and noisy outputs reduce accuracy, and some algorithms perform poorly when data are heterogeneous, redundant, or contain complex feature interactions.^[1] Training also depends on large amounts of high-quality labelled data and is time-consuming to tune.^[1] The bias-variance dilemma itself was analysed by Geman, Bienenstock and Doursat in Neural Computation in 1992.^[1]

Where it stands

Supervised learning matters because it turns learning into an optimisation problem with a clear target: given inputs and their correct answers, find a function that also answers correctly on new data. The costs follow from the same definition. Labels must be collected and must be right, overfitting has to be held back, and algorithms and their settings have to be chosen with care. Success is never the score on the training set; it is generalisation, measured on data the model has not seen.^[1]