Unsupervised learning

What it is

Unsupervised learning is a machine learning framework in which an algorithm learns patterns from data that carry no labels, in contrast to supervised learning ^[1]. In supervised training every example is paired with a desired answer; in unsupervised training only the examples are supplied, so whatever structure the model captures has to come from the data itself.

How it works

Work in this framework falls into a handful of task families: clustering, dimensionality reduction, density estimation, anomaly detection, latent variable models and generative modelling ^[1]. Clustering groups similar examples together; dimensionality reduction replaces many measured variables with fewer; density estimation and generative modelling try to describe how the data were produced; anomaly detection picks out the examples that do not fit the prevailing pattern.

A small example makes the setting concrete. Suppose a table has one row per object and one column per measured property, and no answer column at all. A clustering method reads the rows and returns a group number for each one. The grouping is the output; nothing in the data states whether the grouping is correct.

Where it came from

The early formulations came from statistical physics. W. A. Little proposed an Ising model of cognition in 1974, an early formulation of unsupervised learning ^[1]. John Hopfield described a variant of the Ising model as the Hopfield network in 1982 ^[1]. In 1983 Hinton and Sejnowski introduced the Boltzmann machine, a probabilistic neural network ^[1], and Paul Smolensky published Harmony Theory in 1986, which resembles a restricted Boltzmann machine ^[1]. Autoencoders emerged around 1986, and the Helmholtz machine was proposed by Dayan and Hinton in 1995 ^[1]. The variational autoencoder was introduced by Kingma, Rezende and others in 2013 ^[1]. Read in order, the line runs from physics-inspired networks of binary units to probabilistic models that describe data through hidden variables.

Where it is used today

In software libraries, clustering of unlabeled data is treated as the canonical unsupervised task: scikit-learn documents it that way and implements it in its sklearn.cluster module ^[2]. Each clustering algorithm there comes in two variants — a class that implements fit to learn clusters on training data, and a function that returns integer labels for the training data ^[2]. A user can therefore work through the class interface or simply take the labels the function produces.

Limitations

Training is the recurring difficulty. Boltzmann machines are hard to train because of their lateral connections, and reaching equilibrium needs many iterations ^[1]. Expectation-maximisation, a common fitting method, can get stuck in local optima and does not guarantee convergence to the true parameters ^[1]. And because the data carry no labels, there is no answer key to compare a result against; the structure a method returns has to be judged on its own terms.