Autoencoder

What it is

An autoencoder is an unsupervised neural network that learns two functions: an encoder mapping the input into a low-dimensional code, and a decoder reconstructing the input from that code ^[1]. The narrow layer between them is called the bottleneck, and the values it holds form the latent representation ^[1]. A 2020 survey states the definition in the same terms: a network designed to encode the input into a compressed and meaningful representation and then decode it back so that the reconstruction is as similar as possible to the original ^[2].

How it works

Training minimises a reconstruction loss, commonly the L2 distance between the input and its reconstruction, by gradient descent ^[1]. No labels are needed, because the target is the input itself.

The bottleneck is what makes the result worth having. A network allowed to pass information through unchanged learns nothing useful: if the code is as large as the input, or the model has too much capacity, it can learn the identity function and become useless ^[1]. Design work therefore concentrates on closing that shortcut — keeping the code small, or constraining what the encoder may produce. The main families of autoencoders differ mostly in which constraint they add: a penalty for dense codes, an obligation to rebuild a corrupted input, or a penalty on how strongly the code reacts to small changes in the input.

Where it came from

Oja showed in 1982 that principal component analysis is equivalent to a neural network with one hidden layer of identity activations, which acts as encoder and decoder ^[1]. Rumelhart and McClelland described autoassociation in 1986, and working implementations followed for speech by Elman and Zipser in 1988 and for images by Cottrell and colleagues in 1987 ^[1]. Baldi and Hornik in 1989 and Kramer in 1991 generalised PCA to non-linear PCA, which is the autoencoder; Kramer called his version a robust autoassociative network ^[1]. Early names for the idea included identity mapping, self-supervised backpropagation and Diabolo network ^[1].

Variants

Sparse autoencoders were explored by Hinton and Zemel in 1993 and by Ng in 2011 ^[1]. Denoising autoencoders came from Kramer's robust network in 1991 and were developed by Vincent and Larochelle in 2010 ^[1]. Contractive autoencoders were proposed by Rifai and colleagues in 2011 ^[1]. The 2020 survey covers the different types of autoencoders mainly used today, and describes their applications and use cases ^[2].

What they are used for

Main uses include dimensionality reduction, semantic hashing for information retrieval, anomaly detection, denoising, compression and drug discovery ^[1].

Because the training signal comes from the data itself rather than from human labels, the same recipe applies wherever unlabelled data is plentiful, and the bottleneck gives a compact description that later stages of a system can work with.

Limits

The obvious failure mode is the one the architecture invites: a model with too much capacity simply learns the identity function and becomes useless ^[1]. The reconstruction loss also defines what survives the bottleneck — the code keeps whatever helps rebuild the input, which need not be what a reader or a later task cares about. Nothing in the objective guarantees that the individual values of the latent representation mean anything on their own; that depends on how the code was constrained and on what data the network was trained to reconstruct.