Variational autoencoder

What it is

A variational autoencoder (VAE) is a generative model built from two networks. The encoder maps an input x to a distribution over a latent space z, and the decoder reconstructs x from a value of z^[1]. The latent space is probabilistic — it can be a multivariate Gaussian — so the model can be seen as an autoencoder whose bottleneck holds a distribution rather than a single point^[1]. Methodologically it belongs to probabilistic graphical models and variational Bayesian methods^[1].

How it works

Training does not maximise the likelihood directly. It maximises the evidence lower bound, which equals the expected log-likelihood of the reconstruction minus the KL divergence between the approximate posterior and the prior^[1]. The two terms have distinct jobs: the first is a reconstruction error, often measured with mean squared error or cross-entropy, and the second pulls the posterior towards the prior^[1].

The difficulty sits in the sampling step. Training with gradients requires gradients to pass through the random draw of z, but a random draw is not a differentiable function of the network's parameters. The reparameterisation trick writes the latent sample as a mean plus a scaled noise term^[1]. The randomness is moved outside the parameters — the noise is drawn independently — so gradients can pass through the expectation^[1]. The result is an unbiased gradient estimate, which means ordinary stochastic gradient descent can be used^[1]. The original paper states the same idea from the optimisation side: a reparameterisation of the variational lower bound yields a lower bound estimator that standard stochastic gradient methods can optimise^[2].

A small hypothetical example makes the trick concrete. Suppose the encoder outputs a mean vector and a spread for a given input. Instead of sampling z directly, one draws a noise vector from a fixed, simple distribution, multiplies it by the spread, adds the mean, and sends the result to the decoder. The decoder still receives a random latent code, but the randomness now enters as an input rather than as something buried inside the parameters, so the gradient of the loss with respect to the mean and the spread is well defined.

Where it came from

"Auto-Encoding Variational Bayes" by Kingma and Welling appeared in 2013 and was later presented at ICLR 2014^[1]. First submitted in December 2013, it introduced a stochastic variational inference and learning algorithm that scales to large datasets^[2], and showed that a reparameterisation of the variational lower bound yields a lower bound estimator optimisable with standard stochastic gradient methods^[2]. A related line of work, "Stochastic Backpropagation and Approximate Inference in Deep Generative Models" by Rezende, Mohamed and Wierstra, was published at ICML in 2014^[1].

One of the paper's practical contributions concerns the posterior. For i.i.d. datasets with continuous latent variables per datapoint, posterior inference becomes efficient by fitting an approximate inference model, also called a recognition model, to the intractable posterior^[2]. That recognition model is what is now usually called the encoder.

Variants and limits

Extensions followed. Beta-VAE was proposed by Higgins and colleagues in 2016 and 2017, and the conditional VAE by Sohn and colleagues in 2015^[1]. The basic objective also has a known failure mode: the KL term can drive the model to ignore the latent variables altogether, a problem called posterior collapse^[1]. When that happens the decoder can still produce plausible outputs, but the latent code carries little information about the input.

Read together, these pieces explain the appeal of the model: a principled probabilistic objective, a training procedure that fits ordinary gradient-based pipelines, and a latent space that is a distribution rather than a fixed code.