Glow: Generative Flow with Invertible 1x1 Convolutions

The problem

Generative models differ in what they can compute exactly. Some give an exact likelihood but are expensive to sample from; others sample quickly while optimising only a bound or an approximation. Flow-based models take another route: the exact log-likelihood and exact latent-variable inference are tractable, and both training and synthesis parallelise ^[1]. The paper gives precisely this combination — exactness in both directions, plus hardware-friendly parallelism — as the reason the family is attractive ^[1].

The method

"Glow: Generative Flow with Invertible 1x1 Convolutions" was first submitted to arXiv on 9 July 2018 as 1807.03039, by Diederik P. Kingma working with Prafulla Dhariwal ^[1]. The paper describes Glow as a simple type of generative flow that uses an invertible 1x1 convolution ^[1].

A flow is a stack of reversible transformations that maps an image to a latent vector of the same size. Training pushes images through the stack to obtain latent codes and an exact likelihood; generation runs the same stack backwards, starting from a random draw. Because every layer is a bijection, nothing is discarded: the model represents a distribution over images by warping a simple distribution, usually a Gaussian, into the shape of the data. What makes the arithmetic work is the change-of-variables formula, in which each layer contributes the determinant of its Jacobian; keeping that determinant cheap to compute is the central design constraint of any flow.

Most flow layers act on each channel separately, so something has to mix information across channels. A fixed shuffle of channels does this without parameters but cannot adapt. A 1x1 convolution is a learned linear combination of channels at every pixel — in effect a small matrix applied at each spatial position — and the paper's contribution is to put such a convolution inside the flow while keeping it invertible ^[1]. The model therefore learns how to combine channels, instead of having the combination fixed in advance.

Results

The authors report a significant improvement in log-likelihood on standard benchmarks ^[1]. They also show that a model optimised towards plain log-likelihood can synthesise and manipulate large images efficiently and realistically ^[1]. The code was released in the openai/glow repository ^[1], so that others could train and inspect the model directly.

Why it matters

Flow models make the accounting unusually clean. The training objective is the likelihood of the data itself, latent variables have exact inference, and an image can be mapped to a code and decoded back without loss. Manipulation therefore has a concrete meaning: change the code, run the stack backwards, and look at the result — and the paper shows this working on large images ^[1]. That places Glow among the generative approaches whose appeal is not a sharpened picture alone but a quantity that can be computed and compared.

Limits

Invertibility is a demanding constraint rather than a free lunch. Because each layer must be reversible, the model cannot simply throw information away as it goes deeper, and the dimensions going in must match the dimensions coming out; the computation and memory cost per parameter is therefore typically higher than in architectures that are allowed to discard data. The Jacobian determinant must also stay cheap to evaluate, which restricts what shape a layer can take. These are general costs of the flow family, and they explain why exact likelihood and exact inference come at a price in engineering effort.