Generative adversarial network

What it is

A generative adversarial network (GAN) is a machine-learning method in which two neural networks compete in a zero-sum game.^[1] The first, the generator, samples from a latent space — a vector of numbers that acts as a seed — and produces fake data. The second, the discriminator, receives an input and judges whether it is real, taken from the training set, or generated.^[1] The generator is trained to fool the discriminator, and in doing so it learns a distribution that matches the statistics of the training set.^[1]

How it is trained

The objective is written as the expectation, over real data, of the log of the discriminator's output, plus the expectation, over generated data, of the log of one minus that output.^[1] The discriminator maximises this quantity: it wants to be confident on real samples and unconfident on fakes. The generator minimises the same quantity, so training is a min-max game.^[1] In practice the two networks are updated alternately by backpropagation.^[1]

The authors' analysis says that in the space of arbitrary functions G and D a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere, because it can no longer tell the two apart.^[2] Training is in theory expected to converge to a Nash equilibrium of this kind, where the generated distribution equals the real one and the discriminator outputs one half.^[1] A practical attraction is that when G and D are multilayer perceptrons the entire system can be trained with backpropagation, with no Markov chains and no unrolled approximate inference networks needed.^[2]

A small picture

Imagine the training set is a collection of small grayscale images. The generator begins by emitting something close to noise, and the discriminator has an easy time separating it from the real images. As training proceeds the generator's output drifts toward the pixel statistics of the training set, and the discriminator is pushed to sharpen its judgement. If everything worked as the theory says, the two would end in a standoff in which the discriminator can do no better than answer at random.^[1]

Where it came from

Ian Goodfellow and colleagues at the University of Montreal proposed the method in June 2014, publishing "Generative Adversarial Nets" at NIPS 2014.^[1] The paper describes training a generative model G that captures the data distribution together with a discriminative model D that estimates the probability that a sample came from the training data rather than from G.^[2] G is trained to maximise the probability of D making a mistake, a framework the authors describe as a minimax two-player game.^[2]

Architectures that followed

DCGAN in 2015 introduced a fully convolutional architecture.^[1] Progressive GAN grew the resolution of generated images layer by layer.^[1] Nvidia's StyleGAN-1 in 2018 and StyleGAN-2 in 2019 were based on style transfer ideas, and BigGAN in 2019 scaled up ImageNet generation.^[1] Read together, these lines of work show the same two-network setup being pushed toward larger and higher-resolution images.

What goes wrong

Mode collapse occurs when the generator produces only a few kinds of samples and misses entire modes of the data distribution.^[1] Training is also unstable and can fail to converge, and when the discriminator becomes too strong the generator receives vanishing gradients.^[1] These are properties of the training dynamics rather than of any particular dataset, which is why later work put so much effort into the architecture of the two networks.

Why it matters

The framework needs no Markov chains and no unrolled approximate inference networks; with multilayer perceptrons, backpropagation alone is enough to train the whole system.^[2] Its theoretical claim is clean: in the space of arbitrary functions a unique solution exists, with the generator recovering the data distribution and the discriminator outputting one half everywhere.^[2] What followed showed that the idea carries from that abstract setting to concrete image generation at increasing scale.^[1]