Large Scale GAN Training for High Fidelity Natural Image Synthesis

The problem

A generative adversarial network (GAN) is trained by competition: a generator turns random input vectors into images, a discriminator tries to tell those images apart from real ones, and the generator learns by trying to fool the discriminator. The paper starts from the observation that generating high-resolution, diverse samples from complex datasets such as ImageNet had remained an elusive goal ^[1]. The setting the authors work in is class-conditional synthesis, where the model is asked for an image of a specified category rather than an image in general, so it has to cover many categories at once instead of settling into the few that are easiest to produce.

Pushing the scale

The authors' response was to train GANs at the largest scale attempted up to that point, and to treat the instabilities that appear at that scale as a subject of study rather than as an accident to be tuned away ^[1]. A larger model, with more parameters and wider layers, gives the generator more capacity to represent detail, but it also makes the adversarial training loop harder to keep balanced: the two networks can drift, oscillate or collapse. Making those failure modes explicit is what allows the rest of the work to proceed, because a training recipe that only works at small scale cannot simply be enlarged.

Orthogonal regularisation and the truncation trick

The key finding reported in the paper is that applying orthogonal regularisation to the generator makes it amenable to a simple truncation trick ^[1]. Orthogonal regularisation is a constraint that pushes the rows or columns of a weight matrix towards being mutually orthogonal, so that the network does not lean on redundant directions and each unit carries a somewhat distinct part of the signal.

The truncation trick works on the input side. Instead of sampling the generator's input vector from its full distribution, one samples from a truncated version of that distribution, or equivalently pulls samples towards the mean, which lowers the variance of what the generator receives. With the regularisation in place, the generator tolerates this narrowing well. The trick then gives fine control over the trade-off between sample fidelity and variety by reducing the variance of the generator's input ^[1]. Truncate harder and the images become cleaner and more typical; truncate less and they become more varied but rougher. What matters is that the trade-off becomes a dial the user can set at generation time, rather than a property baked into a trained model.

Results

The resulting models are called BigGANs, and they set a new state of the art in class-conditional image synthesis ^[1].

Why it matters

The contribution is threefold, and none of the three parts is a single number. First, the work is evidence that scale is itself a lever for image quality in adversarial training. Second, it names and studies the instabilities that come with that scale instead of hiding them, which makes large-scale GAN training something one can reason about. Third, it turns the fidelity-versus-variety trade-off into a knob that costs almost nothing to use. Naming the models BigGAN and reporting a new state of the art in class-conditional synthesis marks where the work landed; the durable part is the way of working it lays out: grow the model, watch what breaks, and give the user a control they can turn ^[1].