mixup: Beyond Empirical Risk Minimization

The problem the paper addresses

A deep neural network with many layers and parameters can fit very complicated functions, and that capacity is a large part of why such models are powerful. The mixup paper points to a side effect of that capacity: large networks memorise their training examples, and their predictions are sensitive to adversarial examples, inputs that have been altered so that the model gets them wrong ^[1]. Ordinary training, usually called empirical risk minimisation, only asks the model to make the average loss small on the training set. It says nothing about what the model should do at inputs that are not in the training set. mixup is presented as a simple learning principle meant to alleviate those issues ^[1].

How it works

Instead of training on single examples, mixup trains a neural network on convex combinations of pairs of examples and their labels ^[1]. Concretely, two training examples are taken, a new input is formed as a weighted average of the two inputs, and the target is formed as the same weighted average of the two labels. If the weight is near one, the mixed example is almost the first example; if it is near zero, it is almost the second; in between, the model sees a point on the straight line that joins the two examples.

The label that comes out of this is a blend rather than a single class. A network shown an input that is half one class and half another cannot answer with full confidence for either, so it is pushed towards producing a target that moves smoothly with the mixing weight. The paper describes this as regularising the network to favour simple linear behaviour in between training examples ^[1]. That region between examples is exactly where ordinary training gives the model no guidance at all, so the added constraint changes how the model behaves off the training points.

The principle is deliberately simple: it introduces no new layers or special modules, it changes what the network is trained on, namely mixed inputs and mixed labels instead of the original examples ^[1].

Experiments

The authors tested mixup on ImageNet-2012, CIFAR-10, CIFAR-100, the Google commands dataset and UCI datasets ^[1]. Together these cover large-scale image classification, smaller image benchmarks, speech commands and tabular data, so the claim is not tied to one kind of input. In these experiments mixup improved the generalisation of state-of-the-art architectures ^[1], that is, models trained with the principle did better on data they had not been trained on.

Publication and authors

The paper, "mixup: Beyond Empirical Risk Minimization", was first submitted to arXiv on 25 October 2017 and carries the identifier 1710.09412 ^[1]. Its authors are Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin and David Lopez-Paz ^[1].

Why it matters

The phrase in the title, beyond empirical risk minimisation, states the central claim. Standard training constrains the model only at the training points; mixup adds a constraint on the space between them, asking the model to change in a simple, near-linear way as the input moves from one training example towards another ^[1]. Because the ingredient is a rule for building training inputs and targets rather than a new architecture, the idea is easy to state and easy to combine with existing models. The paper presents it as a simple learning principle and reports gains in generalisation across a range of architectures and datasets ^[1].