Maxout Networks

The setting

Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville and Yoshua Bengio posted *Maxout Networks* to arXiv on 18 February 2013 ^[1]. The paper sits where two questions meet: how a regulariser that switches units off at random should be analysed, and what kind of unit a network should be built from in the first place.

The problem: dropout and its cheap approximation

Dropout is a training trick. On each update, units of the network are switched off at random, so the network cannot lean on any single unit being present, and the weights that survive have to work well in many different configurations. The authors described dropout as a recently introduced approximate model averaging technique ^[1].

The reason for that description is easy to see. Every random switch-off pattern defines a different, thinner subnetwork, so training with dropout is roughly like training a very large family of subnetworks that all share the same weights. Running all of those subnetworks at test time would be far too expensive. Dropout therefore uses a cheap stand-in: a single forward pass whose weights are scaled to stand for the average behaviour of the whole family.

How good that stand-in is depends on the network it is applied to. Rather than treating dropout as something bolted onto an arbitrary architecture, Goodfellow and his co-authors asked what architecture would make dropout both easy to optimise and accurate under the fast approximation ^[1].

Maxout: a unit designed around dropout

Their answer is the maxout unit. The authors defined a model called maxout, named because its output is the maximum of a set of inputs, and described it as a natural companion to dropout ^[1]. A maxout unit is defined by that single operation: it is given several values and keeps only the largest.

The model was designed with two goals: to facilitate optimisation by dropout, and to improve the accuracy of dropout's fast approximate model averaging ^[1]. Mechanically, a maximum is a forgiving operation. If one of the values feeding a unit is removed, the unit still returns the largest of those that remain, so switching units off does not throw its output arbitrarily far away — which is what a regulariser that keeps removing units needs from the pieces it removes.

Verification and results

The authors said they empirically verified that the model accomplishes both tasks ^[1]. They also reported state-of-the-art classification performance using maxout and dropout on four benchmark datasets: MNIST, CIFAR-10, CIFAR-100 and SVHN ^[1]. All four were standard image-classification benchmarks, so the claim was checked on several different problems rather than on one.

Why it matters

The paper's argument is about co-design. Dropout is usually introduced as a general technique that can be applied to a finished network; Maxout Networks works through a case in which the unit itself is chosen to suit the regulariser, and then checks the result on benchmarks ^[1]. For a reader learning the subject, it is a compact example of how a regulariser is analysed as approximate model averaging, how an architectural choice can be judged by whether it helps that approximation, and how such a choice is tested — on several standard datasets, against the best published results at the time.