Neural Architecture Search with Reinforcement Learning

The problem

Deep neural networks were delivering strong results on hard image, speech and language tasks, but they were still hard to design ^[1]. Designing one means deciding how many layers it has, what operation each layer performs, how the layers are wired to one another and where connections skip ahead. Those decisions were made by hand, by researchers drawing on experience and trying out variants. The loop is slow: every candidate design has to be built and trained before anyone knows whether it is any good, and the space of possible designs is far too large to explore by hand.

The paper asks whether the designing itself can be automated, and answers it with a working method.

How it works

The authors use a recurrent network to generate the model descriptions of neural networks ^[1]. A recurrent network (RNN) processes a sequence one element at a time while keeping a running internal state, which makes it a natural fit for producing a structured object step by step. Here the sequence it produces is a description of a network: at each step the controller emits a choice about a layer — which operation it performs, and how it connects to what came before — and the string of choices, read out in order, is the architecture itself.

The controller is not trained on a corpus of known good designs. It is trained with reinforcement learning to maximise the expected accuracy of the architectures it generates on a validation set ^[1]. The procedure is a loop: the controller samples a batch of architectures, each is built and trained, and its accuracy on held-out validation data becomes the reward. The controller's parameters are then updated so that designs that scored well become more likely to be produced again, and designs that scored badly become less likely. Judging candidates on a validation set rather than on the data they were trained on matters, because an architecture that only fits its training data is not a good design.

Results

On CIFAR-10, starting from scratch, the method designs an architecture that rivals the best human-invented architecture in test accuracy ^[1]. That model reaches a test error rate of 3.65, which the authors say is 0.09 percent better and 1.05 times faster than the previous state of the art obtained with a similar scheme ^[1].

The search was not limited to assembling whole networks out of familiar parts. On Penn Treebank the model composes a novel recurrent cell that beats the widely used LSTM cell and other baselines, with a test perplexity of 62.4 ^[1]. Perplexity is a standard measure of how uncertain a language model is about the next token in a sequence; lower is better, so 62.4 means the searched cell predicted held-out text better than the comparison models.

Why it matters

The paper's contribution is a shift in how architecture design is framed. Instead of a person proposing a structure and testing it, the structure becomes the output of a generator that is optimised against measured performance. The same machinery works at more than one scale: it can lay out a whole network for an image task, and it can also invent a single reusable component such as a recurrent cell — the Penn Treebank result shows the search producing a cell that outperforms LSTM, a building block in wide use ^[1]. Because the controller is trained rather than hand-written, the method does not depend on a fixed list of candidate designs assembled by a researcher.

The paper, "Neural Architecture Search with Reinforcement Learning", was first submitted to arXiv on 5 November 2016 as 1611.01578 and was written by Barret Zoph and Quoc V. Le ^[1].