Large Scale Distributed Deep Networks

The problem

Deep networks draw much of their capability from having many layers and many parameters, and that same size is what makes them expensive to train. Training is iterative: the model repeatedly looks at training examples, measures how far its output is from the target, and adjusts its parameters to reduce the error. With a large model and a large data set, the number of adjustments required is very large, and a single machine can spend a long time on the job. The question the paper takes up is whether that work can be spread over a cluster of machines instead of being confined to one.

DistBelief and two training procedures

The authors developed a software framework called DistBelief that could use computing clusters with thousands of machines to train large models ^[1].

On top of that framework the paper describes two different ways of carrying out the optimisation. The first is Downpour SGD, an asynchronous stochastic gradient descent procedure supporting a large number of model replicas ^[1]. Stochastic gradient descent is the standard recipe of adjusting parameters from small batches of data rather than from the entire data set at once; the asynchronous label distinguishes this procedure from arrangements in which every copy of the model must move in strict step with the others. The second is Sandblaster, a framework supporting distributed batch optimisation procedures, including a distributed implementation of L-BFGS ^[1]. L-BFGS belongs to the family of quasi-Newton methods, which use approximate curvature information to choose the direction in which to move the parameters, and it is normally applied to a batch of data rather than to small samples.

Having two procedures matters because the two styles of optimisation make different demands on the network and on the cluster, and the paper reports how each of them was made to work at scale ^[1].

What the experiments showed

The authors said they had trained a deep network 30 times larger than previously reported in the literature ^[1]. They also reported state-of-the-art performance on ImageNet, which the paper describes as a task with 16 million images and 21k categories ^[1]. Beyond that benchmark, the authors said the same techniques dramatically accelerated training of a more modestly sized deep network for a commercial speech recognition service ^[1]. Taken together, the results cover two quite different situations: one deliberately pushed to an extreme scale, and one closer to a service that had to be trained in practice.

Publication

Google Research's page for the paper says it was to appear at NIPS 2012 in Lake Tahoe, Nevada, United States ^[2]. The authors are Jeffrey Dean, Greg S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc'Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang and Andrew Y. Ng ^[1].

Why it matters

The paper separates two questions that had usually been treated as one: how large a model can be, and how fast a single machine is. By describing a framework that spans thousands of machines, and by reporting a model far larger than earlier published ones along with results on a standard benchmark and on a production speech task, it presented distributed training of deep networks as a practical engineering option rather than a thought experiment ^[1]. The two optimisation procedures it set out, an asynchronous replica-based stochastic gradient descent and a distributed batch optimiser, correspond to design choices that remain central whenever the training of a model has to be shared among many machines.