Large-scale Deep Unsupervised Learning using Graphics Processors

The problem it addressed

Many useful machine-learning methods are unsupervised: they look for structure in data without being given labels. This paper considers two such models, deep belief networks and sparse coding ^[1]. A deep belief network is a stack of layers of stochastic units, trained so that each layer builds a representation of the activity in the layer below it; a sparse coding model represents each input as a combination of a small number of learned basis functions. Both are built out of very large numbers of matrix and vector operations applied to large datasets, so the cost of training grows quickly with the size of the model and the amount of data. The practical limit on such models was therefore not the mathematics but the available computing power.

How they did it

The authors' starting point was a claim about hardware: modern graphics processors far surpass the computational capabilities of multicore CPUs, and have the potential to change how widely deep unsupervised learning methods can be applied ^[1]. Graphics processors are built to do the same simple arithmetic on many pieces of data at once, which is exactly the shape of most of the work inside these models. The paper sets out general principles for massively parallelizing unsupervised learning tasks on graphics processors ^[1] — deciding which computations can be run simultaneously, how data should be laid out so that the many parallel units stay busy, and how to organise the training so that the fast hardware is not left waiting on slower steps.

Results

For deep belief networks, the authors' implementation was up to 70 times faster than a dual-core CPU implementation when the models were large ^[1]. In concrete terms, they reduced the time needed to learn a four-layer deep belief network with 100 million free parameters from several weeks to around a single day ^[1]. For sparse coding they wrote a simple algorithm that is inherently parallel, giving a 5 to 15-fold speedup over previous methods ^[1].

Publication

The work was published as "Large-scale Deep Unsupervised Learning using Graphics Processors" by Rajat Raina, Anand Madhavan and Andrew Y. Ng of Stanford University at the 26th International Conference on Machine Learning in Montreal in 2009 ^[1]. The ACM Digital Library records it in the ICML '09 proceedings, pages 873–880, with DOI 10.1145/1553374.1553486 and a publication date of 14 June 2009 ^[2].

Why it matters

The headline numbers are about wall-clock time, but their importance is in what that time buys. A model that takes several weeks to train can be tried once; a model that takes about a day can be tried repeatedly, compared against alternatives and scaled up. By showing that a graphics processor — hardware originally designed for drawing images — could stand in for a multicore CPU on this kind of workload, the paper made a case for a different kind of machine for machine learning: many slow, simple parallel units rather than a few fast general-purpose ones. That argument, made on two standard unsupervised models and backed by speedups of one to two orders of magnitude, helped establish graphics processors as serious tools for training large models, and the practice has since become a routine part of how deep networks are built and trained.