NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning

Background

The NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning belongs to a line of meetings about learning representations from unlabeled data. Its title names two topics: deep learning and unsupervised feature learning. In speech recognition, these ideas were being tested as ways to build acoustic models, the part of a recognizer that maps audio to phones or other sound units. An earlier NIPS Workshop on Deep Learning for Speech Recognition and Related Applications was held in 2009, and Hinton, Osindero and Teh's 2006 paper on a fast learning algorithm for deep belief nets appears in the same citation trail^[2]. These references place the 2010 workshop after earlier work on deep belief networks and after an earlier speech-focused workshop.

What happened

At NIPS 2010, the workshop provided a dedicated venue for work combining deep models with unsupervised feature learning. Yu, Deng and Dahl presented "Roles of pre-training and fine-tuning in context-dependent DBN-HMMs for real-world speech recognition"^[1]. The title points to three ingredients: pre-training, fine-tuning, and context-dependent DBN-HMM acoustic models. The same NIPS 2010 also had a main-conference paper, "Phone recognition with the mean-covariance restricted Boltzmann Machine", published on 06 December 2010^[2]. Its subject, phone recognition with a mean-covariance restricted Boltzmann machine (RBM), shows that RBM-based acoustic modeling was part of the broader NIPS 2010 program, not only the workshop.

Key details

A hidden Markov model (HMM) supplies a sequence model for speech: it represents how the sound units follow one another over time. A deep belief network (DBN) is a stack of stochastic layers that can be trained layer by layer. In a DBN-HMM, the deep network helps estimate the acoustic evidence, while the HMM handles the sequence. Context-dependent modeling makes the unit more specific than a single phone: it can represent a phone in the context of neighboring sounds. Pre-training initializes the deep model from unlabeled data, and fine-tuning adjusts it for the labeled recognition task. The workshop paper's title makes the roles of these two stages its explicit subject in real-world speech recognition^[1].

Citation trail

The IEEE/CAA reference list connects the workshop paper to later journal work. It cites Dahl, Yu, Deng and Acero's 2012 paper in IEEE Transactions on Audio Speech and Language Processing on context-dependent pre-trained deep neural networks for large-vocabulary speech recognition^[1]. It also cites Hinton, Deng, Yu, Dahl, Mohamed, Jaitly, Senior, Vanhoucke, Nguyen, Sainath and Kingsbury's 2012 paper in IEEE Signal Processing Magazine on deep neural networks for acoustic modeling in speech recognition, described as the shared views of four research groups^[1]. A third later item is Seide, Li and Yu's Interspeech 2011 paper on conversational speech transcription using context-dependent deep neural networks^[1]. These are not workshop papers, but they show the vocabulary and methods discussed at the workshop continuing in peer-reviewed venues.

Why it matters

The workshop's importance is as a meeting point. The citation trail runs from the 2006 fast learning algorithm for deep belief nets, to the 2009 speech-focused NIPS workshop, to the 2010 workshop and the NIPS 2010 main-conference paper on phone recognition with a mean-covariance RBM, and then to the 2011 Interspeech paper and the 2012 IEEE journal papers^[1][2]. That trail documents how pre-training and fine-tuning were discussed for speech recognition, and how unsupervised feature learning and RBMs were being connected to phone recognition. For a general reader, the event is a window into a moment when deep networks were being adapted to a difficult sequence problem, with workshop papers, conference papers and journal papers forming a connected record.