Phone recognition with the mean-covariance restricted Boltzmann Machine

Overview

The paper Phone recognition with the mean-covariance restricted Boltzmann Machine was written by George E. Dahl, Marc'Aurelio Ranzato, Abdel-rahman Mohamed and Geoffrey Hinton and appeared in NIPS 2010, volume 1, pages 469 to 477, published on 6 December 2010 ^[1]. It studies acoustic modeling for phone recognition with Deep Belief Nets (DBNs) and a mean-covariance restricted Boltzmann machine (mcRBM) ^[1].

What problem did it address?

Phone recognition is the task of assigning phone labels to speech. A central question for acoustic modeling is how to represent acoustic input so that a recognizer can use it. The paper's abstract describes a straightforward application of DBNs to acoustic modeling: this produces a rich distributed representation of speech data and gives what the authors call impressive results on the speaker-independent TIMIT phone recognition task ^[1]. The work therefore focuses on the acoustic model itself and on how its input representation is built.

How it worked

A DBN is built from stacked restricted Boltzmann machines. An RBM has visible units and hidden units and learns a probability distribution over its visible inputs; stacking RBMs lets the model form layers of features. The paper's first-layer Gaussian-Bernoulli RBM treats different components of the acoustic input vector as conditionally independent given the hidden state ^[1]. This is a modeling choice: once the hidden state is fixed, the visible components are not modeled as dependent on one another in that layer.

The paper also uses an mcRBM. According to the abstract, the mcRBM was first introduced for modeling natural images and offers a more representationally efficient way of modeling the covariance structure of speech data ^[1]. In other words, this component is designed to capture how input components vary together. The reported system combines mcRBM features with DBNs ^[1].

Results

On speaker-independent TIMIT phone recognition, the combination of mcRBM features and DBNs achieved a phone error rate of 20.5 percent, which the abstract describes as superior to all published results on this task to date ^[1]. Phone error rate counts the percentage of phones recognized incorrectly, so a lower number is better.

Why it matters

The paper is a concrete demonstration that a deep belief network can be applied to acoustic modeling and can yield a distributed representation of speech data with strong benchmark results ^[1]. It connects unsupervised feature learning with RBMs to supervised phone recognition. For speech recognition, the interest is not only the final error rate but the representation: the model learns hidden structure from acoustic input, and the paper evaluates that representation on phone recognition ^[1].

Related work

The ACM reference list for the paper includes Mohamed, Dahl and Hinton's Deep belief networks for phone recognition, from the NIPS Workshop on Deep Learning for Speech Recognition and Related Applications in 2009 ^[1]. The IEEE/CAA reference list includes Yu, Deng and Dahl's Roles of pre-training and fine-tuning in context-dependent DBN-HMMs for real-world speech recognition, from the NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning ^[2]. These entries place the TIMIT result in a broader line of work on DBN-based acoustic models and on how pre-training and fine-tuning affect recognition.