Latent Dirichlet Allocation

The problem it addresses

A text corpus is a collection of discrete data — words, counts, occurrences — and the paper's abstract describes latent Dirichlet allocation (LDA) as a generative probabilistic model for exactly that kind of collection ^[2]. The modelling question behind it is how to give a document a compact description that is statistically meaningful. LDA's answer is to describe a document through probabilities attached to topics: in text modelling, the topic probabilities provide an explicit representation of a document ^[2]. A topic here is not a label attached from outside; it is a recurring pattern of word usage that the model picks out of the corpus itself.

How the model is put together

LDA is a three-level hierarchical Bayesian model, and each item of a collection is modelled as a finite mixture over an underlying set of topics ^[2]. The hierarchy runs from the collection as a whole, through the individual document, down to the words that are actually observed. Because a document is a mixture rather than a member of a single class, it can draw on several topics in different proportions, and those proportions are what the model reports. The "Dirichlet" in the name refers to the distribution conventionally used for quantities that must be non-negative and sum to one, which is what a set of mixing proportions is.

Inference and estimation

In a model of this kind the quantities one wants — which topic each word came from, and in what proportion the topics enter a document — are hidden. The paper's route to them is approximate. The authors present efficient approximate inference techniques based on variational methods, together with an EM algorithm for empirical Bayes parameter estimation ^[2]. Variational methods replace the difficult posterior with a simpler family of distributions and then search inside that family for the member closest to the true one; the EM algorithm alternates between estimating the hidden structure under the current parameters and updating the parameters given that estimate. Working together, the two let the parameters of the model be estimated from data rather than fixed by hand.

Results

The paper reports results in document modelling, text classification and collaborative filtering, compared with a mixture of unigrams model and probabilistic LSI ^[2]. Three rather different tasks are handled by one model: the same mixture-over-topics description is used to summarise documents, to serve as features for classification, and to fill in missing entries of a rating matrix in collaborative filtering.

Publication

The Journal of Machine Learning Research Volume 3 lists Latent Dirichlet Allocation by David M. Blei, Andrew Y. Ng and Michael I. Jordan as 3(Jan):993-1022, 2003 ^[1]. In the paper, Blei and Jordan are listed at the University of California, Berkeley, and Ng at Stanford University, with John Lafferty as editor ^[2]. The same volume lists A Neural Probabilistic Language Model by Bengio, Ducharme, Vincent and Jauvin as 3(Feb):1137-1155, 2003 ^[1].

Why it matters

Two things carry the paper's importance. The first is representation: topic probabilities give a document an explicit description instead of leaving it as an undifferentiated bag of words ^[2]. The second is computation: the variational approximate inference and the EM procedure for empirical Bayes estimation are what make a three-level hierarchical model of this sort something that can actually be fitted ^[2]. Because the model is stated for collections of discrete data in general rather than for text alone, the same machinery carries over to other kinds of counts, which is why collaborative filtering appears among the reported experiments alongside the text tasks ^[2].