Building a Large Annotated Corpus of English: The Penn Treebank

The problem the article addresses

Working out the grammatical structure of a sentence — which words group together, which verb takes which arguments — is one of the basic tasks in getting computers to handle language. To train a program on that task, or to measure how well it does, you need text in which people have already marked the structure by hand. The article is the published description of how one such corpus, the Penn Treebank, was assembled and annotated, written so that other researchers could see what the annotation contains and what it was designed to support^[1].

Where it appeared

The paper was published in 1993 in the journal *Computational Linguistics*, volume 19, number 2, pages 313–330, by MIT Press in Cambridge, MA, with Julia Hirschberg as editor. Its authors were Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz^[1].

How the text was chosen

Building the corpus began with a selection decision. From a three-year collection of the *Wall Street Journal* running to 98,732 stories, 2,499 stories were chosen for syntactic annotation^[2]. That choice fixes the character of the main corpus: it is edited American newspaper prose, a single, fairly formal register, rather than a sample of everything written in English.

What the annotation is

The marking used is bracketing. Brackets are placed around groups of words that function together in a sentence, so that the grouping is recorded explicitly in the file rather than left for a reader to infer. The release described in the paper's orbit uses a scheme known as Treebank II style^[2]. The design goal of the bracketing is stated plainly: it should allow simple predicate/argument structure to be extracted from the annotated text^[2]. In other words, a program reading the files should be able to recover the basic pattern of who did what to whom, without the annotation committing to every detail of a full syntactic theory.

What the release contains

Several bodies of material are described. The Treebank-2 release holds one million words of 1989 *Wall Street Journal* material annotated in Treebank II style^[2]. Alongside it sits a small sample of ATIS-3 material, annotated in the same style^[2] — a small window onto a different kind of language than newspaper writing. The release also includes a fully tagged version of the Brown Corpus^[2], giving part-of-speech information over a general corpus. Taken together, over one million words of text come with the Treebank bracketing^[2].

Why it matters

A corpus of this kind is what makes sentence-level work on English checkable. Because the structure is written down for each sentence, two researchers can ask the same question of the same data, and a parsing system can be trained on part of the material and tested on the rest. The annotation choices matter as much as the size: a scheme that supports extraction of predicate/argument structure keeps the data useful for questions about who did what, while a scheme with heavier theoretical commitments would be harder for others to reuse. The published article, with its account of selection and annotation style, is the document that lets people interpret the files correctly.

Limits

The main body of the corpus is one genre from one newspaper over a limited span, and that shapes what can be learned from it: vocabulary and sentence patterns reflect edited journalistic prose. The ATIS-3 sample and the tagged Brown Corpus add material of other kinds, but they are small next to the *Wall Street Journal* portion^[2]. Anyone using the treebank has to keep that in view when drawing conclusions about English in general.