Deep contextualized word representations

The problem

Most neural networks for language start by turning each word into a vector. In the usual setup the vector is looked up in a fixed table, so a word gets the same vector wherever it appears. Language does not work that way: a word's meaning shifts with the sentence it sits in, and its grammatical role shifts too. This paper introduces a type of deep contextualised word representation that models both complex characteristics of word use, such as syntax and semantics, and how those uses vary across linguistic contexts^[1]. The paper, "Deep contextualized word representations", was first submitted to arXiv on 15 February 2018 as 1802.05365^[1]; its first author is Matthew E. Peters, and it has six authors in total^[1].

How it works

The word vectors here are not a table to be looked up. They are learned functions of the internal states of a deep bidirectional language model pre-trained on a large text corpus^[1]. A language model is a network trained on plain text to assign high probability to text that actually occurs, so it has to absorb a great deal about how words combine. "Bidirectional" means the network reads the text in both directions, so the state at a given position can draw on what comes before and what comes after. To represent a word in a sentence, the sentence is run through this pre-trained network and the internal state at that position is read off. Because that state depends on the whole surrounding sentence, the same word ends up with different vectors in different contexts, which is what "contextualised" means here.

What the analysis shows

An analysis in the paper shows that exposing the deep internals of the pre-trained network is crucial, letting downstream models mix different layers of representation^[1]. The useful signal, in other words, is not only the final output of the network. Giving a downstream model access to intermediate layers, and letting it decide how to combine them, is part of why the representation works as well as it does. This makes the internal structure of the pre-trained model something to be used on purpose, rather than a black box whose last layer is the only thing worth keeping.

Results

The authors show the representations can be easily added to existing models^[1]. Doing so significantly improves the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis^[1]. The key word is "added": the downstream systems did not need a newly designed architecture for each task. A general-purpose representation computed from unlabelled text was plugged in as an extra input, and the existing model did the rest.

Why it matters

Two things stand out. First, the work separates what is learned from raw text from what is learned from task labels. A large corpus supplies the language knowledge once, in the form of a pre-trained network; a downstream task then reuses that knowledge instead of starting from scratch. Second, it treats the layers inside a deep network as a resource. The finding that downstream models benefit from mixing several layers suggests that intermediate representations carry different, complementary information, and that choosing how to expose them is a design decision worth making carefully. For anyone building a system for a new language task, the practical message is that a strong contextual representation of words can be borrowed rather than rebuilt, and that the way it is wired into the rest of the model matters as much as the representation itself.