Recurrent neural networks and LSTM

What it is

A recurrent neural network (RNN) is a neural network whose connections loop back. The activity of its hidden units at one step is fed back in at the next step, which gives the network a dynamic memory of earlier inputs.^[1] This suits data that arrives in order, such as the words of a sentence or the sounds of speech, where the meaning of one item depends on what came before.

How it works

Consider reading the sentence "the dog sat". The network takes in one word at a time. After "the" and "dog", its hidden state holds a summary of what it has seen; when "sat" arrives, that state is combined with the new word to produce the next state. The same set of connection weights is reused at every step, so the network can handle sentences of any length, although in practice its memory of the distant past is limited.

Where it came from

Jeffrey Elman's paper "Finding Structure in Time", published in the journal Cognitive Science in March 1990, developed this approach. It built on a proposal first described by Jordan in 1986. Elman's simple recurrent networks learned tasks that ranged from a temporal version of XOR to discovering syntactic and semantic features of words.^[1]

Training uses gradient methods, such as backpropagation through time (BPTT) or real-time recurrent learning (RTRL). Gradient methods nudge each weight in the direction that reduces the error. The catch is that error signals flowing backward through time tend to blow up or vanish, because their size depends exponentially on the weights. Sepp Hochreiter analysed this in 1991.^[2] In plain terms, an error traced back through many steps either shrinks toward nothing or grows without bound, so the network struggles to connect distant events. In 1994 Yoshua Bengio, Patrice Simard and Paolo Frasconi published a paper whose title states the problem directly: "Learning long-term dependencies with gradient descent is difficult".^[3]

The LSTM fix

The long short-term memory (LSTM) network, published by Sepp Hochreiter and Jürgen Schmidhuber in Neural Computation in 1997, addressed this problem with a different kind of memory cell.^[2] Each cell is built around a "constant error carousel", a path along which the error keeps a steady size as it flows back through time. Gates control the cell: an input gate decides when new information may enter the memory, and an output gate decides when stored content may affect the rest of the network. A helpful picture is a conveyor belt with doors, where information rides along unchanged until a gate opens. The result is that an LSTM can bridge time lags of more than 1,000 discrete steps.^[2] In 2000 Felix Gers, Jürgen Schmidhuber and Fred Cummins added an adaptive "forget gate", which lets a cell learn to reset its own state.^[4]

Where it was used

LSTMs became a workhorse for language and speech. Deep LSTMs drove the rise of neural machine translation: in 2014 Ilya Sutskever, Oriol Vinyals and Quoc Le translated English to French with a four-layer LSTM encoder and decoder.^[5] Google's 2016 GNMT system used eight encoder and eight decoder LSTM layers, with attention. In human side-by-side evaluations on isolated simple sentences, it reduced translation errors by an average of 60% compared with Google's phrase-based production system.^[6]

Limits and what came next

An RNN processes a sequence one step after another. Each step needs the result of the step before, which makes it hard to parallelise computation within a single training example. The 2017 Transformer paper still called RNNs and LSTMs the established state of the art for sequence modelling, but it replaced recurrence with attention.^[7] Attention lets every position look at every other directly, and it runs well on modern parallel hardware. Today RNNs are less common in the largest systems, but the ideas of memory, gating and sequence modelling they introduced remain part of the field's vocabulary.