Long Short-Term Memory

The paper at a glance

"Long Short-Term Memory" appeared in Neural Computation, volume 9, issue 8, pages 1735–1780, in November 1997. Its authors were Sepp Hochreiter of the Technische Universität München and Jürgen Schmidhuber of IDSIA in Lugano, Switzerland.^[1]

The problem

Recurrent networks trained with backpropagation through time or real-time recurrent learning had a serious flaw. Error signals flowing backward in time tend to blow up or vanish, so such networks took prohibitively long to learn, or failed, when they had to store information over long intervals. The paper reviews Hochreiter's 1991 analysis of this problem.^[1] In plain terms, a network asked to remember a detail from many steps earlier receives a training signal that is either too weak to matter or too strong to control.

The method

The central device is a memory cell built around a "constant error carousel", a path that keeps the error flowing back through time neither exploding nor vanishing.^[1] Gates regulate the cell. A multiplicative input gate protects the stored content from irrelevant inputs, and a multiplicative output gate protects other units from irrelevant stored content. The gates learn when to open and close.^[1] A helpful picture is a conveyor belt with doors: information rides along the belt unchanged, and the doors decide what gets loaded onto it and what gets taken off.

The results

An LSTM can learn to bridge minimal time lags of more than 1,000 discrete time steps, even with noisy and incompressible input sequences, and it keeps its ability to handle short lags. Its computational cost per time step and per weight is constant.^[1]

In experiments on artificial data, the authors compared LSTM with real-time recurrent learning, backpropagation through time, recurrent cascade-correlation, Elman nets and neural sequence chunking. LSTM succeeded in many more runs, learned much faster, and solved long-time-lag tasks that no earlier recurrent network algorithm had solved.^[1]

Why it mattered

The paper showed that a change in the design of the memory unit, not only in how the network is trained, could make learning across long gaps work. That gave researchers a practical tool for text, speech and other sequences, and the design has been reused and extended in many later models.^[1]

Later influence

In 2000 Felix Gers, Jürgen Schmidhuber and Fred Cummins added an adaptive forget gate, which lets a cell learn to reset its state on continuous input streams.^[2]

Deep LSTMs then powered neural machine translation. The 2014 sequence-to-sequence model of Ilya Sutskever, Oriol Vinyals and Quoc Le, and Google's 2016 GNMT system with eight encoder and eight decoder LSTM layers, both built on this architecture.^[3][4]

The 2017 Transformer paper opens by citing LSTM among the approaches "firmly established as state of the art" in sequence modelling, and then replaces recurrence with attention.^[5]