Sepp Hochreiter

Overview

Sepp Hochreiter is a German computer scientist, born on 14 February 1967 in Mühldorf, West Germany.^[1] He is best known for co-developing the long short-term memory (LSTM) network, a type of recurrent neural network. A recurrent network reads data step by step, such as the words of a sentence, and carries an internal state forward from one step to the next. Training such networks over long sequences is difficult, and the LSTM was designed to deal with that difficulty.

Education and early research

In 1991 Hochreiter wrote a diploma thesis at the Technical University of Munich. The thesis analysed why error signals vanish or explode when recurrent networks are trained by gradient descent, the method that adjusts connection strengths step by step so that the network's error becomes smaller. The thesis also contained the first LSTM results.^[1]

He completed his doctorate at the same university in 1999 under Wilfried Brauer. His dissertation addressed generalisation in neural networks of low complexity, that is, how well a small network performs on examples it was not trained on.^[1]

Major contributions

In 1997 Hochreiter and Jürgen Schmidhuber published the paper Long Short-Term Memory in the journal Neural Computation. Hochreiter wrote it while he was at the computer science faculty of the Technical University of Munich.^[3] The central idea is a memory cell controlled by gates. The gates are learned units that decide how much new information enters the cell, how much stored content is kept, and when content is read out. Because the cell can hold a value across many steps, error signals that would otherwise fade over a long sequence can still reach earlier steps during training.

His other work includes Flat Minimum Search, rectified factor networks and modern Hopfield networks with continuous states. He also developed factor-analysis methods for bioinformatics data, including FABIA and FARMS.^[1]

In May 2024 Hochreiter was the senior author of xLSTM (Extended Long Short-Term Memory). The model scales LSTMs to billions of parameters using exponential gating and new memory structures. The authors report that its language-modelling results are competitive with Transformers, the architecture behind most current large language models.^[5]

Career

Hochreiter has worked at TU Berlin, the University of Colorado Boulder and the Technical University of Munich, and then moved to Johannes Kepler University (JKU) Linz in Austria.^[1] At JKU he led the Institute of Bioinformatics from 2006 to 2018 and has led the Institute for Machine Learning since 2018. In 2017 he became head of the Linz Institute of Technology (LIT) AI Lab. He is also a founding director of the Institute of Advanced Research in Artificial Intelligence (IARAI).

Influence

In 2021 the IEEE Computational Intelligence Society gave him its Neural Networks Pioneer Award for his LSTM work.^[4] Before Transformer models became dominant, LSTM networks were a common choice for learning from sequences such as speech, text and time series. The xLSTM work shows that the approach is still being developed.