CTC-trained LSTM wins three ICDAR 2009 handwriting competitions

Background

Handwriting recognition turns an image of handwritten text into a string of characters. The hard version is *connected* handwriting: cursive scripts in which letters run into one another, so there is no clean boundary between characters. A common approach was to segment first and classify second — cut the line into pieces, then recognise each piece. Such pipelines break down exactly when the segmentation is ambiguous, which in cursive writing is the normal case. Farsi and Arabic are written cursively, so the competitions on those scripts at the International Conference on Document Analysis and Recognition (ICDAR) posed precisely this problem.

What happened at ICDAR 2009

IDSIA reports that Alex Graves won three connected handwriting competitions at ICDAR 2009, for French, Farsi and Arabic ^[1]. Wikipedia's account records that a team led by Graves submitted three models, one of which was the most accurate in the competition and another the fastest ^[3]. Both accounts date the win to 2009 and describe it as the first time a recurrent neural network had won international competitions ^[1][3].

The method: LSTM plus CTC

The system combined LSTM with Connectionist Temporal Classification (CTC), both developed in Schmidhuber's research groups at TU Munich and IDSIA ^[1].

LSTM — long short-term memory — is a recurrent neural network: it reads the input one step at a time and carries a hidden state forward, so what it has already seen shapes what it does next. What separates it from a plain recurrent network is a memory cell regulated by gates: multiplicative units that decide how much new information is written in, how much of the old state is kept, and how much is exposed as output. The gates keep the error signal from vanishing or blowing up as it travels back through many time steps, which is what lets the network use context over long stretches of input.

CTC solves the labelling problem. A recogniser looking at a wide strip of handwriting has to produce a much shorter string of characters, and there is no character-by-character alignment between the two. CTC lets the network emit a distribution over characters, plus a special blank symbol, at every horizontal position, and trains it by summing over all the alignments that collapse to the correct string. At test time repeated labels are merged and blanks dropped. The practical consequence is that training needs only pairs of line images and transcriptions — nobody has to mark where each character begins.

Why it mattered

IDSIA presents the CTC-trained LSTM as the first recurrent neural network to win any international contests ^[1], and Schmidhuber's timeline goes further: to his knowledge this was also the first deep learning system of any kind, recurrent or not, to win such a contest ^[2]. The claim matters for its timing. It places a competitive win for a deep network in a period when hand-designed features and staged pipelines were still the default in document analysis; the neural network had to beat those pipelines on their own benchmark. It also shows the two ingredients working together — a recurrent architecture that can hold context, and an output layer that does not need segmented training data. The fact that the team took both the accuracy and the speed honours with different submitted models ^[3] indicates it was not a single lucky configuration.

Afterwards, pairing a recurrent network with a CTC output layer became a common recipe for sequence problems where input and output differ in length, such as speech recognition and handwritten text recognition. Seen from that later landscape, the ICDAR 2009 competitions were the first public contest in which this line of work was measured against others and came out ahead.