Show and Tell: A Neural Image Caption Generator

A task that joins two fields

Automatic image description means producing a natural-language sentence for a photograph: a system is shown an image and has to say what is in it, in words rather than in labels. The paper frames this as a problem that connects computer vision and natural language processing^[1]. "Show and Tell: A Neural Image Caption Generator" was submitted to arXiv on 17 November 2014 under the identifier 1411.4555^[1]. Its authors are Oriol Vinyals, Alexander Toshev, Samy Bengio and Dumitru Erhan^[1].

How the model works

The authors describe their system as a generative deep recurrent architecture that combines advances in computer vision and advances in machine translation^[1]. The general shape of such a model is easy to state: the image is first turned into a compact numerical representation by a vision network, and a recurrent network then produces the sentence word by word. A recurrent network is one that keeps a running internal state and feeds its own previous output back in at each step, so what it has already written conditions what it writes next. The two halves are trained together rather than assembled from separately designed parts.

Training is stated as a single objective: the model is trained to maximise the likelihood of the target description sentence given the training image^[1]. In other words, the network is shown an image together with the sentence that people wrote for it, and its parameters are adjusted so that this sentence becomes more probable under the model. No extra annotation beyond image–sentence pairs is part of this objective.

Results

On the Pascal dataset, the authors report that the model reaches a BLEU-1 score of 59, against 25 for the then state of the art^[1]. BLEU is a scoring scheme that compares a generated sentence with one or more reference sentences by counting word sequences that the two share; the "1" in BLEU-1 means that only single words are counted, without looking at word pairs or longer phrases. Higher is better, and such scores are meaningful mainly when compared on the same dataset and the same metric setting, which is how the paper uses them: 59 next to 25 on Pascal.

Why it matters

The paper stresses that the language model learns fluent sentences solely from image descriptions^[1]. That point matters because it means the fluency of the output is not supplied by a separately written grammar, a sentence template or a hand-built rule set; it comes out of the training pairs themselves, as the network is pushed to make the human-written description more likely for its image. A single model thus covers what had usually been treated as two problems, seeing and saying, and is trained end to end on one objective.

The framing also matters. By treating captioning as a mapping from one kind of sequence-like input to a sequence of words — the way machine translation maps one language to another — the paper places image description inside the same family of generative models, and presents it as a meeting point of computer vision and natural language processing rather than a specialised vision task with a vocabulary attached^[1]. The reported gap on Pascal, 59 against 25, is the paper's evidence that this way of building a captioner was worth pursuing^[1].