Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

The problem

Describing a photograph in one sentence is a small task with a very large search space. A picture contains far more than a caption can mention: a caption picks out a few objects, perhaps an action and a setting, and leaves everything else out. The model therefore has to decide not only which words to use but, implicitly, which parts of the image each word is about. Getting a sentence that is grammatical is one problem; getting one that is about the right things is another.

How the model works

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel and Yoshua Bengio posted the paper to arXiv on 10 February 2015 ^[1]. It introduced an attention-based model that automatically learns to describe the content of images ^[1].

Attention here means a weighting over parts of the input. Instead of treating the image as one undifferentiated whole, the model can give more weight to some locations than to others at each step of producing a sentence. The weights shift as the sentence is written, so the region that matters for one word need not be the region that matters for the next. The authors said the model can report where it is looking while generating a caption ^[1]. That is the practical consequence of the design: the same weights that shape the sentence can be read out as a location, so a reader can see which part of the picture was emphasised for which word.

Two kinds of attention

The paper described both a hard deterministic attention mechanism and a soft deterministic attention mechanism ^[1]. In the usual vocabulary of attention, a hard mechanism commits to a single location at each step, like a spotlight that must point somewhere; a soft mechanism spreads weight over many locations at once, like a transparency laid over the picture, so every region contributes a little. Both variants in the paper are described as deterministic, meaning the attention is computed from the model's own state rather than sampled at random. Offering both made it possible to compare two ways of doing the same job within one framework.

Results

The authors reported that the attention mechanism produced more descriptive and accurate captions than a comparable model without attention ^[1]. Because the comparison holds the rest of the model roughly fixed, it isolates attention itself as the source of the gain rather than treating it as one ingredient among many.

The second result concerns inspection rather than scores. The paper said that visualising the learned attention showed that the model learned to fix its gaze on salient objects while generating the corresponding words ^[1]. In other words, the alignment between words and image regions was not imposed by hand; it emerged from training, and drawing the attention made that alignment visible. The method was evaluated on the Flickr30k, COCO and Pascal datasets ^[1].

Why it matters

Two things are being claimed at once, and they are different in kind. The first is quality: attention gives better captions than a comparable model without it ^[1]. The second is legibility. A captioning model that takes an image as a single fixed representation produces a sentence and nothing else; a reader can judge the sentence but cannot see how it was arrived at. A model that reports where it is looking supplies an intermediate object that a person can check — here, a map of gaze over the image, tied to individual words ^[1].

That combination is why the paper is easy to describe in its own title. The model shows, in the sense of looking at a region; it attends, in the sense of weighting that region more heavily than others; and it tells, in the sense of emitting a sentence. The three parts are carried out by one network rather than by separate stages, and the middle part is not a hidden detail but something the authors could plot ^[1]. For anyone building or auditing systems that turn images into language, the paper set out a pattern that remains a common way to talk about the problem: a model whose output is a sentence and whose internal weighting can be read as a place in the picture.