An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT)

The question

By the time this paper appeared, the Transformer architecture had become the de-facto standard for natural language processing. In computer vision, however, its use had remained limited: it appeared in conjunction with convolutional networks, or inside them, rather than on its own ^[1]. Convolution — sliding small filters across an image to pick up local patterns — was treated as the default foundation for a vision model. The authors, Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov and nine further co-authors, put the question bluntly: is this reliance on convolutional networks actually necessary? The paper was submitted to arXiv on 22 October 2020, and its answer is that it is not ^[1].

The method

The method is to take the convolution out entirely. An image is cut into small square patches, and each patch is treated the way a word is treated in a language model: as one element of a sequence ^[1]. The sequence of patches is fed into a standard Transformer, and the model is trained to assign the image to a class. Apart from the step that splits the image into patches, the model is an ordinary Transformer — a pure transformer applied directly to sequences of image patches ^[1].

What the Transformer contributes here is attention: when it builds a representation for one element of the sequence, it can weigh all the other elements at once. A patch is therefore not restricted to information from its immediate neighbours; it can draw on any other part of the image. This correspondence between patches and words is what the paper's title points at, describing an image as worth 16x16 words ^[1].

The paper defines three model sizes: ViT-Base with 86M parameters, ViT-Large with 307M parameters and ViT-Huge with 632M parameters ^[1].

The results

ViT was pre-trained on large amounts of data and then transferred to a range of mid-sized and small image recognition benchmarks, among them ImageNet, CIFAR-100 and VTAB. On these it attained excellent results compared with state-of-the-art convolutional networks ^[1]. The authors also report that ViT requires substantially fewer computational resources to train than comparable convolutional networks ^[1].

Why it matters

The paper's claim is narrow and firm: the reliance on convolutional networks in vision is not necessary, because a pure Transformer working on image patches can perform very well on image classification ^[1]. That removes an assumption that had shaped how vision models were built, and it opens a different set of design choices — the machinery developed for sequence models becomes available for images without a convolutional backbone wrapped around it.

The second result matters on its own terms. If a model of comparable quality can be trained with substantially fewer computational resources ^[1], then the cost of building and experimenting with such models is lower, and the range of groups that can do that work is wider.

What the result depends on

The strong numbers were obtained in a specific setting: pre-training on large amounts of data, followed by transfer to other recognition benchmarks ^[1]. The task examined is image classification ^[1]. The three published sizes — 86M, 307M and 632M parameters — show that the comparison with convolutional networks was made at substantial scale ^[1].

Read together, the pieces form a simple argument: cut the image into patches, treat the patches as a sequence, let attention relate them, and the convolution is no longer required to get excellent classification results ^[1].