An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

The assumption the paper attacked

The starting point is stated plainly in the abstract: the Transformer architecture had become the de-facto standard for natural language processing tasks, while its applications to computer vision remained limited ^[1]. Where attention had been used in vision at all, it was either applied in conjunction with convolutional networks or used to replace certain components of convolutional networks while keeping their overall structure in place ^[1]. Convolution, in other words, was still treated as the load-bearing wall. The authors set out to test whether that reliance on CNNs was actually necessary ^[1].

Applying a Transformer to patches

The answer they give is yes, it is not necessary: a pure Transformer applied directly to sequences of image patches can perform very well on image classification tasks ^[1]. The resulting model is called the Vision Transformer, or ViT ^[1]. The construction is easiest to grasp through the paper's own title, which frames an image as a collection of 16×16 words. An image is cut into small fixed-size tiles; each tile is turned into a vector, and the sequence of these vectors is handed to a Transformer in the same way a sentence is handed over as a sequence of word vectors.

The point of using a Transformer here is that it imposes almost no assumption about the structure of its input. Its core operation, attention, lets every element of a sequence look at every other element and combine their information with learned weights, so the model can decide for itself which parts of the image need to be related to which. A convolutional network, by contrast, is built around a fixed local window that slides across the image. ViT keeps the general machinery of the Transformer and drops the convolutional scaffolding altogether.

Results

The paper's central empirical claim is a comparison. When pre-trained on large amounts of data and then transferred to multiple mid-sized or small image recognition benchmarks — ImageNet, CIFAR-100 and VTAB are the ones the abstract names — ViT attains excellent results compared with state-of-the-art convolutional networks, while requiring substantially fewer computational resources to train ^[1]. Both halves matter. The accuracy half says that a model with no convolution at all is competitive on standard recognition tasks; the resource half says that reaching that level does not demand more compute than the convolutional alternatives.

Why it matters

For years, the design of vision models had been organised around convolution, and attention was something added to a CNN or swapped into part of it ^[1]. The paper's contribution is to show that this arrangement is a choice rather than a requirement: the same architecture that had become standard for text can be pointed at pixels and work ^[1]. That makes the boundary between the two fields a matter of data representation rather than of architecture, and it gives vision researchers a second family of models to build on.

The authors also made the work usable. The paper states that fine-tuning code and pre-trained models are available in the google-research vision_transformer repository ^[1], so others could reproduce the results and fine-tune the models on their own data rather than only reading about them.

The record

The paper is by Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit and Neil Houlsby ^[1]. It was submitted to arXiv on 22 October 2020 as 2010.11929, and is classified under computer vision and pattern recognition, with cross-lists in artificial intelligence and machine learning ^[1]. The arXiv record shows a second version dated 3 June 2021, described as the ICLR camera-ready version with two small modifications ^[1].

What the paper does not do is settle the question of scale. Its claim is conditional on pre-training on large amounts of data before transfer to smaller benchmarks ^[1]; the comparison it reports is for image classification ^[1]. Those conditions define both the strength of the result and the questions it leaves for later work.