Segment Anything

What the paper is after

Image segmentation is the job of carving a picture up into meaningful regions — separating an object from its background, a road from the cars on it, one organ from another in a scan. For years the standard recipe was to train a separate model for each such task, on a dataset built for that task, with every pixel labelled by hand. The Segment Anything paper asks a different question: what would it take to have a single segmentation model that, like a language model, can be pointed at a new situation and simply work?

The authors frame their answer as a package of three things that have to be designed together — a task, a model, and a dataset — rather than a model alone ^[1].

A promptable task and a promptable model

The task the paper defines is promptable segmentation: the model is given an image together with a prompt that says what to segment, and returns a segmentation mask. Crucially, the model is designed and trained to be promptable, which is what lets it transfer zero-shot to new image distributions and tasks — that is, to handle kinds of images and kinds of segmentation problems it was never explicitly trained on, without further tuning ^[1].

That is a deliberate departure from the usual arrangement, in which a model's competence is tied to its training distribution. A promptable model is meant to be reused: the same weights, aimed at whatever problem the user has in front of them.

Building the data with the model

A model like this needs an unusually large and varied body of segmentation data, and hand-labelling at that scale is not realistic. The team's solution was to put a fast, efficient model inside the data-collection loop, so that annotators could work with model assistance instead of drawing every mask from scratch. Using this loop, they built what they describe as the largest segmentation dataset to date: over 1 billion masks on 11 million licensed and privacy-respecting images ^[1].

The model and the data therefore grew together, each making the other more useful — the model speeds up annotation, and the resulting data improves the model.

Results

The paper reports that, across numerous tasks, the model's zero-shot performance is impressive, often competitive with or even superior to prior fully supervised results ^[1]. In other words, a system that was not trained on a given benchmark can match systems that were trained specifically for it. That is the central empirical claim of the work, and it is what makes the foundation model framing plausible for segmentation rather than merely aspirational.

What was released, and why it matters

The paper releases both the Segment Anything Model and the SA-1B dataset of 1 billion masks and 11 million images, explicitly to foster research into foundation models for computer vision ^[1]. Making the data public matters as much as making the model public: progress on vision foundation models depends on having large, diverse segmentation data that researchers can build on and measure against, and such data had previously been scarce.

The paper itself

Segment Anything is by Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar and Ross Girshick, and was submitted to arXiv on 5 April 2023 as 2304.02643 ^[1].