Meta releases the Segment Anything Model and SA-1B dataset

Background

Image segmentation means marking which pixels of an image belong to which object. Traditional segmentation tools are usually built for a fixed set of object types. Meta set out to build a general-purpose model that could work with objects it had not been trained on.^[1]

What happened

On 5 April 2023 Meta released the Segment Anything Model (SAM), a promptable image segmentation model. Along with it, Meta released SA-1B, a dataset with over 1.1 billion masks on about 11 million images. Meta said this was 400 times more masks than any earlier segmentation dataset.^[1]

Key details

SAM segments objects from points, boxes or masks given as prompts. It can segment object types that were not seen in training.^[1]

Once the image embedding has been computed, SAM produces a mask in about 50 milliseconds, which makes it practical for interactive use.^[1]

The model was released under the Apache 2.0 licence, together with the code, the dataset and an online demo. The paper is arXiv:2304.02643.^[1][2]

Meta presented SAM as a step toward a foundation model for image segmentation, meaning one general model that can be prompted for many different segmentation tasks.^[1]

In SAM, a prompt is an input that tells the model where to look, such as a click on a point or a box drawn around an object. The model then returns a mask, a set of pixels that outlines the object. Masks are the output of image segmentation, so the dataset of over 1.1 billion of them is a large set of labelled examples.^[1]

Impact

Meta used SAM in Instagram's Backdrop feature, announced in September 2023.^[3]

Because the code, dataset and paper were all published, other researchers and developers could build on SAM directly. The dataset's scale also gave the field a large public resource for segmentation work.^[1][2]