SAM 2: Segment Anything in Images and Videos

What the paper sets out to do

Segment Anything Model 2 (SAM 2) is a foundation model for promptable visual segmentation in images and videos, presented in a paper credited to Meta FAIR and submitted to arXiv on 1 August 2024 as 2408.00714 ^[1]. The aim is a single model that can be pointed at an object in a still picture or in a clip and return a mask for it, and that keeps that mask consistent as the object moves through the video ^[1].

The earlier Segment Anything Model, referred to in the paper as SAM, handled images only. SAM 2 is designed to cover both media with one system, which is why the paper's title speaks of images and videos together ^[1].

The task: promptable visual segmentation

The paper formalizes the problem as Promptable Visual Segmentation, or PVS. A user places a prompt — a point, a box or a mask — on any frame of a video, and the model predicts a spatio-temporal mask, which the paper calls a masklet. The prediction is not fixed: prompts on later frames can be used to refine it ^[1].

This is more demanding than segmenting a single frame. A masklet has to be right about where an object's boundary lies and also about how that boundary carries forward in time, including when the object is partly hidden or changes shape.

How the model works

The architecture is a simple transformer with streaming memory that processes video in real time ^[1]. The memory is what lets information about an object persist from earlier frames into later ones, so the masklet stays attached to the same object across the clip.

A central design choice is to treat an image as a video with a single frame, so one model covers both cases. On images the memory is simply empty, and the model behaves like SAM ^[1]. The video capability is therefore not a separate module bolted onto an image model; images and video share the same machinery, and the difference between them is only whether there is anything in memory.

Building the data

A model of this kind needs training data at a scale that did not exist for video segmentation, so the team built a data engine that improves both the model and the data through user interaction, and used it to collect the largest video segmentation dataset to date ^[1].

The released SA-V dataset contains 35.5 million masks across 50.9 thousand videos, which is 53 times more masks than any existing video segmentation dataset ^[1]. The gap in mask count is the point: video segmentation had been limited less by model design than by the absence of annotated footage, and the data engine was the method for producing it.

Results

On video segmentation, SAM 2 is more accurate than prior approaches while requiring 3 times fewer interactions ^[1]. On image segmentation, it is more accurate than SAM and 6 times faster ^[1]. The two results follow from the same design: the video setting gains from carrying memory across frames and from a model trained on far more annotated video, while the image setting gains from an architecture that is quicker to run than the original SAM.

What was released and why it matters

The main model, the dataset, the training code and a demo were all released with the paper ^[1]. Releasing the weights, the data and the training code together means other groups can reproduce the model and adapt it rather than only compare against reported numbers.

The paper's contribution is therefore twofold. On the modelling side, it shows that promptable segmentation can be expressed as a single task over images and video by treating an image as a one-frame video and adding streaming memory. On the data side, it produces a video segmentation dataset an order of magnitude beyond what existed, and a data engine for extending it. Both are aimed at the same end: making segmentation with prompts a general capability that works on moving footage, not only on still pictures ^[1].