Meta releases SAM 2 for video and image segmentation

Background

Promptable segmentation means that a user gives a model a hint, such as a click or a box drawn around an object, and the model returns the outline of that object as a mask. The task is harder in video than in still pictures. The model has to keep following the same object as it moves, changes shape and reappears from one frame to the next. Meta's earlier Segment Anything Model (SAM) handled images. SAM 2 extended the approach to video.

What happened

On 29 July 2024 Meta released SAM 2, a unified model for real-time promptable segmentation in both images and videos.^[1] "Unified" here means one model handles both kinds of input, so people do not need one tool for pictures and another for clips. The model is designed to follow the selected object across the frames of a video.

Key details

Speed. Meta said SAM 2 runs at about 44 frames per second. It does this with a streaming memory design, which processes frames one after another and keeps a memory of what it has already seen, instead of treating every frame as a separate problem.^[1]

Fewer corrections. Meta said SAM 2 needs about three times fewer interactions than earlier video approaches. An interaction here means a point, click or other correction a person must supply to get the right mask. Meta also said SAM 2 is six times faster than the original SAM when working on images.^[1]

Training data. Meta released the SA-V dataset with the model. It contains about 51,000 videos and over 600,000 masklets, where a masklet is an object mask followed across the frames of a video. The videos come from 47 countries. The dataset is licensed under CC BY 4.0, which allows reuse as long as the creator is credited.^[1]

Licences. The code and the model weights were released under the Apache 2.0 licence.^[1]

Follow-up. An improved checkpoint, SAM 2.1, followed in autumn 2024.^[1]

Impact

Because the code, weights and dataset were all published under open terms, researchers and developers could download the model, test it and adapt it to their own work. The combination of speed and open terms made SAM 2 a practical base for video tasks that need to find an object and keep following it, such as editing, analysis and labelling. Its successor line of work is covered in the later SAM 3 entry in this archive.^[1]