Meta unveils Movie Gen media generation models

Background

Generating a video from a text description is harder than generating a single image. Each frame has to match the frames around it, and motion has to look believable over time. Video models therefore learn both appearance and movement from large collections of examples. Movie Gen was Meta's attempt to cover video and sound together in one family of models. A foundation model is a large model trained on broad data that can be used for many tasks, rather than one narrow task.

What happened

On 4 October 2024 Meta premiered Movie Gen, a set of media foundation models for video, image and audio.^[1] Meta presented the set as covering three media.^[1]

Key details

Video model. The main video model is a 30-billion-parameter transformer. It generates videos up to 16 seconds long at 16 frames per second.^[1] A transformer is the type of neural network architecture that most large models use today. Parameters are the numbers inside the model that are set during training, and a larger count usually means a bigger model.

Audio model. The audio model has 13 billion parameters. It generates up to 45 seconds of sound that is synchronised to the video.^[1] Synchronised audio means the sound lines up with what happens on screen, such as a door closing at the moment it appears to close.

Personalised video. Movie Gen can make personalised videos from a photo of a person.^[1] This is different from a general text-to-video prompt, because the person's appearance comes from the photo the user supplies.

Editing with text. The model also supports text-based video editing.^[1] This means changing an existing clip through written instructions, rather than by editing each frame by hand.

Evaluation. Meta said that human evaluators preferred Movie Gen over competing industry models.^[1] Human preference tests ask people to compare outputs side by side. They are a common way to judge generated media, which is hard to score with a single number.

Not released. Meta did not release Movie Gen. It said the company would work with filmmakers and creators before any future release.^[1] A public release would let others download and run the models, so keeping them private meant they stayed under Meta's control.

Impact

Movie Gen showed Meta's work extending from images into video and sound. The company chose to show the models without making them public, and said it would work with filmmakers and creators before any future release.^[1]