Stability AI releases Stable Diffusion 2.0

Background

Stable Diffusion turns a text prompt into an image through several parts working together. One of them is the text encoder, which converts the prompt into numbers. Changing the encoder changes how the image model understands the words it is given. Another part is the training data, which decides what kinds of pictures the model learns to draw. Both choices shape the result, and the release of version 2.0 was a change to each of them.

What happened

On 24 November 2022 Stability AI released Stable Diffusion 2.0 as open source. ^[1] The release replaced the CLIP text encoder with OpenCLIP, a text encoder developed by LAION with Stability's support. ^[1]

The new version generates images at 512x512 and 768x768 pixels by default. It added a 4x upscaler, a depth2img model that is guided by depth information, and an updated inpainting model. ^[1]

Training data filters work as classifiers. A filter scores each image or caption in a dataset and drops the ones below a chosen threshold. Because a filter removes material before training, its settings shape both what the model learns to draw and how often certain subjects appear in its training. Changing the filter is therefore a decision about the model's output, not only about the dataset.

Key details

The training data was an aesthetic subset of LAION-5B. The subset was filtered for adult content with LAION's NSFW filter. ^[1]

Stable Diffusion 2.1 followed on 7 December 2022. Its NSFW filtering was looser, which was intended to improve how the model depicts people. ^[2]

Impact

Version 2.0 showed that an open release could combine a new text encoder, a new filter and new capabilities such as upscaling and depth-guided generation in one package. The 2.1 follow-up then adjusted the filter after the first version's approach was seen to have removed too many people from the data. ^[1][2]