Stability AI publicly releases Stable Diffusion

Background

Text-to-image generation is the task of producing a picture from a written description. Stable Diffusion belongs to the family of diffusion models, which learn to turn random noise into an image through many small denoising steps. Its design is a latent diffusion model: instead of working on every pixel, it runs the diffusion inside a compressed mathematical representation of the image, which lowers the computing power needed. The method was described in a research paper presented at CVPR 2022, a computer vision conference.

What happened

On 22 August 2022 Stability AI publicly released Stable Diffusion. The release made the model weights, a model card and the code available on Hugging Face. ^[1][2] Before this, Stability had announced a first-stage release to researchers on 10 August. ^[1][2]

Hugging Face described the model as created by researchers and engineers from CompVis, Stability AI and LAION. Stability's launch announcement also credited Runway, the Machine Vision & Learning group at LMU Munich and EleutherAI. ^[3][2]

Key details

The model has an 860-million-parameter UNet, a neural network that predicts and removes noise at each step, and a frozen CLIP ViT-L/14 text encoder with 123 million parameters. The text encoder turns the prompt into numbers that guide the image generation. Frozen means its weights are not changed during training. ^[4][5]

The training images were 512x512 pixels, taken from a subset of the LAION-5B database. Stability named LAION-Aesthetics as the core dataset. ^[4][2]

Stability said the model runs on consumer GPUs with less than 10 GB of VRAM. It was trained on the Ezra-1 cluster, a set of 4,000 A100 GPUs. Before the public release, more than 10,000 beta testers were creating 1.7 million images a day. ^[2]

The weights were released under the CreativeML OpenRAIL-M license. The license allows commercial and non-commercial use, with use restrictions that must be passed on to anyone who redistributes the model. ^[1]

The public release included an AI-based safety classifier, enabled by default. Stability also offered its DreamStudio service. ^[1]

The method behind the model comes from the paper High-Resolution Image Synthesis with Latent Diffusion Models, by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Bjorn Ommer. It was a CVPR 2022 paper. ^[6]

Impact

The significance of the release lies in the fact that the weights, the code and the license were all public. Anyone with suitable hardware could run the model locally instead of relying only on a hosted service, and the license set out clear terms for both commercial and non-commercial use. The number of beta testers and their daily image output show that interest was already large before the public launch. ^[1][2]