Stable Diffusion 3

What it is

Stable Diffusion 3 is a text-to-image model in the Stable Diffusion family from Stability AI. Stability announced it in early preview on 22 February 2024, calling it the company's most capable text-to-image model. Access during the preview was through a waitlist. ^[1]

Origins

The announced suite ranged from 800 million to 8 billion parameters. The model combines a diffusion transformer architecture with flow matching. A diffusion transformer uses a transformer, the attention-based network design widely used in language models, in place of the convolutional U-Net that earlier Stable Diffusion models used. Flow matching is a training approach that learns a direct path from noise to image. ^[1]

The technical paper, Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, was posted to arXiv on 5 March 2024. The authors are Patrick Esser, Robin Rombach and 15 others. ^[2]

Rectified flow is a way of training a generative model so that it learns to follow nearly straight paths from random noise to data. Straighter paths can allow generation with fewer steps. A transformer splits its input into tokens, and in this design image tokens and text tokens form separate streams that exchange information.

Main features

Stability said the model improved on multi-subject prompts, image quality and spelling. ^[1]

The paper describes a transformer with separate weights for image and text tokens, and two-way information flow between them. The paper reports better text comprehension, typography and human preference ratings. Stability calls this design the Multimodal Diffusion Transformer, or MMDiT. ^[2][4]

Its place in the development

The open weights of Stable Diffusion 3 Medium, a 2-billion-parameter model, were released on 12 June 2024 on Hugging Face, where the model is listed as stabilityai/stable-diffusion-3-medium. ^[3] Stable Diffusion 3 is thus both a preview of a large suite and the basis of a smaller open release. ^[3]