Stability AI previews Stable Diffusion 3

Background

Text-to-image systems turn a written prompt into a picture. Most of them are diffusion models: networks trained to remove noise step by step, so that starting from random noise they gradually arrive at an image that fits the prompt. The quality of such a system depends on the network that does the denoising, on how that network is trained, and on how well it ties the words of the prompt to the pixels it draws.

The announcement

On 22 February 2024 Stability AI announced Stable Diffusion 3 in an early preview, with a waitlist rather than an open release.^[1] The company presented it as a suite of models spanning 800M to 8B parameters, so that the same generation could be offered at several sizes.^[1]

Parameter count is a rough measure of a model's size: the number of weights it has learned. A larger model can hold more capacity but costs more to run, so a range from 800M to 8B gives users a choice between speed and capability.

How it works

Stability said the model combines a diffusion transformer architecture with flow matching.^[1] A diffusion transformer uses transformer blocks — the kind of building block also used in language models — as the network that removes noise, in place of the convolutional networks that diffusion image models had typically relied on. Flow matching is a way of training that network: instead of only learning to predict the noise that was added to an image, the model learns a path that carries noise toward data and is trained to follow it.

What Stability said it improved

The company named three areas: prompts that describe several subjects at once, general image quality, and spelling — rendering legible words inside the generated picture.^[1] Multiple subjects are hard because the model has to keep several described things distinct and place them consistently; spelling is hard because letters are fine detail that has to be drawn exactly rather than plausibly.

Safety

Stability said it considered safety from the training stage onward, and that it took reasonable steps during testing, evaluation and deployment to prevent malicious misuse; the preview was also released with added safeguards.^[1]

The technical paper

A technical paper followed on arXiv on 5 March 2024, "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis".^[2] It describes rectified flow as a generative model formulation that connects data and noise in a straight line, and presents a transformer-based architecture for text-to-image generation that allows bidirectional flow of information between image and text tokens.^[2] Bidirectional here means the text tokens can attend to the image tokens and the image tokens to the text tokens, so that wording and pixels are adjusted together rather than one side being fixed while the other is produced.

Why it matters

The preview put two ideas at the centre of Stability's next generation of image models — a transformer backbone and a flow-based training objective — and coupled the announcement with a paper explaining the method shortly afterwards.^[1][2] Releasing it as a staged preview with a waitlist and added safeguards also set the tone for how the company introduced the model: limited access first, with safety measures described as part of the process from training onward.^[1]