Tencent open-sources the Hunyuan-DiT text-to-image model

Background

Text-to-image models produce pictures from written descriptions. Hunyuan-DiT is Tencent's entry in this area: a diffusion transformer designed to handle prompts in English and Chinese ^[2].

A diffusion model learns to reverse a process that gradually adds noise to images, so it can start from noise and produce a picture that follows a text prompt. A text encoder turns the words of the prompt into numbers the image model can use. CLIP is a widely used encoder that learns to match images with their captions, and T5 is a text-only language model encoder. Using two encoders in one system gives the image model two different views of the same prompt.

What happened

On 14 May 2024 Tencent posted the Hunyuan-DiT technical report to arXiv and publicly released the code and pretrained models ^[1]. The model is a 1.5-billion-parameter diffusion transformer for text-to-image generation in English and Chinese ^[2]. It encodes prompts with a bilingual CLIP encoder and a multilingual T5 encoder ^[2].

Key details

A human evaluation with more than 50 professional evaluators set, by the authors' account, a new state of the art for Chinese-to-image generation among open-source models ^[1]. The overall human-evaluation score was 59.0%. That trailed MidJourney v6 (63.3%) and DALL-E 3 (71.0%) ^[2]. Evaluation scores like these come from the authors' own study, so they are best read alongside other independent comparisons.

Follow-up versions 1.1, released on 13 June 2024, and 1.2, released on 8 July 2024, were published in the same repository ^[2].

Impact

The release put an open, bilingual image generation model in the hands of researchers and developers, who could obtain the code and pretrained models from the repository ^[1][2]. Because the model accepts both languages, its prompts do not have to be written in English.

Human evaluation results also depend on the prompts and evaluators chosen, so readers may want to compare these numbers with other independent tests.