Hunyuan-DiT

What it is

Hunyuan-DiT is a text-to-image diffusion transformer from Tencent, designed with fine-grained understanding of both English and Chinese prompts. The technical report was posted to arXiv on 14 May 2024, and the code and pretrained models were released publicly at the same time^[1].

A diffusion model learns to produce an image by starting from noise and removing it step by step, with each step guided by the text prompt. Hunyuan-DiT does this inside a latent space, meaning it works on a compressed representation of the image rather than on raw pixels, and it uses a transformer as its backbone rather than the U-Net design common in earlier image models^[2].

How it works

The text prompt reaches the model through two encoders. A pre-trained bilingual CLIP encoder with 350 million parameters handles the alignment between English or Chinese text and images, and a multilingual T5 encoder with 1.6 billion parameters supplies additional text representation^[2]. The generative model itself has 1.5 billion parameters, a size it keeps across versions v1.0, v1.1 and v1.2^[2]. Splitting the prompt work between a comparatively small alignment encoder and a larger multilingual text encoder is what lets the same model handle prompts in either language without a separate Chinese-only build.

On the data side, Tencent trained a multimodal large language model to refine image captions, so that the descriptions used in training carry more detail than raw web captions typically do^[1]. A separate captioning model, Hunyuan-Captioner, was released on 27 June 2024^[2].

The system also supports multi-turn multimodal dialogue, in which images are generated and then refined using the context of the conversation rather than a single fixed prompt^[1]. A companion DialogGen model with 7 billion parameters provides prompt enhancement for this kind of interaction^[2]. A TensorRT version and a multi-turn text-to-image demo were released on 22 May 2024^[2].

Evaluation

In a human evaluation with more than 50 professional evaluators, Hunyuan-DiT scored 59.0% overall. That put it ahead of the open models SDXL (42.7%), PixArt-α (45.5%) and Playground 2.5 (54.3%), and ahead of the closed SD 3 (56.7%). It trailed MidJourney v6 at 63.3% and DALL-E 3 at 71.0%^[2]. The authors also claim a new state of the art for Chinese-to-image generation among open-source models^[1].

Read together, these numbers describe a model that was competitive with the strongest openly available image generators of its moment and clearly behind the two leading closed systems, while being unusually strong on Chinese prompts relative to other open releases.

Versions and requirements

Hunyuan-DiT was updated several times in its first months. Version 1.1 arrived on 13 June 2024 to mitigate image oversaturation, and version 1.2 followed on 8 July 2024^[2]. Support in the Diffusers library was added on 5 June 2024, which made the model usable through a widely adopted inference toolkit rather than only through its own repository^[2].

Hardware demands are moderate for this class of model. The GitHub repository reports a peak of 11 GB of GPU memory for Hunyuan-DiT alone on an A100 at batch size 1, and 14 GB on an RTX 3090 or 4090^[2]. Full-parameter training needs at least 20 GB on a single GPU, with about 30 GB recommended^[2].

Where it sits

Hunyuan-DiT belongs to the generation of image models that replaced U-Net diffusion with transformer backbones, and it pairs that architecture with an explicit bet on bilingual text understanding. Releasing weights, code and a technical report alongside the announcement made the model usable and checkable by outside researchers, and the sequence of version updates and library integrations over its first two months shows the release was maintained rather than dropped as a one-off demo. The human evaluation numbers give a concrete picture of where it stood: better than the open competition, short of the closed frontier, and distinctive mainly in how well it handled Chinese.