Tencent open-sources Hunyuan-Large, a 389B-parameter MoE model

Background

Hunyuan-Large is a mixture-of-experts (MoE) model. Its parameters are divided among many specialized sub-networks, and only some of them are active for each input. That design lets a model carry a very large total parameter count while using only a fraction of it for each step ^[1].

The total parameter count of a mixture-of-experts model is not the number of parameters used at each step. Storing the full model still requires memory, so the design trades memory for lower computing cost per token. Open weights mean the trained parameter values are published, so anyone with enough hardware can load and inspect the model.

What happened

Tencent submitted the Hunyuan-Large technical report to arXiv on 4 November 2024 ^[1]. On 5 November 2024 it open-sourced the Pretrain, Instruct and Instruct-FP8 versions on Hugging Face ^[2]. The code was released on GitHub, and the weights were published on Hugging Face under the name tencent/Tencent-Hunyuan-Large ^[1].

Key details

The model has 389 billion total parameters and 52 billion activated parameters. The paper called it the largest open-source Transformer-based MoE model ^[1]. It supports up to 256K tokens and was pre-trained on 7 trillion tokens, nearly 1.5 trillion of them synthetic ^[1]. Tencent reported 88.4% on MMLU, above the 85.2% of Llama 3.1 405B, with only 52 billion activated parameters ^[1].

MMLU is a multiple-choice test covering many academic subjects, and the scores below are Tencent's reported figures on that test.

Impact

The open release made the code and weights available to outside users, who could study and test a model of this size directly ^[2]. The results above are Tencent's own reported figures, and readers may want to compare them with independent evaluations.

Models of this size need substantial hardware to run, which limits direct use to organizations with significant computing resources.