Hunyuan-Large

What it is

Hunyuan-Large is an open-weight, Transformer-based mixture-of-experts (MoE) model from Tencent. It has 389 billion total parameters, of which 52 billion are activated ^[1]. Tencent described it as the largest open-source Transformer-based MoE model at the time ^[1]. Its weights are published under Tencent's own license rather than a standard open-source license ^[3].

Origins

Tencent submitted the technical report to arXiv on 4 November 2024 ^[1]. On 5 November 2024 it released the Hunyuan-A52B-Pretrain, Hunyuan-A52B-Instruct and Hunyuan-A52B-Instruct-FP8 versions on Hugging Face ^[2]. The Instruct and FP8 versions were updated on 18 November 2024 ^[2].

Main features

The model uses a routing scheme that mixes a shared expert, which handles general knowledge, with specialized experts, and it applies expert-specific learning-rate scaling ^[1]. In a mixture-of-experts model, the shared expert is always active, while the specialized experts are chosen for each input, so the number of parameters working at any moment is much smaller than the total.

The pretrained model supports sequences of up to 256K tokens, and the Instruct model supports up to 128K tokens ^[2]. The context length is how much text the model can take into account at once. The model was pre-trained on 7 trillion tokens, including nearly 1.5 trillion tokens of high-quality synthetic data ^[1].

To compress the key-value (KV) cache, the model combines grouped-query attention with cross-layer attention. The KV cache is the memory a model keeps for earlier tokens while it generates text, and in long conversations it takes up a large share of memory. Tencent says this design cuts the KV-cache portion by 50% ^[2].

Its tokenizer has about 100K tiktoken-derived tokens plus 29K added Chinese tokens. Compared with the LLaMA3 tokenizer, this raises the compression from 2.78 to 3.13 characters per token ^[2].

Reported results

For the base model, Tencent reported MMLU 88.4, against 85.2 for Llama 3.1 405B. It also reported GSM8K 92.8, MATH 69.8, HumanEval 71.4, CMMLU 90.2 and C-Eval 91.9 ^[1]. For the Instruct model it reported MMLU 89.9, MATH 77.4, HumanEval 90.0, Arena-Hard 81.8 and MT-Bench 9.4 ^[2]. These are Tencent's self-reported results.