DeepSeek introduces the DeepSeekMoE architecture

Background

Mixture-of-experts is a way to make a language model larger without making every token pay the full computing cost of the model. In such a model, a router sends each input to a subset of expert sub-networks, so only part of the model runs for each token. A design question follows: how to keep the experts from learning the same things.

DeepSeek, a Chinese AI company, had released code and general models before this point, and its architecture work was aimed at this design question.

What happened

On 9 January 2024 DeepSeek released the DeepSeekMoE architecture ^[3]. The paper introducing it was posted to arXiv on 11 January 2024 ^[1]. The release came with an open-weights model, the DeepSeekMoE 16B, so the architecture could be tested by people outside the company ^[2].

The architecture rests on two ideas. Fine-grained expert segmentation splits experts into smaller ones and activates more of them in combination ^[1]. Shared expert isolation keeps a few experts active for every token, to capture common knowledge, so that the remaining experts can focus on more specific material ^[1].

Key details

DeepSeek reported that DeepSeekMoE 16B performed comparably to LLaMA2 7B while using only about 40% of the computation ^[1]. A preliminary 145B DeepSeekMoE model was reported as comparable to DeepSeek 67B while using 28.5% of its computation ^[1]. The company described the 145B result as preliminary.

The 16B weights were public, while the 145B figure is an early result reported by the company.

Impact

DeepSeek-V2, released in May 2024, used DeepSeekMoE for economical sparse training ^[4]. Sparse training means that only part of a network is active for each token, which keeps costs down. The architecture therefore became part of the foundation of a later, larger model in the company's line.

An open-weights release of this kind lets other researchers check the reported efficiency claims themselves. Readers who want to understand how mixture-of-experts models differ from dense models can use this event as a concrete reference point.