DeepSeekMoE

What it is

DeepSeekMoE is a mixture-of-experts (MoE) architecture for language models, released by DeepSeek on 9 January 2024; the paper was posted to arXiv on 11 January 2024.^[3][1] The aim it sets out is expert specialization: making each expert hold knowledge that is focused and does not overlap much with the knowledge held by other experts.^[1]

A mixture-of-experts model replaces one large feed-forward network with several parallel "experts" plus a router that sends each token to only a few of them. Because only part of the parameters is used for any single token, the model can hold far more parameters than it spends computation on. Capacity therefore grows faster than per-token cost — which is why the design is interesting for scaling, and also why it is hard to get right: if the router spreads work evenly for no good reason, the experts end up duplicating each other instead of dividing the job.

Two design changes

The first change is fine-grained expert segmentation. Rather than cutting the feed-forward layer into a handful of large experts, DeepSeekMoE splits it into more, smaller experts and activates a larger number of them in each combination.^[1] Each expert then covers a narrower slice of knowledge, and the combinations the router can form are more varied.

The second is shared expert isolation. A small number of experts are kept always active for every token and are meant to capture knowledge that is common across contexts, leaving the routed experts free to specialize on what is specific.^[1]

The paper's motivation is that conventional MoE designs such as GShard face challenges in ensuring expert specialization.^[1] Routing alone does not guarantee a clean division of labour among experts, and the two changes above are the paper's answer to that problem.

The released model and reported results

The first model released with the architecture was DeepSeekMoE 16B, published with open weights.^[2] It has 16.4 billion parameters and was trained on 2 trillion English and Chinese tokens.^[2] DeepSeek reported that it performed comparably to LLaMA2 7B while using only about 40% of the computation.^[1]

The paper also reported results at other scales. At small scale, a 2B DeepSeekMoE model matched a larger GShard model.^[1] A preliminary 145B DeepSeekMoE model was reported as comparable to DeepSeek 67B while using 28.5% of its computation.^[1] Taken together, the pattern the paper points to is that the gap between an MoE model and a larger conventional model widens as the design is scaled up, because a bigger parameter budget can be spread over more finely divided experts.

Where it sits

The paper argues that mixture-of-experts is a promising architecture for managing computational costs when scaling parameters.^[1] That argument carried into later work: DeepSeek-V2, from May 2024, used DeepSeekMoE for economical sparse training.^[4] Wikipedia lists DeepSeek-MoE as discontinued.^[3]