Megrez2-3x7B-A3B

What it is

Megrez2-3x7B-A3B is a mixture-of-experts (MoE) language model designed for device-native deployment, meaning it is intended to run on the device itself rather than in a data center^[2]. The name spells out its two numbers: the model activates about 3 billion parameters for each token it processes, while holding roughly 7.5 billion parameters in total^[2]. The weights were uploaded to Hugging Face on 22 July 2025 and are published under the Apache-2.0 licence^[1].

Mixture-of-experts is a way of getting more capacity out of a fixed computing budget. Instead of passing every token through every part of the network, a router picks a small subset of "experts" — separate small subnetworks — for each token. Only the chosen experts do work, so the amount of computation per token stays low even when the model as a whole is large. The catch is memory: all the experts still have to be stored somewhere, and on a phone or a laptop that is often the binding constraint.

How it is built

Megrez2's central trick attacks exactly that constraint through cross-layer expert sharing. Adjacent layers reuse a single pool of experts rather than each layer keeping its own, which cuts the total number of parameters that must be stored while leaving the number of activated parameters per token unchanged^[2].

Concretely, the released preview network has 31 layers, with a dense first layer followed by groups of three layers that share a pool of 64 experts. Routing is top-6, and each layer also carries 4 shared experts^[2]. Two further pieces of engineering support the design. Pre-gated routing lets the router fetch the experts needed by the next group of layers early, so that loading weights from memory overlaps with computation instead of stalling it^[2]. And the tokenizer is the one already used by the Megrez-3B series, with a vocabulary of 128,880 tokens^[2].

Training

The model was pre-trained on 5 trillion tokens in three stages: 1.5 trillion tokens, then 3 trillion tokens, and finally more than 600 billion tokens aimed at long context^[2]. Post-training had two parts — supervised fine-tuning on millions of samples, followed by reinforcement learning with verifiable rewards^[2]. The latter is a technique in which the model is trained against answers that can be checked automatically, such as a maths result or a code execution outcome, rather than against human preference judgements alone.

Reported results

The released Megrez2-3x7B-A3B reports 85.4 on MMLU and 58.8 on GPQA, in both cases while activating 3 billion parameters per token^[1]. Its context length is 32K tokens, with a RoPE base frequency of 5,000,000^[1]. MMLU is a broad multiple-choice benchmark spanning many school and professional subjects; GPQA is a set of deliberately hard, graduate-level science questions.

Where it sits

The interesting claim behind Megrez2 is not raw benchmark position but the trade-off it targets. Running a language model on a device is limited less by arithmetic than by memory capacity and memory bandwidth: weights must be moved from storage to the processor for every token, and that movement costs time and energy. Cross-layer expert sharing reduces how much has to be stored, and pre-gated routing reduces the time spent waiting for weights to arrive^[2]. Whether this works well enough in practice depends on the hardware and on how often the router's choices match what the next layers actually need — a general weakness of expert routing, since a token that misses the experts it would have benefited from cannot recover the loss. The model card and technical report give the architecture and the headline numbers; the longer test is how the approach holds up as device-side models continue to grow.