Megrez-3B-Instruct

What it is

Megrez-3B-Instruct is an open-weights language model released by Infinigence AI. The company published the weights on Hugging Face on 15 December 2024 under the Apache-2.0 licence, which permits commercial use and modification without a separate agreement^[1]. It sits at the small end of the size range: 2.92 billion parameters in total, of which 2.29 billion remain once the embedding and softmax layers are excluded^[1]. The two figures are worth keeping apart, because the non-embedding count is closer to the amount of computation actually spent per token inside the transformer stack.

Architecture and tokenizer

The model uses a Llama-2 architecture with grouped-query attention rather than a custom block design^[1]. Grouped-query attention sits between full multi-head attention and multi-query attention: several query heads share one set of key and value heads, which shrinks the key-value cache that has to be kept in memory while text is generated. The smaller the model, the larger that cache looms relative to everything else, especially in long conversations, so the choice is a practical one for running locally.

The context window is 32K tokens and the vocabulary size is 122,880^[1]. A large vocabulary means a given piece of text is usually cut into fewer tokens, which leaves more of the window for actual content and shortens the sequence the model has to process. This matters more for Chinese, where an English-oriented vocabulary tends to fragment characters into many pieces.

Training

The model was pre-trained on 3 trillion tokens covering Chinese and English^[1]. Infinigence describes the outcome as compressing the capability of its previous 14B generation into a 3B model^[1] — a statement about relative quality rather than about any single measurement.

Reported results

The model card's own OpenCompass run reports C-EVAL 84.8, CMMLU 74.7, MMLU 72.8, MMLU-Pro 46.1, HumanEval 78.7 and GSM8K 65.5^[1]. The spread looks like that of a model tuned with Chinese in mind: C-EVAL comes out above MMLU, and the harder MMLU-Pro falls well below plain MMLU, while code generation at 78.7 on HumanEval holds up better than multi-step arithmetic at 65.5 on GSM8K. Because these numbers come from the publisher's own harness, they should be read as the company's claim rather than as an independent measurement.

The same table reports a decoding speed of 2,329.4 tokens per second, against 1,279.4 for Qwen2-7B^[1]. Throughput of this kind depends heavily on the hardware, the batch size and the serving stack used, so the useful reading is the comparison drawn inside one table, not the absolute figure on its own.

Why the plain structure

The team chose the plain LLaMA structure over sparse or ReLU-based variants so that developers can deploy the model without code changes^[1]. Models that depart from standard blocks usually need modified kernels, a particular inference engine or patched libraries; a checkpoint that loads into the same pipeline as any other Llama-2 model removes that step. For a small model intended to run locally, deployment friction is often what decides whether it gets used at all.

Demo and ecosystem

Alongside the weights, Infinigence published InfiniWebSearch, a web-search demo built on this model that decides on its own when to call search^[2]. The design is a small agent loop: the model reads the question, judges whether its own knowledge is enough, and if not issues a search query and folds the results back into the answer. Keeping that decision on a 3B model rather than a large server-side model makes the loop cheaper and lets it stay on the local machine.