Megrez-3B-Omni

What it is

Megrez-3B-Omni is an open-source multimodal model published on 16 December 2024, one day after the text-only Megrez-3B-Instruct.^[1] Its developer calls it the first open-source omni-modal understanding model built for end-side devices — that is, for running on local hardware such as phones and laptops instead of in a data centre.^[1] Omni-modal here means the model accepts three kinds of input: images, text and audio.^[1] It is an extension of the text-only model rather than a separate architecture, so the language part is inherited and the new work sits in attaching the other two senses.^[1]

The name is also a fair description of the design goal. A 3B-class model is small enough to be considered for a local device, and the interesting question is how much multimodal capability can be packed into that budget without wrecking the language ability the model already had.

How it is put together

The model is assembled from three modules. The language backbone has 2.29 billion parameters and keeps the Llama-2 design with grouped-query attention.^[1] Images are encoded by SigLip-SO400M, a vision encoder with 0.42 billion parameters, and reach the language model through a cross-attention connector.^[1] Audio is encoded by the encoder half of Whisper-large-v3, with 0.64 billion parameters, through a linear connector.^[1] The three modules add up to roughly 4 billion parameters, and the context length is 4K tokens.^[1]

A connector is the piece that translates an encoder's output into a form the language model can use. A linear connector applies a simple projection, mapping audio features into the language model's representation space. A cross-attention connector instead lets the language model query the visual representation directly during computation, rather than only pasting image features into the input as prefix tokens. Grouped-query attention lets several query heads share one set of keys and values, which shrinks the key-value cache that has to be kept in memory during inference — on memory-constrained end-side hardware, that kind of choice directly affects whether a model runs at all.

Results

On the eight multimodal benchmarks in OpenCompass, it scores 66.2 on average.^[1] Individual results include MME 2,315, MMMU validation 51.89, OCRBench 82.8 and MathVista 62.^[1] These benchmarks probe different things: general image-text understanding, reasoning with subject knowledge, reading text in documents and scenes, and maths problems posed with pictures. According to the developer, text ability stays within 2% of the text-only Megrez-3B-Instruct.^[1] That matters for an end-side model: if language quality degrades noticeably once vision and audio are added, everyday question answering suffers.

A single score says little on its own. Judging the level requires putting the same benchmarks, under the same evaluation settings, next to other models; because benchmarks have different scales and scoring rules, adding scores across them is not meaningful, so an average is a summary rather than a ranking.

Where it sits

On the end-side multimodal track, Megrez-3B-Omni follows a common recipe: rather than making the model bigger, keep a language backbone in the low billions of parameters and attach relatively specialised vision and audio encoders, so the model still handles three input types at a few billion parameters in total. The combination of roughly 4 billion parameters and a 4K context sets the scale of task it can take on — enough for image question answering and reading text off a picture, less suited to long documents or long recordings.

Its open-source status is part of the picture. The accompanying Infini-Megrez-Omni repository was created on 14 December 2024 under Apache-2.0.^[2] For a model meant to be installed on a phone or a laptop, the licence is what decides whether someone else can integrate it into their own product.