Kolibri-1

What it is

Kolibri-1 is an open-weight large language model from Aleph Alpha, released under the Apache 2.0 licence^[1]. It is a mixture-of-experts (MoE) model: it holds 78.1 billion parameters in total, but only 3.46 billion of them are activated for each token it processes^[1]. The idea behind a mixture of experts is to keep most of the parameters in separate "expert" blocks and send each token through only a few of them, so that a model with a big parameter count can still be cheap to run.

Kolibri-1 handles German and English only. The model card describes that two-language scope as a deliberate choice of depth over breadth^[1].

From Kolibri Origin

Before scaling up, Aleph Alpha validated the design as Kolibri Origin, a 30B-parameter model with 3B active parameters and a 65K context window^[2]. Kolibri-1 is the larger version of that recipe.

Architecture and training

The network is a 50-layer transformer. Each layer holds 384 experts, of which one is shared and six are routed^[1]. Attention alternates between sliding-window and global attention at a ratio of 4:1^[1]: most layers look only at a nearby stretch of text, and a minority look at the whole context, which is what keeps very long inputs affordable.

Pre-training used 20 trillion tokens, about 62.5 percent English, 23.9 percent German and 13.6 percent code^[1]. Two further stages followed: 3.44 trillion tokens of mid-training and 201 billion tokens of long-context extension^[1]. The pre-training run took 768 Nvidia B200 GPUs working for 21 days, about 392,000 GPU-hours and 6.4e23 FLOPs^[1]. Aleph Alpha estimates 9.5e2 MWh of energy for pre-training, mid-training and long-context work together, a figure that includes data-centre overhead^[1].

Using it

The context window is 1,048,576 tokens, though the card recommends at most 262,144 for complex tasks^[1]. A window of this size lets a model read whole documents in one pass; the lower recommended figure is a practical limit rather than a hard one. Weights are FP8 in 128 by 128 blocks and need about 78 GB, so the model fits on one H200 or one B200^[1]. The knowledge cutoff is 18 June 2026 in both languages^[1].

Results

On MMLU-Pro with chain-of-thought prompting the model scores 80.0 in English, and it scores 75.5 on MMLU-ProX in German^[1]. On Global MMLU in German it reaches 77.5, against 81.0 on MMLU in English^[1]. These are multiple-choice knowledge benchmarks, and chain-of-thought prompting means the model is allowed to write out its reasoning before giving an answer. The German and English results sit fairly close together, which is what a model trained on two languages instead of many is meant to achieve^[1].

Where it fits

The licence is Apache 2.0, and the model card states that Aleph Alpha signed the EU GPAI Code of Practice^[1]. Open weights, a two-language focus and a memory footprint that fits a single accelerator go together: this is a model built to be downloaded and run by its users, not only called through an API.