DBRX

What DBRX is

DBRX is a large language model that Databricks announced on 27 March 2024.^[1] It is a decoder-only transformer — it produces text by predicting one token after another — and it is built as a mixture of experts rather than as a single dense block of computation. The model holds 132 billion parameters in total, of which 36 billion are active for any given token.^[1] It accepts up to 32,768 tokens of context.^[1]

Stated that way, the two numbers describe different things. The total is how much the model has learned to store; the active count is how much arithmetic it performs per token. Keeping the second much smaller than the first is the whole point of the architecture.

How the mixture of experts works

In a mixture-of-experts (MoE) model, the feed-forward portion of each layer is divided into several parallel "experts", and a routing network decides which of them process each token. DBRX sends every token to 4 of its 16 experts, where Mixtral and Grok-1 pick 2 of 8.^[1] Databricks says the finer split gives 65 times more possible expert combinations.^[1] A routing decision can therefore combine experts in many more distinct ways, so the model can hold far more parameters than it uses at any one moment and spend storage where it buys capability without paying for it on every token.

The rest of the architecture is conventional. It uses rotary position embeddings (RoPE), which encode word order by rotating the vectors that represent tokens; gated linear units (GLU) in the feed-forward blocks; grouped-query attention, a variant that shares key and value heads across query heads to save memory during generation; and the GPT-4 tokenizer.^[1]

Training

DBRX was pretrained on 12 trillion tokens of text and code.^[1] The run used 3,072 Nvidia H100 GPUs joined by 3.2 Tbps InfiniBand.^[1] The whole build took about three months, covering pretraining, post-training, evaluation, red-teaming and refinement.^[1] Red-teaming here means deliberately probing the model for harmful or unsafe behaviour before release, and post-training covers the stages that turn a raw pretrained model into one that follows instructions.

Results

DBRX Instruct scores 73.7 on MMLU, 70.1 on HumanEval and 66.9 on GSM8K.^[1] Those three benchmarks test broad knowledge, code generation and school-level mathematics respectively. On the Open LLM Leaderboard the model scores 74.5, and on the Databricks Gauntlet, an internal evaluation suite, 66.8.^[1]

Databricks notes that DBRX is about 40 percent of Grok-1's size in both total and active parameters, while scoring higher on these benchmarks.^[1] That comparison is the clearest statement of what the design is meant to show: the number of active parameters, not the total, is roughly what a token costs to process.

Efficiency and availability

Inference is up to twice as fast as LLaMA2-70B, and Databricks estimates that training a mixture-of-experts model takes about half the FLOPs of a dense one.^[1] The base and instruct weights are published on Hugging Face under an open licence.^[1]

Where it sits

Two design decisions shape how DBRX is used. The first is the mixture-of-experts split: it separates the cost of storing a parameter from the cost of using it, though the memory needed at serving time tends to follow the total parameter count rather than the active one, so a large MoE model is cheaper to run per token than its size suggests but not necessarily easy to fit on small hardware. The second is the open release: weights on Hugging Face mean an organisation can deploy and adapt the model inside its own infrastructure rather than only through an outside service.^[1]

The build itself — 16 experts with 4 active per token, a 32,768-token window, 12 trillion training tokens, and roughly three months on 3,072 H100s — marks the scale at which such a model was assembled.^[1]