Databricks releases the 132B mixture-of-experts model DBRX

What was released

On 27 March 2024, Databricks published DBRX, a large language model released in two versions: a base model and an instruct model ^[1]. The distinction is a common one. A base model is trained on large amounts of text to continue it — given a passage, it predicts what comes next — and it is the raw material from which other things are built. An instruct model has been tuned further so that it answers questions and follows directions rather than simply continuing text. Releasing both lets one version serve as a research artifact and the other as something an application can call directly.

DBRX is a mixture-of-experts model with 132 billion parameters. In a mixture-of-experts (MoE) architecture, a layer does not hold one large feed-forward network but many smaller 'expert' networks, plus a router that decides, token by token, which few experts should handle it. The consequence is that the model's total parameter count can be much larger than the computation spent on any single token, because most experts stay idle for most tokens. That is why a mixture-of-experts model's parameter count and its running cost are separate numbers, and why comparisons across architectures need care.

Which models it was measured against

Databricks positioned DBRX against Mixtral and Grok-1 rather than against closed frontier models ^[1]. The choice of comparison set is itself a claim about what kind of model this is. Instead of measuring itself against the largest systems reachable only through someone else's API, the company placed DBRX in the field of models that others can obtain and build on. It also tells a reader what question the release is answering: not whether this is the most capable model in the world, but how it compares with the alternatives a team could actually adopt.

The benchmark results

The most concrete number is on HumanEval, a benchmark that asks a model to write a working function from a signature and a docstring and scores the share of problems solved. There DBRX edged past CodeLLaMA-70B Instruct, 70.1 against 67.8 ^[1]. The margin is small — under three points — and a single benchmark measures one narrow skill, so the result is better read as competitive than as a wide lead.

Databricks also compared the model with GPT-3.5, which scored 48.1 on HumanEval and 57.1 on GSM8K ^[1]. GSM8K is a set of grade-school arithmetic word problems, and reporting it next to a code benchmark gives two different views: writing programs, and carrying out multi-step arithmetic stated in prose. The GPT-3.5 scores appear as reference points in the same table.

Against LLaMA2-70B Chat, DBRX won on every benchmark the post tabulated ^[1]. A clean sweep across a table is a stronger claim than a win on one row, though it still depends on which benchmarks were chosen and on how each model was prompted. All of these numbers come from the releasing company's own evaluation, which is the normal situation at a model launch and is also the reason such tables are read as evidence rather than proof.

How customers could get it

The model was available to Databricks customers through APIs on the same day ^[1]. Serving a model and announcing it together connects the release to something usable: a customer does not have to wait for third-party hosting or set up infrastructure before trying it.

Databricks said customers could pretrain their own DBRX-class models from scratch or keep training on top of it ^[1]. These are two quite different offers. Pretraining from scratch means running the whole training process on one's own data to produce a model of the same class; continuing training means taking the released model and pushing it further with additional or domain-specific data. The second is far cheaper, and it is the usual route for teams adapting a model to their own material.

Why it mattered

The release marks a platform company building and shipping its own model rather than only hosting other companies', and it shows the open-model field with a competitive table of its own. The emphasis on how customers could retrain the model also shows where the value was claimed to sit: not only in the artifact, but in the pipeline around it.