OpenELM

A family of small models

OpenELM is a family of small open language models from Apple. It comes in four sizes — 270M, 450M, 1.1B and 3B parameters — and each size is published in two forms: a pretrained base model and an instruction-tuned version^[2]. All four sit well below the scale of frontier systems, in the range where a model can plausibly be run and fine-tuned outside a large data centre.

Where it came from

The family was described in a paper first posted to arXiv on 22 April 2024. The paper has 11 authors and is led by Sachin Mehta^[1]. The model card on Hugging Face lists Mehta together with Mohammad Hossein Sekhavat and Qingqing Cao among those 11 authors^[2].

Layer-wise scaling

A standard transformer stacks layers of identical shape, so every layer holds the same number of parameters. OpenELM changes that: it uses layer-wise scaling, allocating parameters unevenly across the transformer's layers rather than dividing them evenly^[1]. The paper reports that this uneven allocation improves accuracy^[1]. The underlying idea is that a fixed parameter budget is a design choice, not a law — how it is split between layers can itself be tuned, in the same way as depth or width.

What it was trained on

The models were pretrained on about 1.8 trillion tokens drawn from public datasets: RefinedWeb, a deduplicated version of the PILE, a subset of RedPajama and a subset of Dolma v1.6^[2]. Pretraining was carried out with Apple's CoreNet library^[2].

Reported results

At about 1 billion parameters, OpenELM reported accuracy 2.36% higher than OLMo while using half as many pre-training tokens^[1]. That comparison is the kind small-model work tends to centre on: not raw capability at any cost, but how much accuracy comes out of a given amount of training compute and data.

What "open" covers

Apple released the full training and evaluation framework, along with training logs, multiple checkpoints and the pre-training configurations — not just the final weights^[1]. The code and training recipes are in the apple/corenet repository on GitHub, and the models are on Hugging Face under the apple-sample-code-license^[1][2].

The release also includes code to convert the models to MLX, Apple's library for inference and fine-tuning on Apple devices, so the same weights can be run and adapted there^[1]. The published models are loaded through the transformers library with remote code enabled^[2].

Where it sits

OpenELM belongs to the strand of language-model work that treats the training process itself as something worth publishing. Anyone can download a set of weights; releasing the framework, the logs and the intermediate checkpoints means an outside researcher can see how the model got to that point, rerun parts of the pipeline, and compare design choices such as layer-wise scaling against their own^[1]. Pairing that with a conversion path into MLX ties the family to Apple's own hardware and software stack, while the Hugging Face release keeps it usable from the tooling most practitioners already have^[1][2]. The models are distributed under the apple-sample-code-license, so the terms attached to them are the ones stated in that licence^[2].