Apple releases the OpenELM open language models

Background

Most large language models released by companies come with their final weights and a short description. Apple's OpenELM release was different in its scope. It was a set of small models, released together with the means to examine and reproduce how they were made. The release was announced through a paper on arXiv, a public preprint server where researchers share work, and through model files on Hugging Face, a platform where open models are hosted.

What happened

On 22 April 2024 Apple researchers posted the OpenELM paper to arXiv and released the models ^[1][2]. Eight checkpoints were published on Hugging Face: 270 million, 450 million, 1.1 billion and 3 billion parameters, each as a base model and as an instruction-tuned model ^[2]. A parameter is a number inside the network that is set during training, and the parameter count gives a rough idea of a model's size.

The models were pretrained on about 1.8 trillion tokens of public data, using Apple's CoreNet library ^[2]. A token is a small piece of text, often a word or part of one. Public data means material that anyone can obtain, which lets the training be described and checked by outsiders.

Key details

Apple released more than the weights. The package included training logs, checkpoints and training configurations, so that the stages of training can be followed and repeated ^[1]. A configuration file records the settings used for a run, such as the size of the model and the data it learned from. Without it, a result is hard to reproduce.

The release also included MLX conversion code for running the models on Apple devices ^[1]. MLX is Apple's machine learning library for its own chips. Conversion code changes the model into a format the library can read, so that the models can be run and fine-tuned on a Mac or iPhone.

The weights were released under Apple's sample-code license ^[2]. The license sets the terms under which the weights can be used. Readers who plan to use the models should read the license text on the Hugging Face page before doing so.

Impact

The release gave researchers and students a complete small model to study. Many open releases publish only the final weights, which makes it hard to see how the model was trained or to repeat the process. OpenELM showed a fuller approach, with the logs and configurations included, and it put that approach in Apple's name.

The release also showed the value of small models. A model with a few hundred million or a few billion parameters can run on everyday hardware, which makes it useful for experiments that a very large model would make expensive. Publishing the logs and checkpoints alongside the weights is the part of this release that makes the process visible.