Tülu 3

What Tülu 3 is

Tülu 3 is an open post-training recipe together with a pair of models, released by the Allen Institute for AI (Ai2) on 21 November 2024. The models, Llama-3.1-Tulu-3-8B and Llama-3.1-Tulu-3-70B, are built on Meta's Llama 3.1 and come in 8-billion and 70-billion parameter sizes.^[1] Post-training is the stage that turns a model which has only read large amounts of text into one that follows instructions and behaves usefully. Ai2 notes that a pretrained model is neither safe nor able to follow instructions until it has been post-trained.^[4]

How the recipe works

The recipe has three parts: instruction fine-tuning, preference fine-tuning with on-policy preference data, and reinforcement learning with verifiable rewards (RLVR).^[1] Instruction fine-tuning trains the model on examples of an instruction paired with a good response. Preference fine-tuning then shows the model pairs of answers and teaches it to prefer the better one; "on-policy" means the preference data is produced by the model being trained rather than collected from somewhere else. RLVR is a reinforcement-learning method that works without a reward model, so the signal used to improve the model does not come from a separate scoring model trained alongside it.^[1]

Ai2 said the recipe closes the performance gap between open and closed fine-tuning recipes.^[1] The blog also points out that teaching specialised skills such as coding can erode a model's other abilities, and that the recipe was designed to avoid this trade-off.^[4]

What was released

Alongside the two models, Ai2 released checkpoints from each training stage, decontaminated datasets, training code (open-instruct), an evaluation framework, and a technical report that includes negative results.^[1] Decontaminated datasets are ones screened so that test material does not leak into training. Publishing the data, the code and the intermediate checkpoints together means others can rerun and inspect each stage rather than only see the final model. The technical report was posted to arXiv on 22 November 2024.^[2]

Where it sits in the lineage

Tülu 3 is the third generation of the line, following Tülu in June 2023 and Tülu 2 in November 2023.^[3] Its recipe was also used to post-train the OLMo 2 Instruct models, so the method travels beyond the Llama-based models it was first applied to.^[1]

Why it matters

Post-training is what makes a raw pretrained model usable, and the details of that stage — which data, which objectives, in what order — matter a great deal for how a model behaves.^[4] Tülu 3's contribution is to describe that stage fully and put the pieces in public hands: the recipe itself, the checkpoints at each step, the datasets, the code and an evaluation framework.^[1] The technical report also records negative results, documenting approaches that did not work, which is information usually left out of published descriptions.^[1] Because the recipe was reused for OLMo 2 Instruct, the release also served as a building block for other openly developed models.^[1] The line has continued across generations, with Tülu in June 2023 and Tülu 2 in November 2023 preceding this release.^[3]