Ai2 releases the Tülu 3 post-training recipe

Background

Large language models are trained in more than one stage. Pretraining exposes a model to large amounts of text and gives it a general grasp of language; post-training is the set of steps that follows, shaping a base model into something that follows instructions, answers questions and produces the kind of output users expect. On the open side, post-training has often been less documented than pretraining, and Ai2 pointed to the gap between open post-training and closed recipes on public leaderboards^[2]. The company also contrasted its own release with closed labs whose post-training data and recipes are not disclosed^[2].

What was released

On 21 November 2024, Ai2 released Tülu 3 in two sizes, 8B and 70B, both fine-tuned from Llama 3.1^[1]. The release is a post-training recipe together with models trained by it, rather than a new base model: the starting point is Llama 3.1, and what Ai2 contributed is the procedure applied on top of it^[1].

The recipe

The recipe has three parts. The first is supervised fine-tuning (SFT): the model is trained on example instructions paired with good responses and learns to imitate them. The second is preference tuning: the model is shown pairs of candidate answers with an indication of which one is preferred, and is trained to favour the preferred one. The third, which Ai2 presents as the addition this recipe makes, is reinforcement learning with verifiable rewards (RLVR)^[1]. In RLVR the model generates answers and is scored by a check that can be run automatically, so the reward does not rest on a human judgement or a learned judge for every sample. Ai2 describes the recipe as combining RLVR with SFT and preference tuning^[1].

What was made public

Ai2 published the datasets, the code, intermediate checkpoints and evaluations^[1]. Intermediate checkpoints are the models saved partway through training; releasing them lets others see how each stage of the recipe changes the model rather than only inspecting the finished result, and publishing the evaluations lets the reported numbers be checked.

Why it matters

Ai2's stated claim is that the release closes the gap between open and closed post-training recipes^[1]. The framing is about transparency as much as about scores: the company contrasts its disclosure with closed labs that keep post-training data and recipes private^[2], and describes the work as targeting the gap between open post-training and closed recipes on public leaderboards^[2]. Because the data, code, checkpoints and evaluations are all public, other groups can inspect and reuse the procedure rather than only its outputs^[1]. That is the point the release is meant to make: the method itself, not just the model, is the deliverable.