SmolLM2

What SmolLM2 is

SmolLM2 is the second generation of Hugging Face's family of small language models, released in November 2024 in three parameter sizes: 135M, 360M and 1.7B^[1][3]. Every size was published under the Apache 2.0 license, and each came in two forms — a base model intended as a starting point for further training, and an instruction-tuned model meant to be used directly^[1]. The "Smol" in the name refers to the parameter counts, not to a stripped-down design: these are ordinary Transformer decoder language models, trained on very large amounts of text with far fewer parameters than the largest systems of the same period^[4].

The three sizes serve different purposes. The smallest ones are cheap enough to run in places where a large model would be impractical, while the 1.7B model is the one the team documented most thoroughly and compared against other small models.

Where it sits in the family

SmolLM2 is the middle generation of a line of models. It succeeds SmolLM, released in July 2024, and is followed by SmolLM3, released in July 2025^[3]. Each generation kept the same basic idea — small, openly licensed models trained on carefully selected data — and the numbering makes the progression easy to follow.

How the 1.7B model was trained

The largest model, SmolLM2-1.7B, was trained on about 11 trillion tokens using a multi-stage data mix^[1][2]. The model card describes a Transformer decoder architecture trained in bfloat16 precision on 256 H100 GPUs with the nanotron framework^[4].

A multi-stage data mix means the composition of the training corpus is changed over the course of training instead of staying fixed from beginning to end: early stages may lean on broad web text, later stages shift toward higher-quality material. The corpus combined FineWeb-Edu, DCLM and The Stack with newly curated mathematics and coding datasets^[4]. The overall scale — trillions of tokens for a model of under two billion parameters — reflects a deliberate choice to spend effort on data rather than on size.

Three new datasets

The work introduced three new datasets: FineMath, Stack-Edu and SmolTalk^[2]. They are documented alongside the training recipe in the accompanying paper, and they are part of the same argument the authors make throughout — that for small models, what goes into training matters as much as how many parameters come out.

Instruction tuning

The instruction-tuned version was built in two steps. First came supervised fine-tuning on public and curated datasets, in which the model is trained on example prompts paired with good answers. Then came Direct Preference Optimization using UltraFeedback, a method that adjusts the model using comparisons between responses that are preferred and responses that are not^[4]. The resulting instruct model additionally supports text rewriting and summarization^[4].

Results and the paper

The technical report, titled "SmolLM2: When Smol Goes Big", appeared on arXiv on 4 February 2025 with Loubna Ben Allal as first author. It reports that SmolLM2-1.7B outperforms Qwen2.5-1.5B and Llama3.2-1B^[2]. Those are comparisons against other small models of roughly similar size, which is the relevant field for a model in this class.

Why it matters

SmolLM2 is a documented example of data-centric training for small models. Rather than treating a small model as a compressed version of a large one, the releases show a full pipeline: curated and newly built datasets, a staged training schedule, then supervised fine-tuning and preference optimization, all described in public. The Apache 2.0 license makes the weights usable in other projects, and the three sizes give users a way to pick a model that fits their hardware^[1]. Its position between SmolLM and SmolLM3 also makes the family a continuous, checkable record of how one group's approach to small models changed from one release to the next^[3].