Falcon3

What Falcon3 is

Falcon3 is a family of open-weight language models released by TII (Technology Innovation Institute). The family ranges from 1B to 10B parameters, and each size comes in two forms: a base model trained on text, and an instruct model tuned to follow instructions^[1]. The weights of Falcon3-10B-Instruct, the largest instruct model in the family, were uploaded to Hugging Face on 14 December 2024^[1]. "Open weights" means the trained parameters can be downloaded and run locally; what may be done with them is set by a separate licence rather than by the download itself.

How the 10B model was built

The 10B model was built by depth up-scaling a 7B base model and training it over 2 trillion tokens on 1,024 H100 GPUs^[1]. Depth up-scaling is a way of growing a model by making it deeper — adding layers — rather than only making each layer wider, and it starts from a smaller model that has already been trained, so that earlier training is reused instead of discarded. The two numbers attached to the run describe its scale: 1,024 H100 accelerators working together, and 2 trillion tokens of text passed through the model during training.

Architecture

The 10B model has 40 decoder blocks^[1]. This is the decoder-only transformer shape used by most chat-oriented language models: each block attends to the tokens before it and passes the result upward.

Attention uses grouped-query attention with 12 query heads and 4 key-value heads^[1]. In ordinary multi-head attention every query head has its own key and value head; in grouped-query attention several query heads share one key-value head. That shrinks the cache of keys and values that has to be kept in memory while text is generated, which matters when many conversations are served at once, at some cost in what each head can express on its own.

The head dimension is widened to 256, and the rotary position embedding (RoPE) base is set to 1,000,042^[1]. RoPE encodes position by rotating query and key vectors by angles that depend on where a token sits in the sequence; the base value sets how quickly those angles change from one dimension to the next. A larger base stretches the range of positions the model can tell apart, which is the usual reason to raise it when long context is the target. The context window is 32K tokens and the vocabulary is 131K^[1]. The context window is how much text — prompt plus answer — the model can hold at once; the vocabulary is the number of distinct units the tokenizer can produce.

Languages and post-training

The model supports four languages: English, French, Spanish and Portuguese^[1]. Post-training used 1.2 million samples covering STEM, conversation, code, safety and function calling^[1]. Post-training is the stage after pretraining in which a model is tuned on selected example exchanges; the mixture named here is what shapes the instruct version — subject knowledge, ordinary chat, programming, refusing harmful requests, and emitting structured calls to outside tools.

Recorded benchmark scores

The Open LLM Leaderboard entry for the 10B instruct model records IFEval 78.17, BBH 44.82, GPQA 10.51 and MMLU-Pro 38.1^[1]. These tests measure different things. IFEval checks whether a model obeys instructions whose compliance can be verified automatically; BBH is a set of hard reasoning tasks; GPQA consists of graduate-level science questions written so that they are difficult to answer by lookup; MMLU-Pro is a broad multi-subject knowledge test with harder questions than the benchmark it replaces. Read together they give a profile rather than a single position.

Licence

Falcon3 is released under the Falcon-LLM Licence, a custom licence rather than an OSI-approved one, published on TII's site^[1][2]. OSI-approved licences are the standard open-source set; a custom licence can attach conditions that open-source licences rule out, so anyone deploying the model has to read the terms instead of assuming the usual open-source permissions apply.

Where it sits

The release covers a full ladder of sizes, 1B to 10B, in both base and instruct form, so users can pick a size that fits the hardware they have. The largest instruct model is documented with a 32K context window, four languages, and post-training data that includes function calling.