TII releases Falcon-H1, a hybrid Transformer-Mamba family

Two ways to handle a long context

Most large language models are built on attention: to produce each token, the model looks back over what came before, and it keeps a cache of previously computed keys and values so that work is not repeated. That cache grows as the context grows, and on long documents or long conversations it becomes one of the most expensive parts of running the model. A state space model takes a different route. Instead of comparing every new token against all earlier ones, it carries a compressed running state forward, so the cost of a longer input rises much more gently.

Each approach has a trade-off. Attention is good at reaching back to a specific earlier detail; state space layers are cheap over long spans but hold the past only in summarised form. A hybrid model tries to get both.

What TII released

On 1 May 2025, TII published the weights of Falcon-H1^[1]. Six sizes shipped at once, from 0.5B to 34B parameters^[2]. According to TII, it is the first model family in the Falcon line to combine attention layers with state space models in parallel^[2].

Shipping six sizes in one release is a practical choice as much as a technical one: the smallest models can run on modest hardware, while the largest are meant for serving. It also means the family can be compared across scales on the same architecture rather than across generations of different designs.

What "parallel" means here

Combining the two kinds of layer can be done in sequence, with one sort of layer stacked after another, or in parallel, with both kinds working on the same input at the same time and their results merged. Falcon-H1 takes the second route^[2].

The argument TII makes for the design is about cost at long context: the hybrid keeps the memory advantage of handling long inputs while cutting the compute needed^[2]. In plain terms, the state space branch handles the bulk of the sequence cheaply, and the attention layers supply the precise recall that the compressed state loses. Whether that balance holds up in practice is exactly what the published weights let outside researchers test.

Files, serving, and licence

The release is not only weights. It includes official GGUF files for llama.cpp, the widely used runtime for running models locally, often on consumer hardware and often in quantised form^[1]. It also includes support for vLLM, the serving engine many teams use to run models for many users at once^[1]. Between the two, the same family can be tried on a laptop and deployed behind an API without waiting for third-party conversions.

The licence stayed the Falcon-LLM Licence^[1]. That is the same custom licence the family has used rather than a switch to different terms, so anyone planning commercial or redistributed use should read the terms rather than assume a standard open-source licence applies.

The technical report

The technical report followed on 30 July 2025^[2], roughly three months after the weights. That ordering matters for readers trying to judge the model: the files were usable first, and the detailed account of the architecture, training and evaluation arrived later. Claims about the hybrid design are best checked against that report rather than against the launch announcement alone.

Where it sits

Falcon-H1 is a sign of how the hybrid idea moved from research papers into shipped model families: attention is no longer treated as the only building block, but as one component to be mixed with cheaper sequence models. The release also shows what a usable launch now looks like — several sizes, converters for common runtimes, and a licence stated up front. What the family's long-context behaviour actually costs, and where the hybrid helps or hurts, is a question the open weights were published to answer.