llama.cpp

What it is

llama.cpp is an inference engine: the layer of software that loads a trained language model and produces text from it. It is written in C/C++ with no external dependencies, and it is built on the ggml tensor library^[2]. Both properties matter in practice. A tensor library supplies the basic numerical machinery a neural network needs — arrays of numbers and the operations performed on them. Having no external dependencies means the project can be compiled on its own, without first installing a stack of other libraries, which makes it easier to build and run on many kinds of machines.

Where it came from

Georgi Gerganov created llama.cpp in March 2023 in order to run Meta's LLaMA model on consumer hardware^[2]. The repository on GitHub was created on 10 March 2023; it was originally published under Gerganov's own account and later moved to the ggml-org organization^[1]. The goal — running a model on ordinary machines rather than on a cluster of servers — shaped the design, because the code doing the work has to stay small and the number of moving parts has to stay low. The project is credited with catalyzing local large-language-model inference on laptops^[2].

Quantization

To fit models into the memory of everyday machines and to make them respond faster, llama.cpp uses quantization, which reduces memory usage and increases speed at the cost of some model accuracy^[2]. A language model is, at bottom, an extremely large collection of numbers — its weights. Quantization means storing those numbers in a more compact form, using fewer bits for each one, so the whole collection takes up less space and can be moved through memory faster. The trade-off is that the numbers are no longer exact: each stored value is a rounded approximation, and the errors add up across a model, which is why some accuracy is lost. Making that trade-off adjustable is what lets the same engine serve both a small laptop and a larger machine.

GGUF model files

Model files are distributed in the GGUF format, a single-file format that stores tensors and metadata together; the GGUF specification is maintained in the ggml repository^[2][3]. The tensors are the weight arrays themselves; the metadata is the information that describes the model — how it is structured, what settings it expects. Keeping both in one file means a model can be passed around as a single object instead of a directory of separate pieces, which simplifies downloading, sharing and loading.

Licence and place in the ecosystem

The repository is released under the MIT license and is free to use, modify and distribute^[1]; it has roughly 130,000 stars on GitHub^[1]. That permissive licence, together with the fact that the engine has no external dependencies, is part of why the project spread beyond its original author: anyone can take the code, change it, and ship it, including inside other software.

llama.cpp sits at the point where a trained model meets a person's own computer. The engine itself does not create models; it takes a model file and runs it. Its contribution is to make that step cheap enough to be routine — by keeping the code small, by compressing the weights through quantization, and by settling on one file format for distributing them. In the short period after its release, running a language model locally on a laptop went from an unusual experiment to something a great many people did, and llama.cpp was one of the tools that made that shift possible^[2].