GGUF

What GGUF is

GGUF, short for GGML Universal File, is a binary file format for storing model data. Its defining trait is that tensors and metadata live together in a single file, an arrangement designed so that model data can be saved and loaded quickly^[1]. A tensor is an array of numbers that holds part of a model's weights, and metadata is the information a loader needs in order to interpret those arrays correctly. Keeping both of these in one place means software does not have to look in several files to reconstruct a model.

Where it came from

The format was introduced in August 2023 by the llama.cpp project^[1]. The project's motivation was compatibility: as llama.cpp added support for more model architectures, the maintainers needed a way to keep older files usable, so the format was designed with backwards compatibility in mind and superseded the earlier GGML format^[1]. GGUF was developed by Georgi Gerganov together with the community^[1]. Its specification is formalized in the ggml repository, which defines the file's magic number, its versioning, blocks holding metadata key-value pairs, and blocks holding tensor information^[2]. A magic number is a fixed marker at the start of a file that lets software recognise which format it is looking at; a version field lets a loader tell which revision of the format produced the file, which is what makes a stable compatibility story possible.

What it holds

GGUF supports quantized integer types from 2-bit to 8-bit, as well as common floating-point formats such as float32, float16 and bfloat16^[1]. Quantization means storing numbers with fewer bits than a full floating-point representation, which makes a model file smaller and cheaper to hold in memory, at some cost in numerical precision. By covering both low-bit integer types and ordinary floating-point types, the format can carry models at several points along that trade-off.

GGUF files are usually not written by hand, and models are not trained from scratch in the format. A GGUF file is typically produced by converting a model that was developed with a different machine-learning library, such as PyTorch^[1]. The conversion step reads the original weights, records the metadata and the tensor layout, and writes out one GGUF file.

Where it sits in the ecosystem

The format has become a standard for distributing quantized large language models for local inference^[1]. Local inference means running a model on a user's own machine rather than on a remote server, a setting where file size, load time and memory use matter directly. GGUF is natively supported by tools including llama.cpp, Ollama, LM Studio, GPT4All, Jan and koboldcpp^[1]. Because several of these programs can read the same file, a single download can serve users who prefer different front ends.