Zhipu open-sources ChatGLM-6B, a chat model that runs on consumer GPUs

Background

For most of the period in which large chat models became publicly usable, the strongest ones were reached through a vendor's interface: the weights stayed on the company's servers, and a researcher who wanted to inspect, fine-tune or simply run the model on their own machine could not do so. A model that is small enough to fit on a single graphics card, and whose weights are published, changes that situation for students and academic groups.

What was released

ChatGLM-6B is a bilingual Chinese–English chat model released in 2023 by Zhipu AI together with the Knowledge Engineering Group (KEG) at Tsinghua University. It has about 6.2 billion parameters, and both its code and its weights were made public.^[1] Parameters are the numbers a network learns during training; the count is a rough measure of capacity, and 6.2 billion is small next to the largest chat models — which is precisely what makes it runnable on modest hardware.

How it was trained

The model was pre-trained on roughly 1 trillion Chinese and English tokens, then put through an alignment stage: supervised fine-tuning, feedback bootstrapping, and reinforcement learning from human feedback.^[1] Supervised fine-tuning trains the model on example instructions paired with good answers. Reinforcement learning from human feedback goes a step further: people compare or rank candidate answers, and the model is then optimised towards the answers that were preferred. Feedback bootstrapping uses the model's own outputs to keep extending the material available for training and feedback.

Running it on one GPU

The repository gives memory figures for inference: about 13 GB of GPU memory at FP16, 8 GB at INT8 and 6 GB at INT4.^[1] FP16, INT8 and INT4 describe how many bits store each weight: half precision keeps 16 bits per number, while the 8-bit and 4-bit settings compress the weights so they occupy less memory, at some cost in numerical precision.

For fine-tuning, the repository's hardware table gives about 14 GB at FP16 and 9 GB at INT8 when using P-Tuning v2.^[1] P-Tuning v2 is a parameter-efficient method: instead of updating every weight, it trains a small set of additional parameters and leaves the base model largely untouched, which keeps memory and storage costs down.

Licence

The code is released under Apache-2.0. The weights are fully open for academic research, and commercial use is free after registering through a questionnaire.^[1] The two are separate: the code uses a standard open-source licence, while the weights carry their own terms that ask commercial users to register first.

Point releases

The repository's update log records several steps after the first release. On 15 May 2023 a v1.1 checkpoint added English instruction-tuning data.^[1] On 17 May 2023 the log lists VisualGLM-6B, a multimodal model that supports image understanding.^[1] On 25 June 2023 came ChatGLM2-6B, pre-trained on 1.4 trillion tokens, with the base context length extended from 2K to 32K.^[1] Context length is how much text the model can take in at once; moving from 2K to 32K means longer documents and longer conversations can be read without being cut off.

Why it mattered

The combination that made ChatGLM-6B notable is the memory footprint plus the open weights. A 6.2-billion-parameter model that needs about 13 GB at half precision, and as little as 6 GB at 4 bits, can be run by one person on one GPU rather than only through an API.^[1] Fine-tuning within 9 to 14 GB with P-Tuning v2 puts adaptation within reach of the same hardware.^[1] For Chinese-language work in particular, a bilingual model with published weights gives students a system they can inspect, reproduce and modify — the practical difference between reading about a chat model and actually working with one.