Tsinghua and Zhipu publish the open bilingual GLM-130B

Background

GLM-130B came from a collaboration between the KEG group at Tsinghua University and Zhipu AI.^[1][2] The model was built on the GLM architecture, which uses a pre-training method called autoregressive blank infilling.^[2][1]

What happened

The paper on GLM-130B was released on 6 October 2022.^[1][2] The project was published as an open bilingual model with 130 billion parameters, and its code and weights were made available under separate licenses.^[1]

Key details

By 3 July 2022 the training data had reached more than 400 billion text tokens, with 200 billion in Chinese and 200 billion in English.^[1] Equal shares of the two languages in the training data are what make the model bilingual.

Inference, which means running a trained model to produce output, needs eight A100 (40G) or eight V100 (32G) GPUs on a single server. With INT4 quantization the model runs on four RTX 3090 (24G) GPUs with almost no performance loss.^[1] The project reported that GLM-130B beat GPT-3 175B on LAMBADA by 4.0 percent, OPT-175B by 5.5 percent and BLOOM-176B by 13.0 percent, and was slightly ahead of GPT-3 175B on MMLU by 0.9 percent.^[1]

On Chinese benchmarks the model outperformed ERNIE TITAN 3.0 260B on seven zero-shot CLUE datasets, by 24.26 percent, and on five zero-shot FewCLUE datasets, by 12.75 percent.^[1]

The paper was accepted to ICLR 2023 and announced on 21 January 2023.^[1]

Impact

The release gave researchers an open bilingual model at a large scale, with weights and code that could be studied directly. For readers, the event is the starting point of the GLM line, which the ChatGLM chat models of 2023 followed.^[2]

What a bilingual open model offers

A bilingual model is trained on two languages, so one system can read and write in both. Open weights let researchers at other institutions download the model, test it on their own tasks and build on it. The release used two separate licenses, one for the code and one for the weights, and each sets its own terms.