GLM-130B

What it is

GLM-130B is an open bilingual Chinese–English dense language model with 130 billion parameters, released by Tsinghua University's KEG group together with Zhipu AI; its paper came out on 6 October 2022.^[1][2] It is built on the GLM architecture, whose 2022 ACL paper introduced a pre-training objective known as autoregressive blank infilling, and the project repository describes GLM-130B itself as a bidirectional dense model.^[2][1] "Dense" here means every token passes through the whole network at inference time, in contrast to sparse mixture-of-experts designs that activate only part of their parameters for each token. By 3 July 2022 the model had been trained on more than 400 billion text tokens, split evenly between the two languages: 200 billion in Chinese and 200 billion in English.^[1]

Where it came from

The GLM-130B paper was submitted to arXiv on 5 October 2022 and lists 18 authors, among them Aohan Zeng and Jie Tang.^[3] The authors stated the goal plainly: to open-source a model at the 100B scale that is at least as good as GPT-3 (davinci).^[3] The comparison targets named in the project's own reporting are the large open models of the period — GPT-3 175B, OPT-175B and BLOOM-176B — which is a fair picture of the field the model was entering.^[1]

Running it

A recurring theme in the project is that a model of this size should be usable without a data centre. Inference runs on a single server with 8 A100 (40G) GPUs or 8 V100 (32G) GPUs.^[1] INT4 quantization for GLM-130B was released on 24 August 2022, and in that form the model runs on 4 RTX 3090 (24G) GPUs with almost no performance loss.^[1] With FasterTransformer, the repository reports up to 2.5X faster generation on a single A100 server.^[1] Training and inference are listed as supported on NVIDIA hardware as well as on Hygon DCU and Ascend 910, with Sunway support marked "Will be released soon".^[1] The code is released under Apache-2.0, while the model weights fall under a separate model license.^[1]

Results

On the LAMBADA benchmark the authors report beating GPT-3 175B by 4.0%, OPT-175B by 5.5% and BLOOM-176B by 13.0%, and edging past GPT-3 175B by 0.9% on MMLU.^[1] The Chinese results were the more lopsided ones: on 7 zero-shot CLUE datasets the model outperformed ERNIE TITAN 3.0 260B by 24.26%, and on 5 zero-shot FewCLUE datasets by 12.75%.^[1] These are the authors' own reported numbers rather than a neutral third-party evaluation, and benchmarks of this kind age quickly as models and test sets change.

Where it sits

The work was accepted to ICLR 2023, announced on 21 January 2023.^[1] It also preceded the ChatGLM chat models released in 2023, which carried the same line of research from a base language model into conversational products.^[2] Read as a whole, GLM-130B is best understood as an argument about access: a hundred-billion-parameter model whose weights, code and inference recipe were public, small enough in its quantized form to run on consumer graphics cards, and strong enough on the benchmarks its authors chose to be compared with the largest open models then available.^[1][3]