NVIDIA and Microsoft unveil the 530-billion-parameter Megatron-Turing NLG

Background

By 2021 language models had grown quickly in size, and each new generation of very large models needed new engineering to train. Training a model with hundreds of billions of parameters means spreading the work across many processors, because no single chip can hold the model or finish the calculations in a practical time. The announcement that NVIDIA and Microsoft made was about both the size of the model and the way it was trained.

A parameter is one of the numbers inside a neural network that are set during training. A language model with 530 billion of them is therefore very large, and the cost of training and running it is correspondingly high.

What happened

On 11 October 2021 NVIDIA and Microsoft announced Megatron-Turing NLG 530B, a generative language model with 530 billion parameters ^[1]. The two companies described it as the largest and most powerful model of its kind, three times the size of the largest existing comparable model ^[1]. The announcement was made through an NVIDIA technical blog post, which explained the training work in detail ^[1].

The paper reported new state-of-the-art results across zero-, one- and few-shot settings on several benchmarks ^[2]. Zero-shot means the model answers without examples, one-shot means it sees one example, and few-shot means it sees a few. Results in these settings show how well a model can handle a task it was not specially trained for.

Key details

The model was trained on 560 NVIDIA DGX A100 servers in the Selene supercomputer ^[1]. A DGX A100 is a server that holds several of NVIDIA's A100 GPUs. A GPU, or graphics processing unit, is a chip that carries out many calculations at the same time, and a supercomputer like Selene links many such servers to work on one task.

Training combined NVIDIA Megatron-LM and Microsoft DeepSpeed ^[1]. Megatron-LM is NVIDIA's library for splitting a large model across many GPUs. DeepSpeed is Microsoft's library for training large models efficiently. Using both together let the team split the work in several ways at once: across copies of the model, across stages of the network and within each layer. This combination is what made a model of this size workable.

The full technical report, "Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model", was posted to arXiv on 28 January 2022 ^[2]. arXiv is a public server where researchers share papers. The gap between the 2021 announcement and the report shows that the public launch came before the detailed record of how the model was built.

Impact

The announcement made the scale of the model a public reference point. Its size was three times that of the largest comparable model, and the companies presented it as the largest of its kind. For researchers, the most useful part was the report, which recorded the training methods and the benchmark results in one place.

Outside researchers could read about the model and its methods in the paper, but the announcement did not make the model itself available. Readers who study the history of large language models will find this announcement useful as a clear case of how the field pushed model size up in the early period of very large models, and how the engineering of training became as important as the model itself.