BigScience releases BLOOM, a 176B-parameter open multilingual model

Background BigScience was a research workshop that brought together a large number of researchers to build an open language model. Hugging Face coordinated the effort. Large language models of this size are expensive to train, and the project relied on shared computing resources and on a common set of rules about how the model and its data would be released.^[1] A language model learns patterns in text so that it can predict and generate language, and a larger number of parameters generally gives the model more capacity to learn those patterns. ## What happened On 12 July 2022 the BigScience workshop, coordinated by Hugging Face, released BLOOM, a language model with 176 billion parameters.^[1] The model was released as an open model, and its weights were made available under the Responsible AI License developed during the BigScience project.^[1] ## Key details BLOOM covers 46 natural languages and 13 programming languages.^[1] Covering many languages means that the model is not limited to English, which is a point of interest for readers who study how language technology is distributed. More than 1,000 researchers from over 70 countries and 250+ institutions took part in building the model.^[1] This large group is part of why the project is described as collaborative rather than the work of a single company. Training ran for 117 days, from 11 March to 6 July 2022. It took place on France's Jean Zay supercomputer, under a compute grant from CNRS and GENCI estimated at €3 million.^[1] A compute grant is an allocation of computing time to a project, and the cost figure is an estimate from the sources rather than a precise accounting. The weights are under the Responsible AI License (RAIL), which was developed during BigScience. The license permits use of the model and building on it, subject to use restrictions.^[1] Restrictions of this kind set out uses that the licensors wanted to exclude, and they apply to anyone who uses the weights. Hugging Face called BLOOM the first multilingual LLM trained in complete transparency. It also released intermediate checkpoints and optimizer states, so that other researchers could follow how the model developed during training.^[1] Intermediate checkpoints are saved snapshots of the model taken during training. The ROOTS training corpus held about 1.6 TB of text.^[2] This is the data from which the model learned, and its size is one of the figures that readers use to judge the scale of the work. ## Impact The release showed that a large multilingual model could be developed in public by a broad group of researchers. Its documentation, license and training record are part of the lasting value of the project, because they let later researchers see what was done and under what terms. Readers who look into the model should consult the original documents and not rely only on summaries.