BLOOM

What it is BLOOM is an open multilingual large language model, released on 12 July 2022 by BigScience. BigScience was a year-long open research workshop coordinated by Hugging Face.^[1][2] A large language model is a neural network trained on a very large body of text to predict and generate language. BLOOM has 176 billion parameters, which are the numerical values the model learns during training. It is a Transformer-based autoregressive model of the decoder-only kind, which means it generates text one token at a time, each token conditioned on those before it. The model was built with Megatron and DeepSpeed, two software frameworks for training very large models efficiently.^[1][2] ## Origins The project began with BigScience, a workshop that started on 28 April 2021 and brought together a very large group of researchers. The model was built by more than 1,000 researchers from over 70 countries, drawn from more than 250 institutions.^[1][4] The aim of the workshop was to produce a model that was open and transparent, and the release was the result of that work. Training ran for 117 days, from 11 March to 6 July 2022. It took place on the French Jean Zay supercomputer, under a compute grant from CNRS and GENCI estimated at €3 million. The model card lists 384 A100 80GB GPUs as the hardware used.^[1][3] An A100 is a data-centre graphics processor made by NVIDIA, and the 80GB figure is the memory of each chip. A supercomputer is a very large computer system, and a compute grant is an allocation of its processing time to a project. ## Languages and data BLOOM generates text in 46 natural languages and 13 programming languages.^[1] The training data was the ROOTS corpus, about 1.6 TB of text, which is roughly 366 billion tokens. OSCAR supplied 38% of it.^[2] A token is the unit of text a model reads and writes, and it is typically a word or a fragment of a word. A corpus is a collection of text used for training. ## Licence and access The weights were released under the BigScience Responsible AI License (RAIL) v1.0. This license allows reuse, but it places use restrictions on the model.^[1][2] The source code is licensed under Apache 2.0, a permissive open-source license.^[1][2] At launch the model could be downloaded or used through a hosted inference API.^[1] ## Transparency BLOOM was described as the first multilingual LLM “trained in complete transparency”. The team also published intermediate checkpoints and optimizer states, which allow other researchers to see how the model changed during training.^[1] An optimizer state records the values used to adjust the model's weights as training proceeds. ## Follow-on models BLOOMZ is an instruction-following version fine-tuned on the xP3 dataset. BLOOM has powered chatbots including BLOOMChat and HuggingChat.^[2] Fine-tuning means training an existing model further on a narrower set of examples, here to make it follow instructions. In February 2023 Hugging Face partnered with AWS. The company said that the next generation of BLOOM would run on AWS's Trainium chip.^[4] Trainium is Amazon's custom chip for training machine-learning models. ## Impact BLOOM is a reference point for open, multilingual model development. It was built in public, with its data, licence and training process documented, and it shows that a large model can be trained by a collaboration of researchers rather than by one company. Readers who want to use or study the model should check the licence terms and the model card before relying on it, because the restrictions attached to the weights matter for what can be done with them.