Meta releases OPT-175B to researchers

Background

Large language models are trained on large amounts of text. At the time, models of this size were typically built by a small number of well-resourced organisations. Meta AI made its model of this size available to research labs.^[1]

What happened

On 3 May 2022 Meta AI announced OPT-175B, a language model with 175 billion parameters, trained on publicly available data sets. Meta said it was the first release of a model of this size with both the pretrained weights and the code needed to train and use them.^[1]

The paper introducing the model, with Susan Zhang as first author, was posted to arXiv on 2 May 2022.^[2]

Key details

Smaller models, from 125 million to 30 billion parameters, were fully released. OPT-175B was available on request to academic, government, civil-society and industry research labs.^[1]

All the models were under a noncommercial licence focused on research use.^[1]

The paper said OPT-175B was comparable to GPT-3 with one seventh of the carbon footprint.^[2]

Meta also published the team's day-to-day training logbook. The logbook records what happened during training, which is useful for people studying how such models are built.^[1]

Model size is usually measured by the number of parameters, which are the numerical values a network learns during training. Training a model of 175 billion parameters needs large amounts of computing power, so releasing the weights lets groups that could not train such a model from scratch study it.^[1]

The announcement and the published paper together gave outside readers both the model and an account of how it was built.^[1][2]

Impact

OPT-175B gave researchers outside large companies access to a model of GPT-3 scale and let them study its behaviour. Its noncommercial licence set limits on how it could be used. The published logbook let outsiders examine the training process.^[1][2]