OPT-175B (Open Pre-trained Transformer)

What it is

OPT, short for Open Pre-trained Transformer, is a family of language models built by Meta AI. Each model is a decoder-only Transformer, the neural network design used by most modern language models, trained to predict the next token in a passage of text. A parameter is one of the numbers a model adjusts during training, so the name OPT-175B refers to a model with 175 billion of them. The family ranges from 125 million to 175 billion parameters, and the largest member is OPT-175B.^[1][2]

Origins

Meta AI announced the family on 3 May 2022. The accompanying paper, *OPT: Open Pre-trained Transformer Language Models*, was posted to arXiv on 2 May 2022. It had 19 authors, and the first author was Susan Zhang.^[1][2] Meta's announcement carried the title "Democratizing access to large-scale language models with OPT-175B", which states the aim of the release: to put models at this scale in the hands of more researchers.^[1]

Within Meta's own sequence of releases, OPT was the first openly shared large language model at GPT-3 scale. Galactica followed in November 2022, and LLaMA in February 2023.^[1][3]

Main features

Model sizes. Models with 125 million, 350 million, 1.3 billion, 2.7 billion, 6.7 billion, 13 billion and 30 billion parameters were fully released at launch. A 66-billion-parameter model was announced as coming soon.^[1]

Who could get OPT-175B. Access to OPT-175B was by request form. The form was aimed at academic researchers, people at government, civil society and academic organisations, and industry research labs worldwide.^[1]

Licence. All pretrained models were licensed under the OPT-175B License Agreement, a noncommercial license focused on research use.^[1]

Training method. Training combined Fully Sharded Data Parallel with NVIDIA Megatron-LM tensor parallelism. It reached about 147 TFLOP/s per GPU on 80 GB A100 GPUs. The released codebase can train and deploy the model on 16 NVIDIA V100 GPUs.^[1]

Training data. The model was trained on publicly available data sets.^[1]

Compute and openness. The paper states that OPT-175B is comparable to GPT-3 while requiring only one seventh of the carbon footprint to develop.^[2] Meta also released development notes and the full day-to-day training logbook. They record the compute used and the human effort needed when training became unstable.^[1][2]

Its place in development

OPT shows one way to release a very large model to researchers: with the weights, the code to run them, and a record of how the training went. The logbook is the most unusual part of that package, because it gives outside readers a concrete account of the compute and the troubleshooting that large training runs require.^[1][2]

Its successors show how the approach changed. Galactica, released in November 2022, was aimed at scientific material, and LLaMA, released in February 2023, became one of the best-known open model families of its time.^[1][3] Read together, OPT, Galactica and LLaMA show Meta moving from a general-purpose open model at GPT-3 scale toward narrower science models and then toward a broader open family.^[1][3]