Models

ERNIE 3.0 Titan is a Chinese pre-trained language model with over 260 billion parameters, described by its authors as the largest dense Chinese model of its time and reported to beat state-of-the-art systems on 68 NLP datasets.

Large language modelsResearch historyChina

Gopher is DeepMind's 280-billion-parameter language model, announced in December 2021 alongside a study of what scale does to language models and a companion taxonomy of their risks.

Large language modelsResearch historyEurope

Megatron-Turing NLG 530B is a 530-billion-parameter language model announced by NVIDIA and Microsoft on 11 October 2021, then the largest monolithic transformer language model, trained on NVIDIA's Selene supercomputer with Megatron-LM and DeepSpeed combined.

Large language modelsFoundation modelsHardware & compute

Jurassic-1 is the first family of large language models from AI21 Labs, announced with the AI21 Studio developer platform in August 2021. Its largest model, Jumbo, has 178 billion parameters.

Large language modelsFoundation modelsDeveloper tools

ERNIE 3.0 is a 10-billion-parameter pre-trained language model from 2021 that trains on plain text plus a large-scale knowledge graph and fuses auto-encoding and auto-regressive networks, so one model handles both language understanding and generation.

Large language modelsResearch historyChina

GPT-J-6B is a six-billion-parameter open language model released by EleutherAI in June 2021 under the Apache 2.0 license. Its published scores sit close to GPT-3 Curie, making it a widely used free baseline for large-model research.

Large language modelsResearch historyOpen source

LaMDA is a family of conversational language models that Google announced in 2021 and described in a 2022 paper, notable for its work on safety and factual grounding in dialogue and for later powering Bard.

Large language modelsResearch historyConsumer apps

GPT-Neo is a family of open text-generation models released by EleutherAI in March 2021 at 125M, 1.3B and 2.7B parameters, built to replicate the GPT-3 architecture and distributed with freely downloadable weights under an MIT license.

Large language modelsResearch historyOpen source

Meena is a 2.6-billion-parameter open-domain chatbot from Google, trained end-to-end on filtered social media conversations. Its 2020 paper introduced the SSA human-evaluation metric and linked next-token perplexity to human-judged chat quality. The work was later renamed LaMDA.

Large language modelsResearch historyConsumer apps

T5 is a family of Google Transformer models that recasts every language task as text in, text out. Its 11-billion-parameter version set state of the art on several benchmarks, and the code, models and C4 dataset were released openly.

Large language modelsOpen sourceFoundation models

ERNIE 2.0 is Baidu's pre-trained language model framework, published in 2019, which builds pre-training tasks incrementally through continual multi-task learning and reported results above BERT and XLNet on English and Chinese benchmarks.

Large language modelsResearch historyOpen source

Baidu's first ERNIE pre-trained language model, released in March 2019, added entity- and phrase-level masking to BERT-style pre-training and reported state-of-the-art results on five Chinese NLP tasks.

Large language modelsResearch historyOpen source

IBM Watson is a question-answering computer system built by an IBM research team led by David Ferrucci; in February 2011 it defeated the two foremost all-time champions of the televised quiz show Jeopardy!

Research historyEnterpriseReasoning