Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 77 · Clear filters
- 2021
Finetuned Language Models Are Zero-Shot Learners
arXiv 2109.01652
A 2021 paper showing that finetuning a language model on many tasks described by instructions — instruction tuning — makes it much better at unseen tasks without examples, surpassing a larger model on most benchmarks tested.
- 2021
Program Synthesis with Large Language Models (MBPP)
arXiv
A 2021 arXiv paper by Austin, Odena, Nye and eight other authors that introduced MBPP, a benchmark of 974 short Python programming tasks, and measured how language models from 244M to 137B parameters turn natural-language descriptions into working code.
- 2021
Evaluating Large Language Models Trained on Code (Codex / HumanEval)
arXiv
A 2021 paper from OpenAI introduces Codex, a GPT model fine-tuned on public GitHub code, and HumanEval, an evaluation set that judges generated Python programs by whether they work rather than by how closely they match a reference.
- 2021
LoRA: Low-Rank Adaptation of Large Language Models
arXiv
LoRA is a fine-tuning method that freezes a pre-trained model's weights and trains small low-rank matrices injected into each layer, cutting trainable parameters and memory by orders of magnitude.
- 2021
Measuring Mathematical Problem Solving With the MATH Dataset
arXiv
MATH is a dataset of 12,500 challenging competition mathematics problems, each with a full step-by-step solution. Released in 2021, it was built to measure and train mathematical reasoning in models.
- 2021
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
FAccT '21
A 2021 FAccT paper by Bender, Gebru, McMillan-Major and Shmitchell that asks whether language models can be too large and sets out recommendations on costs, data, research goals and alternative directions.
- 2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
arXiv
A 2021 paper by Fedus, Zoph and Shazeer that simplifies Mixture-of-Experts routing so that language models can hold up to a trillion parameters while keeping per-example computation roughly constant.
- 2020
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
arXiv
The Pile is an 825 GiB English text dataset of 22 combined sources, released by EleutherAI on 31 December 2020 with an arXiv paper. It became the training data for several of the group's open models.
- 2020
Measuring Massive Multitask Language Understanding (MMLU)
arXiv
A benchmark released in 2020 that tests language models across 57 subjects, from elementary mathematics to law, and showed that even the largest models of the time were far from expert-level.
- 2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
arXiv
GShard, submitted to arXiv in June 2020, is a set of annotation APIs and an XLA compiler extension from Google that lets existing model code express parallel computation, scaling a sparsely gated mixture-of-experts translation model past 600 billion parameters.
- 2020
Language Models are Few-Shot Learners (GPT-3)
NeurIPS
The GPT-3 paper showed that a language model with 175 billion parameters could perform many tasks from a few examples in its prompt, with no fine-tuning, and that this ability grew with model size.
- 2020
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (RAG)
arXiv
A 2020 paper that pairs a pre-trained seq2seq model with a dense Wikipedia index fetched by a neural retriever, setting state of the art on three open-domain QA tasks and producing more factual text than a parametric-only baseline.
- 2020
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
arXiv
A 2020 paper by Clark, Luong, Le and Manning that pre-trains text encoders by detecting replaced tokens instead of filling in masked ones, matching much larger baselines with far less compute.
- 2020
Scaling Laws for Neural Language Models
arXiv
A 2020 paper finding that language model loss falls predictably, as a power law, with model size, data and compute. It gave labs a quantitative reason to build much larger models and shaped the scale-up that followed.
- 2019
C4: Colossal Clean Crawled Corpus (T5 paper)
JMLR
A 2019 paper that casts every language task as text-to-text, systematically compares transfer-learning choices, releases the C4 corpus and models, and reports state-of-the-art results on summarization, question answering, and classification.
- 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
JMLR
A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.
- 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach
arXiv
A replication study of BERT pretraining that measures how much key hyperparameters and training data size matter, finds BERT was significantly undertrained, and shows a properly trained BERT matches or beats later models, reaching state of the art on GLUE, RACE and SQuAD.
- 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding
arXiv
XLNet is a generalized autoregressive pretraining method from a 2019 paper. It learns bidirectional contexts by maximizing expected likelihood over all factorization-order permutations and outperforms BERT on 20 tasks under comparable settings.
- 2019
HellaSwag: Can a Machine Really Finish Your Sentence?
arXiv
HellaSwag is a 2019 benchmark in which a machine must pick the most plausible continuation of a short event description. Its wrong answers are generated and filtered to fool models rather than people.
- 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
arXiv
SuperGLUE is a benchmark for general-purpose language understanding, released in 2019 as a harder successor to GLUE. It bundles eight tasks, a single-number metric, an analysis toolkit and a public leaderboard.