Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 71 · Clear filters
- 2023
RedPajama: open reproduction of the LLaMA training dataset
Together AI blog / GitHub
RedPajama is an open-source project that began by reproducing the LLaMA training dataset, releasing a 1.2-trillion-token corpus as a clean-room, fully open implementation.
- 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4
arXiv
A March 2023 paper by Microsoft researchers testing an early GPT-4, arguing it shows strikingly general, near-human performance across many domains and could be seen as an early but incomplete step toward AGI.
- 2022
Holistic Evaluation of Language Models (HELM)
arXiv
A 2022 paper by Percy Liang, Rishi Bommasani, Tony Lee and many co-authors that ran 30 language models through the same 42 scenarios and seven metrics, turning scattered benchmarks into one standardized evaluation.
- 2022
LAION-5B: An open large-scale dataset for training next generation image-text models
arXiv
LAION-5B is an openly available dataset of 5.85 billion CLIP-filtered image-text pairs, released so that researchers outside a handful of labs could study and reproduce large vision-language models.
- 2022
Emergent Abilities of Large Language Models
arXiv 2206.07682 (TMLR 2022)
A 2022 paper arguing that some abilities of large language models appear only once models are large enough, and cannot be predicted by extrapolating from smaller models — so further scaling may keep widening what these models can do.
- 2022
Beyond the Imitation Game (BIG-bench)
arXiv
BIG-bench is a large community-built benchmark of 204 tasks designed to test language models on problems believed to lie beyond their current abilities, described in a paper posted in June 2022.
- 2022
Training Compute-Optimal Large Language Models (Chinchilla)
NeurIPS
A DeepMind study showing that, for a fixed computing budget, language models should be trained on far more data than was then standard; its 70-billion-parameter Chinchilla beat larger contemporaries.
- 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
NeurIPS
A 2022 paper by Jason Wei and colleagues at Google Research showing that prompting a large language model with examples that include intermediate reasoning steps, rather than answers alone, substantially improves its performance on arithmetic, commonsense and symbolic reasoning tasks.
- 2021
Training Verifiers to Solve Math Word Problems (GSM8K)
arXiv
GSM8K is an OpenAI benchmark of 8.5K grade school math word problems, released with a paper showing that even the largest Transformer models scored poorly, and that training verifiers to select among many candidate solutions improved results.
- 2021
Finetuned Language Models Are Zero-Shot Learners
arXiv 2109.01652
A 2021 paper showing that finetuning a language model on many tasks described by instructions — instruction tuning — makes it much better at unseen tasks without examples, surpassing a larger model on most benchmarks tested.
- 2021
Program Synthesis with Large Language Models (MBPP)
arXiv
A 2021 arXiv paper by Austin, Odena, Nye and eight other authors that introduced MBPP, a benchmark of 974 short Python programming tasks, and measured how language models from 244M to 137B parameters turn natural-language descriptions into working code.
- 2021
Evaluating Large Language Models Trained on Code (Codex / HumanEval)
arXiv
A 2021 paper from OpenAI introduces Codex, a GPT model fine-tuned on public GitHub code, and HumanEval, an evaluation set that judges generated Python programs by whether they work rather than by how closely they match a reference.
- 2021
Measuring Mathematical Problem Solving With the MATH Dataset
arXiv
MATH is a dataset of 12,500 challenging competition mathematics problems, each with a full step-by-step solution. Released in 2021, it was built to measure and train mathematical reasoning in models.
- 2020
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
arXiv
The Pile is an 825 GiB English text dataset of 22 combined sources, released by EleutherAI on 31 December 2020 with an arXiv paper. It became the training data for several of the group's open models.
- 2020
Measuring Massive Multitask Language Understanding (MMLU)
arXiv
A benchmark released in 2020 that tests language models across 57 subjects, from elementary mathematics to law, and showed that even the largest models of the time were far from expert-level.
- 2019
On the Measure of Intelligence (ARC-AGI)
arXiv
A 2019 paper by Francois Chollet that defines intelligence as skill-acquisition efficiency and introduces the Abstraction and Reasoning Corpus (ARC), a benchmark of grid tasks that assumes no specialised knowledge or language.
- 2019
C4: Colossal Clean Crawled Corpus (T5 paper)
JMLR
A 2019 paper that casts every language task as text-to-text, systematically compares transfer-learning choices, releases the C4 corpus and models, and reports state-of-the-art results on summarization, question answering, and classification.
- 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
JMLR
A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.
- 2019
HellaSwag: Can a Machine Really Finish Your Sentence?
arXiv
HellaSwag is a 2019 benchmark in which a machine must pick the most plausible continuation of a short event description. Its wrong answers are generated and filtered to fool models rather than people.
- 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
arXiv
SuperGLUE is a benchmark for general-purpose language understanding, released in 2019 as a harder successor to GLUE. It bundles eight tasks, a single-number metric, an analysis toolkit and a public leaderboard.