Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 67 · Clear filters
- 2022
Scalable Diffusion Models with Transformers (DiT)
arXiv
A 2022 paper by William Peebles and Saining Xie that replaces the U-Net backbone of diffusion models with a transformer operating on latent patches, showing that image quality improves predictably as compute grows and reaching a state-of-the-art FID of 2.27 on ImageNet 256×256.
- 2022
Holistic Evaluation of Language Models (HELM)
arXiv
A 2022 paper by Percy Liang, Rishi Bommasani, Tony Lee and many co-authors that ran 30 language models through the same 42 scenarios and seven metrics, turning scattered benchmarks into one standardized evaluation.
- 2022
LAION-5B: An open large-scale dataset for training next generation image-text models
arXiv
LAION-5B is an openly available dataset of 5.85 billion CLIP-filtered image-text pairs, released so that researchers outside a handful of labs could study and reproduce large vision-language models.
- 2022
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Imagen)
arXiv
Imagen is a text-to-image diffusion model that uses a large pretrained language model to encode prompts; it reached a zero-shot FID of 7.27 on COCO and introduced the DrawBench benchmark.
- 2022
Training Compute-Optimal Large Language Models (Chinchilla)
NeurIPS
A DeepMind study showing that, for a fixed computing budget, language models should be trained on far more data than was then standard; its 70-billion-parameter Chinchilla beat larger contemporaries.
- 2021
High-Resolution Image Synthesis with Latent Diffusion Models
arXiv 2112.10752 (CVPR 2022)
A 2021 paper by Rombach, Blattmann, Lorenz, Esser and Ommer that runs diffusion models inside the latent space of pretrained autoencoders, cutting the compute needed for high-resolution image synthesis while adding cross-attention for text and bounding-box control.
- 2021
Program Synthesis with Large Language Models (MBPP)
arXiv
A 2021 arXiv paper by Austin, Odena, Nye and eight other authors that introduced MBPP, a benchmark of 974 short Python programming tasks, and measured how language models from 244M to 137B parameters turn natural-language descriptions into working code.
- 2021
Evaluating Large Language Models Trained on Code (Codex / HumanEval)
arXiv
A 2021 paper from OpenAI introduces Codex, a GPT model fine-tuned on public GitHub code, and HumanEval, an evaluation set that judges generated Python programs by whether they work rather than by how closely they match a reference.
- 2021
Zero-Shot Text-to-Image Generation (DALL·E)
arXiv
A 2021 OpenAI paper describing a single 12-billion-parameter autoregressive transformer trained on 250 million image-text pairs, which generates images from text and, evaluated zero-shot, is competitive with domain-specific models.
- 2021
Learning Transferable Visual Models From Natural Language Supervision (CLIP)
ICML
CLIP is an OpenAI model trained on 400 million image-text pairs to map pictures and sentences into one shared space, so it can classify images zero-shot from a written description.
- 2020
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
arXiv
The Pile is an 825 GiB English text dataset of 22 combined sources, released by EleutherAI on 31 December 2020 with an arXiv paper. It became the training data for several of the group's open models.
- 2020
Denoising Diffusion Probabilistic Models
NeurIPS
A 2020 paper by Jonathan Ho, Ajay Jain and Pieter Abbeel that trained a U-Net to reverse a gradual noising process, making diffusion models a practical rival to GANs and paving the way for DALL-E 2, Imagen and Stable Diffusion.
- 2020
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
arXiv
A 2020 paper by Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng that represents a scene as a neural radiance field: a network mapping 5D coordinates to density and colour, optimized from a sparse set of images with known camera poses.
- 2019
C4: Colossal Clean Crawled Corpus (T5 paper)
JMLR
A 2019 paper that casts every language task as text-to-text, systematically compares transfer-learning choices, releases the C4 corpus and models, and reports state-of-the-art results on summarization, question answering, and classification.
- 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
JMLR
A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.
- 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
arXiv
SuperGLUE is a benchmark for general-purpose language understanding, released in 2019 as a harder successor to GLUE. It bundles eight tasks, a single-number metric, an analysis toolkit and a public leaderboard.
- 2018
A Style-Based Generator Architecture for Generative Adversarial Networks
arXiv
This 2018 paper introduces a style-based GAN generator that separates high-level attributes from stochastic variation and allows scale-specific control, improving distribution quality, interpolation, and disentanglement.
- 2018
Large Scale GAN Training for High Fidelity Natural Image Synthesis
arXiv
A 2018 paper by Andrew Brock, Jeff Donahue and Karen Simonyan that trained generative adversarial networks at the largest scale attempted at the time, introduced BigGANs and set a new state of the art in class-conditional image synthesis.
- 2018
Glow: Generative Flow with Invertible 1x1 Convolutions
arXiv
A 2018 paper by Diederik P. Kingma and Prafulla Dhariwal presenting Glow, a flow-based generative model built on invertible 1x1 convolutions, which improved log-likelihood on standard benchmarks and could synthesise and manipulate large images.
- 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
arXiv
GLUE is a benchmark and analysis platform that evaluates natural language understanding models across a diverse set of existing tasks, with an online comparison platform and a hand-crafted diagnostic suite for linguistic analysis.