Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 61 · Clear filters
- 2024
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
arXiv
FrontierMath is a benchmark of hundreds of original, unpublished mathematics problems written and vetted by expert mathematicians, introduced by Epoch AI to test advanced mathematical reasoning in AI. At release, state-of-the-art models solved under 2% of the problems.[^1][^2]
- 2024
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
arXiv
FlashAttention-3 is an attention algorithm for NVIDIA Hopper (H100) GPUs that reaches up to 75% utilization in FP16 and close to 1.2 PFLOPs/s in FP8, using asynchronous execution and low-precision techniques.
- 2024
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
ICML
Tri Dao and Albert Gu's paper introduces state space duality, a framework showing that structured state space models and attention variants are closely related, and uses it to derive Mamba-2, whose core layer runs 2 to 8 times faster than Mamba's.
- 2024
Accurate structure prediction of biomolecular interactions with AlphaFold 3
Nature
AlphaFold 3, published in Nature in May 2024, extends structure prediction to biomolecular interactions, reporting greater accuracy than docking tools for protein–ligand binding and than AlphaFold-Multimer v.2.3 for antibody–antigen complexes.
- 2023
Scaling deep learning for materials discovery
Nature
A 2023 Nature paper from researchers at Google DeepMind presenting a scaled-up deep-learning approach to predicting stable crystal structures, screening 2.2 million candidates and reporting a large jump in stability-prediction accuracy.
- 2023
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
arXiv
GPQA is a benchmark of 448 expert-written multiple-choice questions in biology, physics and chemistry, built so that skilled non-experts cannot look the answers up, and hard enough that the strongest GPT-4 baseline scored 39% while domain experts reached 65%.
- 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
arXiv
FlashAttention-2 is a 2023 revision of the FlashAttention attention kernel that reorganizes work across GPU thread blocks and warps, roughly doubling speed and reaching up to 73% of the A100's theoretical peak throughput.
- 2023
QLoRA: Efficient Finetuning of Quantized LLMs
arXiv
QLoRA is a finetuning method that trains a 65B-parameter language model on a single 48GB GPU while matching full 16-bit finetuning quality, using 4-bit quantization, double quantization and paged optimizers.
- 2022
Learning skillful medium-range global weather forecasting
Science
GraphCast is a graph neural network trained on four decades of reanalysis data that produces 10-day global weather forecasts at 0.25 degree resolution in under a minute, outperforming the leading operational deterministic systems on most verification targets.
- 2022
Scalable Diffusion Models with Transformers (DiT)
arXiv
A 2022 paper by William Peebles and Saining Xie that replaces the U-Net backbone of diffusion models with a transformer operating on latent patches, showing that image quality improves predictably as compute grows and reaching a state-of-the-art FID of 2.27 on ImageNet 256×256.
- 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
arXiv
FlashAttention is an exact attention algorithm built around GPU memory traffic: by tiling computation so intermediates stay on chip, it cuts reads and writes to high bandwidth memory, training Transformers faster and enabling longer sequences.
- 2022
Magnetic control of tokamak plasmas through deep reinforcement learning
Nature
A 2022 Nature paper by Degrave, Felici and colleagues showing that a deep reinforcement learning controller can learn on its own to command all of a tokamak's magnetic control coils, producing and holding a range of plasma shapes.
- 2021
Highly accurate protein structure prediction with AlphaFold
Nature
The Nature paper described AlphaFold's methods, system design, and evaluation for protein-structure prediction.
- 2021
LoRA: Low-Rank Adaptation of Large Language Models
arXiv
LoRA is a fine-tuning method that freezes a pre-trained model's weights and trains small low-rank matrices injected into each layer, cutting trainable parameters and memory by orders of magnitude.
- 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT)
arXiv
A 2020 paper showing that a plain Transformer applied directly to sequences of image patches, with no convolution at all, can match state-of-the-art convolutional networks on image classification.
- 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
arXiv 2010.11929 (ICLR 2021)
A 2020 paper by Alexey Dosovitskiy and colleagues showing that a pure Transformer applied directly to sequences of image patches — the Vision Transformer, or ViT — matches state-of-the-art convolutional networks on image classification when pre-trained on large amounts of data.
- 2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
arXiv
GShard, submitted to arXiv in June 2020, is a set of annotation APIs and an XLA compiler extension from Google that lets existing model code express parallel computation, scaling a sparsely gated mixture-of-experts translation model past 600 billion parameters.
- 2019
Mastering Atari, Go, chess and shogi by planning with a learned model
Nature
MuZero is a DeepMind reinforcement-learning algorithm that learns a model of its environment and plans inside that model, mastering Atari, Go, chess and shogi without being given the rules of those games.
- 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5)
JMLR
A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.
- 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
NAACL
Google's 2018 BERT paper introduced an encoder-only Transformer that reads text in both directions and is pre-trained by hiding words. It set new results on eleven language tasks and made download-and-fine-tune the default workflow.