Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 33 · Clear filters
- 2026
Muse Spark Safety & Preparedness Report
arXiv
Meta's safety and preparedness report for Muse Spark, the model behind Meta AI, sets out the catastrophic-risk evaluations behind its launch and argues that, with mitigations applied, residual risk is acceptable.
- 2026
GLM-5: from Vibe Coding to Agentic Engineering
arXiv
Zhipu's GLM-5 report describes a 744-billion-parameter mixture-of-experts model built on sparse attention and asynchronous reinforcement learning, aimed at moving AI-assisted programming from vibe coding toward agentic engineering.
- 2026
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
arXiv
Terminal-Bench 2.0 is a curated benchmark of 89 hard tasks carried out in computer terminal environments, built to measure how well AI agents handle realistic command-line work; frontier models and agents score below 65% on it.
- 2024
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
arXiv
FlashAttention-3 is an attention algorithm for NVIDIA Hopper (H100) GPUs that reaches up to 75% utilization in FP16 and close to 1.2 PFLOPs/s in FP8, using asynchronous execution and low-precision techniques.
- 2024
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
ICML
Tri Dao and Albert Gu's paper introduces state space duality, a framework showing that structured state space models and attention variants are closely related, and uses it to derive Mamba-2, whose core layer runs 2 to 8 times faster than Mamba's.
- 2024
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
arXiv
A benchmark for code LLMs that keeps collecting fresh contest problems from LeetCode, AtCoder and CodeForces, so that scores reflect ability rather than memorised test data, and that also tests code repair and execution reasoning.
- 2023
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
arXiv
A benchmark of 2,294 real software engineering problems taken from GitHub issues and the pull requests that resolved them, in which a model must edit a codebase to fix an issue; on release the best model solved under 2% of them.
- 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
arXiv
FlashAttention-2 is a 2023 revision of the FlashAttention attention kernel that reorganizes work across GPU thread blocks and warps, roughly doubling speed and reaching up to 73% of the A100's theoretical peak throughput.
- 2023
QLoRA: Efficient Finetuning of Quantized LLMs
arXiv
QLoRA is a finetuning method that trains a 65B-parameter language model on a single 48GB GPU while matching full 16-bit finetuning quality, using 4-bit quantization, double quantization and paged optimizers.
- 2022
Scalable Diffusion Models with Transformers (DiT)
arXiv
A 2022 paper by William Peebles and Saining Xie that replaces the U-Net backbone of diffusion models with a transformer operating on latent patches, showing that image quality improves predictably as compute grows and reaching a state-of-the-art FID of 2.27 on ImageNet 256×256.
- 2022
Constitutional AI: Harmlessness from AI Feedback
arXiv
Constitutional AI, a 2022 paper from Anthropic, proposes training a harmless AI assistant with AI-generated feedback guided by a written set of principles, instead of human labels of harmful outputs.
- 2022
Holistic Evaluation of Language Models (HELM)
arXiv
A 2022 paper by Percy Liang, Rishi Bommasani, Tony Lee and many co-authors that ran 30 language models through the same 42 scenarios and seven metrics, turning scattered benchmarks into one standardized evaluation.
- 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
arXiv
FlashAttention is an exact attention algorithm built around GPU memory traffic: by tiling computation so intermediates stay on chip, it cuts reads and writes to high bandwidth memory, training Transformers faster and enabling longer sequences.
- 2022
Training language models to follow instructions with human feedback (InstructGPT)
NeurIPS
InstructGPT used human-written demonstrations, human rankings of outputs and reinforcement learning to make GPT-3 follow user instructions better. Labelers preferred its outputs even though it was far smaller than GPT-3.
- 2021
Program Synthesis with Large Language Models (MBPP)
arXiv
A 2021 arXiv paper by Austin, Odena, Nye and eight other authors that introduced MBPP, a benchmark of 974 short Python programming tasks, and measured how language models from 244M to 137B parameters turn natural-language descriptions into working code.
- 2021
Evaluating Large Language Models Trained on Code (Codex / HumanEval)
arXiv
A 2021 paper from OpenAI introduces Codex, a GPT model fine-tuned on public GitHub code, and HumanEval, an evaluation set that judges generated Python programs by whether they work rather than by how closely they match a reference.
- 2021
LoRA: Low-Rank Adaptation of Large Language Models
arXiv
LoRA is a fine-tuning method that freezes a pre-trained model's weights and trains small low-rank matrices injected into each layer, cutting trainable parameters and memory by orders of magnitude.
- 2021
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
FAccT '21
A 2021 FAccT paper by Bender, Gebru, McMillan-Major and Shmitchell that asks whether language models can be too large and sets out recommendations on costs, data, research goals and alternative directions.
- 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (ViT)
arXiv
A 2020 paper showing that a plain Transformer applied directly to sequences of image patches, with no convolution at all, can match state-of-the-art convolutional networks on image classification.
- 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
arXiv 2010.11929 (ICLR 2021)
A 2020 paper by Alexey Dosovitskiy and colleagues showing that a pure Transformer applied directly to sequences of image patches — the Vision Transformer, or ViT — matches state-of-the-art convolutional networks on image classification when pre-trained on large amounts of data.