Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 29 · Clear filters
- 2026
Muse Spark Safety & Preparedness Report
arXiv
Meta's safety and preparedness report for Muse Spark, the model behind Meta AI, sets out the catastrophic-risk evaluations behind its launch and argues that, with mitigations applied, residual risk is acceptable.
- 2026
GLM-5: from Vibe Coding to Agentic Engineering
arXiv
Zhipu's GLM-5 report describes a 744-billion-parameter mixture-of-experts model built on sparse attention and asynchronous reinforcement learning, aimed at moving AI-assisted programming from vibe coding toward agentic engineering.
- 2026
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
arXiv
Terminal-Bench 2.0 is a curated benchmark of 89 hard tasks carried out in computer terminal environments, built to measure how well AI agents handle realistic command-line work; frontier models and agents score below 65% on it.
- 2024
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
arXiv
FlashAttention-3 is an attention algorithm for NVIDIA Hopper (H100) GPUs that reaches up to 75% utilization in FP16 and close to 1.2 PFLOPs/s in FP8, using asynchronous execution and low-precision techniques.
- 2024
LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
arXiv
A benchmark for code LLMs that keeps collecting fresh contest problems from LeetCode, AtCoder and CodeForces, so that scores reflect ability rather than memorised test data, and that also tests code repair and execution reasoning.
- 2023
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
arXiv
A benchmark of 2,294 real software engineering problems taken from GitHub issues and the pull requests that resolved them, in which a model must edit a codebase to fix an issue; on release the best model solved under 2% of them.
- 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
arXiv
FlashAttention-2 is a 2023 revision of the FlashAttention attention kernel that reorganizes work across GPU thread blocks and warps, roughly doubling speed and reaching up to 73% of the A100's theoretical peak throughput.
- 2023
QLoRA: Efficient Finetuning of Quantized LLMs
arXiv
QLoRA is a finetuning method that trains a 65B-parameter language model on a single 48GB GPU while matching full 16-bit finetuning quality, using 4-bit quantization, double quantization and paged optimizers.
- 2022
Constitutional AI: Harmlessness from AI Feedback
arXiv
Constitutional AI, a 2022 paper from Anthropic, proposes training a harmless AI assistant with AI-generated feedback guided by a written set of principles, instead of human labels of harmful outputs.
- 2022
Holistic Evaluation of Language Models (HELM)
arXiv
A 2022 paper by Percy Liang, Rishi Bommasani, Tony Lee and many co-authors that ran 30 language models through the same 42 scenarios and seven metrics, turning scattered benchmarks into one standardized evaluation.
- 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
arXiv
FlashAttention is an exact attention algorithm built around GPU memory traffic: by tiling computation so intermediates stay on chip, it cuts reads and writes to high bandwidth memory, training Transformers faster and enabling longer sequences.
- 2022
Training language models to follow instructions with human feedback (InstructGPT)
NeurIPS
InstructGPT used human-written demonstrations, human rankings of outputs and reinforcement learning to make GPT-3 follow user instructions better. Labelers preferred its outputs even though it was far smaller than GPT-3.
- 2022
Magnetic control of tokamak plasmas through deep reinforcement learning
Nature
A 2022 Nature paper by Degrave, Felici and colleagues showing that a deep reinforcement learning controller can learn on its own to command all of a tokamak's magnetic control coils, producing and holding a range of plasma shapes.
- 2021
Program Synthesis with Large Language Models (MBPP)
arXiv
A 2021 arXiv paper by Austin, Odena, Nye and eight other authors that introduced MBPP, a benchmark of 974 short Python programming tasks, and measured how language models from 244M to 137B parameters turn natural-language descriptions into working code.
- 2021
Evaluating Large Language Models Trained on Code (Codex / HumanEval)
arXiv
A 2021 paper from OpenAI introduces Codex, a GPT model fine-tuned on public GitHub code, and HumanEval, an evaluation set that judges generated Python programs by whether they work rather than by how closely they match a reference.
- 2021
LoRA: Low-Rank Adaptation of Large Language Models
arXiv
LoRA is a fine-tuning method that freezes a pre-trained model's weights and trains small low-rank matrices injected into each layer, cutting trainable parameters and memory by orders of magnitude.
- 2021
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
FAccT '21
A 2021 FAccT paper by Bender, Gebru, McMillan-Major and Shmitchell that asks whether language models can be too large and sets out recommendations on costs, data, research goals and alternative directions.
- 2020
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
arXiv
GShard, submitted to arXiv in June 2020, is a set of annotation APIs and an XLA compiler extension from Google that lets existing model code express parallel computation, scaling a sparsely gated mixture-of-experts translation model past 600 billion parameters.
- 2020
Scaling Laws for Neural Language Models
arXiv
A 2020 paper finding that language model loss falls predictably, as a power law, with model size, data and compute. It gave labs a quantitative reason to build much larger models and shaped the scale-up that followed.
- 2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
arXiv 1909.08053
An NVIDIA paper from 2019 presenting an intra-layer model parallelism method, implemented with a few communication operations in native PyTorch, that trained transformer models of up to 8.3 billion parameters and improved several benchmarks.