Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 82 · Clear filters
- 2024
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
arXiv
Dolma is an openly released English corpus of three trillion tokens, built from web text, papers, code, books, social media and Wikipedia, with a paper documenting both the dataset and the pipeline used to curate it.
- 2024
Mixtral of Experts
arXiv
Mixtral 8x7B is a sparse mixture-of-experts language model from Mistral AI. A router sends each token to two of eight experts per layer, so the model holds 47B parameters but uses 13B during inference.
- 2023
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
arXiv
Mamba is a sequence-model architecture introduced by Albert Gu and Tri Dao that replaces attention with selective state space models, giving linear-time scaling, faster inference, and language-model quality competitive with Transformers twice its size.
- 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models
arXiv 2307.09288
A July 2023 paper from Meta led by Hugo Touvron that introduces and releases Llama 2, a family of pretrained and fine-tuned language models from 7 to 70 billion parameters, together with a detailed account of how the chat versions were tuned and made safer.
- 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
arXiv
FlashAttention-2 is a 2023 revision of the FlashAttention attention kernel that reorganizes work across GPU thread blocks and warps, roughly doubling speed and reaching up to 73% of the A100's theoretical peak throughput.
- 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
arXiv 2305.18290
A 2023 paper by Rafailov and colleagues that rewrites the reinforcement-learning-from-human-feedback objective so a language model can be aligned to preferences with a simple classification loss, without a separate reward model or reinforcement learning.
- 2023
QLoRA: Efficient Finetuning of Quantized LLMs
arXiv
QLoRA is a finetuning method that trains a 65B-parameter language model on a single 48GB GPU while matching full 16-bit finetuning quality, using 4-bit quantization, double quantization and paged optimizers.
- 2023
RedPajama: open reproduction of the LLaMA training dataset
Together AI blog / GitHub
RedPajama is an open-source project that began by reproducing the LLaMA training dataset, releasing a 1.2-trillion-token corpus as a clean-room, fully open implementation.
- 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4
arXiv
A March 2023 paper by Microsoft researchers testing an early GPT-4, arguing it shows strikingly general, near-human performance across many domains and could be seen as an early but incomplete step toward AGI.
- 2023
GPT-4 Technical Report
arXiv 2303.08774
OpenAI's technical report on GPT-4, submitted to arXiv in March 2023, describes a large multimodal Transformer model, reports human-level results on several professional benchmarks, and explains how the team made scaling behaviour predictable.
- 2023
LLaMA: Open and Efficient Foundation Language Models
arXiv
LLaMA is a family of foundation language models from 7B to 65B parameters, trained only on public datasets; its 13B model beat GPT-3 on most benchmarks, and all models were released to researchers.
- 2023
Toolformer: Language Models Can Teach Themselves to Use Tools
arXiv
A 2023 paper by Schick, Dwivedi-Yu, Dessì and co-authors showing that a language model can teach itself when to call a calculator, search engine, Q&A system, translation system or calendar, and how to use the results.
- 2022
Scalable Diffusion Models with Transformers (DiT)
arXiv
A 2022 paper by William Peebles and Saining Xie that replaces the U-Net backbone of diffusion models with a transformer operating on latent patches, showing that image quality improves predictably as compute grows and reaching a state-of-the-art FID of 2.27 on ImageNet 256×256.
- 2022
Constitutional AI: Harmlessness from AI Feedback
arXiv
Constitutional AI, a 2022 paper from Anthropic, proposes training a harmless AI assistant with AI-generated feedback guided by a written set of principles, instead of human labels of harmful outputs.
- 2022
Holistic Evaluation of Language Models (HELM)
arXiv
A 2022 paper by Percy Liang, Rishi Bommasani, Tony Lee and many co-authors that ran 30 language models through the same 42 scenarios and seven metrics, turning scattered benchmarks into one standardized evaluation.
- 2022
ReAct: Synergizing Reasoning and Acting in Language Models
arXiv
ReAct is a prompting method that has a language model interleave written reasoning with calls to outside tools or environments, improving question answering, fact verification and interactive decision-making.
- 2022
Emergent Abilities of Large Language Models
arXiv 2206.07682 (TMLR 2022)
A 2022 paper arguing that some abilities of large language models appear only once models are large enough, and cannot be predicted by extrapolating from smaller models — so further scaling may keep widening what these models can do.
- 2022
Beyond the Imitation Game (BIG-bench)
arXiv
BIG-bench is a large community-built benchmark of 204 tasks designed to test language models on problems believed to lie beyond their current abilities, described in a paper posted in June 2022.
- 2022
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
arXiv
FlashAttention is an exact attention algorithm built around GPU memory traffic: by tiling computation so intermediates stay on chip, it cuts reads and writes to high bandwidth memory, training Transformers faster and enabling longer sequences.
- 2022
Training Compute-Optimal Large Language Models (Chinchilla)
NeurIPS
A DeepMind study showing that, for a fixed computing budget, language models should be trained on far more data than was then standard; its 70-billion-parameter Chinchilla beat larger contemporaries.