Home›Papers
Papers
Papers
Papers that changed AI, with a plain-language explainer for each.
Total 76 · Clear filters
- 2024
Mixtral of Experts
arXiv
Mixtral 8x7B is a sparse mixture-of-experts language model from Mistral AI. A router sends each token to two of eight experts per layer, so the model holds 47B parameters but uses 13B during inference.
- 2023
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
arXiv
Mamba is a sequence-model architecture introduced by Albert Gu and Tri Dao that replaces attention with selective state space models, giving linear-time scaling, faster inference, and language-model quality competitive with Transformers twice its size.
- 2023
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
arXiv
A benchmark of 2,294 real software engineering problems taken from GitHub issues and the pull requests that resolved them, in which a model must edit a codebase to fix an issue; on release the best model solved under 2% of them.
- 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models
arXiv 2307.09288
A July 2023 paper from Meta led by Hugo Touvron that introduces and releases Llama 2, a family of pretrained and fine-tuned language models from 7 to 70 billion parameters, together with a detailed account of how the chat versions were tuned and made safer.
- 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
arXiv 2305.18290
A 2023 paper by Rafailov and colleagues that rewrites the reinforcement-learning-from-human-feedback objective so a language model can be aligned to preferences with a simple classification loss, without a separate reward model or reinforcement learning.
- 2023
QLoRA: Efficient Finetuning of Quantized LLMs
arXiv
QLoRA is a finetuning method that trains a 65B-parameter language model on a single 48GB GPU while matching full 16-bit finetuning quality, using 4-bit quantization, double quantization and paged optimizers.
- 2023
RedPajama: open reproduction of the LLaMA training dataset
Together AI blog / GitHub
RedPajama is an open-source project that began by reproducing the LLaMA training dataset, releasing a 1.2-trillion-token corpus as a clean-room, fully open implementation.
- 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4
arXiv
A March 2023 paper by Microsoft researchers testing an early GPT-4, arguing it shows strikingly general, near-human performance across many domains and could be seen as an early but incomplete step toward AGI.
- 2023
GPT-4 Technical Report
arXiv 2303.08774
OpenAI's technical report on GPT-4, submitted to arXiv in March 2023, describes a large multimodal Transformer model, reports human-level results on several professional benchmarks, and explains how the team made scaling behaviour predictable.
- 2023
LLaMA: Open and Efficient Foundation Language Models
arXiv
LLaMA is a family of foundation language models from 7B to 65B parameters, trained only on public datasets; its 13B model beat GPT-3 on most benchmarks, and all models were released to researchers.
- 2023
Toolformer: Language Models Can Teach Themselves to Use Tools
arXiv
A 2023 paper by Schick, Dwivedi-Yu, Dessì and co-authors showing that a language model can teach itself when to call a calculator, search engine, Q&A system, translation system or calendar, and how to use the results.
- 2022
Constitutional AI: Harmlessness from AI Feedback
arXiv
Constitutional AI, a 2022 paper from Anthropic, proposes training a harmless AI assistant with AI-generated feedback guided by a written set of principles, instead of human labels of harmful outputs.
- 2022
Holistic Evaluation of Language Models (HELM)
arXiv
A 2022 paper by Percy Liang, Rishi Bommasani, Tony Lee and many co-authors that ran 30 language models through the same 42 scenarios and seven metrics, turning scattered benchmarks into one standardized evaluation.
- 2022
ReAct: Synergizing Reasoning and Acting in Language Models
arXiv
ReAct is a prompting method that has a language model interleave written reasoning with calls to outside tools or environments, improving question answering, fact verification and interactive decision-making.
- 2022
Emergent Abilities of Large Language Models
arXiv 2206.07682 (TMLR 2022)
A 2022 paper arguing that some abilities of large language models appear only once models are large enough, and cannot be predicted by extrapolating from smaller models — so further scaling may keep widening what these models can do.
- 2022
Beyond the Imitation Game (BIG-bench)
arXiv
BIG-bench is a large community-built benchmark of 204 tasks designed to test language models on problems believed to lie beyond their current abilities, described in a paper posted in June 2022.
- 2022
Training Compute-Optimal Large Language Models (Chinchilla)
NeurIPS
A DeepMind study showing that, for a fixed computing budget, language models should be trained on far more data than was then standard; its 70-billion-parameter Chinchilla beat larger contemporaries.
- 2022
Training language models to follow instructions with human feedback (InstructGPT)
NeurIPS
InstructGPT used human-written demonstrations, human rankings of outputs and reinforcement learning to make GPT-3 follow user instructions better. Labelers preferred its outputs even though it was far smaller than GPT-3.
- 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
NeurIPS
A 2022 paper by Jason Wei and colleagues at Google Research showing that prompting a large language model with examples that include intermediate reasoning steps, rather than answers alone, substantially improves its performance on arithmetic, commonsense and symbolic reasoning tasks.
- 2021
Training Verifiers to Solve Math Word Problems (GSM8K)
arXiv
GSM8K is an OpenAI benchmark of 8.5K grade school math word problems, released with a paper showing that even the largest Transformer models scored poorly, and that training verifiers to select among many candidate solutions improved results.