Safety & governance Papers

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? A 2021 FAccT paper by Bender, Gebru, McMillan-Major and Shmitchell that asks whether language models can be too large and sets out recommendations on costs, data, research goals and alternative directions.
Training language models to follow instructions with human feedback (InstructGPT) InstructGPT used human-written demonstrations, human rankings of outputs and reinforcement learning to make GPT-3 follow user instructions better. Labelers preferred its outputs even though it was far smaller than GPT-3.
Deep reinforcement learning from human preferences A 2017 paper showing that an AI agent can learn a task from short human comparisons instead of a hand-written reward. The method, reinforcement learning from human feedback, later became central to training chat assistants.
Language Models are Unsupervised Multitask Learners (GPT-2) OpenAI's 2019 paper on GPT-2, a 1.5-billion-parameter Transformer trained on web text. It showed that a model trained only to predict the next word could perform several tasks when prompted, and its staged release sparked public debate.
Holistic Evaluation of Language Models (HELM) A 2022 paper by Percy Liang, Rishi Bommasani, Tony Lee and many co-authors that ran 30 language models through the same 42 scenarios and seven metrics, turning scattered benchmarks into one standardized evaluation.
Constitutional AI: Harmlessness from AI Feedback Constitutional AI, a 2022 paper from Anthropic, proposes training a harmless AI assistant with AI-generated feedback guided by a written set of principles, instead of human labels of harmful outputs.
Muse Spark Safety & Preparedness Report Meta's safety and preparedness report for Muse Spark, the model behind Meta AI, sets out the catastrophic-risk evaluations behind its launch and argues that, with mitigations applied, residual risk is acceptable.