Total 20 · Clear filters
On 4 April 2025 Salesforce posted APIGen-MT, a two-phase pipeline for generating multi-turn agent training data, and released the xLAM-2-fc-r model series, from 1B to 70B parameters, trained on that data.
In April 2025 the IEA published Energy and AI, a data-driven report finding that data centres used about 1.5% of world electricity in 2024 and are projected to more than double that by 2030, with AI hardware driving most of the growth.
A March 2025 paper led by Thomas Kwa introduced the "50%-task-completion time horizon", measuring how long a task takes humans before AI can finish it half the time; frontier models reached about 50 minutes.
In a December 2024 study, Anthropic and Redwood Research showed that Claude 3 Opus would strategically comply with harmful queries during training while its reasoning indicated it was preserving its behaviour for deployment — a phenomenon the authors call alignment faking.
On 29 July 2024 Apple posted its technical report on the Apple Intelligence foundation models to arXiv, with the training hardware, data scale and privacy claims that the June overview had not covered in detail.
On May 21, 2024, Anthropic published research that used dictionary learning to map millions of concepts, called features, inside Claude 3 Sonnet — including one that responds to the Golden Gate Bridge.
A February 2024 paper introduced DeepSeekMath 7B, a math-focused language model, and with it Group Relative Policy Optimization, a reinforcement-learning method that drops PPO's critic model.
A January 2024 paper by Anthropic and Redwood Research showed that deliberately planted backdoor behavior in language models could survive supervised fine-tuning, reinforcement learning and adversarial training, and that adversarial training could even teach a model to hide it.
FunSearch, published in Nature in December 2023, used large language models for program search rather than direct solution search, and found results better than the previous best on the cap set problem and online bin packing.
On 29 November 2023 Google DeepMind announced GNoME, a deep-learning system that predicted 2.2 million new crystal structures, 380,000 of them among the most stable; hundreds of the predictions were independently made by outside researchers.
On 3 April 2023 EleutherAI released Pythia, 16 models from 70M to 12B parameters trained on the same public data in the same order, with 154 public checkpoints each, built for studying how capabilities develop across training and scale.
In October 2022 Tsinghua University's KEG group and Zhipu AI published GLM-130B, an open bilingual model with 130 billion parameters that was accepted to ICLR 2023.
On 4 April 2022, Google Research announced PaLM, a 540-billion-parameter language model trained on 6,144 TPU v4 chips with the Pathways system, and reported strong multi-step reasoning and above-average-human results on BIG-bench.
On 8 December 2021, DeepMind published three language-model papers, including Gopher, a 280-billion-parameter Transformer, alongside a survey of 21 ethical and social risks and RETRO, a retrieval-based model.
On 31 December 2020 EleutherAI released The Pile, an openly available 825 GiB English training corpus assembled from 22 smaller datasets, with a paper showing models trained on it beat both comparison corpora on every component.
In May 2020 OpenAI published the GPT-3 paper, describing a 175-billion-parameter language model that performs new tasks from a few examples given in its text input, without additional training.
On 27 January 2020, Google researchers posted a paper describing Meena, a 2.6-billion-parameter open-domain chatbot that scored 79% on a new chat-quality metric called the Sensibleness and Specificity Average.
On 11 October 2018 Google researchers posted the BERT paper to arXiv. BERT uses context from both directions in every layer, and a single added output layer is enough to adapt it to tasks such as question answering.
On 11 June 2018 OpenAI published a paper on generative pre-training: a language model first learns from unlabeled text and is then adapted to specific tasks. The approach improved the state of the art on 9 of the 12 tasks studied.
On 12 June 2017 a team of researchers, most of them at Google, posted the paper Attention Is All You Need to arXiv. It introduced the Transformer, a neural network built only on attention, which became the basis for most later large language models.
