Open source Papers

LLaMA is a family of foundation language models from 7B to 65B parameters, trained only on public datasets; its 13B model beat GPT-3 on most benchmarks, and all models were released to researchers.

A July 2023 paper from Meta led by Hugo Touvron that introduces and releases Llama 2, a family of pretrained and fine-tuned language models from 7 to 70 billion parameters, together with a detailed account of how the chat versions were tuned and made safer.

Mixtral 8x7B is a sparse mixture-of-experts language model from Mistral AI. A router sends each token to two of eight experts per layer, so the model holds 47B parameters but uses 13B during inference.

Meta's 2024 paper describing Llama 3 as a family of foundation models — 8B, 70B and 405B parameters — trained on about 15 trillion tokens and reported to match leading models such as GPT-4 across a wide range of tasks.

A technical report from DeepSeek-AI on DeepSeek-V3, a mixture-of-experts language model with 671 billion total parameters, describing its architecture, training pipeline, and evaluations, including a full training run of 2.788 million H800 GPU hours.

The DeepSeek-R1 report described how a reasoning model was trained largely with reinforcement learning, rewarding correct answers, and reached performance comparable to OpenAI's o1 on math, code and reasoning benchmarks. Its weights were released openly.

The Qwen3 Technical Report presents Alibaba's Qwen3 family of dense and mixture-of-experts language models, which unify thinking and non-thinking modes, add a controllable thinking budget, and are released under the Apache 2.0 licence.

Kimi K2 is Moonshot AI's trillion-parameter mixture-of-experts model, pre-trained with the MuonClip optimizer and post-trained on large-scale agentic data plus reinforcement learning; the report reports state-of-the-art results among open-source non-thinking models and releases base and post-trained checkpoints.

A preview report on the DeepSeek-V4 series, describing a 1.6-trillion-parameter Pro model and a 284-billion-parameter Flash model, both with one-million-token context, plus the attention, connection and optimization changes that make that length affordable.

Kimi K3 is Moonshot AI's 2.8-trillion-parameter open mixture-of-experts model with native vision and a one-million-token context, released with full weights. Its report describes the architecture, training and serving work behind it, and reports frontier-level results that still trail the strongest proprietary models.

A 2011 workshop paper by Ronan Collobert, Koray Kavukcuoglu and Clement Farabet describing Torch7, a Lua-based numeric computing framework and machine learning library built for flexible experimentation on fast low-level kernels.

A 2011 Journal of Machine Learning Research paper by Collobert, Weston, Bottou, Karlen, Kavukcuoglu and Kuksa, presenting one neural network architecture and learning algorithm for part-of-speech tagging, chunking, named entity recognition and semantic role labeling, trained on mostly unlabeled data with little task-specific engineering.

Submitted to arXiv in January 2013 by Tomas Mikolov and colleagues at Google, this paper introduced CBOW and Skip-gram, two simple architectures that learn word vectors quickly from very large text collections. They are the basis of the word2vec project.

The Pile is an 825 GiB English text dataset of 22 combined sources, released by EleutherAI on 31 December 2020 with an arXiv paper. It became the training data for several of the group's open models.

LAION-5B is an openly available dataset of 5.85 billion CLIP-filtered image-text pairs, released so that researchers outside a handful of labs could study and reproduce large vision-language models.

FineWeb is a 15-trillion-token pretraining dataset built from 96 Common Crawl snapshots and released in 2024 with its curation code and ablation models; its educational subset, FineWeb-Edu, improves results on knowledge- and reasoning-heavy benchmarks.

A 2025 paper proposing Absolute Zero, a reinforcement-learning paradigm in which one model writes its own code-reasoning tasks and then solves them, with a code executor as the only judge, reaching state of the art on coding and maths without external training data.

The Qwen3.5-Omni Technical Report documents Alibaba's Qwen3.5-Omni, a hundreds-of-billions-parameter model that understands long audio and video and generates speech across 10 languages, reporting state-of-the-art results on 215 audio and audio-visual benchmarks.

A 2019 Google paper that frames every language task as text in, text out, using one encoder-decoder Transformer and one training setup. It also introduced the C4 dataset, and the released T5 models became common research starting points.

RedPajama is an open-source project that began by reproducing the LLaMA training dataset, releasing a 1.2-trillion-token corpus as a clean-room, fully open implementation.

Dolma is an openly released English corpus of three trillion tokens, built from web text, papers, code, books, social media and Wikipedia, with a paper documenting both the dataset and the pipeline used to curate it.

An eight-terabyte collection of public domain and openly licensed text for pretraining language models, released together with two 7B-parameter models trained on it to show the data is enough.