Multimodal Papers

OpenAI's technical report on GPT-4, submitted to arXiv in March 2023, describes a large multimodal Transformer model, reports human-level results on several professional benchmarks, and explains how the team made scaling behaviour predictable.

A report from Google's Gemini Team introducing the Gemini 1.5 family, multimodal models built around very long context, with Gemini 1.5 Pro recalling information across up to 10 million tokens.[^1]

The Gemini 2.5 technical report, submitted to arXiv on 7 July 2025, documents Google's Gemini 2.5 Pro and 2.5 Flash models: their reasoning, long-context video handling, and cost-versus-capability positioning.

Kimi K3 is Moonshot AI's 2.8-trillion-parameter open mixture-of-experts model with native vision and a one-million-token context, released with full weights. Its report describes the architecture, training and serving work behind it, and reports frontier-level results that still trail the strongest proprietary models.

A 2014 paper by Vinyals and colleagues that trained a single deep network to turn a photograph into a sentence, reporting a BLEU-1 score of 59 on Pascal against 25 for the previous state of the art.

WaveNet is a deep neural network that generates raw audio waveforms one sample at a time, predicting each sample from all the ones before it. Used for text-to-speech, it was rated by listeners as more natural than the systems it was compared with.

CLIP is an OpenAI model trained on 400 million image-text pairs to map pictures and sentences into one shared space, so it can classify images zero-shot from a written description.

A 2021 OpenAI paper describing a single 12-billion-parameter autoregressive transformer trained on 250 million image-text pairs, which generates images from text and, evaluated zero-shot, is competitive with domain-specific models.

Imagen is a text-to-image diffusion model that uses a large pretrained language model to encode prompts; it reached a zero-shot FID of 7.27 on COCO and introduced the DrawBench benchmark.

LAION-5B is an openly available dataset of 5.85 billion CLIP-filtered image-text pairs, released so that researchers outside a handful of labs could study and reproduce large vision-language models.

WebArena is a reproducible environment of four fully functional websites — e-commerce, a social forum, collaborative software development and content management — built for testing language-guided web agents. Its best GPT-4 agent completed 14.41% of tasks; humans completed 78.24%.[^1]

GAIA is a benchmark of real-world questions that are easy for humans but hard for AI assistants, testing reasoning, multimodality, web browsing, and tool use. It compares AI progress against average human performance.

OSWorld is a benchmark and environment for testing multimodal agents on 369 real computer tasks in Ubuntu, Windows and macOS; humans complete over 72.36% of the tasks, while the best model reaches only 12.24%.

SAM 2 is a Meta FAIR foundation model that segments objects in images and videos from simple prompts such as points, boxes or masks, released together with a large video segmentation dataset and training code.

Humanity's Last Exam (HLE) is a 2,500-question, multi-modal benchmark spanning dozens of academic subjects, built by experts worldwide so that frontier language models can still be measured after saturating older tests.

The Qwen3.5-Omni Technical Report documents Alibaba's Qwen3.5-Omni, a hundreds-of-billions-parameter model that understands long audio and video and generates speech across 10 languages, reporting state-of-the-art results on 215 audio and audio-visual benchmarks.