Multimodal

RT-1 is a transformer-based model for real-world robot control, trained on over 130,000 episodes collected with 13 robots and reported to perform more than 700 language instructions at a 97% success rate.

Google's consumer AI assistant, announced as Bard in 2023 and renamed Gemini in 2024. It is where ordinary users meet Google's Gemini models, first as a chat box and later through voice, research agents and subscriptions.

GPT-4 is OpenAI's large multimodal model, released on 14 March 2023. It accepts image and text input, scored near the top of human test-takers on a simulated bar exam, and was succeeded by GPT-4 Turbo, GPT-4o and GPT-5.

OpenAI's technical report on GPT-4, submitted to arXiv in March 2023, describes a large multimodal Transformer model, reports human-level results on several professional benchmarks, and explains how the team made scaling behaviour predictable.

Gemini 1.0 is the first generation of Google DeepMind's multimodal model family, announced on 6 December 2023 in Ultra, Pro and Nano sizes; Google reported its Ultra version as the first model to beat human experts on MMLU.

Claude 3 Opus, announced on 4 March 2024, was the most capable model in the Claude 3 family. It added vision and launched with a 200K-token context window and a Needle In A Haystack recall above 99%.

A report from Google's Gemini Team introducing the Gemini 1.5 family, multimodal models built around very long context, with Gemini 1.5 Pro recalling information across up to 10 million tokens.[^1]

GPT-4o is a multimodal model from OpenAI, released in May 2024, that takes text, audio, image and video as input and generates text, audio and image output within one neural network. It was removed from ChatGPT in February 2026.

π0 is a generalist robot policy from Physical Intelligence, built by adapting a 3-billion-parameter vision-language model with flow matching so one model can output fast, continuous motor commands for dexterous manipulation.

Gemini 2.5 Pro is Google's first model in the Gemini 2.5 family, released in March 2025 as a "thinking model" with a 1-million-token context window, debuting at number one on LMArena.

The Gemini 2.5 technical report, submitted to arXiv on 7 July 2025, documents Google's Gemini 2.5 Pro and 2.5 Flash models: their reasoning, long-context video handling, and cost-versus-capability positioning.

GPT-5 is OpenAI's fifth GPT foundation model, released on 7 August 2025. It pairs a fast model with a deeper reasoning model and a router that chooses between them for each request.

Gemini 3 Pro is the first model of Google's Gemini 3 generation, released in preview in November 2025 with a 1-million-token context window and top-ranked results on arena leaderboards and reasoning, coding and multimodal benchmarks.

ByteDance's video generation model released in February 2026, built to produce picture and sound together from text, image, audio or video inputs, and known for a copyright dispute that paused its global rollout.

Kimi K3 is a 2.8-trillion-parameter open-weight multimodal model from Moonshot AI, released on 16 July 2026 with weights following on 27 July under the Kimi K3 License.

Kimi K3 is Moonshot AI's 2.8-trillion-parameter open mixture-of-experts model with native vision and a one-million-token context, released with full weights. Its report describes the architecture, training and serving work behind it, and reports frontier-level results that still trail the strongest proprietary models.

A multimodal model handles more than one kind of input or output, such as images together with text, inside a single model. Work on CLIP and Flamingo showed how such models are trained and how they can be steered toward new tasks.

A vision-language-action model (VLA) is a robot policy that takes in camera images and a language instruction and outputs actions directly, usually by adapting a pretrained vision-language model to control.

SenseTime is a Chinese AI company founded in October 2014 and listed in Hong Kong at the end of 2021. It develops the SenseNova large models and, in 2026, reported its first IFRS net profit since listing.

A 2014 paper by Vinyals and colleagues that trained a single deep network to turn a photograph into a sentence, reporting a BLEU-1 score of 59 on Pascal against 25 for the previous state of the art.

WaveNet is a deep neural network that generates raw audio waveforms one sample at a time, predicting each sample from all the ones before it. Used for text-to-speech, it was rated by listeners as more natural than the systems it was compared with.

Runway is a New York-based AI video company founded in 2018. It co-released Stable Diffusion in 2022, has built a series of video models from Gen-1 to Gen-4.5, and in February 2026 raised $315 million at a $5.3 billion valuation.

An image-text model from OpenAI that learns to match pictures with their captions. Trained on about 400 million image-caption pairs, it can classify images by comparing them against written prompts, with no task-specific training.

CLIP is an OpenAI model trained on 400 million image-text pairs to map pictures and sentences into one shared space, so it can classify images zero-shot from a written description.

DALL·E is OpenAI's text-to-image model unveiled on 5 January 2021: a 12-billion-parameter transformer trained on 250 million text-image pairs that generates a picture token by token from a caption.

A 2021 OpenAI paper describing a single 12-billion-parameter autoregressive transformer trained on 250 million image-text pairs, which generates images from text and, evaluated zero-shot, is competitive with domain-specific models.

Adept is an AI company founded in 2022 that set out to train large multimodal models and turn them into agents, open-sourced the Fuyu-8B model in 2023, and shifted focus in 2024 when its co-founders joined Amazon.

Imagen is a text-to-image diffusion model that uses a large pretrained language model to encode prompts; it reached a zero-shot FID of 7.27 on COCO and introduced the DrawBench benchmark.

Whisper is OpenAI's open-source speech recognition system, released in September 2022 and trained on 680,000 hours of audio, which transcribes speech in many languages and translates it into English.

LAION-5B is an openly available dataset of 5.85 billion CLIP-filtered image-text pairs, released so that researchers outside a handful of labs could study and reproduce large vision-language models.

Microsoft Copilot is the company's AI chatbot and assistant, which started in 2023 as Bing Chat and became a single consumer and business AI brand that now includes in-house models.

Runway Gen-2 is a multimodal video generation system announced in March 2023 that turns text, images or video clips into new video, and was among the first commercially available text-to-video models.

StepFun (阶跃星辰) is a Shanghai AI company founded in April 2023 by former Microsoft staff. It builds multimodal Step-series models and has open-sourced several of them.

SenseNova is SenseTime's family of foundation models, unveiled on 10 April 2023 with a chatbot, an image generator and an avatar video generator, first offered to government and corporate clients rather than the public.

WebArena is a reproducible environment of four fully functional websites — e-commerce, a social forum, collaborative software development and content management — built for testing language-guided web agents. Its best GPT-4 agent completed 14.41% of tasks; humans completed 78.24%.[^1]

Doubao is ByteDance's AI chatbot, opened for invitation testing in August 2023 and later offering text, image and voice generation, AI search and video generation.[^1][^2]

GPT-4V is a version of GPT-4 that accepts image input alongside text, documented in OpenAI's system card of 25 September 2023 and released to developers through preview API models; image understanding later became native in GPT-4o.

Meta AI is Meta's conversational assistant, introduced in US beta in September 2023 on WhatsApp, Messenger and Instagram. It was rebuilt on Llama 3 in 2024 and on Muse Spark in 2026.

Fuyu-8B is a decoder-only multimodal model that Adept open-sourced in October 2023; it drops the image encoder, accepts arbitrary resolutions, and reads screens and charts.

GPT-4 Turbo is a cheaper, more capable version of OpenAI's GPT-4, announced on 6 November 2023 with a 128,000-token context window and lower API prices.

GAIA is a benchmark of real-world questions that are easy for humans but hard for AI assistants, testing reasoning, multimodality, web browsing, and tool use. It compares AI progress against average human performance.

A mid-size multimodal model in Google's Gemini 1.5 generation, announced in February 2024, known for a context window that reached one million tokens and later two million.

Sora was OpenAI's text-to-video model, previewed in February 2024 and released that December, able to generate clips up to a minute long. It was succeeded by Sora 2 and discontinued in 2026.

Claude 3 Sonnet was the middle model of Anthropic's Claude 3 family, announced on 4 March 2024. It powered the free tier of claude.ai, offered a 200K-token context window, and was retired on 21 July 2025.

Claude 3 Haiku is the smallest and fastest model in Anthropic's Claude 3 family, announced in March 2024 with low per-token pricing, a 200K-token context window and image input; retired in April 2026.

Microsoft AI is the consumer AI organisation Microsoft formed in March 2024 to advance Copilot and its other AI products and research, led by Mustafa Suleyman. It develops the MAI family of in-house models.

OSWorld is a benchmark and environment for testing multimodal agents on 369 real computer tasks in Ubuntu, Windows and macOS; humans complete over 72.36% of the tasks, while the best model reaches only 12.24%.

Phi-3 is Microsoft's family of small language models, launched in April 2024 with Phi-3-mini, a 3.8-billion-parameter model small enough to run on a phone.

SenseNova 5.0 is SenseTime's large language model released in April 2024, built on a mixture-of-experts architecture with a roughly 200K-token context window, which the company positioned as matching or surpassing GPT-4 Turbo.

Veo is a generative video model from Google DeepMind, announced at Google I/O on 14 May 2024. Google described it as its most capable video model, able to produce 1080p clips longer than a minute and later succeeded by Veo 2.

Kling (可灵) 1.0 is Kuaishou's video generation model. Text-to-video opened for testing on 6 June 2024, and Kling could generate videos of up to two minutes at up to 1080p.

OpenVLA is a 7-billion-parameter open-source vision-language-action model trained on 970,000 real robot demonstrations, released with checkpoints and code so that other groups can fine-tune and run it on their own robots.

Runway's 2024 video generation model, trained jointly on video and images, with finer temporal control such as keyframing; later extended with an API, video-to-video and Act-One performance capture.

A small, low-cost language model OpenAI released on 18 July 2024, priced far below GPT-4o and reported at 82% on MMLU; it replaced GPT-3.5 Turbo in ChatGPT on launch day.

SAM 2 is a Meta FAIR foundation model that segments objects in images and videos from simple prompts such as points, boxes or masks, released together with a large video segmentation dataset and training code.

xAI's Grok-2 and Grok-2 mini, released in August 2024 as a beta for X subscribers, added image input, posted competitive benchmark scores and later had their weights published.

Llama 3.2, announced on 25 September 2024, added vision models at 11 billion and 90 billion parameters and small text-only models at 1 billion and 3 billion for edge and mobile devices.

Computer use is an Anthropic capability, released in public beta in October 2024, that lets an AI model operate a computer the way a person does — reading the screen, moving a cursor, clicking and typing — so it can drive ordinary software built for humans.

HunyuanVideo is Tencent's open-source text-to-video generation model, with over 13 billion parameters, released in December 2024. Running it locally needs at least 60GB of GPU memory.

Amazon Nova is Amazon's family of foundation models, introduced on 3 December 2024 and offered exclusively in Amazon Bedrock. It spans text understanding models from Micro to Pro, plus Canvas for images and Reel for video, and was succeeded by Nova 2 in December 2025.

Gemini 2.0 Flash is the first model of Google's Gemini 2.0 generation, released as an experimental model in December 2024 with multimodal output and native tool calling, and made generally available in February 2025.

MiniMax-01 is an open-weight model series released in January 2025. It includes the language model MiniMax-Text-01 and the vision-language model MiniMax-VL-01, and uses a hybrid attention design that supports very long context.

Kimi k1.5 is a multimodal reasoning model released by Moonshot AI on 20 January 2025, which the company said matched OpenAI's o1 in mathematics, coding and multimodal reasoning and which was trained with reinforcement learning rather than tree search or process reward models.

Doubao-1.5-pro, released by ByteDance on 22 January 2025, is a closed multimodal model built on a sparse mixture-of-experts design. It was offered through Volcano Engine at low API prices.

Humanity's Last Exam (HLE) is a 2,500-question, multi-modal benchmark spanning dozens of academic subjects, built by experts worldwide so that frontier language models can still be measured after saturating older tests.

Grok 3 and the cheaper Grok 3 mini are xAI's reasoning models from February 2025. Grok 3 added a Think mode, a 1-million-token context window and the DeepSearch research agent.

Helix is Figure's vision-language-action model for humanoid control, pairing a 7-billion-parameter vision-language model with a fast 80-million-parameter action policy, trained on about 500 hours of teleoperated data and run entirely on onboard GPUs.

A general-purpose embodied foundation model released by AgiBot on 10 March 2025, built on a Vision-Language-Latent-Action architecture. In the company's tests across five tasks it raised average success rate from 46% to 78%.

Gemini Robotics is a vision-language-action model from Google DeepMind, built on Gemini 2.0, that outputs physical actions to control robots. It was announced on 12 March 2025 alongside Gemini Robotics-ER.

Gemma 3 is Google's third generation of open Gemma models, released on 12 March 2025 in 1B to 27B sizes, with a 128,000-token context window, image input on the larger variants, and support for more than 140 languages at the pretrained level.

ERNIE X1 is Baidu's deep-thinking reasoning model released on 16 March 2025 alongside ERNIE 4.5, notable for tool use, multimodal input and low pricing, and later followed by X1 Turbo and X1.1.

ERNIE 4.5 is Baidu's native multimodal foundation model, released on 16 March 2025 and handling text, images, audio and video. In June 2025 Baidu open-sourced ten models from the family under the Apache License 2.0.

OpenAI's native image generator built into GPT-4o, released in ChatGPT on 25 March 2025 and later offered in the API as gpt-image-1. It replaced DALL·E 3 and was noted for accurate text rendering and close prompt following.

Runway Gen-4 is a video generation model released on 31 March 2025 that keeps characters, locations and objects consistent across shots, working from reference images and text instructions rather than per-project training.

Llama 4 Maverick is a mixture-of-experts model Meta released alongside Scout on 5 April 2025, with 400 billion total parameters and a 1-million-token context. Its ranking on LMArena was later questioned.

Llama 4 Scout is a mixture-of-experts model Meta released on 5 April 2025. It has 109 billion total parameters, 17 billion of them active for each token, and a context window of 10 million tokens.

SenseTime's SenseNova V6 is a native mixture-of-experts multimodal foundation model with over 600 billion parameters, released on 10 April 2025 in general, reasoning and real-time interactive variants.

Kling 2.0 is Kuaishou's AI video generator, launched globally in April 2025 and billed by the company as the world's most powerful, with a multimodal editing feature and the Multi-modal Visual Language concept.

π0.5 is a vision-language-action model from Physical Intelligence, released in April 2025, built to generalize to unfamiliar homes and trained by co-training on many kinds of robot and web data.

Veo 3 is Google's video generation model, announced at Google I/O on 20 May 2025, that generates video with native audio, including dialogue with lip syncing; its outputs carry SynthID watermarks.

ByteDance Seed's general-purpose model series, announced on 11 June 2025 as Doubao Large Model 1.6, combining multimodal input, a 256K context window and a mode that decides per prompt whether to reason before answering.

Seedance 1.0 is a video-generation model from ByteDance Seed, announced in June 2025. One model handles text-to-video and image-to-video, produces 1080p multi-shot clips, and ranked first on two Artificial Analysis leaderboards.

Grok 4 and Grok 4 Heavy are xAI's models from July 2025. Grok 4 has a 256,000-token context and was trained with reinforcement learning at pretraining scale, while Heavy runs parallel test-time compute.

Step 3 is StepFun's open multimodal reasoning model, with 321 billion total parameters and 38 billion active per token. It was released in July 2025 and uses Apache 2.0.

Seedream 4.0 is ByteDance Seed's image model released in September 2025, which puts text-to-image generation, image editing and multi-image composition into a single architecture and can output images from 1K up to 4K resolution.

Qwen3-VL is Alibaba's open vision-language model family, released in September 2025 under Apache 2.0. Its 235B flagship handles long video, GUI operation and 2D and 3D grounding.

An open-source image generation model from Tencent that handles image understanding and generation inside a single autoregressive mixture-of-experts framework; its code, weights and technical report were released in September 2025.

ERNIE 5.0 is Baidu's natively omni-modal foundation model, unveiled in November 2025 and described as having 2.4 trillion parameters. It is proprietary and was made available to enterprises and consumers in January 2026.

Kling O1 is a video model that Kuaishou announced on 1 December 2025. The company described it as the first unified multimodal video model: text, image, video and subject inputs go into one engine used both for generation and for editing.

Mistral AI's flagship model of the Mistral 3 family, released in December 2025: a sparse mixture-of-experts model with 675B total parameters, open weights under Apache 2.0, and image understanding.

Kimi K2.5 is a natively multimodal open-weight model from Moonshot AI, released on 27 January 2026. It has 1 trillion total parameters, a 256K-token context and an Agent Swarm feature for parallel sub-agents.

Kling 3.0 is Kuaishou's video and image model series, launched on 5 February 2026 as four models. It generates up to 15 seconds of video with simultaneous audio, adds multi-shot generation and supports native 4K output.

Seed2.0 is ByteDance's February 2026 Doubao model series, offered in three agent-oriented sizes with low token pricing and reported benchmark results.

Qwen3.5 is a natively multimodal model family from Alibaba released in February 2026. Its open flagship, Qwen3.5-397B-A17B, has 397B total parameters and 17B active per token, under Apache 2.0.

Gemini 3.1 Pro is Google's preview upgrade to Gemini 3 Pro, released on 19 February 2026, scoring a verified 77.1% on ARC-AGI-2 and shipped through the Gemini API, the Gemini app and NotebookLM.

Reka Edge is a 7-billion-parameter vision-language model from Reka AI, released on 11 March 2026, built to take images, video and text and to run locally on laptops, phones and small edge devices.

Gemma 4 is Google's fourth generation of open-weight models, released in April 2026 and built on the same research and technology as Gemini 3, ranging from phone-sized E2B and E4B to a 26B mixture-of-experts and a 31B dense model.

Muse Spark is the first model from Meta Superintelligence Labs, released on 8 April 2026 as a closed model that powers the Meta AI app and meta.ai. An open-weight sibling followed later.

EXAONE 4.5 is LG AI Research's first open-weight vision language model: a 33-billion-parameter model with a 1.2-billion-parameter vision encoder, trained on documents and text, with a 256K-token context window, released in April 2026.

π0.7 is a general-purpose robot manipulation model from Physical Intelligence that matches fine-tuned specialists on a wide range of dexterous tasks while following new language commands and recombining learned skills in unfamiliar settings.

The Qwen3.5-Omni Technical Report documents Alibaba's Qwen3.5-Omni, a hundreds-of-billions-parameter model that understands long audio and video and generates speech across 10 languages, reporting state-of-the-art results on 215 audio and audio-visual benchmarks.

Gemini Omni Flash is a Google DeepMind generative model that makes and edits video from text, image, audio and video prompts. Video creation arrived in the Gemini app on 19 May 2026, and the model later became the recommended replacement for Google's Veo 3.1 API previews.

Command A+ is Cohere's open-source mixture-of-experts language model, released on 20 May 2026 under the Apache 2.0 licence with 218 billion total and 25 billion active parameters, aimed at sovereign and critical-infrastructure deployments.

A mixture-of-experts language model released by MiniMax in June 2026, with about 428B total parameters, a 1M-token context window and a new sparse attention scheme built for long-context and agent-style work. [^1]

ByteDance Seed's Seed2.1, released in June 2026 in Pro and Turbo versions, is a Doubao-family model aimed at real productivity work, with reported gains in agent ability, coding, knowledge, reasoning and multimodal understanding.

Vidu S1 is Shengshu's real-time interactive video generation model, released in July 2026. It takes spoken commands as control input, runs on consumer GPUs at 540p and up to 42 frames per second, and keeps generating without drift.

Inkling is the first model released by Thinking Machines Lab: a 975B-parameter mixture-of-experts model with open weights, native text, image and audio input, and a context window of up to 1M tokens.

FLUX 3 is a multimodal model family from Black Forest Labs, announced on 23 July 2026, covering images, video, audio and action prediction. Its video model became generally available on 4 August 2026.

Step 5 Preview is a preview release of StepFun's flagship foundation model: a sparse mixture-of-experts system with 600 billion total parameters, a 1-million-token context window and published weights, released in September 2026.

Vidu S2 is a two-part video generation system: S2-Avatar renders interactive digital characters at 720p in real time, and S2-Editing transforms live video streams. Both also produce stereo views for immersive displays.

DeepSeek-V4.1-Flash, released on 10 September 2026 under the API name deepseek-flash, is a 552-billion-parameter multimodal mixture-of-experts model with a causal encoder-decoder design and a 1M-token context.

PixVerse R2 is a real-time world model announced on 22 September 2026, an upgrade to R1 that keeps a navigable world generating while the user moves inside it and holds state across a session.

Mistral Large 4 is Mistral AI's flagship open-weight model, with about 1 trillion total parameters, a 1M-token context window and native multimodality. It was previewed on 6 October 2026, and an improved version's weights followed on 29 October 2026.

Muse Spark Open is Meta's open-weight large language model, announced on 15 October 2026. With 816 billion total parameters, it is built for coding and agentic work, and its weights can be downloaded, unlike the closed Muse Spark that started the family.

Kling 4.0 is a text-to-video generation model released to all users in October 2026, adding longer shots, multi-image references and more convincing physical motion, alongside a faster Flash variant built for high-volume work.

LFM3-Omni-2B is Liquid AI's first omni-modal model in the LFM3 family: one 2.17-billion-parameter model that reads text, images, video and audio and answers in text or speech.

Alec Radford is a machine learning researcher who worked at OpenAI from 2016 to 2024. He was first author of the GPT-2, CLIP and Whisper papers, and lead author of the 2018 paper that introduced the GPT approach.

Liquid AI is an MIT CSAIL spin-out that calls itself an efficiency-first foundation model company, arguing that models should run on local devices; its LFM3-Omni-2B is an omni-modal model published in October 2026.

Reka AI is an AI research lab building models and infrastructure for the physical AI era. Its releases include reka-flash-3, a 21-billion-parameter reasoning model, and Reka Edge, a 7-billion-parameter vision-language model designed to run on devices at the edge.

Raj Reddy is an Indian-born computer scientist who built the first system capable of continuous speech recognition and led major computing institutions at Carnegie Mellon University. He shared the 1994 ACM Turing Award with Ed Feigenbaum.

LG AI Research is the organisation behind EXAONE, a family of large language models it documents in technical reports and releases under its own EXAONE licence.

Physical Intelligence is a company that develops foundation models and learning algorithms for robots, aiming to bring general-purpose AI into the physical world. It publishes generalist robot policies such as π0 and π0.5, along with open weights and code.

Cristóbal Valenzuela is a Chilean-born technologist and the co-founder and CEO of Runway, the New York company behind the Gen series of AI video models.

Skype Translator is a Microsoft feature that automatically translates speech and instant-message conversations. A public preview opened on 15 December 2014 with Spanish and English speech translation and more than 40 messaging languages.[^1]

Luma AI is a generative AI company founded in 2021 in Redwood City, California, known for the Dream Machine video app, the Ray3 video model and the Luma Agents platform, with more than $1.1 billion raised.

Flamingo is a DeepMind visual language model announced in April 2022. It takes prompts that mix images, videos and text and replies in language, learning new tasks from only a few examples.

Gato is a generalist agent from DeepMind, announced on 12 May 2022. A single network with one set of weights handles text, images, Atari games and robot control, aiming at multi-task, multi-embodiment learning.

Runway Gen-1 is a video-to-video diffusion model released in February 2023 that restyles existing footage from text or image prompts while keeping the source video's structure. It was among the first commercially available video-generation models.

RT-2 is a vision-language-action model from Google DeepMind, announced on 28 July 2023, that turns images and text instructions into robot actions written as text tokens. In over 6,000 trials it reached 62% on unseen scenarios, up from 32% for RT-1.

Qwen-VL is a family of vision-language models released in August 2023 that builds on the Qwen language model to caption images, answer questions about them, locate objects and read text, with open Base and Chat versions.

Stable Video Diffusion is Stability AI's image-to-video model, released on 21 November 2023 as a research preview that turns a single still image into a clip of a few seconds.

Yi-VL-34B and Yi-VL-6B are open vision-language models from 01.AI, released on 23 January 2024. They read images and text together and ranked first among open models on the MMMU and CMMMU benchmarks.

SenseNova 4.0, announced on 8 February 2024, upgraded SenseTime's model family with 4K, 32K and 128K context versions, specialised models and a first multimodal tool-calling API.

Step-1V is a roughly 100-billion-parameter multimodal model from StepFun, unveiled in March 2024, that reads text, data and charts in images and supports video understanding; it ranked first on OpenCompass at launch and was succeeded by Step-1.5V.

Grok-1.5 is xAI's upgrade to Grok-1, with better reasoning and a 128,000-token context window. Its companion Grok-1.5V was xAI's first multimodal model. Neither model's weights were released publicly.

Gemini 1.5 Flash is a lightweight, fast model in Google's Gemini 1.5 family, announced on 14 May 2024 and trained by distillation from 1.5 Pro, with a 1-million-token context window.

Project Astra is a Google DeepMind research prototype of a universal AI assistant that watches and listens while it responds, shown at Google I/O in May 2024 and later rebuilt on Gemini 2.0.

Dream Machine is Luma AI's text- and image-to-video generator, released to the public on 12 June 2024. It later became the home of Luma's Ray3 and Ray3.14 video models.

SenseTime's July 2024 model family, unveiled at WAIC, adding a real-time multimodal model, lightweight and edge versions, and migration incentives aimed at OpenAI customers.

Phi-3.5 is Microsoft's August 2024 update to its small Phi models, released in three versions — mini, vision and the line's first mixture-of-experts model — under the MIT licence.

Qwen2-VL is Alibaba's vision-language model released on 29 August 2024. It understands videos longer than 20 minutes, reads text in many languages, and can act as an agent on phones and robots.

Kling 1.5 is a September 2024 version of Kuaishou's video generation model, adding 1080p output, a Motion Brush, and quality improvements before being succeeded by Kling 1.6 in December 2024.

Pixtral 12B is Mistral AI's first multimodal model, pairing a 12B language decoder with a 400M vision encoder. Released openly under Apache 2.0 in September 2024, it reads images at native resolution and was retired at the end of 2025.

Molmo is a family of open vision-language models released by Ai2 on 25 September 2024 that can point at objects in images, with weights, code, data and evaluations published openly.

A large language model released by iFlytek in October 2024 and upgraded in January 2025, trained and served on the domestic Feixing No.1 computing platform; iFlytek reported it leading several Chinese and English benchmarks against GPT-4 Turbo and GPT-4o.

Pixtral Large is Mistral AI's open-weight multimodal model announced on 18 November 2024, built on Mistral Large 2, combining a 123B-parameter decoder with a 1B vision encoder and a 128K-token context window. It was retired in May 2026.

Kling 1.6 is a December 2024 upgrade to Kuaishou's Kling video generation model, with better handling of motion and camera instructions in prompts and improved visual quality; it was succeeded by Kling 2.0 in April 2025.

A family of open mixture-of-experts vision-language models from DeepSeek, released in three sizes from 1.0B to 4.5B activated parameters for document, chart and OCR understanding, succeeding DeepSeek-VL.

Veo 2 is a video-generation model Google announced on 16 December 2024, emphasising real-world physics and human movement, with outputs up to 4K that can extend to minutes.

Megrez-3B-Omni is an open-source model from December 2024 that understands images, text and audio on end-side devices, pairing a 2.29B-parameter language backbone with SigLip and Whisper encoders for about 4B parameters in total.

Qwen2.5-VL is Alibaba's vision-language model family released in January 2025 in 3B, 7B, 32B and 72B sizes. It reads charts and documents, localises objects, works as a visual agent and understands videos over an hour long.

Janus-Pro is DeepSeek's January 2025 multimodal model that handles both image understanding and text-to-image generation in one system, released in 1B and 7B sizes as a stronger successor to Janus.

An experimental Google model released in February 2025 as part of the Gemini 2.0 family, described by Google as its best yet for coding and complex prompts, with a 2-million-token context window and tool use.

Step-Audio is an open family of speech-interaction models released by StepFun on 17 February 2025. It includes Step-Audio-Chat, with 130 billion parameters, and Step-Audio-TTS-3B.

Phi-4-mini is a 3.8B-parameter text model and Phi-4-multimodal a 5.6B model that adds images and speech, both released by Microsoft in February 2025 as the small, efficiency-focused members of the Phi-4 family.

Cohere's first vision-language model, released in March 2025 as open-weight 8B and 32B versions. It handles images and text in 23 languages and, in Cohere's comparisons, beats several much larger models, under a non-commercial licence.

Mistral Small 3.1 is an open-weight model released by Mistral AI in March 2025 that adds image understanding and a 128K-token context window to the Small line, runs on a single consumer GPU, and was later superseded by newer Small releases.

Qwen2.5-Omni is Alibaba's end-to-end multimodal model that takes text, images, audio and video as input and answers with text and natural speech. The 7B version was released on 26 March 2025.

Kimi-VL is an open-source vision-language model from Moonshot AI, released in April 2025. It pairs a 16B-parameter mixture-of-experts language model with a native-resolution vision encoder, so only about 3B parameters are active at a time.

Llama 4 Behemoth is a very large Meta model previewed on 5 April 2025 as the teacher for the Llama 4 family. It was never publicly released.

A speech-to-speech foundation model in Amazon's Nova family, announced on 8 April 2025, which handles recognition, understanding and speech generation in a single model rather than a chain of separate ones.

ERNIE 4.5 Turbo is Baidu's multimodal large language model released on 25 April 2025, presented as a faster successor to ERNIE 4.5 with lower hallucination rates and sharply cut API prices.

Amazon Nova Premier is the most capable model of the first Nova generation, available in Amazon Bedrock from 30 April 2025. It handles text, images and video with a 1-million-token context window, and also serves as a teacher model for smaller Nova models.

Kling 2.1 is a Kuaishou video generation model released at the end of May 2025 in three tiers — 720p Standard, 1080p High Quality and 1080p Master — with claimed gains in motion, physical simulation and prompt responsiveness. It was superseded by Kling 2.5 Turbo.[^1][^2][^3]

Apple Foundation Models (2025) are the second generation of Apple's models. They add image understanding and a mixture-of-experts server design, and they open an on-device model to developers through the Foundation Models framework.

Gemma 3n is a mobile-first open model in Google's Gemma family, released in June 2025 in two sizes that run in roughly 2–3 GB of memory and natively accept image, audio, video and text input.

SenseNova V6.5 is SenseTime's enhanced multimodal model, announced in July 2025, built around interleaved image-text reasoning and powering an agent upgrade of the Raccoon assistant.

Voxtral is Mistral AI's family of speech-understanding models, released on 15 July 2025 in 24B and 3B sizes under Apache 2.0; it transcribes long audio, answers questions about it, and turns spoken requests into function calls.

Step-Audio 2 is a speech model from StepFun that can understand audio and hold spoken conversations. Its mini versions were released under the Apache 2.0 license in 2025.

Runway Aleph is a video model released in July 2025 that edits an input video by adding, removing or transforming objects, generating a new angle of a scene, or changing style and lighting, across many visual tasks.

Command A Vision is Cohere's first vision-language model in the Command family, released in July 2025. It pairs a 112-billion-parameter Command A language model with a SigLIP2 image encoder to read images, charts and documents and answer in text.

A vision-language model released by Zhipu on 11 August 2025, built on GLM-4.5-Air and covering image, video, long-document and GUI tasks; Zhipu reports state-of-the-art results at its scale on 42 benchmarks.

MAI-Voice-1 is Microsoft AI's speech generation model, announced in August 2025 as the first model in the MAI family and built to produce expressive, high-fidelity audio quickly.

Qwen3-Omni is an open-weight multimodal model released by the Qwen team in September 2025 under Apache 2.0, able to take in text, audio and video and to answer in text or speech across many languages.

Kling 2.5 Turbo is a Kuaishou video generation model released at the end of September 2025, which Artificial Analysis ranked No. 1 worldwide for text-to-video and image-to-video ten days after launch, and which cut creators' per-video generation cost by nearly 30%.

Ray3 is a video generation model released by Luma AI on 18 September 2025 and described by the company as the first reasoning video model, with HDR output, native 1080p and a neural upscaler to 4K.

A pair of robotics models announced by Google DeepMind in September 2025: Gemini Robotics 1.5, a vision-language-action model that reasons in natural language before it moves, and Gemini Robotics-ER 1.5, an embodied-reasoning planner that can call tools.

Veo 3.1 is an October 2025 update to Google's Veo 3 video model, adding audio across Flow's features plus new editing tools; its API previews were retired in October 2026.

DeepSeek-OCR is an open-source model of about 3B parameters that renders long text as 2D images and reads it back from a small number of vision tokens, testing whether optical compression can stand in for very long text contexts.

Baidu's open-weight multimodal reasoning model from November 2025: 28B total parameters with 3B active per token, "Thinking with Images" zoom-and-tool reasoning, and Apache 2.0 weights.

Runway Gen-4.5 is a video generation model released in December 2025 that topped the Artificial Analysis text-to-video leaderboard and later gained native audio, multi-shot editing and a role as the base for Runway's GWM-1 world model.

Amazon Nova 2 is the second generation of Amazon's Nova models, released at AWS re:Invent in December 2025. Its first member, Nova 2 Lite, is a fast reasoning model available in Amazon Bedrock with a 1-million-token context.

Three small dense models from Mistral AI — 3B, 8B and 14B — released in December 2025 under Apache 2.0, each in base, instruct and reasoning variants with image understanding.

Kling 2.6 is a Kuaishou video generation model released in December 2025 that produces visuals, voiceover, sound effects and ambient audio in a single pass, with clips capped at 10 seconds.

Molmo 2 is the second generation of Ai2's open vision-language models, released on 11 December 2025, which extends the Molmo line from still images to video with pointing, tracking and timestamped answers.

GWM-1 is Runway's first general world model: an autoregressive system that generates 720p video frame by frame in real time and can be steered with camera pose, robot commands or audio.

An audio-video generation model from Volcano Engine, announced on 18 December 2025, that produces picture, voices, music and ambient sound together and keeps them in millisecond-level sync.

Doubao Large Model 1.8 (Doubao-Seed-1.8) is ByteDance's Volcano Engine flagship released on 18 December 2025. It is built for multimodal agent tasks, with stronger tool calling and computer-operation abilities.

Phi-4-reasoning-vision-15B is Microsoft's 15-billion-parameter open-weight vision-language model, released on 4 March 2026 under the MIT licence, built to switch between step-by-step reasoning for math and science and direct answers for image perception tasks.

Mistral Small 4 is an open-weight mixture-of-experts model released in March 2026 that folds instruction-following, step-by-step reasoning and agentic coding into one model, with a switch that controls how much reasoning it does.

A speech-to-text model from Microsoft AI released in April 2026 that transcribes 25 languages, runs Copilot's Voice Mode, and starts at $0.36 per hour of audio.

SenseNova U1 is SenseTime's open-weight multimodal model family, first released in April 2026 under Apache 2.0, which understands and generates images without a separate visual encoder or VAE.

Step 3.7 Flash is a StepFun open-weights model released on 29 May 2026. It adds native image input to Step 3.5 Flash, supports a 256K context, and is licensed under Apache 2.0.

Apple Foundation Models 3 (AFM 3), published on 8 June 2026, is a family of five models: two on-device models, including a sparse 20B multimodal model, and three cloud models, one of which is optimised for NVIDIA GPUs.

GPT-Live-1 is OpenAI's full-duplex voice model family, launched 8 July 2026, which lets ChatGPT listen and speak at the same time and replaced the earlier chained speech pipeline in ChatGPT Voice.

Seedance 2.5, released by ByteDance in July 2026, is an audio-video joint generation model that makes clips of up to 30 seconds per pass, accepts many multimodal references, and supports editing parts of a scene without regenerating the whole clip.

SenseNova U1.5 is SenseTime's open-weight, 8-billion-parameter model that handles image understanding, reasoning and native 4K generation in a single network, without separate vision encoders or VAEs. It is released under the Apache License 2.0.

MiniMax-M3.1-Flash is a sparse mixture-of-experts model from MiniMax with about 428B total parameters, 23B activated per token, text, image and video input and up to 1M tokens of context, released on Hugging Face in October 2026.

PLaMo 3 Translate is a family of translation-specialised language models from Preferred Networks, released in four sizes from 0.8B to 31B parameters, covering 53 languages and shipping in several quantised formats.

Nemotron 3.5 Transcribe is Nvidia's free, open-weight speech transcription model, released on 28 October 2026. It handles recordings up to an hour long, transcribes up to 40 languages and can separate up to eight speakers.

SenseTime's multimodal agent model, announced in Hong Kong in November 2026, aimed at office work and design generation: long-term memory, task execution, and output ranging from PPTX decks to Blender 3D scenes.

Anastasis Germanidis is a co-founder of Runway, the generative AI video company he started in 2018, and a co-author of the research behind Gen-1, Runway's video-to-video generation model.

Jiang Daxin is founder and CEO of the Chinese AI company StepFun and a former Microsoft global vice president. He argues that scaling laws set the ceiling on model intelligence and that unifying multimodal understanding and generation is the path to AGI.

Yuewen is StepFun's AI assistant app, fully opened in March 2024, which reads text, data and charts in images and handles content creation, reasoning and analysis. In December 2024 its team absorbed the Maopaoya team, and the product was later renamed Step AI.

NVLM is a family of multimodal large language models from NVIDIA, described in a paper posted to arXiv in September 2024; its flagship NVLM-D-72B was later released as open weights for non-commercial use.