OpenAI introduces GPT-4o

Background

Many earlier voice assistants were built as a chain of separate steps: speech was turned into text, a language model wrote a reply, and a speech synthesiser read it aloud. Each step adds delay, and the chain can discard cues such as tone of voice or pauses that the text does not carry.

GPT-4 was OpenAI's earlier flagship model, and GPT-4 Turbo was the version GPT-4o was compared against on text tasks. Against that background, the key change in GPT-4o was to bring several kinds of input and output into one model.

What happened

On 13 May 2024 OpenAI announced GPT-4o, which it called its new flagship model. The announcement said the model can reason across audio, vision and text in real time.^[1] The event was livestreamed and presented by Mira Murati, OpenAI's chief technology officer.^[4]

The same day OpenAI introduced a desktop version of ChatGPT, and developers could use GPT-4o through the API.^[4] Microsoft also made GPT-4o available in preview in the Azure OpenAI Studio playground, starting with text and vision inputs in two US regions.^[5]

Key details

OpenAI's system card describes GPT-4o as an autoregressive omni model. It accepts any combination of text, audio, image and video, and can generate text, audio and image outputs. It was trained end-to-end, so one neural network processes all inputs and outputs, rather than chaining separate models together.^[3] Autoregressive means the model produces its output one token at a time; omni refers to the many kinds of input and output.

For audio input, GPT-4o can respond in as little as 232 milliseconds, with an average of 320 milliseconds. OpenAI said this is similar to human response time in conversation.^[1]

On quality, GPT-4o matches GPT-4 Turbo on English text and code, with significant improvement on non-English text. In the API it is faster and 50% cheaper.^[1] Both comparisons are OpenAI's own, so they are best read as the company's reported results.

On access, free ChatGPT users could use GPT-4o with usage limits, and Plus subscribers received five times the message capacity of free users.^[2]

The system card describes the model's design. Individual apps built on the model may expose different inputs and outputs, so a model's capabilities and a product's features should be kept apart when reading announcements.

Impact

GPT-4o mattered for two reasons. It put speech, vision and text into one model, which brought real-time conversation closer to the pace of talking with a person. It also made a flagship model's capabilities available to free users, while opening the same model to developers and cloud customers on the same day.

These figures come from OpenAI's and Microsoft's public announcements and are the companies' own reported results. Independent evaluation of the claims was a separate matter, and readers should keep the two apart.