ChatGPT can now see, hear and speak

What OpenAI announced

On 25 September 2023 OpenAI announced that ChatGPT could see, hear and speak. The assistant gained two kinds of capability at once: it could take in images and discuss them, and it could hold a conversation by voice instead of through typed text ^[1][2]. The image understanding was powered by a model OpenAI calls GPT-4V, and a system card for that model was published the same day ^[1].

A system card is the document a lab releases alongside a model: it sets out what the model is, how it was tested, and what was done to limit harmful use. Publishing one is a way of stating in public what the lab checked before release.

It is worth separating the words in the announcement. "Seeing" means the model accepts images as input and works with them alongside text. "Hearing" and "speaking" mean a conversation can be carried on in voice rather than only in writing. Multimodal, the usual term for this, simply describes a model that does not take just one kind of input: a language-only system handles words, while a multimodal one handles more than words inside the same exchange.

The safety work behind image input

For GPT-4V, OpenAI tied the rollout of image input to the model's safety evaluations ^[1]. The system card says the safety work built on what had already been done for GPT-4, and went deeper into evaluations, preparation and mitigation for image inputs ^[3].

The reason images need their own round of work is that a picture is a different channel from text. Checks written around words do not automatically cover what can arrive inside an image, so testing, preparation before release and mitigations all have to be extended to the new input. The card's account is about that extension: not a fresh start, but more work in the same three areas, aimed at a kind of input the earlier effort had not covered.

Two days later: web access

On 27 September 2023, ChatGPT also gained web access, and with it current information ^[2]. That shifts the boundary of what the assistant can work from: instead of relying only on what it learned earlier, it can reach material about the present. The two updates arrived within days of each other, one changing the kinds of input and output, the other changing the sources a reply can draw on.

Why it mattered

The system card argues that multimodal large language models could expand the impact of language-only systems, through novel interfaces and capabilities ^[3]. This announcement is a point where that argument became a product rather than a research claim: the assistant stopped being only a text box. Voice and images are not simply extra buttons on the same thing; they change who can comfortably use the system and what can be asked of it, and they change what has to be checked before it is handed to users. That is also why the company presented the image launch together with the evaluations behind it, and why the system card spends its space on how the new input was tested, prepared for and mitigated rather than on the feature itself ^[1][3].