Google pauses Gemini's people image generation after historically inaccurate images

Background

In February 2024 the Gemini conversational app could produce pictures as well as text. The image feature had gone live about three weeks earlier and was built on Google's Imagen 2 model ^[1]. Like other text-to-image systems, it takes a written prompt and produces a picture that matches the description: the model has learned from large collections of images and their captions what objects, scenes and styles tend to look like together. It has no independent knowledge of history — it fills in a scene from the words in the prompt — so the people it draws depend heavily on how the model was trained and tuned.

Part of that tuning was deliberate. Google tuned the feature so that the people it generated would show a range of people ^[1]. That is a common goal for a consumer product with a global audience: when many users ask for a picture of "a doctor" or "a family", returning the same kind of face every time is not a good answer. At the time, the feature was not available in the UK and Europe ^[2].

What happened

Users began sharing examples of historically inaccurate images, including depictions of Second World War German soldiers and Vikings as people of colour ^[2]. On 22 February 2024, Google paused Gemini's image generation of people while it worked to improve the accuracy of its responses ^[1][2].

Google's explanation

Google said two things had gone wrong. The tuning meant to show a range of people failed to account for cases that should clearly not show a range ^[1]. A prompt naming a specific historical group is a request for a particular kind of depiction, and the diversity setting was being applied where it did not belong. Separately, over time the model had become more cautious than intended and refused to answer certain prompts entirely ^[1]. That is a different kind of failure: instead of producing a wrong picture, the system produced none at all, so the user's request was still not answered correctly.

Jack Krawczyk, a senior director on Google's Gemini team, said the tool was designed to reflect Google's global user base but was "missing the mark" on historical depictions ^[2].

In a blog post on 23 February 2024, Google acknowledged that some of the images generated were "inaccurate or even offensive" and apologized ^[1].

Aftermath

Google said it had turned the image generation of people off and would work to improve it significantly, including extensive testing, before turning it back on ^[1]. The episode became a widely cited illustration of a general problem in generative AI: a system's defaults are chosen by people, and a default that is helpful in one context can be plainly wrong in another. It is also not a simple thing to patch, because the behaviour comes out of the training data together with the tuning layered on top of it. The case drew attention to the narrow line between a model that refuses too little and one that refuses too much.