Project Astra

What it is

Project Astra is a research prototype of a universal AI assistant from Google DeepMind. Google presented it at Google I/O on 14 May 2024.^[1] The company described it as an advanced seeing and talking responsive agent, that is, a system built to take in what a person shows it and says to it and answer in the moment.^[1]

It is a prototype rather than a finished product. Google DeepMind's Project Astra page describes it as a research prototype that is being used and refined by a limited number of trusted testers.^[3]

How it works

Google said the prototype processes information faster by doing three things together: it continuously encodes video frames, it combines video and speech input into a timeline of events, and it caches that material so it can be recalled efficiently.^[1]

The idea is easier to grasp by contrast with an ordinary text assistant. A chat assistant receives a written question and answers it. Astra receives a live stream: camera frames arriving continuously, plus the user's speech. Encoding the frames turns pictures into a form the model can work with; merging them with speech gives one ordered record of what was seen and said and when. The cache is what makes recall cheap: instead of reprocessing everything each time a question arrives, the system can look back at the stored timeline. That is what allows a question about something shown a moment earlier to be answered without repeating the whole stream.

Astra is not limited to seeing and talking. Google said it can use Search, Gmail, Calendar and Maps, and can use interface control to complete tasks.^[3] So the prototype is meant to reach beyond conversation into carrying out actions across Google's services and on-screen controls.

From the first demo to Gemini 2.0

In December 2024, Google released an updated Astra built on Gemini 2.0.^[2] The update added multilingual conversation, the use of Google Search, Lens and Maps, up to 10 minutes of in-session memory, and testing on prototype glasses.^[2]

Session memory and hardware are closely linked in the design. Google said Astra works on Android phones and prototype glasses, with a memory that lets a conversation continue across devices.^[3] A thread of conversation started on a phone can therefore carry over to a pair of glasses, rather than starting again.

Where it sits in Google's plans

Astra works as a testbed whose parts are then moved into products. Google said it was integrating Astra capabilities such as screen sharing and video understanding into Gemini Live.^[3] It also said it was working to bring Project Astra's capabilities to Gemini Live, to new experiences in Search, and to new form factors such as glasses.^[3]

That arrangement explains why the prototype stays with testers while its features appear elsewhere. The seeing-and-talking assistant is developed under the Astra name, and pieces of it — screen sharing, video understanding, memory, use of Search and Maps — are folded into Gemini Live and other Google products as they become ready, with glasses named as one of the hardware directions being explored.^[3]