Project Astra (original) (raw)

Project Astra

Exploring breakthrough capabilities for Google products — on the way to building a universal AI assistant

We’re working to bring Project Astra’s capabilities to Gemini Live, new experiences in Search, as well as new form factors like glasses.

Some of the latest features in Gemini Live were first explored using Project Astra.

Over the past year, we’ve been integrating capabilities like screen sharing and video understanding into Gemini Live for more people to experience.

Capabilities
Blind and Low Vision
Platforms
Safety
Try Project Astra

Natural interaction

Generates content significantly faster than even our fastest model so far.

Proactive responses

Project Astra can intuitively start conversations, and respond in the moment — without interrupting or time lag.

Context aware dialogue

Ignores distractions, like background conversation and irrelevant speech.

Agent highlighting

Understands objects in context, using on-screen highlights to show you what’s important.

Tool use

Uses tools like Search, Gmail, Calendar Maps, and interface control to complete tasks for you.

Personalized reasoning

Uses deep reasoning and memory to build a rich understanding of your preferences, and tailor shopping recommendations to your needs.

Content retrieval

Retrieves content — like a PDF user manual or a recipe you’ve shared — to provide tailored help for tasks.

Multimodal memory

Integrates different types of data to perform tasks, and remembers key details from past interactions.

Supporting the blind and low-vision community.

We’re developing a version of Project Astra in collaboration with the blind and low-vision community.

Project Astra’s Visual Interpreter research prototype is able to understand objects and unfamiliar spaces.

Slide 1 of 2

It reacts to changes in the visual world, like describing what it sees as the camera view moves.

The prototype also works with Google products like Maps, Photos and Lens to accurately identify objects and the environment they appear in.

We’ve partnered with visual interpreting service Aira, to develop and refine Project Astra for blind and low-vision users. Aira users and interpreters helped us tailor the prototype for their community.

Now we’re offering a Trusted Tester program for early access to the prototype — with live, professional, human oversight. If you’d benefit from this experience, sign up for the waitlist.

Dorsey’s using Project Astra to adapt to this change in his life. Using his phone, Project Astra describes his surroundings, and accesses apps like Lens and Maps to help him explore new places.