Google DeepMind’s AI cursor can understand what users are pointing at on a screen

According to DeepMind, the aim is to make AI assistance more seamless by enabling the computer to interpret both the visual and semantic context of whatever the user is highlighting

by e4m Desk
Published: May 14, 2026 4:21 PM | 2 min read

Google DeepMind has introduced an experimental AI-powered mouse cursor that can interpret user interactions on a computer screen, aiming to enhance AI interaction.
The prototype, named "Reimagining the mouse pointer for the AI era," allows users to issue voice commands by pointing at text, images, videos, or datasets, streamlining the process of engaging with AI.
Demonstrations included generating pie charts from statistics, doubling recipe ingredients, summarizing PDFs, and identifying locations in travel videos, showcasing the cursor's ability to understand visual and semantic context.
DeepMind's goal is to create a more intuitive computing experience by reducing the need for lengthy text prompts and minimizing the friction in user-AI interactions.

Google DeepMind has unveiled an experimental AI-powered mouse cursor that can understand what users are pointing at on a computer screen, offering a glimpse into a more intuitive way of interacting with artificial intelligence.

The research prototype, titled “Reimagining the mouse pointer for the AI era,” was demonstrated this week as part of the company’s ongoing work with Gemini models. The system allows users to point at text, images, videos or datasets and issue short voice commands, eliminating the need to type lengthy prompts.

According to DeepMind, the aim is to make AI assistance more seamless by enabling the computer to interpret both the visual and semantic context of whatever the user is highlighting with the cursor.

In one demonstration, a user hovered over a table of statistics and asked the AI to generate a pie chart. In another, a spoken instruction was used to automatically double the ingredients in a recipe. Additional examples showed the system summarising a PDF into bullet points suitable for an email and identifying locations from paused travel videos.

Google said current AI tools often require users to switch between applications and repeatedly provide context. By allowing Gemini to “see” what the user is focused on, the experimental cursor is designed to reduce friction and make interactions feel more natural.

DeepMind researchers described the concept as a shift away from “text-heavy prompts” toward more intuitive, context-aware computing, where the mouse pointer becomes a direct interface between users and AI.