Google brings photo support in AI Mode: What is it and how to use

Google is enhancing its AI-powered search experience, AI Mode, by integrating multimodal capabilities of its Lens technology. This upgrade will allow users to search using images alongside text within AI Mode, delivering a richer and more contextually aware search experience, the company has said.

Google claims that it has received positive feedback from early adopters of AI Mode, who praised its clean interface, rapid response times, and ability to handle complex and nuanced queries. It is essentially an experimental, opt-in feature in Google Search Labs that provides a more AI-like interface for searching, allowing users to ask new questions and explore topics with AI-powered responses, currently available to Google One AI Premium subscribers.

Google reported that “people are using AI Mode for help with exploratory and open-ended questions, along with more complicated tasks — like comparing two products, exploring how-tos and planning a trip.”

Google brings visual intelligence of Lens directly into AI Mode
This new multimodal understanding empowers users to snap a photo or upload an image and pose questions about its content. AI Mode will then generate comprehensive responses, complete with relevant links for further exploration.

This integration leverages a custom-built version of Google's Gemini model , allowing AI Mode to not only identify objects within an image but also understand the entire scene, including the relationships between objects and their specific attributes like materials, colours, shapes and arrangements.

Google emphasized that this capability builds upon years of work in visual search, taking it a step further. Lens precisely identifies individual elements within an image, and AI Mode then employs a "query fan-out technique" to issue multiple queries about the image as a whole and its constituent parts.

This approach allows AI Mode to access a broader and deeper range of information compared to traditional text-based searches, resulting in more nuanced and contextually relevant answers.