Google Photos will introduce Ask Photos powered by Gemini this summer, providing users with the ability to search their images using voice prompts. Google shared more details about how Ask Photos works, describing it as a powerful example of how Gemini models can function as agents through memory capabilities. Examples of queries include requests like “Show me the best photo of every national park I’ve visited” and “What themes have we had for Lena’s birthday parties?” These conversational queries are passed to a Gemini agent model to determine the best retrieval-augmented generation (RAG) tool for the task.
The agent model begins by understanding the user’s intent and conducting a search through their photos using an updated vector-based retrieval system. This system is better at comprehending natural language concepts than keyword searches. A response model then analyzes the photos and videos returned by the search, leveraging Gemini’s long context window and multimodal capabilities to find the most relevant information, including visual content, text, dates, locations, and other metadata. The response model constructs a useful response based on the studied photographs and videos.
One interesting feature of Ask Photos is the ability to correct it, with the app remembering that information for future conversations. This functionality transforms Ask Photos from a simple search tool to a potential assistant, allowing users to view and manage remembered details at any time. This experimental feature, which may be linked to the rumored Ellman Project, is set to be implemented in the coming months with additional capabilities already in development.
Google Photos continues to innovate and expand its capabilities, with Ask Photos being the latest addition to the platform. Users can look forward to an enhanced search experience using voice prompts and advanced AI technology to retrieve and organize their photos effectively. The integration of Gemini models as agent systems demonstrates Google’s commitment to improving the user experience and making photo management more intuitive and efficient.
In conclusion, Ask Photos powered by Gemini represents a significant advancement in search functionality within Google Photos. By leveraging AI models to understand user queries and provide relevant responses, Ask Photos offers a more personalized and interactive experience for users looking to explore their photo collections. As Google continues to enhance its photo management tools, users can expect further improvements in usability and efficiency, ultimately making it easier to access and enjoy their memories captured in digital form.
Article Source
https://9to5google.com/2024/05/25/google-photos-ask-photos-works/