Explaining Google-Extended

Google-Extended is a robots.txt product token that lets a website tell Google whether pages its crawlers collect may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. It is not a crawler. No request ever arrives from Google-Extended, and no server log will record one. The token exists because Google fetches the web with shared infrastructure for search and for artificial intelligence (AI), and publishers wanted a way to refuse one use without vanishing from the other.

That separation is also the source of the controversy. The token leaves Google Search untouched in both directions: blocking it costs nothing in rankings, and it does nothing to keep content out of AI Overviews or AI Mode, the generative features built into Search.

How the token works

Google’s crawler documentation lists Google-Extended alongside Googlebot, GoogleOther and its other common crawlers, with one caveat. Crawling is done with existing Google user agent strings, and the…