Google-Extended

Crawlers and access

Google-Extended is a robots.txt token that lets a site owner manage whether content Google crawls from the site may be used to train and ground Google's Gemini models, without affecting inclusion in Google Search.

Why it matters

You'll meet Google-Extended the first time someone tells you to block AI from your site. Google says it manages whether content it crawls from your site may be used for training and grounding in Gemini Apps and Vertex AI, and it's easy to misread. It doesn't decide whether you show up in regular Google results.

Knowing what it does and doesn't touch lets you make the call on purpose instead of copying a robots.txt snippet from a forum.

Example

A dentist's web designer adds a Google-Extended rule to the robots.txt file because the practice owner doesn't want her site used to train AI. The owner worries she's just vanished from Google. She hasn't. Her pages still show up in Google Search the same as before. The rule covers training and grounding in Gemini. Grounding means Gemini pulls content from the Google Search index at answer time, so the rule can also limit how her pages are used in Gemini answers, not just training. Google Search inclusion and ranking aren't affected.

What Google-Extended controls

Google-Extended is a token you put in your robots.txt file. It isn't a separate crawler. Google's crawler documentation says it has no separate HTTP request user agent string, that crawling is done with existing Google user agent strings, and that the robots.txt token is used in a control capacity.

Google describes it as a standalone product token that publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini. It also covers grounding, which Google defines as providing content from the Google Search index to the model at prompt time to improve factuality and relevancy, in Gemini Apps and Grounding with Google Search on Vertex AI. That means blocking it isn't only a training opt-out. It can also affect whether your content is used in Gemini's answers, a visibility effect in Gemini.

You use it like any other robots.txt group, with user-agent: Google-Extended followed by allow and disallow lines.

What it does not control

Google says Google-Extended doesn't impact a site's inclusion in Google Search and isn't used as a ranking signal in Google Search.

AI Overviews and AI Mode work differently. Google's page on AI features says AI is built into Search, which is why robots.txt directives for Googlebot are the control site owners have over how their sites are crawled for Search. To limit the information shown from your pages in Search, Google points to nosnippet, data-nosnippet, max-snippet, and noindex. It sends you to Google-Extended only for limiting AI training and grounding in some of Google's other systems.

So the two decisions are separate. Blocking Google-Extended is a choice about Gemini training and grounding. Whether your pages can appear in Search, and in the AI features built into it, is decided by what you allow for Googlebot.

Sources

Ready to see how AI describes your business?

Set up in ten minutes with a 14-day free trial on every self-serve plan, or talk to a specialist first.

Start Free Trial