Source-backed crawler reference
Google-Extended Robots.txt: User-Agent, Allow & Block Guide
Google-Extended robots.txt direct answer
The exact product token is Google-Extended. Google documents it as a standalone control for whether crawled content may be used for specified Gemini model training and grounding uses. It does not control inclusion or ranking in Google Search.
# Allow Google-Extended User-agent: Google-Extended Allow: / # Block Google-Extended User-agent: Google-Extended Disallow: /
Test Google-Extended independently from Googlebot. Blocking Google-Extended does not block Googlebot crawling, remove a page from the Search index, or act as a noindex directive.
Google's current Google-Extended documentation. Last verified: .
Last reviewed: for page guidance. Source verified: .
Googlebot vs Google-Extended
Googlebot and Google-Extended should be tested separately. Googlebot affects normal Google Search crawling and indexing. Google-Extended is a separate Google AI product control token and does not control normal Search crawling.
| Item | Googlebot | Google-Extended |
|---|---|---|
| Primary purpose | Google Search crawling and indexing | Google AI product control token |
| Affects normal Search crawling | Yes | No |
| robots.txt token | Googlebot | Google-Extended |
| Common use | Allow/block Google Search crawler | Allow/block certain Google AI uses |
| Should be tested separately | Yes | Yes |
User agent
The robots.txt user-agent token to test is Google-Extended. Use this exact token when checking allow and block rules for Google-Extended.
User-agent: Google-Extended
What Google-Extended is used for
Use this reference when the policy question is Google AI product use controls rather than Search indexing. A site can allow Googlebot while blocking Google-Extended.
Google-Extended does not control Google Search indexing or eligibility for Google AI Overviews and AI Mode.
Search indexing, AI training, AI retrieval, shopping, or other use
Crawler policy surface. Review its documented purpose separately from unrelated search, training, retrieval, and dataset crawlers.
Impact on crawling, indexing, AI use, or retrieval
Blocking Google-Extended should not be described as blocking Google Search. Review Googlebot for Search crawling and Google-Extended for the separate AI control surface.
Blocking Google-Extended must not be described as blocking Google Search or Google AI feature eligibility.
How to allow Google-Extended
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: Google-Extended Allow: / User-agent: * Allow: /
What allowing Google-Extended enables
Allowing Google-Extended means the deployed robots.txt policy does not block the Google-Extended token on matching public paths. It does not guarantee a visit, indexing, inclusion, ranking, training, or citation.
How to block Google-Extended
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: Google-Extended Disallow: / User-agent: * Allow: /
What blocking Google-Extended affects and does not affect
Blocking Google-Extended asks compliant Google-Extended requests not to fetch matching paths. It does not authenticate the page, remove it from indexes, or control other crawler tokens.
Common verification notes
- Test Google-Extended and Googlebot separately on the same URL path.
- Do not use a Google-Extended block as a noindex strategy.
- Document whether the policy intent is AI product control or Search crawling.
Google-Extended verification and troubleshooting note
Verify Google-Extended with its official source, then test the final canonical URL and the exact path in robots.txt. Server logs can help confirm whether the crawler requested the page after policy changes.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Set this policy independently from Googlebot and use normal Search controls for Google Search visibility.
Official source
Google documentation for Google-Extended
This page bases crawler role descriptions on the linked owner documentation and labels limitations where robots.txt cannot prove downstream product behavior.
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.