Source-backed crawler reference
Google-Extended Robots.txt Checker
Google-Extended is a Google robots.txt control token for certain Gemini and Vertex AI uses. It is separate from Googlebot and does not control normal Google Search crawling.
Last verified:
User agent
The robots.txt user-agent token to test is Google-Extended. Use this exact token when checking allow and block rules for Google-Extended.
User-agent: Google-Extended
What Google-Extended is used for
Use this reference when the policy question is Google AI product use controls rather than Search indexing. A site can allow Googlebot while blocking Google-Extended.
Google-Extended does not control Google Search indexing or eligibility for Google AI Overviews and AI Mode.
Search indexing, AI training, AI answers, or retrieval impact
Blocking Google-Extended should not be described as blocking Google Search. Review Googlebot for Search crawling and Google-Extended for the separate AI control surface.
Blocking Google-Extended must not be described as blocking Google Search or Google AI feature eligibility.
How to allow Google-Extended
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: Google-Extended Allow: / User-agent: * Allow: /
How to block Google-Extended
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: Google-Extended Disallow: / User-agent: * Allow: /
Common verification notes
- Test Google-Extended and Googlebot separately on the same URL path.
- Do not use a Google-Extended block as a noindex strategy.
- Document whether the policy intent is AI product control or Search crawling.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Set this policy independently from Googlebot and use normal Search controls for Google Search visibility.
Official source
Google documentation for Google-Extended
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.