AI search guide
Google-Extended vs Googlebot: What Each Crawler Does
Understand the difference between Google-Extended and Googlebot so robots.txt changes do not accidentally target the wrong Google crawler.
Googlebot is for Google Search crawling
Googlebot is associated with crawling for Google Search. If a page is blocked for Googlebot, that can affect how Google Search discovers or refreshes that page.
Googlebot policy should be reviewed alongside canonical tags, noindex directives, sitemap inclusion, redirects, and server responses.
Google-Extended is a separate AI control token
Google-Extended is not the same crawler as Googlebot. It is a control token used for certain Gemini and Vertex AI purposes. Blocking Google-Extended should not be described as blocking Google Search indexing.
The common mistake is copying an AI opt-out snippet and assuming it changes all Google crawling. Keep the purpose of each user-agent clear in comments and internal documentation.
User-agent: Google-Extended Disallow: / User-agent: Googlebot Allow: /
When to test both tokens
Test both tokens after site launches, migrations, robots.txt rewrites, CMS changes, and compliance reviews. The two tokens can have different rules in the same file.
A checker can identify what the deployed robots.txt says, but policy owners still need to decide whether each allow or block state matches business intent.