AI search guide
Googlebot vs Google-Extended - Robots.txt Rules and AI Search Impact
Compare Googlebot and Google-Extended, see robots.txt examples, and check whether Google crawlers are allowed or blocked.
Direct answer
Googlebot is used for Google Search crawling and indexing, while Google-Extended is a separate robots.txt control token for certain Google AI product uses. Blocking Googlebot can affect Search crawling; blocking Google-Extended is separate from normal Google Search crawling.
Quick answer: Googlebot vs Google-Extended
Googlebot is the crawler token site owners review for Google Search crawling and indexing. Google-Extended is a separate robots.txt token for certain Google AI product uses. They can appear in the same robots.txt file, but they should not be treated as the same crawler.
If your goal is to keep pages visible in Google Search, avoid blocking Googlebot on canonical public URLs. If your goal is to make a separate AI policy decision, review Google-Extended independently and test both tokens before publishing the change.
Googlebot vs Google-Extended quick comparison
Googlebot and Google-Extended are separate robots.txt user-agent tokens. A google robots check should evaluate both when the site owner cares about Search crawling and AI-related Google policy controls.
The most important difference is Search indexing risk. Blocking Googlebot can affect Google Search crawling and indexing. Blocking Google-Extended is separate from normal Google Search crawling.
| Crawler | Purpose | User-agent token | Search indexing impact | AI/training impact | Should most sites block it? |
|---|---|---|---|---|---|
| Googlebot | Google Search crawling and page refresh | Googlebot | High. Blocking Googlebot can affect Search discovery, crawling, and indexing. | Separate from Google-Extended AI policy controls. | Usually no for public pages intended to rank in Google Search. |
| Google-Extended | Separate control token for certain Gemini and Vertex AI uses | Google-Extended | Separate from normal Google Search crawling. | Relevant to certain Google AI product use controls. | It depends on business policy; do not block it by accident. |
Decision table: which Google directive affects what
Use this table when deciding whether a robots.txt change is about Google Search indexing, Google AI product use, or both. Test both crawler tokens before publishing because the same robots.txt file can contain different rules for each token.
| Decision | What it means | What to test |
|---|---|---|
| Which directive affects Google Search indexing? | Googlebot robots.txt access is the crawler permission signal tied to normal Google Search crawling and page refresh. | Test Googlebot on the exact canonical URL and confirm the page is not noindexed. |
| Which directive affects Google AI product use? | Google-Extended is the separate user-agent token for certain Google AI product controls. | Test Google-Extended separately from Googlebot on the same URL path. |
| Googlebot blocked but Google-Extended allowed | Google Search crawling can be restricted while the separate Google-Extended policy remains open. This is usually risky for pages intended to rank. | Fix accidental Googlebot Disallow rules, wildcard blocks, and path-specific directory blocks before requesting indexing. |
| Googlebot allowed but Google-Extended blocked | Google Search crawling can remain open while the separate Google AI product control token is restricted. | Confirm Googlebot is allowed, the page returns 200, canonical is correct, and Google-Extended block is intentional. |
What Googlebot does
Googlebot requests public pages so Google Search can discover, crawl, render, and refresh content. A site that blocks Googlebot from important public pages may make it harder for Google Search to evaluate those pages, even if they appear in sitemap.xml.
Googlebot access is only one part of indexability. A page also needs a successful response, a canonical URL that points to itself or the intended preferred URL, no conflicting noindex directive, useful internal links, and enough crawlable content to justify indexing.
What Google-Extended does
Google-Extended is a separate user-agent token that site owners can use for certain Google AI product controls. It should be documented as an AI policy surface, not as the Google Search crawler.
This separation is the reason a robots.txt file can allow Googlebot and block Google-Extended at the same time. That pattern communicates one policy for Google Search crawling and another policy for the Google-Extended control token.
Which one affects Search indexing
Googlebot is the crawler to review when the concern is Google Search crawling, rendering, discovery, and refresh. If robots.txt blocks Googlebot from a public page, that page may become harder for Google Search to evaluate.
Googlebot should be reviewed alongside status codes, canonical tags, noindex directives, sitemap inclusion, internal links, redirects, and server response behavior.
User-agent: Googlebot Allow: / Sitemap: https://example.com/sitemap.xml
Which one affects AI training or AI product use
Google-Extended is the separate token to review for certain Google AI product uses. It should not be described as the same crawler as Googlebot, and a Google-Extended block should not be treated as a Google Search noindex instruction.
The common mistake is copying an AI opt-out snippet and assuming it changes all Google crawling. Keep the purpose of each user-agent clear in comments, internal documentation, and SEO QA checklists.
User-agent: Google-Extended Disallow: / User-agent: Googlebot Allow: /
When to allow both
Allow both tokens when the business intent is to keep public pages available for Google Search crawling and not restrict the Google-Extended AI control surface. This is common for public documentation, pricing, support, educational guides, and pages intended for broad discovery.
An allow rule does not guarantee Google will crawl, index, rank, use, or cite a page. It only means the deployed robots.txt policy does not block that user-agent for the tested path.
User-agent: Googlebot Allow: / User-agent: Google-Extended Allow: / User-agent: * Allow: /
When to block Google-Extended only
Block Google-Extended only when the policy intent is to restrict that specific AI control token while keeping normal Google Search crawling open. This should be a deliberate business decision, not a copied default.
After publishing this rule, run a google robots check for both Googlebot and Google-Extended on the homepage and high-value paths. The expected result is Googlebot allowed and Google-Extended blocked.
User-agent: Google-Extended Disallow: / User-agent: Googlebot Allow: /
Robots.txt examples that accidentally block Googlebot
The risky pattern is a broad wildcard or Googlebot disallow that was copied from staging, compliance notes, or an AI opt-out discussion without checking Search indexing impact. If this appears on production public pages, it can block Google Search crawling.
Another risky pattern is blocking an entire directory such as /guides/ or /docs/ while submitting those same URLs in sitemap.xml. That sends mixed discovery and crawl signals.
User-agent: Googlebot Disallow: / User-agent: Google-Extended Disallow: / # Risk: Google Search crawling is blocked, not only Google-Extended.
When to test both tokens
Test both tokens after site launches, migrations, robots.txt rewrites, CMS changes, and compliance reviews. The two tokens can have different rules in the same file.
A checker can identify what the deployed robots.txt says, but policy owners still need to decide whether each allow or block state matches business intent. Run the AI crawler robots.txt checker before requesting indexing or changing crawler policy at scale.
FAQ
Is Google-Extended the same as Googlebot? No. Googlebot is associated with Google Search crawling. Google-Extended is a separate user-agent token for certain Google AI product controls.
Can blocking Googlebot affect Search indexing? Yes. Blocking Googlebot can affect Search crawling and indexing for the blocked paths.
Can I block Google-Extended while allowing Googlebot? Yes. Use separate user-agent groups and test the final deployed robots.txt file.
Does allowing either crawler guarantee ranking or AI citation? No. robots.txt only describes crawl permission for compliant crawlers. Ranking, indexing, and citation depend on additional signals.