Source-backed crawler reference
Googlebot Robots.txt Rules: Allow, Disallow & Blocking Guide
Googlebot robots.txt direct answer
The exact robots.txt user-agent token is Googlebot. An applicable Allow rule permits crawling for the matching path, while an applicable Disallow rule blocks compliant Googlebot requests for that path.
# Allow Googlebot User-agent: Googlebot Allow: / # Block Googlebot User-agent: Googlebot Disallow: /
Googlebot controls Google Search crawling. Google-Extended is a separate product token for certain Google AI uses, so blocking Google-Extended is not equivalent to blocking Googlebot. To determine whether a URL is blocked, fetch robots.txt from the canonical host and evaluate the most specific matching user-agent group and Allow or Disallow path rule for the exact final URL.
Google's current Googlebot and robots.txt documentation. Last verified: .
Last reviewed: for page guidance. Source verified: .
Googlebot vs Google-Extended
Googlebot and Google-Extended should be tested separately. Googlebot affects normal Google Search crawling and indexing. Google-Extended is a separate Google AI product control token and does not control normal Search crawling.
| Item | Googlebot | Google-Extended |
|---|---|---|
| Primary purpose | Google Search crawling and indexing | Google AI product control token |
| Affects normal Search crawling | Yes | No |
| robots.txt token | Googlebot | Google-Extended |
| Common use | Allow/block Google Search crawler | Allow/block certain Google AI uses |
| Should be tested separately | Yes | Yes |
User agent
The robots.txt user-agent token to test is Googlebot. Use this exact token when checking allow and block rules for Googlebot.
User-agent: Googlebot
What Googlebot is used for
Use this reference when the question is googlebot robots txt, google robots check, or whether Google Search can crawl a page. Googlebot should be reviewed with canonical tags, noindex, sitemap inclusion, and server status.
Googlebot robots controls affect Google Search crawling. Google AI Overviews and AI Mode use normal Search eligibility controls.
Search indexing, AI training, AI retrieval, shopping, or other use
Crawler policy surface. Review its documented purpose separately from unrelated search, training, retrieval, and dataset crawlers.
Impact on crawling, indexing, AI use, or retrieval
Googlebot is the crawler with the clearest Search indexing impact in this directory. If a page should rank in Google Search, do not block Googlebot by accident.
A page must be indexed and eligible for a Google Search snippet before it can appear as a supporting link in Google AI features.
How to allow Googlebot
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: Googlebot Allow: / User-agent: * Allow: /
For wildcard policies that apply beyond Googlebot, review the robots.txt allow-all examples and caveats.
What allowing Googlebot enables
Allowing Googlebot means the deployed robots.txt policy does not block the Googlebot token on matching public paths. It does not guarantee a visit, indexing, inclusion, ranking, training, or citation.
How to block Googlebot
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: Googlebot Disallow: / User-agent: * Allow: /
What blocking Googlebot affects and does not affect
Blocking Googlebot asks compliant Googlebot requests not to fetch matching paths. It does not authenticate the page, remove it from indexes, or control other crawler tokens.
Common verification notes
- Test Googlebot and Google-Extended separately.
- Blocking Googlebot can affect Search crawling and indexing.
- A Googlebot allow rule does not guarantee ranking or indexing.
Googlebot verification and troubleshooting note
Verify Googlebot with its official source, then test the final canonical URL and the exact path in robots.txt. Server logs can help confirm whether the crawler requested the page after policy changes.
Exact Googlebot robots.txt phrases to verify
The phrase googlebot robots.txt usually means a user wants to know which rule matched Googlebot. The phrase googlebot disallow usually means a Disallow directive may be preventing Googlebot from fetching a URL. The phrase googlebot blocked by robots.txt means the checker should identify the matching rule and tested path.
For a google robots.txt check, test the exact page URL rather than only the homepage. A site can allow / but block /guides/, /docs/, or another directory that contains the page you want Google to crawl.
Googlebot blocked by robots.txt troubleshooting
If the result says Googlebot is blocked by robots.txt, inspect the most specific matching rule first. Common causes include a production User-agent: Googlebotgroup left from staging, a wildcard Disallow: /, path-level blocks such asDisallow: /guides/, conflicting Allow and Disallowrules, robots.txt fetch failures, and confusing crawl blocking with noindex.
- Fetch
/robots.txtfrom the canonical host. - Match the most specific applicable user-agent group for
Googlebot. - Evaluate the matching
AllowandDisallowpath rules. - Check the host, protocol, subdomain, redirects, and final URL.
- Test the exact affected URL and confirm the decisive matching rule.
User-agent: Googlebot Disallow: /guides/ Allow: /guides/public/ # Test the exact final URL path, not only the homepage.
robots.txt controls crawling. A noindex directive controls index eligibility after a page can be fetched. If Googlebot cannot crawl a blocked page, Google may not see a page-level noindex update quickly.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Keep important public pages crawlable when Google Search visibility is intended. Review noindex, snippet controls, and canonicals separately.
Official source
Google documentation for Googlebot
This page bases crawler role descriptions on the linked owner documentation and labels limitations where robots.txt cannot prove downstream product behavior.
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.