Source-backed crawler reference
Googlebot Robots.txt Checker
Googlebot is Google's crawler for Search crawling and indexing. Blocking Googlebot in robots.txt can affect how Google Search discovers, crawls, and refreshes public pages.
Last verified:
User agent
The robots.txt user-agent token to test is Googlebot. Use this exact token when checking allow and block rules for Googlebot.
User-agent: Googlebot
What Googlebot is used for
Use this reference when the question is googlebot robots txt, google robots check, or whether Google Search can crawl a page. Googlebot should be reviewed with canonical tags, noindex, sitemap inclusion, and server status.
Googlebot robots controls affect Google Search crawling. Google AI Overviews and AI Mode use normal Search eligibility controls.
Search indexing, AI training, AI answers, or retrieval impact
Googlebot is the crawler with the clearest Search indexing impact in this directory. If a page should rank in Google Search, do not block Googlebot by accident.
A page must be indexed and eligible for a Google Search snippet before it can appear as a supporting link in Google AI features.
How to allow Googlebot
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: Googlebot Allow: / User-agent: * Allow: /
How to block Googlebot
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: Googlebot Disallow: / User-agent: * Allow: /
Common verification notes
- Test Googlebot and Google-Extended separately.
- Blocking Googlebot can affect Search crawling and indexing.
- A Googlebot allow rule does not guarantee ranking or indexing.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Keep important public pages crawlable when Google Search visibility is intended. Review noindex, snippet controls, and canonicals separately.
Official source
Google documentation for Googlebot
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.