Source-backed crawler reference
ClaudeBot Robots.txt Checker
ClaudeBot is Anthropic's automated web crawler token. Site owners can express an allow or block policy for ClaudeBot in robots.txt.
Last reviewed: for page guidance. Source verified: .
User agent
The robots.txt user-agent token to test is ClaudeBot. Use this exact token when checking allow and block rules for ClaudeBot.
User-agent: ClaudeBot
What ClaudeBot is used for
Use this reference when reviewing Anthropic crawler access for public pages. ClaudeBot policy should be checked against current Anthropic documentation because crawler names and product surfaces can change.
ClaudeBot can be addressed in robots.txt, but one Anthropic user agent should not be assumed to cover every Anthropic product purpose.
Search indexing, AI training, AI retrieval, shopping, or other use
Crawler policy surface. Review its documented purpose separately from unrelated search, training, retrieval, and dataset crawlers.
Impact on crawling, indexing, AI use, or retrieval
ClaudeBot access can matter for AI crawler policy, but an allow rule does not guarantee a visit, inclusion, model use, or citation.
Crawler naming and purpose can change, so policy should be checked against current Anthropic documentation.
How to allow ClaudeBot
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: ClaudeBot Allow: / User-agent: * Allow: /
What allowing ClaudeBot enables
Allowing ClaudeBot means the deployed robots.txt policy does not block the ClaudeBot token on matching public paths. It does not guarantee a visit, indexing, inclusion, ranking, training, or citation.
How to block ClaudeBot
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: ClaudeBot Disallow: / User-agent: * Allow: /
What blocking ClaudeBot affects and does not affect
Blocking ClaudeBot asks compliant ClaudeBot requests not to fetch matching paths. It does not authenticate the page, remove it from indexes, or control other crawler tokens.
Common verification notes
- Use a ClaudeBot-specific rule when the policy should not rely on User-agent: *.
- Test important public content paths, not only the homepage.
- Review source documentation before assuming every Anthropic product uses the same token.
ClaudeBot verification and troubleshooting note
Verify ClaudeBot with its official source, then test the final canonical URL and the exact path in robots.txt. Server logs can help confirm whether the crawler requested the page after policy changes.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Choose an explicit policy and verify the current user-agent documentation before relying on it.
Official source
Anthropic documentation for ClaudeBot
This page bases crawler role descriptions on the linked owner documentation and labels limitations where robots.txt cannot prove downstream product behavior.
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.