AI crawler policy
ClaudeBot Robots.txt Checker
Audit whether a site's robots.txt rules allow, block, or leave ClaudeBotunspecified. Use the result as a policy and crawlability check, not as a visibility guarantee.
Crawler Details
User agentClaudeBot
VendorAnthropic
Policy surfacerobots.txt
FAQ
What does ClaudeBot do?
ClaudeBot is associated with Anthropic and Anthropic crawler access.
Can robots.txt guarantee ClaudeBot behavior?
Robots.txt is a directive for compliant crawlers. It is useful for public policy signaling, but it is not access control.