Source-backed crawler reference
CCBot Robots.txt Checker
CCBot is Common Crawl's crawler token for dataset collection. It is not the same as a direct search crawler or answer-engine retrieval crawler.
Last verified:
User agent
The robots.txt user-agent token to test is CCBot. Use this exact token when checking allow and block rules for CCBot.
User-agent: CCBot
What CCBot is used for
Use this reference when the policy question is Common Crawl dataset collection. Treat CCBot as a dataset policy decision rather than a direct AI search visibility control.
CCBot robots policy controls Common Crawl collection, which is separate from direct search crawler access.
Search indexing, AI training, AI answers, or retrieval impact
A CCBot allow or block rule can affect Common Crawl collection. It should not be used as a proxy for Googlebot, OAI-SearchBot, GPTBot, or PerplexityBot policy.
Common Crawl access is a dataset policy decision and should not be merged with search discovery recommendations.
How to allow CCBot
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: CCBot Allow: / User-agent: * Allow: /
How to block CCBot
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: CCBot Disallow: / User-agent: * Allow: /
Common verification notes
- Decide whether dataset collection matches business policy.
- Do not merge CCBot decisions with search crawler decisions.
- Review whether downstream use of Common Crawl matters for the site.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Choose a dataset collection policy based on business intent. Do not treat CCBot as a direct AI search crawler.
Official source
Common Crawl documentation for CCBot
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.