Source-backed crawler reference
CCBot Robots.txt Guide: Common Crawl Allow and Block Rules
CCBot is Common Crawl's crawler token for dataset collection. It is not the same as a direct search crawler or answer-engine retrieval crawler.
Last reviewed: for page guidance. Source verified: .
User agent
The robots.txt user-agent token to test is CCBot. Use this exact token when checking allow and block rules for CCBot.
User-agent: CCBot
What CCBot is used for
Use this reference when the policy question is Common Crawl dataset collection. Treat CCBot as a dataset policy decision rather than a direct AI search visibility control.
CCBot robots policy controls Common Crawl collection, which is separate from direct search crawler access.
Search indexing, AI training, AI retrieval, shopping, or other use
Common Crawl dataset collection. It is related to public web datasets, not direct Search indexing or live answer retrieval.
Impact on crawling, indexing, AI use, or retrieval
A CCBot allow or block rule can affect Common Crawl collection. It should not be used as a proxy for Googlebot, OAI-SearchBot, GPTBot, or PerplexityBot policy.
Common Crawl access is a dataset policy decision and should not be merged with search discovery recommendations.
How to allow CCBot
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: CCBot Allow: / User-agent: * Allow: /
What allowing CCBot enables
Allowing CCBot permits Common Crawl's crawler token to request matching public paths for dataset collection when the crawler chooses to visit them. It does not guarantee that the page will appear in a Common Crawl release or any downstream AI product.
How to block CCBot
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: CCBot Disallow: / User-agent: * Allow: /
What blocking CCBot affects and does not affect
Blocking CCBot asks Common Crawl not to fetch matching paths. It does not block Googlebot, GPTBot, OAI-SearchBot, PerplexityBot, or other crawlers, and it does not remove content already collected elsewhere.
Common verification notes
- Decide whether dataset collection matches business policy.
- Do not merge CCBot decisions with search crawler decisions.
- Review whether downstream use of Common Crawl matters for the site.
CCBot verification and troubleshooting note
Verify CCBot with the Common Crawl source, then test the exact path in robots.txt. If dataset reuse is the concern, review server logs and policy history instead of treating CCBot as a live search crawler.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Choose a dataset collection policy based on business intent. Do not treat CCBot as a direct AI search crawler.
Official source
Common Crawl documentation for CCBot
The source supports CCBot as a Common Crawl crawler. This page avoids claiming direct AI citation or Search indexing effects from CCBot alone.
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.