Source-backed crawler reference
GPTBot Robots.txt Checker
GPTBot is OpenAI's crawler token for model-related crawling. You can control GPTBot access with robots.txt rules that target the GPTBot user agent.
Last reviewed: for page guidance. Source verified: .
User agent
The robots.txt user-agent token to test is GPTBot. Use this exact token when checking allow and block rules for GPTBot.
User-agent: GPTBot
What GPTBot is used for
Use this reference when the policy question is AI model training or model-related crawling, not ChatGPT search discovery. GPTBot should be reviewed independently from OAI-SearchBot and ChatGPT-User.
OpenAI documents GPTBot separately from OAI-SearchBot; disallowing it indicates content should not be used for model training.
Search indexing, AI training, AI retrieval, shopping, or other use
Crawler policy surface. Review its documented purpose separately from unrelated search, training, retrieval, and dataset crawlers.
Impact on crawling, indexing, AI use, or retrieval
A GPTBot allow or block rule communicates model-related crawler policy. It does not directly control Google Search indexing, OpenAI search discovery, or user-triggered ChatGPT requests.
Training permission is a separate policy decision from ChatGPT search discovery.
OAI-SearchBot is used for OpenAI search discovery. GPTBot is documented separately for model training, and ChatGPT-User is used for user-initiated requests. A site can therefore allow search discovery while making a different policy decision about training.
How to allow GPTBot
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: GPTBot Allow: / User-agent: * Allow: /
What allowing GPTBot enables
Allowing GPTBot means the deployed robots.txt policy does not block the GPTBot token on matching public paths. It does not guarantee a visit, indexing, inclusion, ranking, training, or citation.
How to block GPTBot
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: GPTBot Disallow: / User-agent: * Allow: /
What blocking GPTBot affects and does not affect
Blocking GPTBot asks compliant GPTBot requests not to fetch matching paths. It does not authenticate the page, remove it from indexes, or control other crawler tokens.
Common verification notes
- Do not describe GPTBot as the ChatGPT search crawler.
- Use a specific User-agent: GPTBot group when the training policy is intentional.
- Retest after migrations because wildcard rules can override expectations.
GPTBot verification and troubleshooting note
Verify GPTBot with its official source, then test the final canonical URL and the exact path in robots.txt. Server logs can help confirm whether the crawler requested the page after policy changes.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Choose an explicit training policy independently from OAI-SearchBot. Do not describe GPTBot as the ChatGPT search crawler.
Official source
OpenAI documentation for GPTBot
This page bases crawler role descriptions on the linked owner documentation and labels limitations where robots.txt cannot prove downstream product behavior.
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.