AI search guide
How to Allow All Crawlers in robots.txt
Direct answer
To allow all compliant crawlers to request every path, use a User-agent: * group with an empty Disallow directive. An explicit Allow: / is also widely supported and states the same intent, but crawler-specific groups can still override the wildcard group for their matching user agents.
Allow all with an empty Disallow
User-agent: * Disallow:
Allow all with an explicit root Allow
User-agent: * Allow: /
Use valid robots.txt allow-all examples, understand empty Disallow versus Allow: /, avoid blocking mistakes, and test the deployed file.
Last reviewed: .
Do not use the block-all form by mistake
The following rule asks every compliant crawler matched by the wildcard group not to crawl any path on this host. The slash after Disallow changes the policy from allow all to block all.
Warning: this blocks all crawling for the matched group
User-agent: * Disallow: /
How robots.txt matching affects allow-all rules
A crawler first selects the most specific user-agent group that matches its token. It then evaluates the path rules in that group, with the most specific matching path taking precedence. Do not rely on a simplified rule that the later line always overrides an earlier line.
A User-agent: * allow-all group does not guarantee that every named crawler is allowed when a crawler-specific group blocks it. robots.txt is also scoped to the protocol, hostname, and port where the file is served, so https://example.com, http://example.com, and a subdomain can expose different policies.
User-agent: ExampleBot Disallow: /private/ User-agent: * Disallow:
What allowing all crawlers does not guarantee
An allow-all robots.txt policy removes one crawl restriction for compliant crawlers. It is not a promise that a search engine or AI system will fetch, index, rank, use, or cite the site.
- It does not guarantee search indexing or rankings.
- It does not guarantee inclusion in AI answers or citations.
- It does not grant or prove permission for model training.
- It does not make a page crawlable when the server returns errors, requires authentication, or blocks bots at a firewall or protection layer.
How to verify the deployed robots.txt file
Verify the production response instead of assuming the source file or deployment configuration is active. Test the canonical host and the exact page paths that matter.
- Open /robots.txt on the final canonical hostname.
- Confirm it returns HTTP 200 as plain text and contains the intended production rules.
- Review crawler-specific groups that may take precedence over User-agent: *.
- Run the AI Crawler Robots.txt Checker against important production URLs.
- Use Google Search Console URL Inspection for pages intended for Google Search.
Common allow-all mistakes
Most allow-all failures are deployment or scope problems rather than syntax problems. Check the response a crawler actually receives.
- Leaving a staging Disallow: / rule on the production host.
- Editing robots.txt on the wrong hostname, subdomain, protocol, or port.
- Returning an HTML error or application page from /robots.txt instead of plain text.
- Expecting robots.txt to remove URLs that are already indexed.
- Blocking CSS, JavaScript, images, or other resources needed to understand a page.
- Overlooking a crawler-specific group that blocks a bot despite the wildcard allow.
- Treating crawl permission as consent for AI training or as a guarantee of AI visibility.
AI crawler policies need separate review
Generic allow-all intent is only one part of crawler policy. Search crawlers, AI search retrieval crawlers, training crawlers, and user-triggered agents may use different user-agent tokens and serve different purposes.
Sources and review date
This guide was last reviewed on 2026-08-30 against Google Search Central guidance and the Robots Exclusion Protocol standard.