Source-backed crawler reference
Applebot-Extended Robots.txt Checker
Applebot-Extended is Apple's robots.txt control token for use of web content in certain generative AI models. It is separate from ordinary search crawling decisions.
Last verified:
User agent
The robots.txt user-agent token to test is Applebot-Extended. Use this exact token when checking allow and block rules for Applebot-Extended.
User-agent: Applebot-Extended
What Applebot-Extended is used for
Use this reference when reviewing Apple generative AI model-use policy. Applebot-Extended should be tested as its own user-agent token.
Apple documents Applebot-Extended separately from normal Applebot search crawling.
Search indexing, AI training, AI answers, or retrieval impact
Blocking Applebot-Extended expresses a model-use policy for that token. It should not be treated as a universal search indexing control.
Model-use controls and search crawling should be reviewed as separate policy decisions.
How to allow Applebot-Extended
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: Applebot-Extended Allow: / User-agent: * Allow: /
How to block Applebot-Extended
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: Applebot-Extended Disallow: / User-agent: * Allow: /
Common verification notes
- Test Applebot-Extended separately from wildcard rules.
- Keep policy notes clear when blocking model-use tokens but allowing search crawlers.
- Verify current Apple documentation before large policy changes.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Set the policy according to your content-use intent and verify current Apple documentation.
Official source
Apple documentation for Applebot-Extended
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.