Source-backed crawler reference
Applebot-Extended Robots.txt Guide: Allow or Block AI Use
Applebot-Extended is Apple's robots.txt control token for use of web content in certain generative AI models. It is separate from ordinary search crawling decisions.
Last reviewed: for page guidance. Source verified: .
User agent
The robots.txt user-agent token to test is Applebot-Extended. Use this exact token when checking allow and block rules for Applebot-Extended.
User-agent: Applebot-Extended
What Applebot-Extended is used for
Use this reference when reviewing Apple generative AI model-use policy. Applebot-Extended should be tested as its own user-agent token.
Apple documents Applebot-Extended separately from normal Applebot search crawling.
Search indexing, AI training, AI retrieval, shopping, or other use
AI product/model-use control token. Review it separately from ordinary search indexing and separately from non-Apple crawlers.
Impact on crawling, indexing, AI use, or retrieval
Blocking Applebot-Extended expresses a model-use policy for that token. It should not be treated as a universal search indexing control.
Model-use controls and search crawling should be reviewed as separate policy decisions.
How to allow Applebot-Extended
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: Applebot-Extended Allow: / User-agent: * Allow: /
What allowing Applebot-Extended enables
Allowing Applebot-Extended leaves Applebot-Extended access open for the Apple model-use surface described in Apple's documentation when Apple chooses to request matching public paths.
How to block Applebot-Extended
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: Applebot-Extended Disallow: / User-agent: * Allow: /
What blocking Applebot-Extended affects and does not affect
Blocking Applebot-Extended expresses a restriction for that Apple token. It does not block Googlebot, OpenAI crawlers, Common Crawl, or page indexing by search systems that use different user agents.
Common verification notes
- Test Applebot-Extended separately from wildcard rules.
- Keep policy notes clear when blocking model-use tokens but allowing search crawlers.
- Verify current Apple documentation before large policy changes.
Applebot-Extended verification and troubleshooting note
Test Applebot-Extended explicitly because a wildcard rule may not reflect the intended Apple AI policy. Keep notes clear when Applebot-Extended is blocked but search crawlers remain allowed.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Set the policy according to your content-use intent and verify current Apple documentation.
Official source
Apple documentation for Applebot-Extended
The source supports Applebot-Extended as an Apple control token for certain generative AI model use. This page does not treat it as a universal Apple Search or web indexing control.
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.