Source-backed crawler reference
Bytespider Robots.txt Guide: Allow, Block, and Test Access
Bytespider is a ByteDance crawler token. A dedicated public purpose statement was not verified, so its product purpose should be treated as uncertain.
Last reviewed: for page guidance. Source verified: .
User agent
The robots.txt user-agent token to test is Bytespider. Use this exact token when checking allow and block rules for Bytespider.
User-agent: Bytespider
What Bytespider is used for
Use this reference when you need a conservative robots.txt policy for the Bytespider token. Avoid assigning an unverified AI search or training purpose.
robots.txt can express a policy for the token, but the product purpose should be treated as uncertain.
Search indexing, AI training, AI retrieval, shopping, or other use
ByteDance crawler token with limited verified public purpose detail. Treat search indexing, AI training, retrieval, and other downstream effects as uncertain unless a primary source confirms them.
Impact on crawling, indexing, AI use, or retrieval
A Bytespider allow or block rule communicates crawler policy for that token, but the downstream product effect should be labeled uncertain unless official documentation confirms it.
Unverified crawler purposes should be labeled as uncertain rather than inferred.
How to allow Bytespider
Add an allow rule when this crawler should be permitted to request public pages. Test the deployed robots.txt file on the exact URL path, because a homepage allow can coexist with deeper disallow rules.
User-agent: Bytespider Allow: / User-agent: * Allow: /
What allowing Bytespider enables
Allowing Bytespider permits the Bytespider token to request matching public paths if it crawls the site. Because the public purpose is not fully verified here, an allow rule should not be described as enabling a specific AI or search feature.
How to block Bytespider
Add a block rule only when the policy intent is to restrict this crawler. Blocking is a public directive for compliant crawlers, not authentication and not a ranking control.
User-agent: Bytespider Disallow: / User-agent: * Allow: /
What blocking Bytespider affects and does not affect
Blocking Bytespider asks that token not to fetch matching paths. It does not block Googlebot, CCBot, GPTBot, ClaudeBot, or other crawlers, and it does not prove anything about downstream ByteDance product behavior.
Common verification notes
- Label the purpose as uncertain when documentation is not specific.
- Use explicit robots.txt rules if business policy requires a decision.
- Monitor server logs and documentation updates.
Bytespider verification and troubleshooting note
Use explicit wording when documenting Bytespider because the product purpose can be ambiguous. Test the exact path, monitor logs, and update the policy if ByteDance publishes clearer source documentation.
How to interpret a robots.txt checker result
An allow result means the matched robots.txt directive does not block this compliant crawler on the tested path. It does not guarantee a visit, indexing, inclusion, ranking, or citation. A block is a public crawler directive, not authentication or access control.
Recommended action: Choose a conservative policy based on your own requirements and monitor current ByteDance documentation.
Official source
ByteDance documentation for Bytespider
The current page treats Bytespider conservatively: the token is identified, but unverified downstream product behavior is not presented as fact.
See the AI Index Check methodology for verification, scoring, limitations, and correction policy.