AIAI Search Visibility Tools

AI search tool

AI Crawler / robots.txt Checker

Inspect robots.txt and summarize whether major AI-related crawlers appear allowed, blocked, or unspecified.

Open toolReadiness report

What This Checks

This page supports focused AI search operations while keeping claims precise. The checks look for crawlable public resources, extractable metadata, clean JSON-LD, and clear policy signals. They do not guarantee LLM inclusion, ranking, or citation.

Recommended Workflow

  1. Run the domain report from the homepage tool.
  2. Review robots.txt decisions for AI-related crawlers and search crawlers.
  3. Validate /llms.txt if you publish one as an emerging discovery convention.
  4. Fix homepage metadata, canonical tags, JSON-LD errors, and thin crawlable content.

FAQ

Can robots.txt enforce crawler behavior?

Robots.txt is a public crawling directive followed by compliant bots. It is not an authentication or access-control layer.

Is unspecified the same as allowed?

For robots.txt matching, no applicable disallow normally means crawlable, but explicit policy is easier to audit.