AI search guide
What Is llms.txt? A Practical Guide for AI Search
Learn what llms.txt is, where it lives, what to include, and why it should be treated as an emerging AI discovery convention.
What llms.txt is
llms.txt is an emerging convention for publishing a concise list of public resources that may help AI assistants and answer engines understand a site.
It is not a crawler access rule, a ranking guarantee, or proof that any model will crawl, train on, display, or cite the content.
What llms.txt is not
llms.txt is not an access-control file, a ranking signal guarantee, a training opt-in contract, or proof that a model will cite a page.
Use it as a curated discovery aid. Keep policy decisions in robots.txt, indexability decisions in page metadata and canonical strategy, and URL discovery in sitemap.xml.
Where llms.txt should live
The expected location is the site root, such as https://example.com/llms.txt. The file should be plain text or markdown-like text, easy to fetch, and focused on canonical public resources.
How llms.txt differs from robots.txt
robots.txt is the public crawl policy file for compliant crawlers. It answers whether a crawler may request a path. llms.txt does not allow or block crawlers, and it should not be used as a substitute for robots.txt rules.
If a page is blocked in robots.txt, listing it in llms.txt creates a contradiction. Use robots.txt for policy, then use llms.txt only to summarize public resources that are intentionally available.
How llms.txt differs from sitemap.xml
sitemap.xml is a search discovery file that lists canonical URLs for crawling and indexing workflows. llms.txt is a curated context file. It should explain the most useful public resources, not list every URL on the site.
A sitemap can contain many canonical pages. A strong llms.txt file is usually shorter and more editorial: product overview, documentation, pricing, support, policies, API references, and pages that answer important questions directly.
Who should create one
A site should consider llms.txt when it has public documentation, product explanations, API references, support pages, policies, or guides that AI-assisted systems may need to understand. It is especially useful for SaaS, developer tools, documentation sites, marketplaces, and technical service businesses.
Do not create one merely to claim AI visibility. The file is useful only when it points to substantial, canonical, public pages that are already aligned with robots.txt, metadata, schema, and internal links.
How llms.txt differs from robots.txt and sitemap.xml
robots.txt communicates crawl policy for compliant crawlers. sitemap.xml lists canonical URLs for search discovery workflows. llms.txt is different: it summarizes high-value public context for AI-assisted readers.
The three files should not contradict each other. Do not list a page in llms.txt if robots.txt blocks it or if canonical strategy points elsewhere.
What to include
Strong files usually include product overview pages, documentation, pricing, support, policies, API references, changelogs, and other pages that answer common questions directly.
Avoid private URLs, staging hosts, login-only content, tracking-heavy links, and pages blocked by robots.txt.
How to validate it
Validation starts with fetchability: /llms.txt should return a readable text response from the canonical host. Then review whether the heading names the site, the summary explains the entity, and each link points to a canonical public resource.
A validator should also warn about unsupported claims. A readable llms.txt file still should not say that it guarantees LLM visibility, indexing, ranking, training inclusion, or citation.
After validation, compare the file with the pages it links to. If those pages have weak titles, missing canonicals, thin body copy, or robots.txt blocks, the llms.txt file may be readable but still unhelpful. Treat validation as the start of the review, not the end.
How to maintain llms.txt over time
Update llms.txt when product names, documentation paths, pricing pages, API references, policy pages, or support URLs change. A stale file that points to redirects or retired docs can create the same ambiguity as stale sitemap entries.
Keep the file intentionally small. If every new blog post is added automatically, the file becomes another sitemap instead of a curated guide. Review it during launches, migrations, and major documentation reorganizations.