llms.txt Validator

What This Checks

The validator accepts a domain or pasted llms.txt content and checks whether the file is fetchable, text-based, readable as markdown, useful enough to audit, and free from misleading visibility guarantees.

The result is meant to reduce avoidable crawl and extraction friction. It does not guarantee LLM inclusion, ranking, indexing, training use, or citation.

What The Validator Checks

The validator checks whether an llms.txt file is present, fetchable, text-based, and structured in a way that an automated reader can understand. It looks for practical conventions rather than pretending there is one universal standard. The most important checks are root availability, readable markdown, a clear heading, useful links, and no claims that the file guarantees AI visibility.

A passing result means the file is technically reasonable. It does not mean a crawler has adopted llms.txt, used the links, indexed the pages, or cited the content.

File Location, Encoding, And Content Type

The expected location is the site root: https://example.com/llms.txt. Redirects should be minimal and should resolve to the canonical host. The response should be UTF-8 compatible text, normally served as text/plain or a similar text content type. A file that downloads as binary, returns HTML error pages, or changes by user agent is harder to trust.

The validator also warns when a file is extremely long or when it has too few links to be useful. The goal is not to maximize length; the goal is to make the best public resources easy to identify.

Heading And Link Expectations

A practical llms.txt file should include a top-level heading naming the site, a short summary, and a curated list of links. Links should point to canonical URLs and use labels that describe the destination. Avoid vague labels such as click here, duplicated URL variants, or pages that redirect through tracking systems.

Relative links can work, but absolute canonical links are clearer for auditing. If the site uses multiple domains, language variants, or separate documentation hosts, spell out which URLs are authoritative.

Common Formatting Failures

Common failures include publishing an empty file, returning a 404 or 500 response, serving a marketing landing page instead of plain text, including private URLs, or claiming that llms.txt forces LLM inclusion. Some files pass basic syntax checks but still do little for discovery because they list thin pages, old documentation, or URLs that robots.txt blocks.

After validation, compare the file with robots.txt, sitemap.xml, and the pages themselves. A technically readable file is only useful when the linked resources are also crawlable, canonical, and substantial.

How To Act On Validation Results

Start with hard failures such as missing files, non-200 responses, invalid redirects, or HTML being returned in place of text. Those issues prevent reliable consumption. Next fix clarity problems: missing heading, vague summary, weak link labels, too few useful resources, or too many low-value URLs. Finally, review policy alignment by checking whether the linked pages are allowed by robots.txt and included in the canonical content strategy.

Some files pass validation but still may not help AI visibility. For example, a file that links only to the homepage, a contact page, and a generic blog archive is technically readable but not very informative. A stronger file links to pages that directly explain the entity, product, documentation, pricing, support, policies, and frequently cited facts. Re-run validation whenever the file changes or when major site sections move.

Example Input

URL: https://example.com/llms.txt

# Example Site

Public resources for Example Site.

- [Docs](https://example.com/docs)
- [Pricing](https://example.com/pricing)

Example Output

Status: Pass with warnings

Detected:
- Root file returned 200
- Text content is readable
- Top-level heading found
- 2 useful links found

Warnings:
- Add more canonical resources such as pricing, docs, support, and policies
- Avoid using llms.txt as a visibility guarantee in page copy

Common Errors Detected

• The file is missing from the root path or returns a 404, 500, or HTML error page.
• The content has no clear heading, summary, or useful labeled links.
• Links point to blocked resources, redirected URLs, tracking URLs, or generic archive pages.
• The file is technically readable but too thin to help a reviewer understand the site.

Recommended Fix Steps

1. Fix fetch failures first: root path, redirects, content type, status code, and text encoding.
2. Add a clear H1, short site summary, and labeled links to useful canonical resources.
3. Remove unsupported visibility promises and any URLs that are private, blocked, or not canonical.
4. Re-run validation after each update and whenever major site sections move.

Before Requesting Indexing

Before submitting this page in Search Console, confirm that the page returns 200 on the canonical host, has a self-referencing canonical tag, appears in the reduced sitemap only when it is index-worthy, and contains enough visible text to stand on its own. Check that ad placeholders do not interrupt the main workflow, that structured data matches visible content, and that the page does not claim AI search visibility is guaranteed.

For a new domain, it is better to request indexing for a small group of strong pages than to push every thin route at once. Re-run the relevant AI Index Check tool after publishing and keep a record of changes so future crawler, schema, or content updates can be audited.

Recommended Workflow

1. generate an llms.txt file
2. validate your llms.txt
3. check AI crawler access
4. test schema extractability
5. run an AI citation readiness report

Related Checks And Guides

Related AI Search Guides

FAQ