llms.txt: The Robots.txt for AI — Is Your Site Ready for the New Crawling Standard?
The llms.txt convention is gaining adoption as the machine-readable way to communicate with AI crawlers. What it is, why it matters for GEO, how to implement it, and whether it actually improves AI citation rates.
The llms.txt convention was proposed in late 2024 as a way for website owners to provide AI models with a curated, machine-readable summary of their site's content — analogous to how robots.txt communicates with search engine crawlers. By 2026, over 15,000 websites had adopted the format, including major publishers, software companies, and documentation sites.
Whether to implement llms.txt is not a trivial question. Done correctly, it gives AI systems a direct, authoritative map to your most important content. Done poorly, it can provide a misleading or incomplete picture that undermines your other GEO signals.
What llms.txt actually is
An llms.txt file is a plain text or Markdown file placed at the root of your domain (yourdomain.com/llms.txt). It contains a structured overview of your site, including: a brief description of what your site is and who it is for, a list of your most important pages with their URLs and descriptions, and optionally, a list of pages you prefer AI systems to use or avoid.
The file is designed to reduce the work AI crawlers must do to understand your site hierarchy and content priorities. Instead of inferring your site's purpose from crawling hundreds of pages, the AI can read your llms.txt and immediately understand your structure.
llms.txt vs robots.txt
| Dimension | robots.txt | llms.txt |
|---|---|---|
| Purpose | Instruct crawlers what NOT to crawl | Instruct AI what TO prioritize |
| Audience | All web crawlers (bots) | LLMs and AI systems specifically |
| Standard | Formal RFC standard (mandatory) | Community convention (optional) |
| Enforcement | Crawlers typically honor it | Voluntary — no enforcement mechanism |
| Content type | URL patterns and directives | Descriptions, links, prioritization |
| Format | Specific directive syntax | Markdown — flexible |
Current adoption and support
As of Q4 2026, confirmed support for llms.txt includes Claude (Anthropic) and several smaller AI platforms. Perplexity, ChatGPT, and Gemini have not officially confirmed llms.txt support, though Perplexity has stated it crawls and indexes the file during site discovery. The convention is most widely adopted by documentation sites, developer tools, and SaaS companies targeting technical audiences.
No guaranteed enforcement
Does it actually improve citations?
Evidence from early adopters is mixed but positive. Sites with well-structured llms.txt files report modest improvements in citation consistency — specifically that AI models more reliably cite their most important pages rather than secondary or older content. The mechanism is logical: if you tell the AI what your canonical pages are, it is more likely to retrieve those pages first.
The citation improvement is larger for sites with complex content architectures (documentation with hundreds of pages, news sites, large e-commerce sites) than for simple sites where the structure is already obvious.
How to implement llms.txt
The minimum viable llms.txt contains three sections: a brief site description, a list of your most important pages with descriptions, and optionally a list of pages to prefer or ignore. A basic implementation for a SaaS company:
# YourCompany > AI citation optimization platform for B2B SaaS companies. ## Core pages - [Home](https://yoursite.com/): Main product overview - [How it works](https://yoursite.com/how-it-works): Platform mechanics - [Pricing](https://yoursite.com/pricing): Plans and pricing ## Documentation - [Getting started](https://yoursite.com/docs/getting-started) - [AEO scoring explained](https://yoursite.com/docs/scoring) ## Optional: pages to ignore - /admin/ - /api/
Content best practices for the file
- Keep page descriptions to 1–2 sentences — concise, claim-dense descriptions extract better than long explanations
- Include your 20 most important pages, not all pages — prioritization signals are more useful than exhaustive lists
- Update the file whenever you add or restructure major sections
- Write descriptions as if they will be read by an AI that has never visited your site — be explicit about what each page is for
- Add an llms-full.txt companion file with more detailed page descriptions for AI systems that support extended context