llms.txt: The Robots.txt for AI — Is Your Site Ready for the New Crawling Standard?

The llms.txt convention was proposed in late 2024 as a way for website owners to provide AI models with a curated, machine-readable summary of their site's content — analogous to how robots.txt communicates with search engine crawlers. By 2026, over 15,000 websites had adopted the format, including major publishers, software companies, and documentation sites.

Whether to implement llms.txt is not a trivial question. Done correctly, it gives AI systems a direct, authoritative map to your most important content. Done poorly, it can provide a misleading or incomplete picture that undermines your other GEO signals.

What llms.txt actually is

An llms.txt file is a plain text or Markdown file placed at the root of your domain (yourdomain.com/llms.txt). It contains a structured overview of your site, including: a brief description of what your site is and who it is for, a list of your most important pages with their URLs and descriptions, and optionally, a list of pages you prefer AI systems to use or avoid.

The file is designed to reduce the work AI crawlers must do to understand your site hierarchy and content priorities. Instead of inferring your site's purpose from crawling hundreds of pages, the AI can read your llms.txt and immediately understand your structure.

llms.txt vs robots.txt

Dimension robots.txt llms.txt

• →Purpose
• →Instruct crawlers what NOT to crawl
• →Instruct AI what TO prioritize
• →Audience
• →All web crawlers (bots)
• →LLMs and AI systems specifically
• →Standard
• →Formal RFC standard (mandatory)
• →Community convention (optional)
• →Enforcement
• →Crawlers typically honor it
• →Voluntary — no enforcement mechanism
• →Content type
• →URL patterns and directives
• →Descriptions, links, prioritization
• →Format
• →Specific directive syntax
• →Markdown — flexible

Current adoption and support

As of Q4 2026, confirmed support for llms.txt includes Claude (Anthropic) and several smaller AI platforms. Perplexity, ChatGPT, and Gemini have not officially confirmed llms.txt support, though Perplexity has stated it crawls and indexes the file during site discovery. The convention is most widely adopted by documentation sites, developer tools, and SaaS companies targeting technical audiences.

Does it actually improve citations?

Evidence from early adopters is mixed but positive. Sites with well-structured llms.txt files report modest improvements in citation consistency — specifically that AI models more reliably cite their most important pages rather than secondary or older content. The mechanism is logical: if you tell the AI what your canonical pages are, it is more likely to retrieve those pages first.

The citation improvement is larger for sites with complex content architectures (documentation with hundreds of pages, news sites, large e-commerce sites) than for simple sites where the structure is already obvious.

How to implement llms.txt

The minimum viable llms.txt contains three sections: a brief site description, a list of your most important pages with descriptions, and optionally a list of pages to prefer or ignore. A basic implementation for a SaaS company:

Content best practices for the file

• →Keep page descriptions to 1–2 sentences — concise, claim-dense descriptions extract better than long explanations
• →Include your 20 most important pages, not all pages — prioritization signals are more useful than exhaustive lists
• →Update the file whenever you add or restructure major sections
• →Write descriptions as if they will be read by an AI that has never visited your site — be explicit about what each page is for
• →Add an llms-full.txt companion file with more detailed page descriptions for AI systems that support extended context

Related The llms.txt File Explained: Should You Add One? Related How AI Crawlers Index Your Website