Tell AI Crawlers Exactly How to
Index and Cite Your Website

llms.txt is a proposed standard file format, similar in concept to robots.txt, that allows website owners to communicate directly with AI language model crawlers about their site's content. It was proposed to give website owners more control over how their content is discovered, extracted, and used by AI training and inference pipelines. The file lives at yourdomain.com/llms.txt and contains structured information about the site's purpose, key content pages, attribution preferences, and usage instructions.

Support is expanding rapidly. OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity's PerplexityBot, and Google's Googlebot (as it evaluates content for AI Overview) are all implementing varying levels of support for the llms.txt format. Implementing now provides an early adopter advantage — your site will be correctly understood by AI crawlers as they expand support for the format over 2025 and 2026.

robots.txt controls crawler access — which pages bots are allowed or not allowed to visit. llms.txt provides content guidance — which pages are most valuable for knowledge extraction and how the site owner wants content attributed and used. A page excluded from robots.txt cannot be crawled at all. A page excluded from llms.txt may still be crawled but will be deprioritised for knowledge extraction relative to the pages you have listed as primary content.

List your most comprehensive, authoritative, and citable content — typically 20 to 50 URLs maximum. Include your most comprehensive guides and tutorials, core product or service pages with unique information, original research or data publications, and highest-traffic content pages. Avoid listing navigational pages, thin pages, ecommerce category pages, and administrative pages — these add noise and dilute the signal about what your best content is.

Quarterly updates are recommended. As you publish new high-quality content, retire old pages, or change your content focus, the llms.txt should be updated to reflect your current best pages. An outdated llms.txt that lists pages you have since improved or unpublished is less useful than no llms.txt at all. The LLMs.txt Generator tracks your file age and sends quarterly update reminders.

Yes. Fully available on the free plan with 15 runs per month. Each run produces a complete, validated llms.txt file with all sections included, ready for upload to your root directory.

llms.txt uses a simple structured text format — not JSON or XML. It contains clearly labelled sections: a business description block (2 to 4 sentences explaining what your site is and what it covers), a key pages list (URLs with brief descriptions of each page's content and purpose), an attribution section (your preferred citation format and link requirements), and an optional restrictions section (topics or use cases you do not want AI engines to extract from). The LLMs.txt Generator produces a correctly formatted file that complies with the current specification version.

llms.txt is a guidance file, not an enforcement mechanism. robots.txt can actively block crawler access to specific pages. llms.txt provides voluntary guidance to AI crawlers that choose to respect it — it does not prevent crawlers from accessing or using content from pages not listed in it. Think of llms.txt as a courtesy communication to AI crawlers saying which pages are most valuable and how you prefer attribution, rather than a technical access control mechanism.

The standard llms.txt specification uses a single root-level file at yourdomain.com/llms.txt covering the entire domain. However, some implementations are exploring subdirectory-level files for large sites with distinct content sections. For most sites, a single well-structured root llms.txt is sufficient and more widely supported by current AI crawler implementations. If your site has dramatically different content sections (for example, a blog section and a documentation section), use the restrictions block to differentiate how each section should be used.

You cannot directly track llms.txt reads in standard analytics tools — AI crawlers do not typically register as page views. Indirect indicators include: checking your server access logs for requests to /llms.txt from known AI crawler user agents (GPTBot, ClaudeBot, PerplexityBot), monitoring your AI citation rate over 4 to 8 weeks after implementation for improvement, and using the WebMCP Readiness Checker which simulates AI crawler access and verifies llms.txt is accessible and parseable.