robots.txt tells crawlers what they may fetch. llms.txt tries to solve the opposite problem: given that an AI system will read your site, what should it read first, and how do you keep that reading accurate?
What is llms.txt?
llms.txt is a proposed convention — introduced by Jeremy Howard of Answer.AI in 2024 — for a Markdown file at your domain root (/llms.txt) that gives language models a curated map of your site: what the product is, where the important docs live, and which pages best answer common questions. Where robots.txt is permission, llms.txt is orientation. A number of developer-tool companies have adopted it, and AI crawlers increasingly fetch it alongside robots.txt.
The format
The proposal is deliberately simple — plain Markdown:
# ASHDOCS
> REST API for PDF and document automation: 24 tools for converting,
> editing, securing and extracting documents. Free tier: 100 credits/month.
## Docs
- [API reference](https://www.ashdocs.com/docs): auth, endpoints, errors
- [Pricing](https://www.ashdocs.com/pricing): free tier and paid plans
## Tools
- [HTML to PDF](https://www.ashdocs.com/tools/html-to-pdf): Chromium rendering, full CSS
- [PDF to JSON](https://www.ashdocs.com/tools/pdf-to-data): schema-driven extraction
- [Bank statement to CSV](https://www.ashdocs.com/tools/bank-statement-to-csv): reconciled output
## Optional
- [Blog](https://www.ashdocs.com/blog): guides on PDF automation
One H1 (the project), a blockquote summary (the sentence you want models to repeat), then H2 sections of annotated links. An optional companion pattern serves Markdown versions of key pages (page.html.md) so models read clean text instead of navigation chrome.
Does it actually work?
Honest answer: adoption is real, enforcement is not. No AI vendor guarantees llms.txt changes retrieval or citation. The pragmatic case: the file costs ten minutes, is read by several crawlers, and — most importantly — forces you to write the one-sentence answer to "what is this site?" that answer engines lift. Treat it as cheap insurance inside a broader AEO effort: answer-first page openings, FAQ schema, quotable definitions.
How to add one in 10 minutes
First, write the blockquote summary — one sentence, no marketing adjectives, the thing you'd want ChatGPT to say about you. Second, list only pages that answer questions (docs, pricing, tool pages) — this is a curation signal, not a sitemap dump; leave out login screens and legal boilerplate. Third, serve it as text/plain or text/markdown at /llms.txt and reference nothing that 404s. Finally, keep it in your deploy pipeline so it updates when your product does — a stale llms.txt teaches models stale facts.
llms.txt vs robots.txt vs sitemap.xml
They coexist: robots.txt governs access, sitemap.xml enumerates everything crawlable, llms.txt curates what matters and what it means. None replaces another, and llms.txt does not block training — if you want that, use robots.txt directives for AI user-agents.
FAQ
Where does llms.txt go?
At the domain root: https://yourdomain.com/llms.txt, served as plain text/Markdown.
Is llms.txt an official standard? No — it's a community proposal from 2024 with meaningful voluntary adoption, comparable to how sitemap.xml spread before formalization.
Will it make AI engines cite my site? No guarantees. It improves the odds models read accurate, current information about you, which is a precondition for good citations — pair it with answer-first content and FAQ schema.
Does llms.txt stop my content being used for training? No. It's orientation, not permission. Access control belongs in robots.txt.
ASHDOCS ships its own llms.txt — view it at www.ashdocs.com/llms.txt, and read the docs it points to.