Glossary · How AI reads content
llms.txt
A proposed plain-text standard that tells AI crawlers which of your pages matter and how to read them — like robots.txt, but for large language models.
# YourBrand> The fastest CRM for small sales teams.## Docs- [Getting started](/docs/start): set up in a day- [Pricing](/pricing): plans for small teams## About- [Company](/about): who we are
llms.txt is a proposed standard for a single markdown file, placed at your site’s root, that hands large language models a clean, curated map of the pages that matter most. It is an emerging idea rather than an established rule — but a cheap one to adopt.
What llms.txt is
llms.txt was put forward by Jeremy Howard of Answer.AI in September 2024. The proposal is simple: publish a plain-text, markdown file at /llms.txt that acts as a curated index of your site for large language models. A typical file has an H1 title (your name or brand), a short blockquote summarizing what you do, and H2 sections that list your most important links with a brief description after each.
The idea borrows the familiar “file at the root” convention, but its job is editorial rather than technical: it is a hand-picked table of contents that says here are the pages worth reading, and here is what each one is.
Why it was proposed
Language models work within a limited context window, and much of the modern web is awkward for them to read — pages wrapped in navigation, ads, cookie banners, and JavaScript that has to render before any content appears. Extracting the substance of a site from that clutter is costly and error-prone.
llms.txt sidesteps the problem by offering a clean markdown summary a model can consume directly. Instead of guessing which of a thousand URLs matter, an AI system can read one concise file that points to the canonical, high-value pages and explains them in a sentence each.
llms.txt vs. robots.txt
The two files are easy to confuse because both sit at the root and both concern automated readers — but they do opposite kinds of work.
They are not alternatives. A site can keep its robots.txt for access rules and add an llms.txt to describe the content it wants understood.
How to write one
Because the format is plain markdown, an llms.txt file is trivial to create and maintain by hand. The proposed shape is:
- An H1 title. Your brand or site name at the top of the file.
- A blockquote summary. One or two sentences on what you do, so a model gets the gist without following any links.
- H2 sections of links. Group your most important pages — docs, pricing, about — and list each as a markdown link with a short description after a colon.
- Only what matters. The value is in the curation. A focused list of canonical pages is more useful than an exhaustive dump of every URL.
Where adoption stands
llms.txt is an emerging proposal, not a confirmed part of how any major AI engine ranks or retrieves content. Adoption among sites and tooling is growing, but the large AI providers have not publicly stated that they consume it as a signal. That makes it a measured bet: the effort to add a markdown file is small, the downside is negligible, and if the convention gains traction you are already prepared. Just don’t treat it as a guaranteed path to being cited — clear, credible, machine-readable content on the pages themselves still does the heavier lifting.
Frequently asked
- Is llms.txt the same as robots.txt?
- No. robots.txt is a long-standing file that tells crawlers which paths they may or may not access — it governs permissions. llms.txt is a newer, separate proposal that does the opposite kind of work: instead of restricting access, it proactively curates and describes your most important content in markdown so a language model can find and parse it easily. They can coexist on the same site and serve different purposes.
- Do AI engines actually use llms.txt yet?
- It is an emerging proposal with growing adoption among sites and tooling, but the major AI providers have not publicly confirmed that they read llms.txt as a consumption or ranking signal. Treat it as a low-cost, forward-looking practice rather than a guaranteed visibility win — it is cheap to add and sensible if the standard gains traction, but it is not yet an established requirement.
- How do I create an llms.txt file?
- Write a plain markdown file and place it at your site's root, at /llms.txt. The proposed format is an H1 with your name or brand, an optional blockquote with a short summary, and H2 sections that list your key pages as links with a brief description after each. Because it is just markdown, it is trivial to create by hand and easy to keep up to date as your important pages change.