llms.txt: Does It Help AI Visibility? A Decision Table and Tests

llms.txt is a file written in Markdown (a plain-text format where # marks headings and [text](url) marks links) and served at /llms.txt, giving AI tools a short, curated list of a site's best pages. It has no measured effect on AI visibility: Google says its Search ignores it, and the studies below found no citation effect.

The name comes from large language models (LLMs), the technology behind ChatGPT and similar assistants. Jeremy Howard proposed the format on September 3, 2024, and it remains a voluntary proposal, not a web standard (llmstxt.org). The only required element is a top-level heading (H1) with the site's name. A short summary in a blockquote usually follows, then sections headed by second-level headings (H2) that hold lists of [name](url): note links. By convention, an "Optional" section holds secondary links.

As of October 2026, the format is at version 2, published in August 2026 (changes page). Version 2 formalizes files at any subpath: a file covers the pages under its path, and the most specific file applies. It adds two link relations, which label how one web address relates to another. rel="alternate" points to a page's Markdown version, and rel="describedby" points to the llms.txt file that covers the page. Both can sit in the page's HTML or in an HTTP response header. Version 2 also keeps the Optional section only as a convention. Search Engine Journal reports that updating an existing file needs only a minor change.

It differs from two older files. robots.txt tells crawlers (automated programs that fetch pages) what they may access. sitemap.xml lists URLs for search engines. llms.txt controls nothing: listing a page grants no permission, and leaving one out blocks nothing. For ChatGPT, access runs through robots.txt. OpenAI's crawler documentation states that sites opted out of OAI-SearchBot, OpenAI's crawler for ChatGPT search, "will not be shown in ChatGPT search answers, though can still appear as navigational links."

Google's position is explicit. Its guide to AI features and your website, last updated July 10, 2026, says: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." It adds that creating llms.txt files or similar ones for other services "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." That covers AI Overviews (the AI summary above regular Google results) and AI Mode (Google's conversational search).

Rarely. In Ahrefs' May 2026 server logs for 137,000 domains, 97% of llms.txt files received zero requests. AI bots of all types made about 19.5% of the requests that arrived, and AI retrieval bots, the crawlers that fetch pages to answer a live question, about 1.1% (Ahrefs).

The study puts publication at about 28% of the domains. Only about 1,100 of roughly 38,000 valid files received any traffic (Search Engine Journal). Of the requests that arrived, 96% came from bots. The AI share was mostly AI agents and agentic infrastructure (10.5%; agents are software that takes actions on a user's behalf) and AI training crawlers (5.3%). The rest came from tools such as SEO audit software, which crawls a site to report technical problems. Ahrefs describes every figure as a ceiling on actual consumption, because a fetch does not show the content was read.

Only the SE Ranking model tested citations; the Ahrefs logs measured requests. Our reading is that the file is easy to publish and hard to benefit from. It does not rule out the narrow case that matters: a person who gives an assistant your URL. ChatGPT-User, the fetcher OpenAI uses when a user asks ChatGPT to visit a page, can read a deliberately supplied llms.txt. That is a use of the file, not a visibility channel.

In our October 14, 2026 check of 37 hand-picked hosts, 18 served a valid llms.txt, 15 returned a 404 (not found) and 4 blocked the request. The valid files were mostly on developer-documentation and software hosts, and none of the six news, reference and retail hosts had one.

Method. We ran curl (a command-line tool for fetching URLs) with -sL to follow redirects, sending the user agent Mozilla/5.0 (compatible; llmstxt-check). A user agent is the name a client sends to identify itself. A valid file means HTTP 200, plain text or Markdown, a first line that is a # heading, and at least one Markdown link. A byte order mark, an invisible character some editors add at the start of a file, was allowed. docs.anthropic.com and docs.claude.com redirect to the same file and count once.

The sample was hand-picked across developer platforms, software firms, SEO publishers, news and retail. It shows patterns, not how common the file is on the web; the 28% and 10.13% figures above measure that. We publish this page, and our own domain was in the sample (it has a file). citationresearch.com, our research lab's site, returned a 404.

Three patterns stand out:

• News, reference and retail: all six hosts (nytimes.com, bbc.com, wikipedia.org, amazon.com, walmart.com, nike.com) returned 404.
• Developer hosts: five of seven (Anthropic's docs, developers.openai.com, docs.perplexity.ai, developers.cloudflare.com, GitHub) served a file. developers.google.com and platform.openai.com did not.
• Size: of the five files we sized, NVIDIA's is 2,610 bytes with 19 links and Vercel's is 4,935 bytes with 25. Ours is 1,523 bytes with 16. Stripe's is 69,923 bytes with 308, and Anthropic's docs file is 83,341 bytes with 764. We counted links with a pattern match.

Our inference is that files cluster where developers and agents feed documentation to tools. Several software and SEO hosts in the sample have none, so a company's sector does not predict it.

Publish llms.txt if people or tools will be pointed at your documentation. Treat it as optional for a software marketing site, and skip it for news, retail and local sites until something requests it. This is our recommendation, built on the usage data above.

Run two tests: one confirms the URL returns a real file, the other counts who requests it. A pass on the first and zero hits on the second is the normal result. Our free agent-readiness checker also scores llms.txt quality among its 22 live checks, with no account needed.

Test A: is your llms.txt a real file?

url="https://example.com/llms.txt"
curl -sL -A "Mozilla/5.0 (compatible; llmstxt-check)" -o /tmp/llms.txt \
  -w "status=%{http_code} type=%{content_type}\n" "$url"
sed '1s/^\xEF\xBB\xBF//' /tmp/llms.txt | head -n 1   # first line
grep -ci '<html' /tmp/llms.txt                       # HTML tags
grep -c '](http' /tmp/llms.txt                       # links

The -A flag sets the user agent. Pass: status 200, type text/plain or text/markdown, a first line starting # , zero HTML tags and at least one link. Fail: a 404, or a 200 with text/html. The second is a soft 404: a single-page app or catch-all route can serve the home page for any path, which fools checkers that read only the status code.

Test B: does anything request it?

grep '"GET /llms.txt' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

This reads a standard combined-format web server log. If your site sits behind a content delivery network (CDN), use its logs instead. Run it over 30 days (our suggestion).

Reading the result: zero lines is typical. Lines from SEO audit tools or unidentified bots are expected. A line from OAI-SearchBot, GPTBot (OpenAI's training crawler), ClaudeBot (Anthropic's crawler) or PerplexityBot (Perplexity's crawler) shows the file was fetched, not that it shaped any answer.

A marketing-site llms.txt should list the pages you would send a new customer to, and nothing else. Of the five files we sized, the smallest had 19 links (NVIDIA) and 25 (Vercel), and our own has 16, so a short list is normal. Use canonical URLs, meaning the same address your sitemap lists.

# Acme Analytics

> Acme Analytics is web analytics software for ecommerce teams. This file lists the pages that best describe what we sell, what it costs and how it works.

Prices are in US dollars. Last reviewed: [date].

## Product
- [Product overview](https://example.com/product): What the software does and who it is for
- [Integrations](https://example.com/integrations): Supported platforms and setup steps

## Pricing
- [Pricing](https://example.com/pricing): Current plans, billing periods and limits

## Docs
- [Getting started](https://example.com/docs/start): Setup steps
- [API reference](https://example.com/docs/api): Endpoints and authentication

## Company
- [About](https://example.com/about): Who runs the company and where it is based
- [Contact](https://example.com/contact): Sales and support

## Optional
- [Changelog](https://example.com/changelog): Release notes

Then run a link check. Pass: every line prints 200. Fail: any other code, which means fix or remove the link.

grep -oE '\]\(https?://[^)]+\)' llms.txt | tr -d ']()' | while read u; do
  printf "%s %s\n" "$(curl -s -o /dev/null -w '%{http_code}' -L "$u")" "$u"
done

Add the link check to your release checklist. The file is public, so list only pages you already publish.

Fix crawler access first, publish pages that answer the questions your buyers ask, then measure citations on a fixed set of prompts before and after every change. This is how we work, and llms.txt is just one more change to put through it.

To test llms.txt, record citations for your prompts across several runs, publish the file and change nothing else. Then compare the same number of runs afterward. Because Google says Search ignores the file, movement in Google AI Mode during the test is a rough sign that something else changed. That is our method, not a proven control. Tracking AI citations over time explains how to keep the runs comparable.

If you want that test run on your own prompts, Book a demo.