Should You Add an llms.txt File to Your Site?

What the file is, and what it asks of you

The proposal comes from Jeremy Howard and was published in September 2024, with a second version in August 2026. The spec at llmstxt.org is short enough to read in a coffee break. The file lives at /llms.txt, or at a subpath like /docs/llms.txt for a section of a site, and an agent is meant to use the most specific one it finds.

Only one part is required: an H1 with the name of the project or site. After that you may add a blockquote summarizing what the thing is, then H2 sections holding lists of markdown links, each link optionally followed by a colon and a note about what the page covers. There is a convention that an ## Optional section holds links an agent can skip when it needs a shorter context. That is the whole format.

Nothing about it is technically demanding. The question is not how to write one, it is whether anything reads it, and that is a separate question from the budget question I worked through in what AEO is and whether a small site needs it.

The number that should set your expectations

Ahrefs ran the measurement that most posts on this topic skip. Across 137,000 domains using their analytics software, they found that 28% published a valid llms.txt file, which is 38,360 sites. Of those, 97% received zero requests for the file during May 2026. Not a bot, not a human, nothing.

Two caveats belong with that figure, and Ahrefs states the first one themselves: the sample is made of people who install web analytics, which skews toward exactly the audience most likely to have heard of llms.txt, so adoption across the web generally runs lower. The second is that a request count is a measure of fetching, not of value. A file nobody fetches is doing nothing, but the reverse is not guaranteed either.

Read it as a gap between adoption and use. Tens of thousands of site owners added the file because an article told them to, and the systems it was written for did not come looking. That is the single most useful fact available about llms.txt, and it should decide how much time you give the file rather than whether you give it any.

Google named the file and said it does nothing

This is unusually direct for search documentation. Google's guide on optimizing for generative AI search has a mythbusting section, and llms.txt is in it by name. The wording: “You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.”

The next sentence is the one worth keeping, because it is permission rather than a recommendation. “Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them.” You can read the whole bullet in Google's AI optimization guide. There is no hedge in it and no future tense.

John Mueller put it less diplomatically. Quoted in Ahrefs' write-up, he said none of the AI services have said they use the file, that you can tell from server logs they do not even check for it, and that to him “it's comparable to the keywords meta tag”: a claim by the site owner about what the site is about, when the site itself is right there to be read.

That comparison is the sharpest argument against the file, and it is also the limit of the argument. Google speaks for Google Search. It does not speak for the other systems the file was proposed for.

Nobody has promised to read it

Here is the asymmetry that gets glossed over. Publishing an llms.txt file and consuming one are two different acts, and the companies doing the first are not the ones promising the second. Anthropic publishes a file for its own documentation. That tells you the format is reasonable. It does not tell you that Claude fetches yours.

As of now, no public commitment exists from OpenAI, Anthropic or Google that their crawlers request llms.txt as part of normal operation. The spec describes agents as fetching the file on demand, and says it is meant for inference rather than training. Those are the proposal's intentions, not a vendor's documented behaviour, and the distinction matters when you are deciding where to spend an afternoon.

Compare it to robots.txt, which is the file that does have documented behaviour. The named AI crawlers publish their user agents, state that they obey it, and change what they fetch when you change it. I covered which ones and how in how to get your site mentioned by ChatGPT and Claude. One file has a contract behind it. The other has a convention and some goodwill.

Then why does Chrome audit it?

Because Google is not one opinion. In May 2026 the Chrome team added an llms.txt check to Lighthouse under its agentic browsing audits, which looks on its face like a contradiction of the Search documentation. Worth knowing what the check actually does before you read anything into it.

Per PPC Land's report, the audit flags a server error when it tries to fetch the file. If the file is simply absent and returns a 404, the audit is marked not applicable, because providing it remains optional. So the check does not reward you for having one. It tells you your file is broken, if you have one.

The sane reading is that two teams are building for two different consumers. Search indexes pages and has decided it does not need a site owner's summary of them. Chrome is building tooling for agents that browse on a person's behalf, and an agent arriving at a site cold has a genuine reason to want a map. Neither position implies the other is wrong, and neither amounts to a promise that anything will read your file next month.

The case that actually survives the evidence

Documentation. That is where the file does visible work, and it is where the idea came from. A coding agent pointed at a developer docs site has to guess which of four hundred pages are the ones that matter, and burns context doing it. A single markdown index that names the reference, the quickstart and the API pages saves it from guessing. Mueller, who is dismissive of the file for search, described that use as a crutch to save some tokens, which is faint praise but is still praise for a real job.

Now apply that to a nine-page marketing site. The whole site is smaller than the context window any current model is working with. An index of nine pages saves nobody anything, which is why the honest version of the recommendation is small: write the file because it is cheap and because a convention may firm up later, not because it does something today.

One more real use, and it is not about machines. Writing the blockquote at the top forces you to describe your site in two sentences with no marketing in them. Mine came out vague the first time, which told me something about the homepage copy it was derived from. That exercise is worth the fifteen minutes even if the file is never requested.

What mine looks like, and the honest accounting

ShipWolf has one at /llms.txt. It opens with the site name, a blockquote saying what the product is and who it is for, then sections for what is included, the price, the key pages, and an index of every blog post with a one-line note on each. The post list is appended by the same script that publishes a post, so it has never gone stale and has never cost me a second of maintenance after the first setup.

What I cannot tell you is that it has done anything. I have not seen a request for it, and given the crawl numbers above I have a good guess about what the logs would show. I keep it as documentation of the site rather than as a marketing asset, which means it gets the effort documentation deserves and none of the effort a landing page deserves.

If you take one thing from my setup, take the automation rather than the file. A hand-written index of a site that changes weekly is a liability: it drifts, it starts listing pages that have moved, and a wrong map is worse than no map. Either the file is generated by whatever publishes your pages, or it lists only things that will not move.

The fifteen-minute version

If you want one, here is the whole job. Write an H1 with the site name. Add a blockquote of two sentences saying what the site is and who it serves, in plain language. Add an H2 per group of pages that belong together, and under each one a markdown link per page with a short note after a colon describing what that page answers.

Put anything secondary under an ## Optional heading, since that is the signal an agent uses to decide what it can skip. Save it as plain text at the root as /llms.txt. Check that you have not blocked it in robots.txt, which is easy to do by accident with a broad disallow rule. Then decide, before you close the file, how it stays current: generated on publish, or on a calendar reminder, or not at all.

Two things to skip. Do not write a full-text dump of your site into a second file unless a specific tool you use asks for one, because the spec does not define that variant and nothing obliges anything to read it. And do not pad the index with pages you would not send a customer to. A file listing nine pages that matter is more useful than one listing sixty that do not, and the second kind is how a site ends up with a map it cannot maintain.

The verdict, by what kind of site you run

Developer tool or documentation site: yes, and put real thought into it. Your readers point coding agents at your docs daily, the file has a working job, and the cost of a bad index is support tickets from people whose agent read the wrong page.

Small marketing site or blog: yes if it can be generated, probably not if it has to be maintained by hand. The upside is a convention that might firm up. The downside is a stale file and an hour you could have spent on a page that gets read by people.

Store with thousands of products: no. An index of a catalogue is a feed, and you already have feeds that something actually consumes. A brand new site with four pages: also no, not yet. Write pages worth indexing first, then add the index when there is something to point at. If what you are after is being quoted inside an AI answer, the work is on the page, which I broke down in how to structure a page so AI can quote it, and no file at your root substitutes for it.

What I like about this question is that it has a real answer, which is rare in this corner of the web. The file is cheap, the claims made for it are not supported, and the one measurement anybody has published says almost nothing fetches it. That combination argues for a small bet made in fifteen minutes and then left alone, not for a project and definitely not for a retainer.

Keep an eye on the one thing that would change the answer: a vendor documenting that its crawler fetches the file, in the same way they document robots.txt behaviour today. Until that exists, llms.txt is a reasonable convention with no contract behind it, and the honest way to treat a convention with no contract is to spend accordingly.

Common questions

Does llms.txt help SEO?

No. Google's documentation on generative AI search states that Search ignores llms.txt and that creating one will neither harm nor help your visibility or rankings. It is not a ranking signal, it does not affect crawling of your pages, and no Google representative has suggested it will. Any SEO benefit you have read about is extrapolation, not documented behaviour.

Is llms.txt the same thing as robots.txt?

No, and the difference is the important part. Robots.txt controls which crawlers may fetch which URLs, the major AI crawlers publish their user agents and state that they obey it, so changing it changes what they do. An llms.txt file grants nothing and blocks nothing. It is a markdown index of pages you consider important, with no vendor committed to reading it.

Do I need an llms-full.txt file as well?

Not unless a specific tool you use asks for one. The llms.txt spec defines a single index file and does not specify a full-text variant, though some generators produce one by convention. Publishing a complete dump of your site as a second file costs you maintenance and gains you nothing documented, so skip it until something you rely on requires it.

Will adding llms.txt get my site cited by ChatGPT or Claude?

There is no evidence that it will. No major provider has committed to requesting the file, and the one published crawl study found that the large majority of llms.txt files are never fetched at all. Being quoted inside an AI answer is decided by what is on your pages and whether crawlers are allowed to read them, not by a file at your root.