llms.txt: Definition and Stakes for AI Visibility

Since Google said out loud that it does not use llms.txt, the file has become the most argued-about four kilobytes in SEO. One camp calls it the robots.txt of the AI era. The other calls it cargo cult. Both duck the useful question: what is in the file, who actually reads it, and what does publishing one buy you. Our agence GEO pillar and our entry on what GEO means give the wider picture.

The short version (TL;DR)

llms.txt is a Markdown file at /llms.txt giving AI models a curated map of your site: a summary plus grouped links to the pages that matter.
Proposed in September 2024 by Jeremy Howard of Answer.AI. It is a voluntary convention, not a web standard.
It is not robots.txt. robots.txt controls access and is enforced. llms.txt only suggests, and nobody is obliged to read it.
Google has said publicly it does not use it, comparing it to the old keywords meta tag.
Publish it if you want, but do not budget for it. Citations come from being retrievable, extractable and sourced, not from a file that describes you.

llms.txt: the one-sentence definition

llms.txt is a plain Markdown file placed at the root of a domain, at /llms.txt, that hands large language models a short curated map of the site: a title, a one-paragraph summary of what the site is, and grouped lists of links to the pages that matter most.

The proposal came from Jeremy Howard of Answer.AI in September 2024, and the reasoning behind it is sound. A model reading your site does not get the whole web. It gets a context window, a small budget of text, and your HTML burns that budget fast: navigation, cookie banners, footers, tracking scripts, none of which say anything about you. The pitch of llms.txt is that you spend one hour writing the clean version yourself, in Markdown, and point the model at it (llmstxt.org, 2024)^[1].

Notice what that pitch quietly assumes: that something on the other end bothers to fetch the file. Hold onto that, because it is the whole story.

What the file actually looks like

An llms.txt file is Markdown with a fixed shape: an H1 with the site or product name, a blockquote holding a one-paragraph summary, then H2 sections grouping links, each link followed by a short description of what it contains.

There is no XML, no schema, no validator. If you can write a README, you can write this. Here is the canonical shape, stripped to the bone:

# Acme Analytics

> Acme Analytics is a self-hosted product analytics tool for small
> engineering teams. Open source, privacy-first, no cookies.

## Docs

- [Quickstart](https://example.com/docs/quickstart): install and send
  a first event in five minutes.
- [API reference](https://example.com/docs/api): endpoints, auth, limits.

## Key pages

- [Pricing](https://example.com/pricing): plans and what each one includes.
- [Self-hosting guide](https://example.com/docs/self-host): Docker setup.

## Optional

- [Changelog](https://example.com/changelog): release history.

The canonical llms.txt shape: H1, blockquote summary, H2 groups, annotated links. The Optional section marks content a model may skip when its context budget is tight.

The spec also defines a bigger sibling, llms-full.txt, which concatenates the actual content of those pages into one Markdown file so a model can swallow everything without following a single link (Answer.AI, 2024)^[2]. The index is cheap to maintain by hand. The full version is not: it goes stale the moment you publish anything, so unless you generate it from your build pipeline, it will quietly start lying about your site.

The one that catches people out. An llms.txt file is written by you, about you, and nothing in it is independently verifiable. That is precisely the objection Google raised, and it is a fair one. A model that trusted your self-description over your actual pages would be trivially easy to game.

llms.txt vs robots.txt vs sitemap.xml

robots.txt is access control and it is enforced. sitemap.xml is discovery and search engines genuinely use it. llms.txt is curation, and reading it is entirely optional. Confusing the first with the third is the expensive mistake.

I have watched this go wrong on real sites. Someone reads that llms.txt is "the robots.txt for AI", writes a beautiful curated file, and assumes AI crawler access is now handled. It is not. Access is decided in robots.txt and at the infrastructure layer, and nowhere else. OpenAI documents its crawlers, GPTBot and OAI-SearchBot, and states plainly that they obey robots.txt directives (OpenAI, platform docs)^[6]. Cloudflare's AI Crawl Control does the same job one level lower, at the edge (Cloudflare, developer docs)^[7]. Neither reads llms.txt to decide anything. Side by side, the three files stop blurring:

File	What it is for	Who honours it
`robots.txt`	Allowing or blocking crawlers, including AI crawlers	Google, Bing, GPTBot, OAI-SearchBot and other declared bots
`sitemap.xml`	Listing your URLs so they get discovered and crawled	Search engines, actively and for years
`llms.txt`	Suggesting which content a model should read first	Nobody is obliged to. No major provider documents using it.

Read the third row again. That is not a criticism of the idea, which is genuinely elegant. It is a statement about the world the idea landed in.

Who actually reads it

As of July 2026, no major AI or search provider documents llms.txt as an input to retrieval or ranking. Google has publicly said it does not use it. OpenAI's crawler documentation describes robots.txt, not llms.txt. The file is read mostly by developer tools and by humans.

Google has been unusually blunt here, twice. John Mueller compared llms.txt to the keywords meta tag, the classic case of a site describing itself to a machine with no way for the machine to check (Search Engine Journal, 2025)^[4]. Gary Illyes confirmed that Google does not endorse the format (Search Engine Roundtable, 2025)^[5]. And Google's own documentation on how its AI features use the web runs to some length without mentioning llms.txt once (Google Search Central, 2025)^[3]. Absence in a document that thorough is not an oversight.

Where the file does earn its keep is narrower and less glamorous: developer documentation. If your product has docs and your users paste them into an assistant, a clean Markdown index is a real convenience, which is why so many documentation sites publish one. That is a genuine use case. It is just not a search visibility play, and the two keep getting sold as one.

Do the AIs cite you, or a competitor?

We run your real business questions through the AI assistants, record who gets cited instead of you, and hand back a clear diagnosis of why, across findability, extractability and credibility.

Get my GEO audit →

Should you publish one?

Publish one if it costs you an hour and you treat it as a lottery ticket, not an investment. It breaks nothing, it may pay off if the convention gains traction, and it is genuinely useful on documentation-heavy sites. Just do not move a single euro of budget away from content to fund it.

We publish one on cicero.studio, and I will tell you exactly why, because the reasoning is the useful part. The site carries 502 articles, 264 in French and 238 in English, according to Cicero Studio's internal data. At that volume, a model arriving cold has a genuine orientation problem: which of these 502 pages is the one that answers the question in front of it? A twenty-line index that says "here are the hubs, here is what each one covers" is a reasonable courtesy to leave at the door. It cost an hour. If it is ever read, good. If not, we lost an hour.

That is the entire honest case for it. Notice what is missing from it: any expectation of ranking, citation or traffic. Anyone selling you llms.txt as a visibility lever is selling you an hour of work at the price of a strategy.

What earns AI citations instead

Citations come from three things: retrievability, so an AI crawler can fetch and read the page; extractability, so each section opens with a self-contained answer a model can lift; and credibility, so claims carry named sources and figures. The research that coined the term GEO measured the third and found it lifted visibility in generative answers by roughly 40 percent.

That 40 percent figure deserves a moment, because it is the most actionable number in this entire field. The academic paper behind the term GEO, from a team including researchers at Princeton and the Allen Institute for AI, benchmarked what actually moves content into AI answers. Adding cited statistics and credible sources lifted visibility by around 40 percent. Keyword stuffing moved nothing (Aggarwal et al., arXiv, 2024)^[8]. The machine reader, it turns out, rewards exactly what a good editor always rewarded.

And the places where those citations are won are more specific than most teams expect. Across the 4836 French keywords Cicero Studio has analyzed, 34% get fewer than 100 monthly searches, so the long tail dominates (Cicero Studio internal data). Those are not scraps. They are the precise, high-intent questions someone types just before deciding, the exact phrasing an assistant answers in two sentences with one or two citations. No root-level file wins those. The page that answers the question cleanly, and sources it, does. Our guides on AI crawlers and invisible websites and machine-readable information architecture cover the retrievability and extractability halves in detail.

Alexis Dollé

CEO & Founder of Cicero Studio

I test AI visibility the slow way, by hand, one real business question at a time, across hundreds of sites. On llms.txt my position is boring and I am comfortable with it: publish the file, forget the file, then go and make your pages worth quoting. That second part is where every citation I have ever traced actually came from.

LinkedIn →

What llms.txt does not do

A definition is only as good as its edges, and this one has sharp edges that cost people money. Here are the four claims to refuse whenever you meet them.

Scope and common misreadings

It does not control AI crawler access. That is robots.txt and your edge configuration. An llms.txt file blocks nothing and permits nothing.
It does not get you into Google's AI features. Google has said publicly that it does not use the file, so treat any promise to the contrary as unsupported.
It does not replace a sitemap. Search engines actively consume sitemaps. Do not delete one to make room for the other.
It does not make thin content citable. A curated index pointing at pages a model cannot use is a well-organised list of pages a model cannot use.

Going further

We document our thinking in the open, because that is our best proof. Each resource below takes one thread further, from Google's stated position to the audit method behind it. Pick whichever matches your next question.

Frequently asked questions

What is llms.txt?

llms.txt is a plain Markdown file placed at the root of a domain, at /llms.txt, that hands large language models a short, curated map of a site: a title, a one-paragraph summary of what the site is, and grouped lists of links to the pages that matter most. It was proposed in September 2024 by Jeremy Howard of Answer.AI. It is a voluntary convention, not a web standard, and no search or AI provider is obliged to read it.

Is llms.txt the same as robots.txt?

No, and confusing the two is the most expensive mistake on this topic. robots.txt is an access-control file: it tells crawlers, including AI crawlers such as OpenAI's GPTBot and OAI-SearchBot, what they may and may not fetch, and those crawlers actually honour it. llms.txt is a curation file: it suggests what a model should read first. robots.txt is enforced, llms.txt is a suggestion. If you want to allow or block AI crawlers, you do it in robots.txt or at the infrastructure layer, never in llms.txt.

Does Google support llms.txt?

No. Google has said so repeatedly and publicly. John Mueller compared llms.txt to the old keywords meta tag, a file the site owner writes about itself that gives a model no independent way to verify anything, and Gary Illyes confirmed Google does not endorse the format. Google's own documentation on how its AI features use the web does not mention llms.txt anywhere. Treat any claim that llms.txt improves Google AI visibility as unsupported.

Should I create an llms.txt file for my site?

You can, as long as you understand what you are buying. It costs about an hour, it breaks nothing, and it is genuinely useful for documentation-heavy sites where a developer or an assistant needs a clean entry point. What it will not do is make an AI cite you. Publish it as cheap insurance if the convention gains traction, then spend the rest of your effort on the things that demonstrably drive citations: retrievable pages, extractable answers, and sourced claims.

What is the difference between llms.txt and llms-full.txt?

llms.txt is the short index: a summary of the site and a curated set of links, meant to be read in one pass without burning much of a model's context window. llms-full.txt is the maximal version: the actual content of those pages concatenated into a single Markdown file, so a model can ingest everything without following any links. The index is cheap to maintain, the full version is large and goes stale quickly unless you generate it automatically.

If llms.txt does not get me cited, what does?

Three things, in order. Retrievability, meaning an AI crawler can actually fetch and read the page. Extractability, meaning each section opens with a self-contained answer a model can lift without rewriting it. Credibility, meaning claims are backed by named sources and figures. The academic paper that coined the term GEO tested exactly this and found that adding cited statistics and credible sources lifted visibility inside generative answers by roughly 40 percent, while keyword stuffing did nothing. No root-level file substitutes for that.

Sources

The 8 references behind this page