robots.txt for AI: Definition and Stakes for AI Visibility

Well set, your robots.txt protects what you want to keep while letting through what makes you citable. Set wrong, it makes you invisible in AI answers without you ever knowing. This page gives a plain definition of robots.txt for AI, the syntax you need, the crawlers to know, and how to decide what you allow.

robots.txt for AI, defined in one line

robots.txt for AI is not a separate file: it is the same robots.txt placed at the root of a site, but we speak of robots.txt for AI once you add rules that target artificial-intelligence crawlers. Each AI crawler carries a user-agent name that you allow or refuse line by line. The file therefore becomes the place where the site declares what the AIs are allowed to read.

robots.txt has existed since the 1990s. It is a text file that applies the Robots Exclusion Standard, a good-conduct protocol by which a site tells crawlers which parts it lets them walk. For twenty-five years it mostly served to manage Googlebot and the crawlers of the classic search engines.

What changed is the arrival of AI crawlers. When OpenAI launched GPTBot, then when Google introduced Google-Extended, these new agents also began reading robots.txt before collecting content. The file did not change in nature; new agent names were simply added for it to recognise. That is the whole point of "robots.txt for AI": using an old tool to answer a new question, that of AI access to your content. It sits one step upstream of the AI crawl budget, the question of how much of your site a crawler then actually reads.

How AI crawlers read your robots.txt

An AI crawler first requests the file at your-site.com/robots.txt, at the root of the domain. It looks there for a block carrying its user-agent name. If it finds one, it follows the Allow and Disallow directives that concern it. If not, it applies the generic rules or crawls freely.

The mechanism is deliberately simple. Before it walks your pages, the crawler fetches the file from one fixed location: the root of your domain. Google's documentation on robots.txt is explicit that the file must sit at exactly that spot, failing which it is ignored. A robots.txt tucked into a subfolder does nothing.

Once the file is fetched, the crawler reads it top to bottom and finds the block that carries its name. Directives apply only to the crawler named in the User-agent line just above them. That is what lets you write, in the same file, one rule for Googlebot, then another for GPTBot, then a third for ClaudeBot, without them mixing. Knowing exactly which agent does what is a whole topic of its own, which our definition of an AI crawler unpacks in detail.

The syntax of User-agent directives for AI

A robots.txt block starts with a User-agent line naming the crawler, followed by one or more Allow or Disallow lines. To block GPTBot, you write User-agent: GPTBot then Disallow: / on the next line. To allow it everywhere, you replace that with an empty Disallow: or Allow: /.

The grammar comes down to three keywords. User-agent names the crawler concerned. Disallow forbids a path. Allow permits a path, often as an exception to a wider Disallow. A single forward slash, /, means the whole site. Here is what a block that fully refuses OpenAI's training crawler looks like.

User-agent: GPTBot
Disallow: /

Conversely, to let a crawler reach the entire site, you give it a block with an empty Disallow, which means "nothing is forbidden".

User-agent: PerplexityBot
Disallow:

You can also forbid only part of the site, a private area for instance, while leaving the rest open. The directive then targets a specific folder rather than the root.

User-agent: ClaudeBot
Disallow: /account/
Disallow: /cart/

Each block is independent. You can stack as many as you have crawlers to manage, and end with a User-agent: * block that applies to every other unnamed crawler. The order of the blocks does not matter; it is the agent name that decides which rule applies to whom.

Blocking training is not blocking AI search

AI crawlers do two very different jobs. Some collect content to train the models, others explore it to ground live answers in ChatGPT or Perplexity. Blocking the first protects your content without hurting your visibility; blocking the second cuts you out of citations in AI answers.

This is the most important distinction in the whole guide, and the one most often misread. Confusing the two roles is how you shoot yourself in the foot: you think you are "protecting your content" and you simply make yourself invisible.

On the training side, OpenAI documents GPTBot as the crawler that fetches content to improve future models. Google, for its part, introduced Google-Extended in 2023: on its official blog, the company presents this token as a way to manage whether a site "helps improve Google's generative AI models", independently of its ranking in Search. Blocking those two crawlers protects your content from a training use you did not choose. That is defensible, and it does not make you disappear from AI answers.

On the search side, the role is entirely different. OpenAI documents OAI-SearchBot as the crawler that indexes sites for search inside ChatGPT, and Perplexity documents PerplexityBot as the one that references pages for its answers. Those crawlers decide your presence in generative answers. Blocking them means refusing to be cited.

The AI crawlers to know for your robots.txt

The crawlers you meet most often come from OpenAI (GPTBot for training, OAI-SearchBot for search), Google (Google-Extended), Perplexity (PerplexityBot) and Anthropic (ClaudeBot). CCBot, the Common Crawl agent, is added to the list. Each plays a precise role depending on whether it serves training or search, a role declared by its operator.

Here are the agents you run into most. For each, its operator, then the role that decides the effect of a block.

Two cases deserve a note. Google-Extended is not a crawler in the strict sense: its documentation makes clear it sends no extra request, it is only a token read in robots.txt that tells Google whether it may use your content for its models. You can therefore stay perfectly indexed on Google while refusing to let your content feed Gemini. CCBot, the Common Crawl agent, archives the web openly and publishes its data freely; many models train on those archives. Blocking GPTBot without thinking of CCBot leaves a door ajar for training.

How to decide what you allow

The decision depends on your goal. If you sell access to your content, blocking training protects your asset. If you want reach, let the AI-search crawlers through, since they decide whether to cite you. The middle path refuses training and allows search.

Three typical profiles cover most situations.

• You sell access to your content (publishers, paid databases, proprietary research). Blocking the training crawlers, GPTBot as well as Google-Extended and CCBot, protects your asset without costing you organic visibility. A defensible, deliberate stance.
• You want to build reach. Letting the AI-search crawlers through, OAI-SearchBot and PerplexityBot first, is in your interest: they are the ones that decide whether to cite you in answers. Blocking them makes you invisible exactly where your prospects now search.
• You are unsure. A middle path exists: refuse training, allow AI search. You keep control of your content without sacrificing your citability.

That middle path translates into a few blocks in your robots.txt. The example below refuses training to the three crawlers concerned, from GPTBot to CCBot by way of Google-Extended, while letting the AI-search crawlers reach the site.

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Disallow:

User-agent: PerplexityBot
Disallow:

Adapt it to your strategy, and check each operator's official documentation, because agent names change and new crawlers appear. A robots.txt is never frozen: you reread it every time the AI ecosystem shifts.

What robots.txt does not do

robots.txt is an instruction, not a lock. A crawler can ignore it. It does not delete content already collected, it does not protect personal data on its own, and it has no effect on human visitors or on copies of your content hosted elsewhere.

You have to be clear-eyed about the tool's reach. The Robots Exclusion Standard, as described by robotstxt.org and later formalised as an IETF standard in RFC 9309, is a good-conduct protocol: it works because serious crawlers choose to respect it, not because it would stop them. A malicious crawler simply ignores it.

robots.txt has no retroactive effect either. If your content was already scraped before you set a directive, blocking it today does not pull it out of existing training corpora. The rule holds for the future.

Finally, on personal data, robots.txt is not enough. When a site carries information about people, collection by AI crawlers raises compliance questions. France's data-protection regulator, the CNIL, has published recommendations on bringing AI systems into line with the GDPR, in particular on the legal basis for collecting web content. robots.txt is a useful signal, but it replaces neither server-level protection nor a compliance process.

What robots.txt changes for your AI visibility

Setting your robots.txt right is not an end in itself. It is the first rung of a strategy for visibility in generative answers, the one that decides whether the race even starts. On the sites I audit at Cicero Studio, we go through the server logs page by page, and I have already seen brands made invisible in ChatGPT because a hastily copied robots.txt blocked the AI-search crawler, with nobody meaning to. That case is not rare, and it lines up with a wider shift: automated bots now make up more of web traffic than humans, as our analysis of AI crawler behaviour lays out. Three consequences follow.

1. Access comes before citation

A crawler can only cite what it has been able to read. If your robots.txt refuses the AI-search crawlers by mistake, the quality of your content no longer counts: you are out of the running upstream. The first requirement is mechanical, it is access for the right crawlers. This is also why the notion of AI crawl budget matters, because allowed access still has to be efficient access.

That upstream question is exactly where planning goes wrong most often. Answer engines are sharpest on precise, specific questions, the long-tail queries a person actually types or dictates, and those are the pages most likely to be starved of crawl access in the first place. According to Cicero Studio's internal data, 34% of the French keywords Cicero Studio analyzed get fewer than 100 monthly searches, the long tail dominates. And across the 4283 French keywords Cicero Studio measured, the median volume is 260 searches per month. Those are not vanity terms; they are the granular questions an AI answer loves to resolve, and they live on the deep pages a careless robots.txt is most likely to fence off.

2. robots.txt complements, it does not replace

Allowing the right crawlers is necessary but not sufficient. A crawler reaches the page, and then it still has to find content that is easy to extract and reuse. An llms.txt file can guide AIs toward your key pages, and a page built to be lifted does the rest. robots.txt handles the door; the structure of the content handles the welcome. Being the answer, rather than one blue link among ten, is the subject of answer engine optimization.

3. Citation replaces ranking

In an AI answer there are no longer ten blue links, but a synthesised answer resting on a handful of cited sources. The question is no longer "am I first?" but "am I one of the sources judged worthy of grounding the answer?". That is the object of GEO, Generative Engine Optimization: preparing content to be reused by AI engines rather than only ranked by Google. Our page on getting cited by ChatGPT takes the idea onto the conversational surface.

Where Cicero Studio fits

Cicero Studio treats robots.txt as the first gate of visibility in generative answers: before chasing citations, we confirm the right crawlers can read you. It starts with a free audit that reads your server logs and access rules, then editorial production and automated semantic internal linking, run as one loop.

So this is not just theory, here is how we put it to work at Cicero Studio, in order: a GEO audit, then editorial production, then automated semantic internal linking, run as one loop rather than three disconnected services. Across the 1210 SEO/GEO audits produced by Cicero Studio, the same lesson keeps surfacing, that the biggest wins usually come from making pages you already have reachable and readable, not from writing new ones no crawler reaches.

That is the whole promise in one line: agency-quality work, software-grade productivity. If you prefer the French-language treatment of the model, our agence GEO pillar covers it in depth, and our English breakdown of an SEO and GEO audit shows exactly what we check.

Alexis Dollé

CEO & Founder

Growth and SEO content strategist, I founded Cicéro to help businesses build lasting organic visibility, on Google and in AI-generated answers alike. Day to day, I run our clients' audits and read their server logs to see which pages AI crawlers actually reach, because that is where visibility is won, before any citation. We put AI to work for production, never in place of expertise.

LinkedIn →

Resources to go further

We document our approach in the open, because published work with its sources beats any sales deck. The 508 articles published on cicero.studio (267 FR, 241 EN) are where we work through the answer-visibility puzzle in public. Each link below digs into one piece of it; the pillar is in French, our home market, and is flagged as such.