AI Crawler: Definition and Stakes for AI Visibility

In the news: Since July 2026, Cloudflare has begun blocking AI "training" and "agent" crawlers by default on ad-supported pages, giving AI companies until September 15, 2026 to separate their search crawlers from their training ones (The Cloudflare Blog, July 1, 2026).

Ask ChatGPT or Perplexity a question and it answers in a paragraph, sometimes with two or three sources named underneath. For your brand to be one of those sources, a bot first had to read the page where you say the thing. That bot has a name: the AI crawler. This page defines it plainly, sets it against the search crawler you already know, sorts the bots by the job they do, shows how to see and control them, and explains why being read has quietly become the price of being cited.

The AI crawler, defined in one line

An AI crawler is an automated bot that browses the web and reads the content of pages to feed an artificial-intelligence model or a generated answer. It is the AI-era cousin of the search-engine crawler, except it reads to help a model reply to a question rather than to rank a page in a list of blue links.

Take the two words on their own. Crawling is exploration: a bot follows links from page to page and downloads the content it finds along the way. The AI part names what the content is for, either training a model or grounding a live answer, instead of populating a search index. So an AI crawler is simply a web crawler pointed at a new destination, one where your pages become raw material for a generated reply.

What changes is the stakes, not the mechanics. A search crawler reads your page so Google can decide where to place it among ten results a human then scrolls. An AI crawler reads your page so a model can lift a sentence, rephrase it, and hand it to someone who asked a question, usually without any click coming back to you. If the crawler never read the page, your brand is simply not a candidate for that answer. This is why the AI crawler sits upstream of everything people now call generative engine optimization.

AI crawler vs search crawler

An AI crawler differs from Googlebot on four counts: its purpose (feeding an answer, not ranking a link), its number (dozens of bots instead of one), its technical behaviour (most read only raw HTML and do not run JavaScript), and its status (many sites now block or charge them). Together these make AI crawling far less predictable than the Google crawling you are used to.

Each gap has a concrete consequence, so it is worth taking them one at a time.

The purpose. Googlebot crawls to index and rank. An AI crawler crawls to answer, most often with no click sent back to the site it quotes. It is the same shift behind zero-click answers: you can be read without ever being visited.

The number. Where there was essentially one Googlebot, there is now a crowd: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot and its siblings for Anthropic, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, Bytespider and more. None of them coordinates with the others, and each consumes a slice of your server's patience.

The technical behaviour. Googlebot renders JavaScript and sees a page the way a browser does. Most AI crawlers, by contrast, read mainly the raw HTML the server returns. A page whose text only appears after scripts run risks being seen empty by an AI, even when it looks perfect to a human. It is a classic trap of JavaScript-framework sites, and it decides whether your content exists for a model at all.

The status. Nobody blocks Googlebot on purpose, on pain of vanishing from Google. For AI crawlers the debate is wide open, and the ground is moving under it: as the Cloudflare change above shows, access is becoming something a site grants, refuses, or even charges for, crawler by crawler. How many of your pages a given bot then gets through is a separate question, the one I cover in the companion page on the AI crawl budget.

The three jobs of an AI crawler

AI crawlers fall into three jobs: training crawlers that collect content to build a model, search or retrieval crawlers that fetch pages to ground a live answer, and user-triggered crawlers that grab a specific page the moment someone pastes its link into a chatbot. The same company often runs one bot for each, and blocking one has no effect on the others.

This split is the single most useful thing to understand, because it is where most mistakes happen. Sorting the bots by intent, rather than by brand, is what lets you protect what you want to protect without accidentally erasing yourself from AI answers.

Training crawlers. These read the open web to assemble the data a model learns from. GPTBot at OpenAI and Google-Extended for Gemini are the clearest examples. Blocking them keeps your content out of a training set, which some publishers want, but it does nothing to your presence in live answers.

Search and retrieval crawlers. These fetch pages so an engine can build an answer on the spot and cite its sources. OpenAI documents this split openly: per its crawler documentation, GPTBot trains the models while OAI-SearchBot powers ChatGPT's live search. Block the wrong one and you leave the training door open while shutting the answer door, or the reverse.

User-triggered fetchers. These act only when a person asks for a specific page, for example by pasting a URL into a chatbot. Anthropic makes the whole ladder explicit in its guidance on how it crawls and how to block it, running ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for those real-time, user-initiated fetches, each controllable on its own.

The AI crawlers reading your site

The AI crawlers worth knowing by name are GPTBot and OAI-SearchBot (OpenAI), ClaudeBot, Claude-SearchBot and Claude-User (Anthropic), PerplexityBot (Perplexity) and Google-Extended (Gemini training). Each announces itself with a user agent you can read in your server logs and allow or block in robots.txt.

The point of a table like this is not to memorise it, since the list changes every quarter, but to recognise the job behind each name when you meet it in your logs.

To recognise these strings for certain, go to the source: each operator publishes the exact user agents it uses, and Google's own overview of Google crawlers and fetchers, which now lists Google-Extended alongside Googlebot, is a good model of how clean that documentation should be. Reviewing client robots.txt files, I have watched a single misplaced name shut out a search crawler while a training one strolled straight through, so the exact spelling is not a detail.

How to see which ones visit you

You see AI crawlers in your server logs, which record every request with its user agent. Filtering by GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended and the rest shows which pages each bot visits, how often, and which it never touches. Logs are the most reliable source, because most AI engines offer no public tool comparable to Google Search Console.

This is the part most brands skip, and it is the part that turns the topic from theory into a decision. Your access logs already hold the answer: they list, line by line, who fetched what and when. Group those lines by user agent and a picture appears, which pages GPTBot returns to, which ones PerplexityBot ignores, whether a bot you meant to block is still getting in. No opinion, just the record.

I do this every week on client sites, and the pattern is almost always the same. The home page and a handful of top-level pages get read again and again, while the deep expertise pages, the ones that actually answer a precise question, barely register. Nobody decided to hide them. They simply sit too far down the tree, load too slowly, or fall under a robots.txt line written years ago for a different purpose, and no AI crawler ever reaches them.

How to let the right ones in

You control AI crawlers in the robots.txt file at the root of your site, allowing or forbidding each user agent by name, following the same standard that governs search crawlers. An emerging convention, the llms.txt file, adds a way to point AIs at the content you most want surfaced. The goal is not to block everything, but to let the answer-engine crawlers in while gating the pure training ones.

The mechanics are familiar, because they reuse the tool search crawlers have always obeyed: the robots.txt file, formalised by the IETF as RFC 9309, the Robots Exclusion Protocol. You add a block per user agent and set what it may read. What is new is the judgement each line now demands.

• Decide by job, not by brand. Allow the search and retrieval crawlers you want to be cited by, and gate the training-only ones if that matters to you. Because the bots are separate, blocking GPTBot leaves OAI-SearchBot untouched, and vice versa.
• Serve your content in HTML. If your key text only appears after JavaScript runs, most AI crawlers will see an empty page. Make sure your reference content is in the HTML the server returns.
• Signal your priorities. The llms.txt file, an emerging convention documented at llmstxt.org, lets you flag for AIs the pages you most want them to use. It is not yet universally honoured, but it costs little and states your intent clearly.
• Watch the defaults. As Cloudflare's July 2026 policy shows, your host or CDN may start blocking some AI crawlers for you, by default. Know what your infrastructure decides on your behalf before you assume your robots.txt has the last word.

Why AI crawlers decide your visibility

If an AI crawler never reads your key pages, your brand cannot be cited in generated answers. The AI crawler is therefore the entry condition for any visibility in ChatGPT or Google's AI answers: before you can be chosen as the answer, you first have to have been read.

The logic is unforgiving, and that is what makes it strategic. A model can only cite what it has seen. If your most useful pages, your guides, your expertise sheets, your service pages, are never crawled because they sit too deep, load too slowly, or are blocked by mistake, then they do not exist for the model. The best page in the world, never read, does nothing.

This also explains a pattern most content plans miss. Answer engines are sharpest on precise, specific questions, the long-tail queries a person actually types or dictates, and that is exactly the territory classic keyword planning writes off as too small. According to Cicero Studio's internal data, 34% of the French keywords Cicero Studio analyzed get fewer than 100 monthly searches, the long tail dominates. And across the 4283 French keywords Cicero Studio measured, the median volume is 260 searches per month. Those are not vanity terms; they are the granular questions an AI answer loves to resolve, and they live on the deep, specific pages an AI crawler is least likely to reach in the first place.

Where Cicero Studio fits

Cicero Studio treats the AI crawler as the first rung of visibility in generated answers: before chasing citations, we confirm you are being read. It starts with a free audit that measures which pages AI crawlers actually reach, then editorial production and automated semantic internal linking, run as one loop.

So this is not just theory, here is how we put it to work at Cicero Studio, in order: a GEO audit, then editorial production, then automated semantic internal linking, run as one loop rather than three disconnected services. Across the 1210 SEO/GEO audits produced by Cicero Studio, the same lesson keeps surfacing, that the biggest wins usually come from making the pages you already have readable and reachable, not from writing new ones no bot crawls.

That is the whole promise in one line: agency-quality work, software-grade productivity. If you prefer the French-language treatment of the model, our agence GEO pillar covers it in depth, and our English breakdown of an SEO and GEO audit shows exactly what we check.

The limits of the term

Let me be straight about what "AI crawler" does and does not cover, partly because that honesty is itself a signal AI answers reward, and partly because overselling the term is how it earns its skeptics.

The AI crawler matters most for a business with genuine expertise to publish and a site technically able to be read. For a page with nothing clear to say, no amount of crawl access conjures a citable answer from thin air. Underneath the new vocabulary, letting the right crawlers reach the right pages is still serious technical and editorial work, pointed at the content that actually matters.

Alexis Dollé

CEO & Founder of Cicero Studio

A growth specialist and content strategy consultant, I founded Cicero to help businesses build durable organic visibility, on Google as in AI answers. Day to day, I run our clients' audits and read their server logs to see which pages AI crawlers actually reach, because that is where visibility is won, before any citation. We put AI to work for production, never in place of expertise.

LinkedIn →

Resources to go further

We document our approach in the open, because published work with its sources beats any sales deck. The 508 articles published on cicero.studio (267 FR, 241 EN) are where we work through the answer-visibility puzzle in public. Each link below digs into one piece of it; several are in French, our home market, and are flagged as such.