What is an AI crawler?

Short answer

An AI crawler is a bot that fetches web pages on behalf of an AI system. Same idea as Googlebot, different job. Some crawlers grab content to train a model. Some build a search index that an assistant queries later. And some show up live, in real time, to read a page while the AI is mid-sentence writing you an answer.

Here's the thing most people miss: these are three different jobs, often done by three different bots from the same company. Treating them all as "the scraper" is how sites accidentally make themselves invisible.

The three jobs an AI crawler does

That middle row is the one that decides whether you get cited. The training row is the one everybody argues about online.

Why the difference actually matters

Say you read a hot take that "AI is stealing your content" and you block everything OpenAI in your robots.txt. Feels righteous. But you may have just blocked OAI-SearchBot, which builds the index ChatGPT search reads from.

Per OpenAI's bot documentation, sites that disable OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links," and the system updates within roughly 24 hours of a robots.txt change. So one rushed line can quietly pull you out of an engine where buyers are asking questions about exactly what you sell. Oof.

OpenAI splits the work cleanly: GPTBot is for training, OAI-SearchBot is for the search index, and ChatGPT-User is the user-triggered fetch (someone pastes your link or asks ChatGPT to go read it). You can allow search while blocking training, which is the combo a lot of publishers actually want. We walk through the exact rules in our guide to AI crawlers and robots.txt.

The crawlers worth knowing by name

• GPTBot (OpenAI): training. Block it and you opt out of training, not search.
• OAI-SearchBot (OpenAI): the ChatGPT search index. This is the citation one.
• ChatGPT-User (OpenAI): live, human-triggered page fetch. Not classic crawling.
• PerplexityBot (Perplexity): builds Perplexity's index.
• ClaudeBot (Anthropic): Anthropic's crawler.
• Google-Extended: a robots.txt toggle for Google's generative training, separate from regular Googlebot search.

Small honest caveat: bot names, user agents, and policies change. Treat this as a starting map, not gospel. Always confirm the current user-agent string from the provider's own docs before you write a rule, because a typo'd token blocks nothing (or blocks the wrong thing).

How crawlers connect to getting cited

A crawler reaching your page is step one. Getting picked as a source is the harder part, and it depends on stuff the crawler can read fast: clear headings, a real answer near the top, clean structure, facts it can lift without guessing. If you want the mechanics, how AI engines choose sources breaks down what tends to get pulled into an answer, and how to get cited by ChatGPT is the practical version.

One more lever worth a look: an llms.txt file is a proposed way to hand AI systems a tidy map of your best content. It is not a magic citation button (nobody's promising that), but it is low effort and it signals intent.

The trade-off nobody likes to say out loud

Blocking training crawlers can be a perfectly reasonable choice. Your content, your call. But understand the cost: there's overlap and confusion between "training" and "search" bots, and a sloppy block can cost you visibility in answers people are actually reading. So decide deliberately. Block training if you want. Just keep the search-index crawlers welcome unless you have a specific reason not to.

And then check whether it's working. That's the gap. You can allow every crawler on earth and still never get cited, because crawling is permission, not performance. AI Citation Monitor tracks how often you actually show up across ChatGPT, Perplexity, Gemini, and Google AI Overviews, with confidence intervals so you know a result is real and not noise, plus competitor share of voice and prescriptive fixes. There's a free instant check if you just want to see where you stand today. Crawlers get you in the door. Citations are the thing you can take to the bank.

FAQ