AI Crawlers and robots.txt: The 2026 Field Guide

AI crawlers fall into three buckets: training bots, search and index bots, and user-triggered fetchers. The single most important move is to allow the search and index crawlers (OAI-SearchBot, PerplexityBot, Google-Extended for generative use) so you can actually get cited, and then make a separate, deliberate decision about the training bots like GPTBot. That's it. That's the whole game.

Most people get this wrong because they treat "AI crawler" as one thing and block all of it in a panic. Then they wonder why ChatGPT never mentions them. The bots are not interchangeable. One feeds model training, one feeds the live answer index, one fetches a single page when a user clicks a link. Block the wrong one and you delete yourself from the answer engines while leaving the training bot you actually cared about untouched.

So this is the field guide I wish existed in 2023. Copy-paste robots.txt blocks, a full user-agent table, the propagation gotchas nobody mentions, and how to verify the bots are getting clean 200s instead of quietly eating a 403 at your CDN. Let's get into the machinery.

Key takeaways

• There are three buckets of AI crawler: training, search/index, and user-triggered. They do different jobs and live under different user agents. Treating them as one thing is how people accidentally block themselves out of ChatGPT.
• Blocking GPTBot is a training opt-out, not a citation opt-out. Per OpenAI's bot documentation, ChatGPT search reads from OAI-SearchBot's index. Block OAI-SearchBot and you are not shown in ChatGPT search. Block GPTBot and you only skip training.
• Google-Extended is a separate generative opt-out token. According to Google's publisher controls update, Google-Extended governs Gemini and Vertex generative use, not Search indexing. Different control, different effect.
• ChatGPT web search leans on the Bing index, so Bingbot still matters. Per OpenAI's bot docs, if you block Bingbot you can quietly hurt ChatGPT's ability to find you. Old crawler, new relevance.
• robots.txt is not the only gate. Changes can take roughly 24 hours to propagate, and a WAF or CDN can return a 403 to bots before robots.txt ever applies. Verify with logs, not vibes.

The one move that matters

If you read nothing else: do not block the search and index crawlers if you want to be cited. That single sentence covers about 80% of the value on this page.

Here is the mental model. An answer engine like ChatGPT or Perplexity can only cite a page it has in its index. It builds that index with a specific crawler. If that crawler cannot fetch your page, your page does not exist as far as the engine is concerned, no matter how good your content is. You can write the best comparison post in your category and it will sit there invisible because a Disallow line three years ago told the wrong bot to go away.

So the priority order is simple. First, confirm the search and index crawlers can reach you and get a 200. Second, decide your stance on training bots, which is a real decision with real trade-offs but does not affect whether you get cited today. Third, handle the gotchas (CDN blocks, JS-only content, propagation) that quietly sabotage step one. If you want the broader picture of how engines pick what to cite once they can crawl you, we wrote a whole piece on how AI engines choose sources that pairs with this one.

The reason this move matters so much is that it is binary. Most GEO tactics move a probability: better structure raises your odds of being cited, schema nudges the entity graph, and so on. Crawler access is not a nudge. It is a gate. Either the bot gets in or it does not. There is no partial credit for a beautifully structured page the crawler can't read.

The three buckets, explained

Every AI crawler you care about does one of three jobs. Once you can name the bucket, the robots.txt decision basically writes itself.

Bucket one: training crawlers

Training crawlers collect content to train or fine-tune future models. The canonical example is GPTBot. When GPTBot fetches your page, that content can become part of the dataset for the next model generation. It is not fetching to answer a live query. It is fetching to learn, in bulk, for later.

Blocking a training crawler is a values and rights decision. Some publishers block it because they don't want their work feeding a model for free. Totally fair. The key thing to understand is that blocking a training bot does not remove you from the live answer engine, because the answer engine reads from a different index built by a different crawler. You are opting out of training, not out of citation.

Bucket two: search and index crawlers

Search and index crawlers build the index that the answer engine reads when it responds to a user right now. OAI-SearchBot (ChatGPT search), PerplexityBot (Perplexity's index), and Google's regular crawling pipeline all live here. This is the bucket that decides whether you can be cited at all.

These are the bots you want to allow, full stop, if AI visibility matters to you. Per OpenAI's documentation, opting OAI-SearchBot out means you are not shown in ChatGPT search. That is the line that should make you check your robots.txt right now. If you are trying to fix a specific engine, our guides on how to get cited by ChatGPT and how to rank on Perplexity assume these crawlers can actually reach you first.

Bucket three: user-triggered fetchers

User-triggered fetchers grab a single page in real time because a user did something, like pasting a URL into ChatGPT or clicking a citation. ChatGPT-User is the OpenAI example. These bots are not building a persistent index and, per OpenAI, are not used for search ranking. They fetch one page, use it for that one response, and move on.

You generally want to allow these too, because they power the live "go read this URL" experience your readers trigger. Blocking them mostly just breaks a feature your visitors are actively using. Low risk to allow, mild annoyance to block.

The full crawler reference table

Here is the part you came for. This is the copy-paste reference: which user agent, who owns it, what it does, and whether you should block or allow it if your goal is to be cited.

A few notes the table can't hold. Per OpenAI's bot docs, ChatGPT web search leans on the Bing index, which is why Bingbot belongs on an AI crawler list even though it is older than half the bots above it. If you block Bingbot to save crawl budget, you may be quietly clipping ChatGPT's ability to find you. PerplexityBot and ClaudeBot are documented as index and collection crawlers in public crawler directories like HUMAN Security's known-bots guide, which is a decent place to sanity-check a user agent you don't recognize.

Google-Extended is the odd one out because it is not really a crawler with its own fetching behavior in the way GPTBot is. It is a control token. We'll come back to that, because it confuses people constantly.

The classic mistake: the 2023 blanket GPTBot block

Here is the screw-up I have seen on dozens of sites. Back in 2023, when GPTBot first showed up, a lot of teams added a blanket block. Often it was broader than intended, something that swept up everything OpenAI-shaped. The intent was reasonable: "we don't want to train OpenAI's models for free." The result was not.

Then OAI-SearchBot launched as a separate crawler for ChatGPT search. Sites that had written an overly broad rule, or that later added OAI-SearchBot to the same block out of habit, found themselves excluded from ChatGPT search results without realizing it. They thought they were making a training decision. They had actually made a visibility decision. And nobody got an email about it.

This is why the bucket model matters so much. A 2023-era "block the AI bots" rule made sense when there was basically one bot and one purpose. In 2026 there are several bots with conflicting purposes under one vendor, and a rule that blocks the training crawler is fine while a rule that blocks the search crawler quietly deletes you from the answer engine. Same vendor. Opposite outcomes. If your brand has gone missing and you can't figure out why, start here, then read why your brand is not showing up in ChatGPT for the rest of the checklist.

The fix is almost always to audit your existing robots.txt line by line and ask, for each Disallow, "which bucket is this bot in, and did I mean to do that?" Most of the damage I find is accidental, inherited from a copy-pasted block somebody added in a hurry two years ago.

A clean copy-paste robots.txt for "cite me, do not train on me"

This is the most common stance I see from publishers in 2026: they are happy to be cited and recommended, but they do not want their content used to train models for free. That is a coherent position, and robots.txt can express it. Here is a starting point.

# Allow the search and index crawlers (so you can be cited)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Bingbot
Allow: /

# Opt out of model training only
User-agent: GPTBot
Disallow: /

# Opt out of Google's generative training use (keeps Search indexing)
User-agent: Google-Extended
Disallow: /

# Everyone else
User-agent: *
Allow: /

Read what that actually says. ChatGPT search can index you (OAI-SearchBot allowed). Perplexity can index you. Bing, which ChatGPT web search leans on, can crawl you. But GPTBot, the training crawler, is told to stay out, and Google-Extended is told you do not want your content used for Gemini and Vertex generative training. You stay citable while opting out of the training you object to.

Two honest caveats. First, this is a stance, not a law of physics. A Disallow is a request that well-behaved crawlers honor; it is not a firewall. Second, if your real goal is maximum AI visibility and you have no strong feelings about training, the simplest correct robots.txt is to allow everything and stop overthinking it. Blocking training bots is a values call, and it is fine to decide you don't have one. If you want the bigger strategic frame around all of this, our AI SEO guide and the llms.txt walkthrough cover the layers that sit on top of crawler access.

A stricter variant: no training, anywhere

If you want to block every training crawler you can name and still stay citable, extend the block list. Add the training bots, leave the search bots alone, and keep your User-agent: * permissive so you don't accidentally break a search crawler you forgot to whitelist. The order matters less than the specificity: a named user-agent block takes priority over the wildcard for that bot, so list the search crawlers explicitly with Allow: / and you are protected against your own wildcard.

The gotchas that quietly break everything

Here is where robots.txt stops being the whole story. You can write a perfect file and still be invisible, because three other things sit between your intent and the crawler's experience. I have watched all three burn real sites.

Propagation is not instant (~24 hours)

Per OpenAI's bot documentation, it can take roughly 24 hours for robots.txt changes to be reflected in OpenAI's systems. Other crawlers cache your file on their own cadence. So if you edit robots.txt at 3pm and check for new behavior at 3:05, you have measured nothing. Make the change, confirm the file serves a clean 200, then wait a day and read your logs. Patience is part of the protocol.

A WAF or CDN 403 beats your robots.txt

This one is the silent killer. Your robots.txt can say Allow: / all day, but if a WAF or CDN in front of your origin returns a 403 or a bot challenge to that user agent, the crawler never even reads your robots.txt. The edge layer decides first. Cloudflare, AWS WAF, and similar services ship bot-management rules that block or challenge crawler user agents by default, and those rules do not know or care what your robots.txt says.

So if a bot is "allowed" in robots.txt but still not showing up, check the edge before you touch the robots file again. In your CDN dashboard, look for bot-fight or managed-challenge rules, and explicitly allow the AI search user agents. This is, in my experience, the number one reason an "allowed" crawler still can't reach a site. The robots.txt was never the problem.

JavaScript-only content is a coin flip

If your content only renders after client-side JavaScript runs, you are betting that each crawler executes JS the way a full browser does. Some do a decent job, some do not, and the ones that do not will see an empty shell. The safe move is server-side rendering or static generation for any content you want cited, so the crawler gets real HTML on the first fetch. This overlaps heavily with classic technical SEO, and our entity SEO and schema markup for AI search pieces go deeper on making your content machine-readable once the crawler is actually in the door.

The throughline across all three gotchas: robots.txt is necessary but not sufficient. It grants permission. It does not guarantee delivery.

How Google-Extended and llms.txt fit

Two things that get lumped in with robots.txt but do slightly different jobs. Worth pinning down so you stop misusing them.

Google-Extended is a generative opt-out, not a Search block

Per Google's publisher controls announcement, Google-Extended is a token you use in robots.txt to opt out of having your content used for Google's generative AI products, specifically things like Gemini and Vertex AI. The crucial detail: it is separate from Google Search indexing. Blocking Google-Extended does not remove you from Google Search, and it does not remove you from the regular crawling that feeds AI Overviews surfacing. It only governs generative model use.

So the decision is the same shape as GPTBot. If you object to generative training use, set User-agent: Google-Extended with Disallow: / and you keep your Search presence. If you want maximum reach into Google's AI surfaces, leave it allowed. People panic that Google-Extended will tank their rankings; it won't, because it is not a ranking control at all. For the Google side specifically, how to get cited by Gemini covers the surfaces this token does and doesn't touch.

llms.txt is curation, not access control

llms.txt is a different layer entirely. robots.txt controls what crawlers may fetch. llms.txt is a curated Markdown map that tells models which of your pages are worth reading. One is a permission system, the other is a recommendation. They are complementary, not interchangeable, and you can run both at once. We wrote a full llms.txt guide that covers what it is, how to write one, and the honest data on whether anything reads it yet (short version: adoption is high, usage is not proven). Do not expect llms.txt to fix a crawler that robots.txt or your CDN is blocking. Access first, curation second.

How to verify crawlers are getting 200s

You don't get to assume your robots.txt works. You verify it. And the verification is gloriously low-tech: read your logs and look at the status codes the AI crawlers are getting.

Grep your server logs by user agent

Pull your access logs and filter for the user agents from the table above. You are looking for two things: are the search crawlers showing up at all, and when they do, are they getting a 200? A 200 means the bot fetched your page successfully. A 403 means something (probably your WAF) blocked it. A 404 means the URL is wrong. A 503 means your server choked. Anything that isn't a 200 on a page you want cited is a problem to chase.

# Example: find OAI-SearchBot hits and their status codes
grep "OAI-SearchBot" access.log | awk '{print $9}' | sort | uniq -c

# See which AI crawlers hit you at all, by volume
grep -Ei "OAI-SearchBot|PerplexityBot|GPTBot|Bingbot|ClaudeBot" access.log \
  | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

The first command counts the HTTP status codes OAI-SearchBot received. If you see a pile of 403s, your robots.txt is irrelevant; your edge layer is the gate, go fix that. The second command shows you which AI crawlers are actually visiting and how often, which is a sanity check on whether your "allow" rules are doing anything in the real world.

Confirm robots.txt serves cleanly

Fetch your own robots.txt the way a bot would and confirm it returns a 200 with the rules you expect. A robots.txt that 404s, or that your CDN serves from a stale cache, will cause crawler behavior you can't explain from reading the file in your editor. What you wrote and what the bot receives are two different things, and only one of them matters.

# Confirm your robots.txt actually serves a 200 with the expected rules
curl -sI https://yoursite.com/robots.txt | head -n 1
curl -s https://yoursite.com/robots.txt

Measure the outcome, not just the access

Logs tell you the crawler reached you. They do not tell you whether you actually get cited, because being crawled is necessary but not sufficient. To close that loop you need to track whether the engines mention or recommend you in real answers, with competitor share of voice and confidence intervals so you know a result is real and not noise. That is exactly what AI Citation Monitor does: it runs the prompts across ChatGPT, Perplexity, Gemini, and Google AI Overviews and tells you whether you showed up. There's a free instant check if you want to see your current standing before you touch a single robots.txt line. For the measurement concept itself, our AI citation glossary entry defines what counts as a citation versus a passing mention.

Crawl access gets you eligible. Tracking tells you whether eligibility turned into actual visibility. Do both, in that order.

Putting it together

The whole field guide compresses to a short checklist. Allow the search and index crawlers (OAI-SearchBot, PerplexityBot, Bingbot) without exception if you want to be cited. Make a deliberate, separate decision on the training bots (GPTBot, Google-Extended) based on your values, knowing it does not affect citation today. Then verify the chain end to end: robots.txt serves a 200, the edge layer isn't returning a 403, your content renders without requiring JS gymnastics, and the crawlers show up in your logs with 200s.

Do that and you have done the unglamorous 80% that pays for itself. The fancy stuff (schema, entity work, content structure) only pays off once the crawler can actually get in the door. Access is the foundation everything else stands on, and it is the part most teams skip while they argue about heading structure. Get the door open first.

FAQ

Should I block GPTBot in robots.txt?

Blocking GPTBot only opts you out of OpenAI's model training. It does not stop ChatGPT from citing you, because ChatGPT search uses a different crawler called OAI-SearchBot. So if your goal is to keep your content out of training data, block GPTBot. If your goal is to be cited, leave OAI-SearchBot alone. The mistake is blocking both by accident.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot is OpenAI's training crawler, the one that collects data to improve future models. OAI-SearchBot is the crawler that builds the index ChatGPT search reads from when it answers questions. Per OpenAI's bot docs, if you block OAI-SearchBot you will not show up in ChatGPT search results. They are two different jobs with two different user agents, and conflating them is the single most common robots.txt mistake of the last two years.

How long does it take robots.txt changes to take effect for AI crawlers?

OpenAI says it can take roughly 24 hours for robots.txt changes to be reflected in its systems. Other crawlers cache their own copy of your file on their own schedule, so do not expect instant results. Make the edit, confirm it serves a clean 200, then check your server logs over the next day or two to see the new behavior land.

Does blocking AI crawlers hurt my SEO or AI visibility?

Blocking the search and index crawlers (OAI-SearchBot, PerplexityBot, Google-Extended for generative use) directly removes you from those answer engines. Blocking the training-only crawlers (GPTBot) does not affect whether you get cited today, it only affects model training. So a blanket block hurts visibility, while a targeted training opt-out usually does not. Know which bucket each bot is in before you write a single Disallow line.

Why is my robots.txt allowing a crawler but it still cannot reach my site?

Your robots.txt is only one layer. A WAF or CDN sitting in front of your origin can return a 403 or a challenge page to bot user agents before robots.txt ever matters. Cloudflare, for example, has bot-blocking rules that fire independent of your robots.txt file. Check your server and edge logs for the crawler's user agent and confirm it is getting a 200, not a 403 or a JavaScript challenge.

What is Google-Extended and do I need it?

Google-Extended is a control token that lets you opt out of having your content used for Google's generative AI products like Gemini and Vertex AI. It is separate from regular Search indexing, so blocking Google-Extended does not remove you from Google Search or AI Overviews crawling. It only governs generative model use. If you want maximum AI visibility, leave it allowed.