Free AI Crawler and Robots.txt Checker

Key takeaways

• OAI-SearchBot is the one that bites. It builds the ChatGPT search index, and sites opted out are simply not shown in ChatGPT search, per OpenAI's bots documentation. Block it by accident and you vanish from one of the biggest answer engines.
• GPTBot and OAI-SearchBot are not the same bot. GPTBot is for training, ChatGPT-User is user-triggered (someone asks ChatGPT to open your link), and OAI-SearchBot is search indexing, all three named separately in OpenAI's docs. You can allow one and block another.
• ChatGPT web search leans on Bing's index, so Bingbot quietly matters too (OpenAI). Block Bingbot and you can knock yourself out of search results you never connected to ChatGPT.
• Google-Extended is Gemini's opt-out token, the control Google added so publishers can opt out of generative use in Gemini and Vertex AI (Google). It does not affect normal Google Search ranking, which trips up a lot of people.
• Bot access is the number one cause of AI invisibility. Loads of sites blanket-blocked AI bots back in 2023 and never noticed they also blocked the search crawlers that feed citations. The block is still sitting in robots.txt, quietly costing them.

How to use the AI Crawler and Robots.txt Checker

1. Type your domain into the input. Just the domain works fine (yoursite.com). You don't need to add /robots.txt yourself. The tool appends it and fetches the live file.
2. Hit check. It pulls your current robots.txt over the wire, parses the rules, and matches them against ten named AI crawlers. Takes a second or two.
3. Read the status table. Every bot gets a red or green pill: green means Allowed, red means Blocked. Scan for red where you didn't expect it.
4. Read the one-line verdict. Plain English, no parsing rules in your head. Something like "You're blocking OAI-SearchBot, which removes you from ChatGPT search." That's the headline.
5. Click through to the generator. If something's blocked that shouldn't be, the button drops you into the AI robots.txt generator so you can build a corrected file and paste it back into your site.
6. Re-run after you deploy. Push the new robots.txt, then check the domain again to confirm the pills turned green. (Caching can lag a few minutes, so give it a beat.)

What it checks

The tool fetches your live robots.txt and tests it against the specific user-agents AI engines actually send. Here's the bot lineup and why each one is on the list.

The OpenAI trio. GPTBot (training data collection), ChatGPT-User (fires when a user asks ChatGPT to fetch a page), and OAI-SearchBot (the ChatGPT search index crawler). These behave differently and you can allow or block each one separately. OAI-SearchBot is the one tied directly to whether you appear in ChatGPT search, so the checker flags it loudest.

The Anthropic pair. ClaudeBot and Claude-SearchBot. The tool reports their allow or block status the same way it reports the rest, so you get the full picture of your robots.txt in one table. (We track four engines today: ChatGPT, Perplexity, Gemini, and Google AI Overviews. These bots still show up in your file, so we show you their status anyway.)

Perplexity. PerplexityBot crawls and indexes content for Perplexity's answer engine. If you want Perplexity to cite you, this one needs to be green.

Google's generative token. Google-Extended is not a crawler in the usual sense. It's an opt-out token that governs whether your content is used for Gemini and Vertex AI generative features. Blocking it does not hurt your normal Google Search ranking, but it does pull you out of generative training and grounding.

Bing. Bingbot indexes for Bing search, and because ChatGPT's web search leans on Bing's index, blocking Bingbot can quietly cut your reach inside ChatGPT too. Most people never link those two facts.

Apple and Common Crawl. Applebot-Extended is Apple's generative opt-out token, and CCBot is Common Crawl, a public dataset that many AI models train on. The checker reports both so nothing in your file is a mystery.

Why this matters for AI citations

If an AI engine can't crawl your page, it can't cite your page. Simple as that. A glowing article that perfectly answers the question is invisible if your robots.txt slams the door on the crawler. This is the most boring failure mode in AI search and also the most common, which is a frustrating combination.

The OAI-SearchBot trap is the big one. OpenAI's own docs are blunt: OAI-SearchBot builds the ChatGPT search index, and sites that opt out are not shown in ChatGPT search (OpenAI). So a single Disallow line aimed at the wrong bot can erase you from a place millions of people now ask questions. And because GPTBot, ChatGPT-User, and OAI-SearchBot are listed as three distinct user-agents in OpenAI's documentation, it's easy to block the one you actually wanted to keep.

Then there's the Bing connection. ChatGPT's web search leans on Bing's index (OpenAI), so your Bingbot rule is now an AI search rule whether you meant it to be or not. Plenty of sites tightened up bot access in a panic during 2023, blanket-blocked anything that smelled like AI, and never circled back. The block is still there. They just never noticed it also took out the search crawlers feeding their citations.

Google-Extended is a gentler case but still worth understanding. It's the opt-out token for generative use in Gemini and Vertex AI (Google), and crucially it's separate from your search ranking. Some teams block it on purpose to keep content out of training, which is a legitimate choice. Others block it by copying a robots.txt snippet off a forum and have no idea they opted out of Gemini grounding. The checker just shows you the truth so the choice is actually yours.

Bot	Run by	What it does	Block it and...
OAI-SearchBot	OpenAI	Builds the ChatGPT search index	You're not shown in ChatGPT search
GPTBot	OpenAI	Collects training data	You opt out of OpenAI model training
ChatGPT-User	OpenAI	Fetches a page when a user asks	ChatGPT can't open links to you on request
PerplexityBot	Perplexity	Crawls for Perplexity's answer engine	Perplexity is less likely to cite you
Google-Extended	Google	Opt-out token for Gemini and Vertex AI	You opt out of Gemini generative use
Bingbot	Microsoft	Indexes for Bing search	You can lose reach in ChatGPT web search too
ClaudeBot	Anthropic	Crawls content for Anthropic	You opt out of that crawl
Claude-SearchBot	Anthropic	Search-oriented crawl	You opt out of that crawl
Applebot-Extended	Apple	Generative opt-out token	You opt out of Apple generative use
CCBot	Common Crawl	Builds a public training dataset	You're left out of Common Crawl

Here's the thing about robots.txt: it fails silently. There's no error page, no alert, no email from OpenAI saying "hey, we couldn't reach you." Your site loads fine in a browser, your analytics look normal, and meanwhile the search crawlers are bouncing off a Disallow line. You only find out when someone asks an AI a question you should obviously win, and you're nowhere in the answer. By then you've probably lost weeks. A thirty-second check closes that blind spot, which is the whole reason this tool exists.

For the bigger picture on how access fits into AI search, the explainer on AI crawlers and robots.txt walks through the whole thing, and the AI crawler glossary entry keeps the definitions handy.

Common mistakes

• Blocking OAI-SearchBot while trying to block GPTBot. People want out of training, grab a snippet, and accidentally disallow the search bot too. Now they're gone from ChatGPT search and never wanted that.
• Forgetting Bingbot is part of the AI story. A tidy SEO decision to limit Bing crawling can ripple into ChatGPT's web search results. Most people don't connect the dots.
• Assuming Google-Extended hurts Google ranking. It doesn't. It only governs Gemini and Vertex generative use. Blocking it has zero effect on your blue-link rankings.
• A stale 2023 blanket block. User-agent: * plus a broad Disallow from two years ago, still live, still quietly excluding the new search crawlers. Nobody re-checked.
• A typo in the user-agent name. OAI-Searchbot (wrong case or spelling) won't match. Bots are picky. The checker uses the exact names the engines send.
• Testing the wrong subdomain. robots.txt is per host. The file on www can differ from the apex, or from blog.yoursite.com. Check each host you publish on.
• Never re-checking after a deploy. You fixed it, pushed it, then assumed it stuck. Run the check again to be sure the pills went green.

FAQ

Is my site blocking AI crawlers right now?

Run your domain through the checker above and you'll know in seconds. It fetches your live robots.txt and shows a red or green pill for each AI bot. If you see red on OAI-SearchBot, PerplexityBot, or Bingbot, you're likely blocking crawlers that decide whether AI engines can cite you at all.

What's the difference between GPTBot and OAI-SearchBot?

They're separate bots with separate jobs. GPTBot collects training data, while OAI-SearchBot builds the ChatGPT search index, per OpenAI's bots docs. You can allow one and block the other. The risky move is blocking OAI-SearchBot by accident, because sites opted out of it are not shown in ChatGPT search.

Will blocking Google-Extended hurt my Google ranking?

No. Google-Extended is an opt-out token for generative use in Gemini and Vertex AI, and Google states it's separate from Search (Google). Your normal blue-link rankings stay unaffected. Blocking it only pulls your content from Gemini's generative features, a choice some publishers make on purpose.

Why does Bingbot show up in an AI crawler checker?

Because ChatGPT's web search leans on Bing's index (OpenAI). So your Bingbot rule is effectively an AI search rule. If you've blocked Bingbot for old SEO reasons, you may be quietly reducing your reach inside ChatGPT's web search without realizing the connection exists.

How do I fix a blocked crawler once I find one?

Click the button in the results that sends you to the AI robots.txt generator. Pick which bots to allow, copy the generated file, and replace your site's robots.txt with it. Deploy, wait a few minutes for caches to clear, then re-run this checker to confirm every pill turned green.

Does allowing these bots mean my content gets used for AI training?

Not necessarily. Training and search are different things handled by different bots. You can allow OAI-SearchBot (so you appear in ChatGPT search) while blocking GPTBot (so you opt out of training). The checker shows each bot separately precisely so you can make that distinction instead of an all-or-nothing call.

How often should I check my robots.txt?

Any time you migrate hosting, change CMS, add a subdomain, or copy a robots.txt snippet from somewhere. Those are the moments blocks sneak in. A quick monthly check is a reasonable habit. And always re-run it right after you deploy a fix to confirm the change actually landed.

Once your crawlers are green, the next question is whether the engines actually cite you. Start with the free instant check, then go deeper with the AI visibility checker to see where you stand across engines, and run the AI citation readiness grader to spot what's holding you back on the page itself. If you're still missing, the guide on how to get cited by ChatGPT and the breakdown of how AI engines choose sources cover the fixes beyond robots.txt, and the AI citation glossary entry keeps the terms straight.