Free AI Robots.txt Generator

Key takeaways

• There's a real difference between getting cited and getting trained on. OpenAI runs separate bots for each: OAI-SearchBot powers ChatGPT search visibility, GPTBot is training only, and ChatGPT-User is user-triggered (OpenAI). Block the wrong one and you vanish from ChatGPT answers.
• Google-Extended controls whether Gemini and Vertex use your content for generative answers, and it's separate from your normal Search indexing (Google). So you can stay in Search and still opt out of generative training.
• A robots.txt change isn't instant. OpenAI takes about 24 hours to reflect an edit, and a CDN or WAF block can override even a clean robots.txt (OpenAI). Wait a day before you panic.
• The tool covers 10 bots in one block: OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, ClaudeBot, Claude-SearchBot, Google-Extended, Bingbot, Applebot-Extended, and CCBot. No memorizing user-agent strings.
• If you want AI search to find and quote you, blocking everything is usually the wrong move. See our guide on how to get cited by ChatGPT for the full play.

How to use the AI robots.txt generator

1. Start with a preset. Choose "cite me but do not train on me," "allow all," or "block all AI." The preset flips every toggle for you, and you can tweak from there.
2. Adjust per-bot toggles. Each AI bot gets its own Allow or Disallow switch. Want Perplexity in but Common Crawl out? Flip exactly those two.
3. Read the little "what each bot does" table next to the toggles. It tells you whether a bot is for search, training, or user-triggered fetches before you decide.
4. Watch the code block update live. Every toggle you flip rewrites the robots.txt instantly, so what you see is what you'll ship.
5. Hit the copy button. The full block lands on your clipboard, formatted and ready.
6. Paste it into the robots.txt at your domain root (yourdomain.com/robots.txt), then verify with the AI crawler robots.txt checker.

What it checks (and what each toggle actually controls)

Here's the thing most people miss. "AI crawler" is not one thing. There are search bots that fetch a page so an engine can cite it, training bots that scrape text to feed a model, and user-triggered bots that grab a single URL because a human asked a chatbot about it. The generator treats all three honestly so you don't accidentally trade citations for "privacy" you didn't need.

OAI-SearchBot is the one that decides whether ChatGPT can show and link your pages in its search results. If you Disallow it, you opt out of ChatGPT search visibility entirely (OpenAI). Most brands want this one allowed.

GPTBot is training only. Disallowing it tells OpenAI not to use your pages to train future models. It does not affect whether ChatGPT can cite you in search. This is the toggle for the "do not train on me" crowd.

ChatGPT-User fires when a person inside ChatGPT clicks through or asks the assistant to fetch your specific page. It's a live, user-triggered request, not a bulk scrape. Block it and you break that on-demand fetch for real users.

Google-Extended is the publisher control for Gemini and Vertex generative use, and Google built it to be separate from Search indexing (Google). Turn it off and you can keep ranking in Google Search while opting out of generative training. They don't have to move together.

PerplexityBot is how Perplexity discovers and cites your content. If Perplexity is a channel you care about, leave it on. Applebot-Extended governs Apple's generative AI training, separate from the regular Applebot that powers Siri and Spotlight. CCBot is Common Crawl, the open dataset a lot of models train from secondhand, so blocking it is a broad "keep my text out of training corpora" move.

ClaudeBot and Claude-SearchBot are Anthropic's crawlers (one leans toward training, one toward search-style fetches), and Bingbot is Microsoft's crawler that also feeds Bing's AI features. The generator emits clean Allow or Disallow lines for each, no guesswork on the exact user-agent string.

One more thing the generator gets right: order and grouping. robots.txt matches the most specific user-agent block, so a sloppy file with overlapping wildcards can silently undo a rule you thought you set. The output groups each bot under its own User-agent line with a paired Allow or Disallow, which keeps precedence predictable. You won't get a generic User-agent: * accidentally swallowing a careful per-bot decision.

And it stays copy-paste safe. The block is plain text with no smart quotes, no trailing junk, and the exact casing each crawler expects. That sounds trivial until a stray curly quote or a lowercase token makes a bot ignore your rule. If you want the deeper mechanics behind directives and precedence, the generative engine optimization guide and the generative engine optimization glossary entry both cover how engines weigh access alongside content quality.

Why this matters for AI citations

If you want to show up when someone asks ChatGPT, Perplexity, or Gemini a question, the AI engine has to be allowed to read your page first. robots.txt is the front gate. Slam it shut on the search bots and you're invisible, no matter how good your content is.

The trap is the blunt "block all AI" instinct. People read a scary headline, paste a Disallow-everything block, and quietly delete themselves from AI search. The nuance: OAI-SearchBot opt-out means no ChatGPT search visibility, while GPTBot is training only and ChatGPT-User is user-triggered (OpenAI). Those are three different decisions, not one.

Google made the same separation deliberate. Google-Extended controls Gemini and Vertex generative use and is independent from Search indexing (Google). You can protect your content from generative training and still rank normally. That's a feature, not a bug.

And give it time. After a robots.txt change OpenAI takes about 24 hours to reflect it, and a CDN or WAF block can override a perfectly clean robots.txt (OpenAI). So if a bot still shows up tomorrow, check your edge config before you blame the file. (This catches more people than you'd think.)

Here's the part teams skip. Allowing a bot is permission, not a guarantee. Letting OAI-SearchBot and PerplexityBot through means the engines can read you, but it doesn't make them pick you over a competitor. Access is the floor. After you set the gate, the real question is whether those engines actually cite you, and that's a measurement job, not a robots.txt job. Generating the file and walking away is like propping the front door open and assuming guests will show up. You still have to be worth quoting, and you still have to check whether the quotes are happening. That's exactly the gap our platform closes, and it's why we built the AI visibility checker to sit downstream of this one.

Bot / token	What it's for	Disallow it and...
OAI-SearchBot	ChatGPT search and citations	You drop out of ChatGPT search visibility
GPTBot	OpenAI model training	Your pages aren't used to train new models
ChatGPT-User	User-triggered page fetches	On-demand fetches for real users break
PerplexityBot	Perplexity discovery and citations	Perplexity stops finding and citing you
ClaudeBot	Anthropic crawler (training-leaning)	Anthropic's general crawler can't read you
Claude-SearchBot	Anthropic search-style fetches	Claude search-style fetches are blocked
Google-Extended	Gemini / Vertex generative use	You opt out of generative training, Search stays
Bingbot	Bing index and Bing AI features	You risk Bing and its AI surfaces
Applebot-Extended	Apple generative AI training	Apple won't use your text for generative training
CCBot	Common Crawl open dataset	Your text stays out of a widely used corpus

Want to understand the whole gate, not just the toggles? Our explainer on AI crawlers and robots.txt walks through directives, precedence, and the common gotchas. And the AI crawler glossary entry keeps the definitions handy.

Common mistakes

• Blocking OAI-SearchBot when you meant to block GPTBot. One kills your ChatGPT citations, the other just opts you out of training. Read the table before you toggle.
• Assuming the change is instant. OpenAI needs about 24 hours, per its bots documentation. Don't re-edit in a panic at hour three.
• Forgetting your CDN or WAF. A clean robots.txt won't help if Cloudflare or your firewall is blocking the bot first. The robots.txt only governs bots that obey it and actually reach your origin.
• Putting robots.txt in the wrong place. It has to live at the domain root, not in a subfolder. yourdomain.com/robots.txt or it's ignored.
• Disallowing Google-Extended and expecting to leave Search. It doesn't touch Search indexing (Google). Different control, different outcome.
• Never verifying. Generate, paste, then actually test it. Run the robots.txt checker so you know the live file says what you think it says.

FAQ

What does an AI robots.txt generator do?

It builds the robots.txt directives that tell AI crawlers whether they can read your site. You pick a preset or toggle each bot, and it emits a correct, copy-paste block covering OAI-SearchBot, GPTBot, PerplexityBot, Google-Extended, and more. No memorizing user-agent strings or directive syntax. Copy, paste at your domain root, done.

Will blocking GPTBot remove me from ChatGPT?

No. GPTBot is training only (OpenAI). Disallowing it stops OpenAI from using your pages to train future models, but it does not affect ChatGPT search citations. The bot that controls citations is OAI-SearchBot. Block GPTBot to opt out of training while staying visible in ChatGPT search.

Should I block all AI crawlers?

Usually not. If you want AI engines to cite and recommend you, they have to be allowed to read your pages first. "Block all AI" removes you from ChatGPT, Perplexity, and Gemini answers. Most brands want search bots allowed and training bots optional. See how to get cited by ChatGPT for the strategy.

How long until my robots.txt change takes effect?

OpenAI takes about 24 hours to reflect a robots.txt change (OpenAI). Other engines vary. If a bot still appears after a day, check your CDN or WAF, because an edge block can override a clean robots.txt entirely. Give it time before assuming the file is wrong.

Does Google-Extended remove me from Google Search?

No. Google-Extended only controls whether Gemini and Vertex use your content for generative answers, and it's separate from Search indexing (Google). You can disallow Google-Extended to opt out of generative training while keeping your normal Google Search rankings untouched. Two different controls, two different outcomes.

Where do I put the generated robots.txt?

At your domain root, reachable at yourdomain.com/robots.txt. It cannot live in a subfolder or subpath, or crawlers will ignore it. If you already have a robots.txt, merge the AI directives in rather than overwriting your existing Sitemap and Disallow rules. Then verify the live file with our checker.

How do I confirm the file actually works?

Paste it, wait for caches to clear, then test the live URL. Our AI crawler robots.txt checker reads your published robots.txt and tells you which AI bots are allowed or blocked. It catches the gap between what you generated and what your server is actually serving.

Where to go next

Generating the file is step one. The bigger question is whether the bots you allowed are actually citing you, and that's the job of the full platform. Run a free instant visibility check to see if ChatGPT, Perplexity, Gemini, and Google AI Overviews mention your brand right now, with confidence intervals and competitor share of voice.

If you'd rather keep stacking technical wins first, the llms.txt generator builds the companion file that points AI engines at your best content, and our llms.txt explainer covers why it's worth the ten minutes. For the deeper definition of what a citation even is in this world, the AI citation glossary entry is short and useful. Get the gate right, get the signals right, then go measure.