Is Cloudflare Hiding You From AI Search?

You can do everything right and still be invisible. Write the page, answer the question, earn the authority, and then have a rule you never set refuse the crawler before it reads a single word.

That scenario stopped being hypothetical on 15 September 2026.

The short answer

From 15 September 2026, Cloudflare's default settings block mixed use crawlers from any pages that host ads. A mixed use crawler is one that combines search, agent use and training in a single bot instead of separating those purposes into different agents.

The default applies to new Cloudflare customers, new sites added by existing customers, and all free tier users.

If your site sits behind Cloudflare, carries advertising, and nobody on your team has reviewed bot settings recently, there is a real chance AI engines stopped being able to read you a fortnight ago. Nothing about your site would look different. Your pages still load perfectly for humans.

What Cloudflare actually changed, and why

This did not appear from nowhere. Cloudflare has been moving in this direction for over a year.

In July 2025 it announced what it called Content Independence Day, changing the default to block AI crawlers unless they pay creators, and launching Pay Per Crawl so publishers could charge for each crawl request.

The 2026 round went further in two ways. It set the 15 September default described above, and it introduced Pay Per Use, which is designed to compensate publishers when their content creates value rather than only when it is fetched. Cloudflare said it was working initially with two partners on that, Ceramic.ai and You.com.

The reasoning given by chief executive Matthew Prince was blunt. Most traffic on the internet is no longer people, so in his framing Cloudflare had to go further and move faster.

You can agree with the policy and still get hurt by it. The publishers who lobbied for this have staff who manage bot access deliberately. The much larger group of ordinary sites on a free plan do not, and they inherit the default without ever making a decision.

The distinction that decides whether you get cited

Here is the part that most coverage of this story skips, and it is the part that actually matters for your visibility.

"AI crawler" is not one thing. The major operators run separate bots for separate jobs, and they behave completely differently for your purposes. These are taken from the operators' own documentation.

Read the third and fifth rows again, because they are the whole point.

GPTBot is training. Blocking it is a defensible commercial decision about whether your content should improve someone else's model for free, and it does not remove you from ChatGPT's search citations.

OAI-SearchBot is what puts you in front of people. OpenAI's own documentation describes it as the agent "used to surface websites in search results in ChatGPT's search features". Perplexity describes PerplexityBot the same way, as the crawler "designed to surface and link websites in search results on Perplexity", operating independently of training.

Block the training bot and you have made a choice about licensing. Block the search bot and you have removed yourself from the answer.

One further wrinkle worth knowing. The user triggered agents behave differently again. Perplexity documents that Perplexity-User typically disregards robots.txt, on the reasoning that a human asked for the page. That does not help you if the block is at the edge, because an edge rule refuses the connection regardless of what any file says.

Why "block AI bots" is the wrong instruction

For two years the standard advice in a lot of publishing circles has been some version of keep the AI companies out. It made sense when the only thing the bots did was ingest content for training.

That advice has not kept up. The same companies now run the surfaces where your customers ask questions, and the crawlers that feed those surfaces are different bots with different names and different jobs.

The result is a large number of sites carrying blanket blocks that were written for a training era problem and are now quietly removing them from the answer layer. The block is doing something its author never intended, and because nothing visibly breaks, nobody notices.

There is also a measurement trap here that makes it worse. If you are blocked from the search crawlers, your AI visibility score does not show an error. It shows zero, which looks exactly like being unpopular. Teams then respond by writing more content, which cannot possibly work, because the new pages are equally unreachable.

The edge decides before robots.txt

This is the technical point people get wrong most often.

Your robots.txt file is a request that a crawler reads after it reaches your server. An edge rule at Cloudflare, or any comparable service, decides whether the request reaches your server at all. When the two disagree, the edge wins, because it acts first.

So a site can have a completely open robots.txt that explicitly welcomes every AI crawler by name, and still return a refusal to all of them. Checking the file tells you what you intended. It tells you nothing about what is actually happening.

The only reliable method is to make a request as the crawler and read the response. Cloudflare's own controls make this concrete: blocked AI crawlers can be served a 403 Forbidden or a 402 Payment Required, and a charge for crawl option exists in closed beta that bills the crawler operator per successful request.

A 402 is particularly worth recognising. It is not an error in the ordinary sense. It means a price has been set on your content, and unless the operator has agreed to pay it, the practical outcome for you is identical to a block.

How to check whether you are blocked

Four checks, in order, cheapest first. None of them take long.

One. Request your own page as each search crawler. Fetch a normal article on your site while identifying as OAI-SearchBot, then as PerplexityBot. Look only at the status code. A 200 means reachable. A 403 means blocked at the edge. A 402 means a payment requirement is in the way. Our free AI crawler and robots.txt checker runs this for you across the major agents if you would rather not do it by hand.

Two. Read your robots.txt with the bot split in mind. Look specifically for blanket rules. A User-agent: * with a broad Disallow catches search crawlers along with everything else. If you find a deliberate block, check whether it names a training bot or a search bot, because those are different decisions.

Three. Check your Cloudflare bot settings directly. Look at whether AI crawlers are being allowed, blocked or charged, and confirm which categories those rules cover. If your site is on a free plan, or was added recently, assume the new default applies until you have seen otherwise.

Four. Check whether the pages carry ads. The 15 September default is scoped to ad supported pages. A site with no advertising is outside that specific rule, though it can still be affected by any blocking you configured yourself or inherited earlier.

How we got here, briefly

The policy makes more sense if you follow the sequence, because each step was a reasonable answer to the step before it.

The training era. When large models were first trained on the open web, crawling was invisible and uncompensated. Publishers found their archives had been ingested wholesale with nothing in return. The reaction was blunt refusal, and the tool available was a robots.txt block.

The infrastructure response. In July 2025 Cloudflare changed its default to block AI crawlers unless they paid, and shipped Pay Per Crawl so a publisher could put a price on each request. That moved the decision from a file on your server, which crawlers may or may not respect, to the edge, where refusal is enforced.

The answer engine era. Meanwhile the same companies built search products. Citations became a real referral channel, and suddenly access was not only a cost, it was distribution. The operators split their bots to reflect this, which is why GPTBot and OAI-SearchBot are separate agents with separate documentation.

The 2026 default. Cloudflare pushed further with the 15 September scoping and Pay Per Use, which prices the value created rather than the fetch itself.

The friction today comes from the fact that most site owners are still operating with instincts formed in the first phase, while the crawlers have moved into the third. Refusal made sense when access only fed a model. It makes much less sense when access also decides whether a buyer hears your name.

The robots.txt patterns that quietly catch search crawlers

Even leaving the Cloudflare default aside, a lot of sites are blocking themselves through rules written years ago. These are the patterns worth grepping your own file for.

The blanket wildcard. A User-agent: * paired with a broad Disallow: / catches every crawler that respects the file, including the search agents that produce citations. Sites in development that went live without the rule being removed are a classic example.

The copied block list. Lists of AI user agents circulated widely in 2024 and got pasted into files without review. Many of those lists predate the split between training and search bots, so they name agents that no longer exist while missing the ones that now matter, and they block search crawlers alongside training ones without distinguishing.

The path block that happens to cover your best content. A Disallow on a directory added for an unrelated reason, perhaps to keep a staging path or a print view out of search, that now covers the blog or docs section where your most citable material lives.

The conflicting specific rule. A permissive wildcard block followed by a more specific rule for a named agent. Precedence in robots.txt is by specificity of the user agent match, not file order, and people regularly get this backwards and assume a later line overrides an earlier one.

Worth repeating, because it is the part that defeats careful reading: none of this matters if the edge refuses first. Audit the file, then test the actual response, and trust the response.

If you are a publisher and you do want to charge

Not every site should open the gates, and it would be dishonest to pretend otherwise.

If your content is the product rather than an advertisement for the product, access is genuinely your leverage. An archive of original reporting, a proprietary dataset, a reference work people pay for. In those cases the calculation flips: being cited without compensation transfers value from you to an engine that monetises the answer, and the referral you get back may not cover it.

For that group the current options are worth understanding rather than dismissing. Pay Per Crawl prices each request. Pay Per Use aims at the value created instead. Charge for crawl in Cloudflare's controls bills the operator per successful request, though it is in closed beta.

Two cautions before anyone rushes at this. The first is that a price is only revenue if someone agrees to pay it. Where an operator has not agreed, the practical result of setting a price is a block, and your content simply stops appearing.

The second is that the two strategies do not mix well within one site. Charging on your archive while wanting citations on your marketing pages is coherent in principle, but it requires per path rules that are correctly scoped, and a mistake in either direction is invisible from the outside. If you go down this route, test both outcomes deliberately: confirm the paid paths return the payment requirement and confirm the open paths return a normal response.

What to allow and what to block

There is no universally correct answer here, because it depends on whether you want the traffic more than you want the licensing leverage. But for most businesses selling something, the calculation is not close.

The one group with a genuine dilemma is large publishers whose entire business is the content itself. For them, withholding access is real leverage and Pay Per Use is a plausible route to being compensated for it.

For a software company, an agency, a local business or a retailer, the content is marketing rather than the product. Blocking the crawler that would recommend you to a buyer, in order to protect content you published to attract buyers, is a trade that makes no sense. Yet plenty of sites in that second group are doing exactly that right now because somebody applied publisher advice to a business that is not a publisher.

What this means for measurement

If you track AI visibility, add crawler reachability to the things you check. A visibility score is only meaningful if the engines can actually read you, and reachability is binary in a way that scores are not.

A sensible order of operations when a score drops to nothing:

1. Confirm the search crawlers get a 200. If they do not, stop. Nothing else matters until that is fixed.
2. Confirm the page is still returning normally to ordinary requests too, in case a wider rule is at fault.
3. Only then look at content and competition.

We put crawler access first in our own checks for this reason. A brand with a genuine content problem and a brand that is simply unreachable produce identical looking scores, and the fixes for them have nothing in common. Telling the two apart on day one saves months.

A worked check, on this site

It seems fair to run the test we are recommending, on ourselves, and publish the result.

This site sits behind Cloudflare. It carries no advertising, so the 15 September default is not scoped to it, but that scoping is the kind of thing worth confirming rather than assuming. Requesting one of our own articles while identifying as each of the four agents that matter returned the following.

Four 200s, so the engines can read us. Our robots.txt names the AI agents explicitly rather than relying on a wildcard, and restricts only the paths that should never be crawled anyway: the API, the dashboard, the admin area and billing.

The whole check took under a minute. That is the point. The cost of confirming this is trivial, and the cost of not confirming it is a year of content nobody can read.

The bottom line

On 15 September 2026, Cloudflare started blocking mixed use AI crawlers on ad supported pages by default, for new customers, new sites and every free tier user. Plenty of site owners inherited that default without deciding anything.

The important nuance is that AI crawlers are not interchangeable. GPTBot trains a model. OAI-SearchBot decides whether ChatGPT can cite you. PerplexityBot does the same for Perplexity. Blocking the first is a business decision. Blocking the second and third removes you from the answers your buyers read, silently, with no error anywhere to tell you.

Go and check your status codes. It takes a few minutes, it costs nothing, and the failure mode it catches is the kind that quietly wastes a year of content budget.

Sources and references

• TechCrunch on Cloudflare's 2026 policy. The 15 September 2026 default blocking mixed use crawlers, meaning bots combining search, agent use and training, on ad supported pages, applied to new customers, new sites from existing customers and all free tier users. Introduction of Pay Per Use with initial partners Ceramic.ai and You.com, and the quote from Matthew Prince. techcrunch.com
• Cloudflare, Content Independence Day, July 2025. The original change of default to block AI crawlers unless they pay, and the launch of Pay Per Crawl. blog.cloudflare.com
• Cloudflare AI Crawl Control documentation. Allow and block behaviour, custom responses of 403 Forbidden or 402 Payment Required, enforcement of robots.txt, and the closed beta charge for crawl option. developers.cloudflare.com
• OpenAI bot documentation. GPTBot for training, OAI-SearchBot for surfacing websites in ChatGPT search features, ChatGPT-User for live visits when a user asks, and OAI-AdsBot for ad landing page checks. developers.openai.com
• Perplexity bot documentation. PerplexityBot for surfacing and linking websites in search results independently of training, and Perplexity-User for live visits, which typically does not apply robots.txt restrictions. docs.perplexity.ai

Cite this article

According to AI Citation Monitor's 2026 guide to AI crawler access, Cloudflare began blocking mixed use AI crawlers on ad supported pages by default on 15 September 2026, applying to new customers, new sites from existing customers and all free tier users. The critical distinction for visibility is that AI crawlers are not interchangeable: OpenAI's GPTBot crawls for model training, while OAI-SearchBot is what surfaces websites in ChatGPT's search features, and Perplexity's PerplexityBot surfaces and links websites in Perplexity results independently of training. Blocking a training crawler has no direct effect on citations, but blocking a search crawler removes a site from those answers entirely, and because an edge rule refuses the request before robots.txt is read, a permissive robots.txt file is not evidence that crawlers can reach the site.

Source: AI Citation Monitor. Is Cloudflare Hiding You From AI Search? (2026). https://aicitationmonitor.com/blog/cloudflare-ai-crawler-block