Is Cloudflare blocking ChatGPT and other AI crawlers from my site?
It may be. Cloudflare changed its default to block AI crawlers in July 2025, and since September 15, 2026 new domains block training and agent bots on pages with ads. Check Security Settings, Configure AI bot policies: allow the Search and Agent categories so ChatGPT, Claude and Perplexity can read and cite you, and block Training if you prefer.
Facts
- On July 1, 2025 Cloudflare announced it was changing its default to block AI crawlers unless they pay creators for their content. (Cloudflare: Content Independence Day (Jul 2025))
- Cloudflare's documentation says that from September 15, 2026 new domains block bots classified as Training or Agent on pages that display ads, while Search stays allowed. (Cloudflare docs: Block AI bots)
- Cloudflare says crawlers that combine Search and Training are blocked by every configuration meant to block AI training. (Cloudflare docs: Block AI bots)
- OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, while blocking GPTBot only opts out of training. (OpenAI: Overview of OpenAI crawlers)
- Anthropic runs three bots: ClaudeBot for training, Claude-User for user requests and Claude-SearchBot for search indexing; blocking the last two can reduce your visibility in Claude. (Anthropic: Does Anthropic crawl data from the web?)
Where do I see what Cloudflare blocks?
In the Cloudflare dashboard, open Security Settings and then Configure AI bot policies. Each category, Search, Agent and Training, has three options: Block, Block on pages with ads, and Allow. The older Block AI bots switch is being deprecated.
Which AI crawlers should I allow to be cited?
Allow the crawlers that answer users and build search indexes; training crawlers are a separate decision.
| Crawler | Purpose | Cloudflare category | To be cited |
|---|---|---|---|
| OAI-SearchBot | ChatGPT search | Search | Allow |
| ChatGPT-User | Fetches a page a user asks about | Agent | Allow |
| Claude-SearchBot | Claude search indexing | Search | Allow |
| Claude-User | Fetches a page a Claude user asks about | Agent | Allow |
| GPTBot, ClaudeBot | Model training | Training | Your choice |
Does robots.txt still matter if I use Cloudflare?
Yes: Cloudflare's rules decide who gets through the network, and robots.txt tells each crawler what it may read. Both have to agree, or the more restrictive one wins.
# Search and user-request crawlers: allowed User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / # Training crawlers: your choice User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: /
Why can blocking training also block search?
Because some crawlers do both jobs. Cloudflare evaluates a mixed Search and Training crawler under both policies, so blocking Training on all pages can shut it out even with Search allowed. Check the Search category after changing Training.
How do I confirm AI crawlers are reading my site again?
Look for their visits after the change, not just at the setting. easyIndex records every AI crawler read per page, so you see when OAI-SearchBot or Claude-SearchBot comes back and notice if the reads stop.
Step by step
- Open the AI bot policies. In Cloudflare go to Security Settings, Configure AI bot policies, and note the option set for each category.
- Allow Search and Agent. Set Search and Agent to Allow so search indexes and user requests from ChatGPT, Claude and Perplexity get through.
- Decide on Training. Block or allow Training as you prefer, then recheck that Search is still allowed for mixed crawlers.
- Align robots.txt. Have your coding agent allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot in robots.txt.
- Watch the crawler reads. Confirm reads from those crawlers appear in the next days; OpenAI says robots.txt changes take about 24 hours.
Common mistakes
- Fixing robots.txt while Cloudflare still blocks the crawler at the network.
- Blocking Training on all pages without checking what happens to mixed crawlers.
- Allowing GPTBot and blocking OAI-SearchBot, which leaves you out of ChatGPT search.
- Trusting the setting without checking that crawler reads actually return.
How easyIndex solves it
easyIndex records which AI crawlers read each page and shows when they stop, so a block at Cloudflare is visible; your coding agent fixes robots.txt over MCP and easyIndex checks it live. US$ 30 per website a month, with a 7-day free trial.
Frequently asked questions
Does Cloudflare block Googlebot too?
Googlebot crawls for search and other uses, and Cloudflare treats mixed crawlers under the Training policy as well, so check that setting if Google traffic drops.
Is the default the same on old and new domains?
No. Cloudflare's September 15, 2026 defaults apply to new domains; older domains keep the options already set, so check yours.
Do I need Cloudflare's pay per crawl to be cited?
No. Pay per crawl is for publishers who want to charge crawlers; to be cited you only need to let search and agent crawlers in.
How long until ChatGPT sees my site after unblocking?
OpenAI says its systems follow robots.txt changes in about 24 hours; being cited after that depends on your pages.
Continue reading
- How does easyIndex work, what do I need, and how long until results?
- How much does easyIndex cost?
- Which AI crawlers exist and how do I allow them in robots.txt?
- Can ChatGPT and Claude see my JavaScript website?
- How do I get my company recommended by ChatGPT?
- How do I measure visitors who come from ChatGPT and other AI assistants?
Last verified on .