Docs

Is Vercel blocking GPTBot, ClaudeBot and other AI crawlers from my site?

Not by default. Vercel's AI Bots Managed Ruleset is off unless someone turns it on, and its Bot Protection and Attack Mode let verified bots such as GPTBot and ClaudeBot through. Your site blocks AI crawlers on Vercel only if the AI Bots ruleset is set to Deny, a custom firewall rule matches their names, or your robots.txt disallows them.

Facts

  • Vercel launched a one-click AI bot managed ruleset on May 13, 2025 that can block GPTBot, ClaudeBot, PerplexityBot and Bytespider, free on all plans. (Vercel changelog: New one-click AI bot managed ruleset (May 13, 2025))
  • Vercel's docs, updated 2026-09-10, say the AI Bots Managed Ruleset is inactive by default, shown as "Allow", with two actions: Log or Deny. (Vercel docs: WAF Managed Rulesets)
  • Vercel's Bot Protection ruleset is also off by default and, when on, excludes verified bots; Vercel's directory lists GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot as verified. (Vercel docs: Bot Management)
  • Attack Mode is free on all Vercel plans and lets known bots from the verified directory through without a challenge. (Vercel docs: Attack Mode)
  • OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and a robots.txt change takes about 24 hours to apply. (OpenAI: Overview of OpenAI crawlers)

Where do I see if Vercel blocks AI bots on my project?

Open the project in the Vercel dashboard, then Firewall and Rules. Three places decide it:

SettingDefaultBlocks AI crawlers when
"AI Bots Ruleset" (Bot Management)"Allow" (inactive)Set to "Deny"
"Bot Protection" (Bot Management)"Off"Rarely: verified bots are excluded, but unverified ones get a challenge
Custom rulesNoneA rule matches a user agent such as GPTBot or uses the "Block AI Bots" template with Deny or Challenge
Attack ModeDisabledNot for verified bots, which pass without a challenge

Why would my Vercel site block ChatGPT without me knowing?

Usually because someone turned on the one-click block to save usage. In a February 2026 thread on Vercel's community forum, a free-plan account was paused after an OpenAI crawler used up its resources, and a Vercel employee suggested blocking AI bots in the firewall.

That fixes the bill and also removes the site from what those crawlers can read. Check the firewall after any cost or traffic incident.

How do I test it from the command line?

Read robots.txt on the production domain and request the home page with each crawler's name. A 200 is a good sign; a 403 or a challenge page means a rule matches that name.

# 1. robots.txt on the production domain
curl -s https://www.example.com/robots.txt

# 2. Status code per AI crawler name (anything but 200 means a rule matches that name)
for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot PerplexityBot; do
  echo "$ua $(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://www.example.com/)"
done

Yes, by blocking the crawlers by name instead of all at once. OpenAI separates GPTBot, used for training, from OAI-SearchBot, used for ChatGPT search, and Anthropic separates ClaudeBot from Claude-SearchBot.

Vercel's AI Bots ruleset treats them as one group. To block training only, disallow GPTBot and ClaudeBot in robots.txt or write a custom firewall rule for those user agents, and leave the search crawlers allowed.

No, they are meant to stay out. Vercel adds an X-Robots-Tag: noindex header to every preview deployment automatically.

The exception: when a custom domain points to a non-production branch, Vercel omits that header, so a staging domain can be indexed unless you add it yourself.

Step by step

  1. Check the AI Bots ruleset. In Firewall, Rules, Bot Management, confirm the "AI Bots Ruleset" shows "Allow" or "Log", not "Deny".
  2. Check custom rules. Look for rules that match GPTBot, ClaudeBot, PerplexityBot or the "Block AI Bots" template, and note their action.
  3. Check robots.txt. Ask your coding agent to show the robots.txt the framework generates and confirm OAI-SearchBot and Claude-SearchBot are allowed.
  4. Test each crawler name. Run the curl loop against the production domain and look for anything other than 200.
  5. Block by name if you must. If cost is the reason, block only the training crawlers or start with "Log" before choosing "Deny".

Common mistakes

  • Turning on the one-click AI block to cut usage and forgetting it also stops ChatGPT search and Perplexity crawlers.
  • Checking only robots.txt while a firewall rule returns 403 to the same crawlers.
  • Testing the preview URL, which carries noindex, instead of the production domain.
  • Leaving a staging custom domain indexable because Vercel omits noindex on it.

How easyIndex solves it

easyIndex never touches your Vercel project: your coding agent makes every change, guided step by step over MCP, and easyIndex checks each page on the live site instead of trusting the agent's report. Its dashboard shows which pages AI crawlers read and when, so a firewall change that stops them shows up as reads that stop. US$ 30 per website a month, with a 7-day free trial.

Frequently asked questions

Does Vercel block AI crawlers on the free Hobby plan?

Not by default. The AI Bots ruleset is available on all plans, including Hobby, and stays inactive until someone sets it to Log or Deny.

Does Vercel's Bot Protection block GPTBot?

No. Vercel says Bot Protection excludes verified bots, and its directory lists GPTBot, OAI-SearchBot and ClaudeBot as verified.

Will Attack Mode hurt my visibility in AI answers?

Vercel says verified bots pass Attack Mode without a challenge, and the AI crawlers of OpenAI, Anthropic and Perplexity are in that directory.

How long until ChatGPT reads my site again after I unblock it?

OpenAI says a robots.txt change takes about 24 hours to reach its systems; a firewall change applies when you publish it.

Continue reading

Last verified on .

Is Vercel blocking GPTBot, ClaudeBot and AI crawlers?