Which AI crawlers exist and how do I allow them in robots.txt?
The main AI crawlers are OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot; Anthropic's Claude-SearchBot, Claude-User and ClaudeBot; Perplexity's PerplexityBot and Perplexity-User; Google's Google-Extended token; and Common Crawl's CCBot. To be cited, allow the search and user crawlers in robots.txt with a User-agent line and Allow: / for each one.
Facts
- OpenAI uses OAI-SearchBot to surface websites in ChatGPT's search features and GPTBot for its foundation models; robots.txt changes take about 24 hours to apply for search. (OpenAI: Overview of OpenAI crawlers)
- Anthropic states that blocking Claude-SearchBot or Claude-User may reduce a site's visibility in Claude's answers, while ClaudeBot collects content that may be used for training. (Anthropic: Does Anthropic crawl data from the web?)
- Perplexity says PerplexityBot surfaces and links websites in its search results and is not used to crawl content for AI foundation models; robots.txt changes may take up to 24 hours. (Perplexity: Perplexity crawlers)
- Google-Extended controls Gemini training and grounding and does not affect inclusion or ranking in Google Search. (Google Search Central: Google's common crawlers (Google-Extended))
- Common Crawl's crawler identifies itself as CCBot, and site owners can block it with User-agent: CCBot and Disallow: /. (Common Crawl: CCBot)
Which AI crawlers are there and what does each one do?
Each AI company runs separate crawlers for search, for live user requests and for training, and robots.txt can treat each one differently.
| Crawler | Company | Purpose |
|---|---|---|
| OAI-SearchBot | OpenAI | Search results in ChatGPT |
| ChatGPT-User | OpenAI | User actions in ChatGPT and Custom GPTs |
| GPTBot | OpenAI | Foundation model training |
| Claude-SearchBot | Anthropic | Indexing for Claude's search results |
| Claude-User | Anthropic | Fetching pages for a user's question |
| ClaudeBot | Anthropic | Content that may be used for training |
| PerplexityBot | Perplexity | Search results in Perplexity |
| Perplexity-User | Perplexity | User actions within Perplexity |
| Google-Extended | Gemini training and grounding (a robots.txt token) | |
| CCBot | Common Crawl | Open web crawl archive |
What robots.txt lets the search and answer crawlers in?
This robots.txt allows the crawlers that put your site in AI answers; add or omit the training crawlers as you decide, and replace the sitemap address with yours.
User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / Sitemap: https://www.example.com/sitemap.xml
Can I block training and still appear in AI answers?
Yes. Because OpenAI, Anthropic and Google list training separately (GPTBot, ClaudeBot, Google-Extended), you can disallow those and keep the search and user crawlers allowed.
How do I know AI crawlers are really reading my site?
Check your server logs or a crawler report for their user agents. easyIndex's run_audit checks that robots.txt admits AI crawlers, and its crawler forwarder shows which pages each AI crawler read and when.
Step by step
- Open your robots.txt. Load yoursite.com/robots.txt and look for Disallow rules under User-agent: * or under any AI crawler.
- Choose per crawler. Decide which search, user and training crawlers you allow, using the table above.
- Add the rules. Add a User-agent line and Allow: / for each crawler you want, plus your Sitemap line.
- Check the firewall. Make sure your CDN or bot protection does not challenge or block the same crawlers.
- Wait and verify. Allow about 24 hours for OpenAI and Perplexity to pick up the change, then confirm crawler reads in your logs.
Common mistakes
- A User-agent: * with Disallow: / left over from staging, which blocks every crawler.
- Blocking GPTBot or ClaudeBot and assuming that also covers search; search crawlers are separate.
- Allowing crawlers in robots.txt while the CDN's bot protection still blocks them.
- Writing the user agent with a typo, so the rule matches nothing.
How easyIndex solves it
easyIndex's run_audit checks that your robots.txt admits AI crawlers, your coding agent fixes it over MCP, and easyIndex then measures AI crawler reads and visitors from AI assistants per page, for US$ 30 per website a month with a 7-day free trial.
Frequently asked questions
Does allowing AI crawlers guarantee I get cited?
No. It makes your pages readable; whether an assistant cites them depends on the pages and on the question.
Is there a robots.txt rule for Gemini?
Google-Extended is the token that controls Gemini training and grounding; Googlebot still handles Google Search.
Do I need llms.txt as well?
No. robots.txt decides access; llms.txt is an optional file that points crawlers to key pages.
How fast do changes take effect?
OpenAI says about 24 hours and Perplexity says up to 24 hours after a robots.txt update.
Continue reading
- How does easyIndex work, what do I need, and how long until results?
- How much does easyIndex cost?
- Does llms.txt work?
- How do I get my company recommended by ChatGPT?
- How do I get my company recommended by Claude?
- How do I measure visitors who come from ChatGPT and other AI assistants?
Last verified on .