How does AI search work?
AI search works in five steps: the assistant decides whether the question needs the web, splits it into several searches, retrieves pages from a search index or its own crawler, picks the passages that answer best, and writes a reply that cites them. A page can only be cited if a crawler could read it first.
Facts
- Google's guide, last updated on 10 July 2026, says its AI features use retrieval-augmented generation on top of its core ranking systems and the Search index. (Google Search Central: Optimizing for generative AI features (July 2026))
- Google announced AI Mode on March 5, 2025, describing a query fan-out technique that runs several related searches at once and combines the results. (Google: Expanding AI Overviews and introducing AI Mode (March 2025))
- OpenAI's OAI-SearchBot is the crawler that surfaces websites in ChatGPT search, and robots.txt changes take about 24 hours to apply to it. (OpenAI: Overview of OpenAI crawlers)
- Perplexity says PerplexityBot surfaces and links sites in its results, and robots.txt changes may take up to 24 hours. (Perplexity: Perplexity crawlers)
- Anthropic gave Claude web search with direct citations on March 20, 2025. (Claude: Claude can now search the web)
What happens between the question and the answer?
The same pipeline runs in ChatGPT, Google's AI features, Perplexity and Claude, with different indexes behind it:
- Decide: the model judges whether it can answer from memory or needs fresh sources
- Fan out: it rewrites the question into several narrower searches
- Retrieve: it fetches candidate pages from a search index or its own crawler
- Select: it keeps the passages that answer each sub-question most directly
- Write and cite: it composes the reply and links the pages it used
Where does each assistant get its pages?
Each assistant relies on a documented crawler or index, which decides whether your page can be found at all.
| Assistant | Where pages come from | Crawler to allow |
|---|---|---|
| Google AI Overviews and AI Mode | Google's Search index | Googlebot |
| ChatGPT search | OpenAI's search crawler | OAI-SearchBot |
| Perplexity | Perplexity's own index | PerplexityBot |
| Claude | Web search with citations | Claude-SearchBot |
Why does my page get retrieved but not cited?
Retrieval only puts a page in the candidate pool; the model then cites the passage that answers a sub-question most plainly, often a short sentence with a number or a direct claim near the top of the page.
What can a website owner actually measure at each step?
You cannot see the model's choice, but you can see the two ends: whether AI crawlers read your page (server logs) and whether people arrive from AI assistants (referrers). easyIndex records both per page, which tells you whether a page is being read and whether it brings visitors.
Common mistakes
- Blocking the search crawlers in robots.txt while hoping to be cited.
- Serving the main text only through client-side JavaScript that crawlers do not run.
- Burying the answer below long introductions, so no passage answers a sub-question directly.
- Treating a training crawler and a search crawler as the same thing.
How easyIndex solves it
easyIndex guides your coding agent over MCP to build pages that answer each sub-question directly, audits each one, checks it on the live site, notifies Bing through IndexNow and measures AI crawler reads and AI visitors, and records which sources ChatGPT, Claude, Perplexity and Gemini cite for your tracked questions in the monthly citation scan your agent runs, with a 7-day free trial.
Frequently asked questions
Does AI search use Google or Bing?
It depends on the assistant: Google's AI features use Google's index, while ChatGPT, Perplexity and Claude run their own search crawlers.
Does the AI read my whole site?
No. It retrieves a few candidate pages per sub-question and reads the passages it needs.
Is AI search the same as the model's training?
No. Training happens months earlier; search fetches pages at the moment of the question.
Can I pay to be included?
No. Google states there is no paid placement in its AI features, and no assistant sells citations.
Continue reading
- How do AI assistants choose which sources to cite?
- Does ChatGPT use Bing to find websites?
- Which AI crawlers exist and how do I allow them in robots.txt?
- What is the difference between AI Overviews and AI Mode?
- How does easyIndex work, what do I need, and how long until results?
- How much does easyIndex cost?
Last verified on .