The most common AEO failure is a robots.txt line
If an engine cannot fetch your page, that page can never be part of the source set it writes from, no matter how well it answers the question. A surprising number of sites are still blocking AI crawlers because of a blanket rule added during the first wave of concern about model training, and never revisited since.
Check yours before anything else: open your-domain.com/robots.txt and search for GPTBot, ClaudeBot, PerplexityBot and Google-Extended. A Disallow on any of them is a decision worth re-making deliberately, not inheriting.
Training crawlers and answering crawlers are different decisions
The most useful distinction when setting policy is what the crawler feeds. Broadly, vendors run three kinds, and you can allow or block them independently.
- Training crawlers collect content used to train or improve models. Blocking these does not remove you from answers today, and it is the decision most often made on principle.
- Search and indexing crawlers build the retrieval index the engine searches when it answers. Blocking these does remove you from citations, and this is the one most sites block by accident.
- User-triggered fetchers retrieve a specific page because a user's prompt pointed at it. Blocking these breaks the case where a prospect pastes your own URL into a chat and asks about you.
The user agents worth knowing
Vendors publish and change these, so treat the list as a starting point and verify against current vendor documentation before writing policy.
- OpenAI: GPTBot crawls content for model training and improvement, OAI-SearchBot builds the index behind ChatGPT search, and ChatGPT-User fetches a page when a user's prompt points at it.
- Google: Google-Extended is not a crawler but a control that governs whether content already crawled by Googlebot may be used for Gemini and for grounding generative answers. It does not affect Search ranking.
- Anthropic: ClaudeBot crawls content, and user-triggered and search-oriented agents fetch pages in response to a conversation.
- Perplexity: PerplexityBot indexes pages for retrieval, and Perplexity-User fetches a page a user's query pointed at.
- Apple: Applebot serves Siri and Spotlight, while Applebot-Extended separately governs use in generative model training.
- Microsoft: Copilot grounds largely in the Bing index, so Bingbot access matters more for Copilot visibility than any AI-specific agent.
- Others worth listing explicitly: meta-externalagent, Amazonbot, MistralAI-User, cohere-ai, DuckAssistBot, YouBot and Bytespider.
- In Chinese-speaking markets: Baiduspider, Sogou web spider, 360Spider and PetalBot feed engines your buyers there actually use.
A defensible default policy
For most commercial sites that want to be discovered, allow everything and rely on the controls that actually protect you: authentication for private content, licensing terms for reuse, and not publishing what you do not want repeated.
If you do want to draw a line, the coherent version is to block training crawlers while allowing search, indexing and user-triggered agents. That keeps you citable today without contributing to model training. What is rarely coherent is blocking the indexing agents while running an AEO programme, which is the combination we see most often.
robots.txt is a request, not an enforcement mechanism, and it is public. Never use it to hide sensitive paths: listing them tells everyone exactly where they are. Use authentication instead.
Allowing the crawler is necessary, not sufficient
- Serve substance in the initial HTML. Several AI crawlers do not execute JavaScript, so a client-rendered page can arrive functionally empty.
- Do not block AI agents at the CDN or WAF layer. Rules there override robots.txt entirely and are easy to forget about.
- Keep a sitemap and clean canonicals, so the crawler spends its budget on pages you want cited.
- Consider an llms.txt summary of your site. Support is not universal, but it is cheap and removes ambiguity about what you do.
Common questions
For most commercial sites that want to be discovered, no. GPTBot relates to training rather than to answering, but blocking AI crawlers as a group is what usually happens in practice, and blocking the search and indexing agents removes you from citations entirely. If you want to draw a line, block training crawlers specifically and keep search, indexing and user-triggered agents allowed.
No. Google-Extended is a separate control governing whether already-crawled content may be used for Gemini and for grounding generative answers. It does not affect how pages rank in Google Search, though it does affect generative surfaces.
No. robots.txt is a request that well-behaved crawlers honour, not an enforcement mechanism, and the file itself is public. Anything that genuinely must not be read by a model needs authentication, not a disallow rule.
Bingbot. Copilot grounds largely in the Bing index, so Bing crawl access and Bing indexing status affect Copilot visibility more than any Copilot-specific user agent does.