// guide
How do you set up robots.txt for AI crawlers?
Updated: 2026-08-19
// short answer
Setting up robots.txt for AI crawlers means making three separate decisions, not one: whether to admit training crawlers, search-index crawlers, and user-initiated fetchers. Every vendor runs different user-agents for each job. Most sites should allow search and user fetches, decide deliberately about training — and know that some fetchers ignore robots.txt entirely.
One decision is actually three
"Should I block AI bots?" has no answer, because it isn't one question. Every major AI company runs separate crawlers for separate jobs: training crawlers feed model corpora, search-index crawlers feed the answers that cite you, and user-initiated fetchers read a page because a person just asked the assistant about it. Block them all with one line and you haven't taken a stance on AI training — you've removed yourself from AI answers.
The costs are asymmetric. Blocking a training crawler is a philosophical choice with, in most cases, no visibility price: Google and Apple document that explicitly, and for OpenAI and Anthropic it follows from the bot separation — search runs on separate agents with separately documented consequences. Blocking a search or user-fetch agent has a direct, documented price — OpenAI, for instance, states that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." The per-engine mechanics are covered in our guides on getting cited by ChatGPT, Claude and Perplexity; this page is the full map.
Two numbers frame the trade. Cloudflare measured how many pages AI companies crawl for every visit they send back in return: per Cloudflare Radar data for June 2025, roughly 1,700 crawls per referral for OpenAI and about 70,900 for Anthropic. The ratios have been falling since — Cloudflare's AI Insights dashboard tracks them live — but the shape of the deal is clear: you trade heavy crawling for citations. The table below is how you set the terms.
The full AI crawlers list, verified against vendor docs
Every row below is checked against the operator's own documentation as of August 2026. "Token" means the name only works as a robots.txt group — there is no separate crawler with that user-agent.
| User-agent | Job | Respects robots.txt | Blocking costs you |
|---|---|---|---|
| GPTBot — OpenAI | Training | Yes | Future content excluded from OpenAI model training; ChatGPT search runs on OAI-SearchBot |
| OAI-SearchBot — OpenAI | Search index | Yes | "Will not be shown in ChatGPT search answers" |
| ChatGPT-User — OpenAI | User fetch | "Rules may not apply" (official) | Declarative only — fetches are user-initiated |
| ClaudeBot — Anthropic | Training | Yes | Future content excluded from Anthropic training datasets |
| Claude-SearchBot — Anthropic | Search index | Yes | Not indexed — "may reduce your site's visibility" |
| Claude-User — Anthropic | User fetch | Yes | Page can't be fetched on a Claude user's request |
| PerplexityBot — Perplexity | Search index | Yes | Out of Perplexity's search index and citations |
| Perplexity-User — Perplexity | User fetch | "Generally ignores robots.txt" (official) | Declarative only |
| Google-Extended — Google (token) | Training + grounding control | Honored via Googlebot | Out of Gemini training and grounding; Search and AI Overviews unaffected |
| Applebot — Apple | Search index (Siri, Spotlight, Safari) | Yes — follows Googlebot rules if not named | Out of Apple search surfaces |
| Applebot-Extended — Apple (token) | Training control | Honored via Applebot | Out of Apple foundation-model training; stays in Siri and Spotlight |
| Meta-ExternalAgent — Meta | Training + indexing | Yes | Out of Meta AI training and its index |
| Meta-ExternalFetcher — Meta | User fetch | "May bypass robots.txt" (official) | Declarative only |
| CCBot — Common Crawl | Open web archive | Yes | Out of the public datasets many AI labs train on — a wholesale opt-out |
| Amazonbot — Amazon | Product improvement; may train Amazon AI models | Yes | Out of Amazon crawling, including any model training |
| Amzn-SearchBot — Amazon | Search index (Alexa search experiences; no training) | Yes — follows generic search-bot rules if not named | Out of Alexa search experiences |
| Amzn-User — Amazon | User fetch (Alexa requests) | "May not follow all robots.txt directives" (official) | Declarative only |
| DuckAssistBot — DuckDuckGo | Real-time answers (no training) | Yes (applies within 72h) | Out of DuckAssist answers |
| Bytespider — ByteDance | Training (undocumented) | No — ignores it in measured practice | Nothing via robots.txt; needs a server-level block |
How to block AI crawlers — or not: three ready-made configs
The table maps to three workable policies.
Config 1 — visibility-first. Allow everything, explicitly: a per-agent Allow group for every documented bot in the table (all but Bytespider — an Allow for a bot that ignores the file is noise), plus a catch-all. Right for sites that live on being found and cited — content businesses, agencies, most B2B. This is what we run ourselves; see "How we run ours" below.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /…and so on for each agent in the table. Our production file at get-geo.ai/robots.txt is the full version.
Config 2 — no-training. The most common deliberate choice: stay out of model corpora, stay in the answers. Disallow the training crawlers and tokens, allow the rest. Two caveats before you paste: the Bytespider line is declarative (the next section explains why), and dropping Meta-ExternalAgent is the one training opt-out with a visibility price — it also exits Meta's index:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: *
Allow: /Config 3 — lockdown. Legitimate for paywalled or proprietary content. Robots.txt lets several User-agent lines share one rule block, so the whole thing stays compact — we leave Applebot out here because blocking it also removes you from Siri and Spotlight search, which is a bigger decision than AI answers:
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: CCBot
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: DuckAssistBot
User-agent: Bytespider
Disallow: /Read the next section before shipping this: for several of the agents above, the file is a request, not a barrier.
Which AI crawlers ignore robots.txt
Robots.txt is a published policy, not an enforcement mechanism — and for one whole class of agents, the vendors say so themselves. OpenAI on ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Meta on Meta-ExternalFetcher: it "may bypass robots.txt because it performs fetches that were requested by the user." Perplexity's user fetcher "generally ignores robots.txt rules." Amazon on Amzn-User: it "may not follow all robots.txt directives." The logic is consistent: a human asked, so the fetch is treated as the human's visit, not a crawl.
Then there is Bytespider, which has no documentation at all and ignores the file in measured practice: Cloudflare found it hitting more protected sites than any other AI crawler (40.4% in mid-2024), and HAProxy measured it at close to 90% of their AI-crawler traffic while noting it ignores robots.txt instructions.
So enforcement lives a layer down. Verify crawlers against the IP lists most vendors publish — OpenAI, Anthropic, Apple, Common Crawl and DuckDuckGo maintain JSON lists, Amazon publishes its ranges on its developer page — and block at the CDN or WAF what you actually want stopped. The verification matters more than it might seem: HUMAN Security measured that one in eighteen requests carrying a known AI-crawler user-agent is spoofed.
How we run ours
Our own robots.txt is the visibility-first config: nineteen explicit per-agent Allow groups — every agent from the table that honors the file, the user fetchers we welcome anyway, and classic Bingbot — followed by a catch-all Allow. We reworked it into this shape in August 2026, dropping the legacy anthropic-ai group in the same pass — and, while fact-checking this guide, a speculative cohere-ai group too: Cohere's own docs state it runs no crawlers at this time.
The obvious objection: a one-line "User-agent: * / Allow: /" would produce identical crawler behavior. True. The explicit groups buy three things the one-liner doesn't. Each vendor gets an unambiguous signal of intent rather than an absence of objection. Future edits are safer — a Disallow added for one agent can't silently apply to agents nobody thought about. And the file doubles as our review checklist: when a vendor ships a new agent, as OpenAI and Anthropic both did within the last two years, the gap is visible in the file itself and in our crawl logs.
The file is one piece of the setup we document in our guide on how we do GEO on our own site. That's the policy of a site that earns its living from AI citations. A publisher with paywalled archives would reasonably run config 2 or 3 — the point of this guide is not our answer but the table that lets you pick yours. Whichever config you choose, put a quarterly review on the calendar: the roster keeps moving — OpenAI added OAI-SearchBot in 2024, Anthropic added Claude-SearchBot in 2025, Amazon split its crawling into three agents by 2026 — and a robots.txt written in 2024 is already wrong about today's crawlers.
Related questions
Should I block AI crawlers?
Split the question by job. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot) costs no search visibility — explicitly documented by Google and Apple, structural for the rest. The exception is Meta-ExternalAgent, which trains and indexes in one agent: blocking it exits both. Blocking search-index and user-fetch agents removes you from AI answers, which is where a growing share of buying questions gets asked. Decide the training question on principle; decide the visibility question on where your citations and traffic come from.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot only feeds model training. ChatGPT's search runs on OAI-SearchBot, and user-requested page reads go through ChatGPT-User — block GPTBot and both keep working. It's the same separation Anthropic runs with ClaudeBot versus Claude-SearchBot and Claude-User.
Does Google-Extended affect AI Overviews?
No — and this is the most common mistake in the genre. Google documents that Google-Extended controls Gemini training and grounding, and "does not impact a site's inclusion in Google Search." AI Overviews and AI Mode are part of Search itself: they're governed by Googlebot access and the snippet controls — nosnippet, data-nosnippet, max-snippet, noindex. The full mechanics are in our guide on appearing in Google AI Overviews.
How do I verify a crawler is genuine?
Never trust the user-agent string alone — one in eighteen requests claiming to be a known AI crawler is spoofed, per HUMAN Security's 2026 measurements. Check the source IP against the vendor's published list: OpenAI, Anthropic, Apple, Common Crawl and DuckDuckGo publish JSON IP lists, Amazon publishes its ranges on its developer page, and Google and Apple support reverse-DNS verification. Anything that fails the check gets treated as a scraper, whatever it calls itself.
Is robots.txt enough, or do I need llms.txt too?
They do different jobs. Robots.txt controls access — who may fetch what. llms.txt is a navigation aid — a machine-readable map that helps models find and interpret your key pages once they're in. Access first: an llms.txt behind a blanket Disallow helps nobody. llms.txt gets its own guide in this series.
Related guides
Sources
- 01Our guide: How do you appear in Google AI Overviews?
- 02Our guide: How do you do GEO on your own site in practice?
- 03OpenAI — bot documentation: GPTBot, OAI-SearchBot, ChatGPT-User
- 04Anthropic — crawler documentation: ClaudeBot, Claude-SearchBot, Claude-User
- 05Perplexity — crawler documentation: PerplexityBot, Perplexity-User
- 06Google — common crawlers and Google-Extended
- 07Google — AI features in Search and site owner controls
- 08Apple — Applebot and Applebot-Extended
- 09Meta — web crawlers: Meta-ExternalAgent, Meta-ExternalFetcher
- 10Common Crawl — CCBot documentation
- 11Amazon — Amazonbot documentation
- 12DuckDuckGo — DuckAssistBot documentation
- 13Cloudflare — the crawl-to-refer ratio of AI platforms (July 2025) and live AI Insights
- 14Cloudflare — Bytespider leads AI-bot traffic and blocks (July 2024)
- 15HAProxy — ~90% of our AI-crawler traffic is Bytespider (Oct 2024)
- 16HUMAN Security — 2026 State of AI traffic: crawler spoofing benchmarks
// share