Many site owners want to block AI crawlers or let specific ones in, and face a confusing list of bots. Some block everything that sounds like AI and disappear from AI answers they would actually like to appear in. Others leave everything open even though they do not want their content used to train models. Both can be avoided, because the major providers use different bots for different purposes.
After reading, you can tell AI crawlers apart by purpose, write rules that match your intent and check that they work.
Three purposes, three kinds of bots
Before you write a single line in robots.txt, one distinction helps. AI companies visit websites for three different reasons:
- Training. Content is collected to train future language models. Blocking here keeps your text out of training.
- Search and answers. Content is collected for an AI assistant's search feature, similar to a classic search engine. Blocking here means you no longer appear as a source in those answers.
- Fetches on behalf of a user. Someone asks an assistant to open or summarize a specific page. The bot then fetches that page once, much like a browser.
For most businesses the second group matters most, because it decides visibility in AI answers. The first group is a policy question about how you want your content used.
The main AI crawlers at a glance
OpenAI lists three relevant agents in its overview of bots. GPTBot collects content for training models. OAI-SearchBot collects pages for search in ChatGPT, and OpenAI states explicitly that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. ChatGPT-User fetches pages when users trigger it in ChatGPT or a custom GPT. Because these fetches are initiated by users, OpenAI says robots.txt rules may not apply to them.
Anthropic also separates by purpose. ClaudeBot collects content for model development, Claude-SearchBot improves search results and Claude-User fetches pages on behalf of users. According to Anthropic, its bots respect robots.txt. ClaudeBot additionally supports the Crawl-delay directive.
Perplexity runs PerplexityBot for its own search index and Perplexity-User for fetches triggered by users.
Google is a special case. Google-Extended is not a separate crawler but a token in robots.txt. It controls whether content Google crawls anyway may be used to train future Gemini models and for grounding in Gemini apps and Vertex AI. According to Google, it does not affect inclusion in Google Search or ranking. AI Overviews and AI Mode are part of Google Search and depend on the regular Googlebot.
There are other training crawlers as well, such as CCBot from Common Crawl, whose datasets many model providers use, or Applebot-Extended for Apple's use of content. The list changes regularly. Check the providers' current documentation before any larger change.
Writing the rules: three typical setups
The right configuration depends on what you want to achieve.
Setup 1: Allow everything. For many B2B companies, software vendors and service providers, visibility in AI answers is an advantage. They do not need to add anything as long as there is no general block.
Setup 2: Block training, allow AI search. This is the most common compromise for publishers and owners of original content:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Setup 3: Protect specific sections. If only certain content should stay out of training data, such as premium articles, block only those paths for the training crawlers and leave the rest open.
Whichever setup you choose, make sure rules for individual bots do not accidentally affect Googlebot. A "Disallow: /" line under "User-agent: *" blocks every crawler without its own group, including search engines.
What robots.txt cannot do
robots.txt is a request, not a technical lock. Reputable providers follow it, but not every crawler does. If you really need to protect content, you need server-side rules or a login as well.
A new rule also only works going forward. Content a provider has already collected does not disappear automatically. And it does not work instantly: OpenAI says it can take about 24 hours for OAI-SearchBot to reflect a change.
For Google Search, robots.txt is also the wrong tool if you want to control what appears in snippets and AI answers. Page-level directives such as nosnippet or max-snippet exist for that. Blocking Googlebot would instead remove the page from search altogether.
Checking that the rules work
After every change, a short check in three steps pays off.
- Test the rules. For every relevant bot, check whether it may fetch a sample URL. The free robots.txt checker for AI crawlers shows which AI and search crawlers your robots.txt allows or blocks and which rule applies.
- Look at your server logs. The logs show which bots actually visit. Anthropic and OpenAI publish the IP ranges of their bots, so you can tell real visits from spoofed user agents.
- Check your firewall and CDN. Many CDNs now offer their own switches for blocking AI bots. They work independently of robots.txt. If robots.txt allows AI search but the CDN switch blocks all AI bots, the switch wins.
Which setup fits you?
There is no universal answer. This orientation helps with the decision.
Online shops. Product pages benefit from appearing as a source in AI answers, for example in buying advice. Blocking the search bots is more likely to cost visibility than to protect anything. Whether you allow training is a policy question. Product descriptions are rarely the part of your content that needs special protection.
Software vendors and service providers. If you sell products that need explaining, you want to appear in answers to questions like "Which tool is best for ...". For this group, setup 1 is often the most sensible. More important than robots.txt is that documentation, pricing and comparison pages are crawlable and clearly structured.
Publishers and media. Original reporting is the product. Setup 2 keeps it out of training without giving up reach through AI search. Some publishers also block the search bots because they want clicks rather than summaries. That is a deliberate trade-off between reach and control.
Platforms with paid content. Content behind a paywall should not be freely accessible anyway. robots.txt complements the technical protection there, it does not replace it.
Common mistakes
- Blocking every bot with "AI" in its name and giving up visibility in AI search along the way.
- Blocking Google-Extended in the belief that it prevents AI Overviews.
- Maintaining rules only in robots.txt while a CDN switch has long been blocking all AI bots.
- Misspelling user agent names. They must be entered exactly as the provider documents them.
- Not checking robots.txt after a relaunch and carrying over a general block from the test environment.
AI crawlers and classic crawlability belong together
An AI search bot can only cite what it can fetch and understand. The same basics as in classic crawling apply: reachable URLs, a clean status code, content in the delivered HTML rather than only after JavaScript, a clear structure. Many AI crawlers do not execute JavaScript. If your main content is only built in the browser, they may see an empty page.
The glossary entry on retrieval-augmented generation explains how AI systems retrieve suitable sources. The entry on generative engine optimization covers what you can do so your content gets selected.
Crawl Foundry Site Audit shows for every crawled URL whether robots.txt blocks it, whether it carries noindex and whether its content can be captured without rendering. That also surfaces blocks nobody intended, such as a rule from the staging environment that went live with the last website migration.
Block training and still appear in AI answers
With AI crawlers, distinguish between training, search and fetches on behalf of users. If you want to block training and still appear in AI answers, block GPTBot, ClaudeBot, Google-Extended and CCBot and allow OAI-SearchBot, Claude-SearchBot and PerplexityBot. Then check the rules, your logs and your firewall, because robots.txt is only as effective as the crawlers that follow it.
Start by listing which bots currently reach your site, then decide which of the three options fits your business model.