The server log says GPTBot, the CDN report says “AI”, and the sales team wants to know whether the new documentation appears in AI answers. These describe different things. No individual record tells the whole story.
A useful investigation needs four separate fields: the provider's documented purpose, your intended rule, the checked identity and the actual response delivered. Only then consider an observed result in a search feature or assistant.
Our guide to blocking or allowing AI crawlers explains the corresponding robots.txt rules. This article concerns evidence in ongoing operations. Its example is entirely hypothetical. It contains no observed bot visits or search results from a real website.
A purpose directory is not an access report
A provider can use different agents for search, model development and user-triggered fetches. That distinction supports policy design. It does not measure what a particular document subsequently contributes to a system.
The current OpenAI overview describes OAI-SearchBot for ChatGPT search, GPTBot for content that may be used in model training, and ChatGPT-User for certain actions on a user's behalf. Search and training settings are independent. Where both bots are allowed, OpenAI may reuse one crawl for both purposes to avoid duplicate requests.
GPTBot log counts therefore cannot precisely count all training uses. Likewise, an OAI-SearchBot record does not establish a later citation of that page. The documented purpose explains a possible processing role, not an individually observed outcome.
Add a source field beside the bot name, linking to the provider description read on the review date. A broad CDN label remains evidence of that service's classification until you understand its exact mapping.
Distinguish names, policy tokens and products
robots.txt addresses named agents or control tokens. Not every such name must appear as a separate HTTP user agent in an access log.
Google documents precisely this exception for Google-Extended. It has no separate HTTP user agent. Its token controls certain uses of already crawled content for Gemini training and grounding. It affects neither inclusion in Google Search nor ranking there.
In its October 30, 2026 announcement, Google said it plans to support the shared __usage_ai-training token in 2027. Its current crawler documentation now gives early 2027 as the target. The token is described in the Internet-Draft on the machine-readable training preference signal. That document remains a work in progress, not an IETF-adopted standard. If a site uses this signal to opt out of training and does not explicitly specify Google-Extended, Google plans to interpret that as a Google-Extended disallow. This is announced future behavior, not an active enforcement result. Before a future robots.txt migration, record separately whether you want Google-Extended grounding in Gemini Apps.
A dashboard searching only for Google-Extended in user-agent strings therefore cannot supply a useful success count for this policy. Zero matches would establish neither that Google ignored the rule nor that it had no effect. The independently named request was never promised.
Do not interpret AI Overviews through this token either. Google-Extended preferences and Google Search controls have different product boundaries. Keep that boundary in the record instead of assuming one company name means one access setting.
User fetches do not share a universal bot rule
For ChatGPT-User, OpenAI says robots.txt rules may not apply because the actions are user initiated. This agent also does not determine whether content can appear in search. Blocking its name is therefore not equivalent to setting the search-bot preference.
Anthropic similarly describes three roles: ClaudeBot for potential training content, Claude-SearchBot for search and Claude-User for user-directed fetches. Anthropic also states that its bots respect robots.txt directives. Do not generalize the OpenAI exception to every agent with “User” in its name.
| Entry | Documented role | What the name alone does not establish |
|---|---|---|
| OAI-SearchBot or Claude-SearchBot | Search-related retrieval | A particular later answer or source selection |
| GPTBot or ClaudeBot | Collection for potential model development or training | That a specific passage actually entered a model |
| ChatGPT-User or Claude-User | User-related retrieval with different provider rules | Which person requested which action |
| Google-Extended | Control of certain content uses | A separate HTTP visit under this name |
This is not an exhaustive directory of agents. OpenAI also lists OAI-AdsBot for checking submitted advertising pages, for example. Keep unfamiliar names unclassified until checked rather than automatically counting them as training crawlers.
Perform two checks for each log finding
First, investigate identity. A user agent can be copied. Check the observed source address using the provider's current documented method where one is available. OpenAI links agent-specific IP lists; Anthropic links a public bot-address list.
An address match supports an origin finding within that verification method. It does not replace purpose classification or prove training. Record which list and version you checked. A generic cloud provider address is not equivalent evidence.
Then inspect the response. Which URL was requested? Did the CDN send a challenge, did the application send a login page, or did the origin deliver the intended article? Status, content type and, where appropriate, a sanitized body sample belong together. A 200 alone cannot answer this.
For this crawling, the decisive record may be upstream. An absent origin-log entry does not establish that no request occurred when the CDN served a cache hit or blocked it first. Record the log layer and period for which you actually have complete visibility.
A hypothetical public-documentation acceptance case
A fictional software company wants search features to access its public API guide while opting out of potential training. Account pages must require authentication. The invented public URL is https://example.com/docs/public-api; the private area is /account/.
The team first records its intended decision for each provider. It then plans these checks. None of the rows reports an executed test.
| Planned case | Evidence to preserve | Limit of the conclusion |
|---|---|---|
| Search bot requests the public guide | Matching policy, checked origin and actual content response | No proof of later selection as a source |
| Training bot is excluded for the guide | Available policy and corresponding provider rule | No retrospective deletion of previously collected content established |
| User-directed request for the guide | Provider-specific rule and observed response | The bot name does not identify a particular user |
Unauthenticated request to /account/ | Access control delivers no private account content | robots.txt does not provide this protection |
The private area has a different technical requirement from public search discoverability. The Robots Exclusion Protocol explicitly says its rules are not access authorization. A disallow entry cannot replace authentication or server-side access control.
Distinguish content types in the policy too. A public guide, a product pricing page and a personal account area need not share one acceptance case. Inspect the actual response content before reporting reachability as the intended success.
A self-issued user agent does not test the bot's decision
A request with an OAI-SearchBot header you set yourself can show which response your server selects for that string. It does not establish an OpenAI visit. An ordinary HTTP client also does not automatically follow your robots.txt simply because you change its user-agent name.
If that local request receives an article despite Disallow, it does not establish misconduct by the real provider. You have initially observed a request you triggered yourself. Check rule evaluation separately and exclude synthetic probes from counts labelled as provider visits.
This distinction also matters for retrieval-augmented generation. A system can retrieve available sources and subsequently select or reject them. Successful transfer is only part of the process. An access log usually cannot observe later answer generation inside another system.
Store intended rules separately from unresolved evidence
One record for the fictional documentation could look like this. These fields are a proposed internal checklist, not a provider-defined format. The values deliberately describe an untested state:
json{
"provider": "OpenAI",
"documented_role": "search",
"policy_token": "OAI-SearchBot",
"scope": "https://example.com/docs/",
"desired_policy": "allow",
"identity_check": "not_observed",
"delivery_check": "not_run",
"source_selection": "not_observed"
}
Here, “allow” expresses only the intended preference. It does not establish that the provider visited or received the page. After a check, update only the field supported by evidence and attach that record. Confirming origin must not automatically resolve the unknown source-selection outcome.
Give the training preference its own entry with the appropriate token. This makes changes comparable without hiding search and training decisions behind one switch. Preserve earlier versions when a current log record needs comparison with an older rule.
Investigate differences with the next appropriate check
A recognized name with unresolved origin initially remains a claimed identity. A request from a plausible provider address receiving an unexpected response instead requires checking the URL, policy version and delivery. These are different assignments.
If a verified search bot receives a block page despite the intended preference, investigate the access path. The business decision to allow search need not change. If a synthetic training-bot probe receives a body despite the intended restriction, first investigate what the probe tested and how rules were evaluated.
When no visits appear, name the observed period and log coverage. “Not observed in the inspected CDN log” differs from “this provider never requests the site”. That distinction keeps missing evidence from quietly becoming either a success claim or an accusation in the handover.
Turn the policy into a testable handover
For each rule, record the host and path, documented purpose, exact token, intended decision and responsible person. Add the review date and expected technical evidence. “AI blocked” is too vague to serve as an acceptance record.
After release, recheck the same example URLs. Preserve the previous rules and identify a transition period. OpenAI gives approximately 24 hours for search-bot changes to be reflected. This is not a universal deadline for every provider, cache or product.
Include these cases in the SEO migration checks during a broader change. A new domain, additional host or inherited default configuration can alter the intended rules. Recording the selected paths makes that difference inspectable.
Crawl Foundry's Site Audit organizes technical findings around affected URLs and verifiable fixes. For the narrower question of whether a supported AI crawler can fetch a website, use the AI Crawler Checker. Log analysis still needs your own provider, policy and response evidence. Neither tool establishes the identity of a bot in your logs or a later use of its content.
For generative engine optimization, visible sources and search results remain separate observations. The initial acceptance statement is narrower: decision recorded, rule checked, origin verified where possible and actual response understood. What another system selects afterwards needs its own evidence.