Skip to main content
Interpret AI bot logs: Separate purpose, policy and access
AI & Modern SEOIntermediate

Interpret AI bot logs: Separate purpose, policy and access

A bot name in a log establishes neither training nor a later citation. Record documented purpose, effective policy, checked identity and the delivered response separately.

Matthias RamahiMatthias Ramahi
8 min readNovember 5, 2026Legacy score: 95/100Updated Nov 17, 2026

The server log says GPTBot, the CDN report says “AI”, and the sales team wants to know whether the new documentation appears in AI answers. These describe different things. No individual record tells the whole story.

A useful investigation needs four separate fields: the provider's documented purpose, your intended rule, the checked identity and the actual response delivered. Only then consider an observed result in a search feature or assistant.

Our guide to blocking or allowing AI crawlers explains the corresponding robots.txt rules. This article concerns evidence in ongoing operations. Its example is entirely hypothetical. It contains no observed bot visits or search results from a real website.

A purpose directory is not an access report

A provider can use different agents for search, model development and user-triggered fetches. That distinction supports policy design. It does not measure what a particular document subsequently contributes to a system.

The current OpenAI overview describes OAI-SearchBot for ChatGPT search, GPTBot for content that may be used in model training, and ChatGPT-User for certain actions on a user's behalf. Search and training settings are independent. Where both bots are allowed, OpenAI may reuse one crawl for both purposes to avoid duplicate requests.

GPTBot log counts therefore cannot precisely count all training uses. Likewise, an OAI-SearchBot record does not establish a later citation of that page. The documented purpose explains a possible processing role, not an individually observed outcome.

Add a source field beside the bot name, linking to the provider description read on the review date. A broad CDN label remains evidence of that service's classification until you understand its exact mapping.

Distinguish names, policy tokens and products

robots.txt addresses named agents or control tokens. Not every such name must appear as a separate HTTP user agent in an access log.

Google documents precisely this exception for Google-Extended. It has no separate HTTP user agent. Its token controls certain uses of already crawled content for Gemini training and grounding. It affects neither inclusion in Google Search nor ranking there.

In its October 30, 2026 announcement, Google said it plans to support the shared __usage_ai-training token in 2027. Its current crawler documentation now gives early 2027 as the target. The token is described in the Internet-Draft on the machine-readable training preference signal. That document remains a work in progress, not an IETF-adopted standard. If a site uses this signal to opt out of training and does not explicitly specify Google-Extended, Google plans to interpret that as a Google-Extended disallow. This is announced future behavior, not an active enforcement result. Before a future robots.txt migration, record separately whether you want Google-Extended grounding in Gemini Apps.

A dashboard searching only for Google-Extended in user-agent strings therefore cannot supply a useful success count for this policy. Zero matches would establish neither that Google ignored the rule nor that it had no effect. The independently named request was never promised.

Do not interpret AI Overviews through this token either. Google-Extended preferences and Google Search controls have different product boundaries. Keep that boundary in the record instead of assuming one company name means one access setting.

User fetches do not share a universal bot rule

For ChatGPT-User, OpenAI says robots.txt rules may not apply because the actions are user initiated. This agent also does not determine whether content can appear in search. Blocking its name is therefore not equivalent to setting the search-bot preference.

Anthropic similarly describes three roles: ClaudeBot for potential training content, Claude-SearchBot for search and Claude-User for user-directed fetches. Anthropic also states that its bots respect robots.txt directives. Do not generalize the OpenAI exception to every agent with “User” in its name.

EntryDocumented roleWhat the name alone does not establish
OAI-SearchBot or Claude-SearchBotSearch-related retrievalA particular later answer or source selection
GPTBot or ClaudeBotCollection for potential model development or trainingThat a specific passage actually entered a model
ChatGPT-User or Claude-UserUser-related retrieval with different provider rulesWhich person requested which action
Google-ExtendedControl of certain content usesA separate HTTP visit under this name

This is not an exhaustive directory of agents. OpenAI also lists OAI-AdsBot for checking submitted advertising pages, for example. Keep unfamiliar names unclassified until checked rather than automatically counting them as training crawlers.

Perform two checks for each log finding

First, investigate identity. A user agent can be copied. Check the observed source address using the provider's current documented method where one is available. OpenAI links agent-specific IP lists; Anthropic links a public bot-address list.

An address match supports an origin finding within that verification method. It does not replace purpose classification or prove training. Record which list and version you checked. A generic cloud provider address is not equivalent evidence.

Then inspect the response. Which URL was requested? Did the CDN send a challenge, did the application send a login page, or did the origin deliver the intended article? Status, content type and, where appropriate, a sanitized body sample belong together. A 200 alone cannot answer this.

For this crawling, the decisive record may be upstream. An absent origin-log entry does not establish that no request occurred when the CDN served a cache hit or blocked it first. Record the log layer and period for which you actually have complete visibility.

A hypothetical public-documentation acceptance case

A fictional software company wants search features to access its public API guide while opting out of potential training. Account pages must require authentication. The invented public URL is https://example.com/docs/public-api; the private area is /account/.

The team first records its intended decision for each provider. It then plans these checks. None of the rows reports an executed test.

Planned caseEvidence to preserveLimit of the conclusion
Search bot requests the public guideMatching policy, checked origin and actual content responseNo proof of later selection as a source
Training bot is excluded for the guideAvailable policy and corresponding provider ruleNo retrospective deletion of previously collected content established
User-directed request for the guideProvider-specific rule and observed responseThe bot name does not identify a particular user
Unauthenticated request to /account/Access control delivers no private account contentrobots.txt does not provide this protection

The private area has a different technical requirement from public search discoverability. The Robots Exclusion Protocol explicitly says its rules are not access authorization. A disallow entry cannot replace authentication or server-side access control.

Distinguish content types in the policy too. A public guide, a product pricing page and a personal account area need not share one acceptance case. Inspect the actual response content before reporting reachability as the intended success.

A self-issued user agent does not test the bot's decision

A request with an OAI-SearchBot header you set yourself can show which response your server selects for that string. It does not establish an OpenAI visit. An ordinary HTTP client also does not automatically follow your robots.txt simply because you change its user-agent name.

If that local request receives an article despite Disallow, it does not establish misconduct by the real provider. You have initially observed a request you triggered yourself. Check rule evaluation separately and exclude synthetic probes from counts labelled as provider visits.

This distinction also matters for retrieval-augmented generation. A system can retrieve available sources and subsequently select or reject them. Successful transfer is only part of the process. An access log usually cannot observe later answer generation inside another system.

Store intended rules separately from unresolved evidence

One record for the fictional documentation could look like this. These fields are a proposed internal checklist, not a provider-defined format. The values deliberately describe an untested state:

json{
  "provider": "OpenAI",
  "documented_role": "search",
  "policy_token": "OAI-SearchBot",
  "scope": "https://example.com/docs/",
  "desired_policy": "allow",
  "identity_check": "not_observed",
  "delivery_check": "not_run",
  "source_selection": "not_observed"
}

Here, “allow” expresses only the intended preference. It does not establish that the provider visited or received the page. After a check, update only the field supported by evidence and attach that record. Confirming origin must not automatically resolve the unknown source-selection outcome.

Give the training preference its own entry with the appropriate token. This makes changes comparable without hiding search and training decisions behind one switch. Preserve earlier versions when a current log record needs comparison with an older rule.

Investigate differences with the next appropriate check

A recognized name with unresolved origin initially remains a claimed identity. A request from a plausible provider address receiving an unexpected response instead requires checking the URL, policy version and delivery. These are different assignments.

If a verified search bot receives a block page despite the intended preference, investigate the access path. The business decision to allow search need not change. If a synthetic training-bot probe receives a body despite the intended restriction, first investigate what the probe tested and how rules were evaluated.

When no visits appear, name the observed period and log coverage. “Not observed in the inspected CDN log” differs from “this provider never requests the site”. That distinction keeps missing evidence from quietly becoming either a success claim or an accusation in the handover.

Turn the policy into a testable handover

For each rule, record the host and path, documented purpose, exact token, intended decision and responsible person. Add the review date and expected technical evidence. “AI blocked” is too vague to serve as an acceptance record.

After release, recheck the same example URLs. Preserve the previous rules and identify a transition period. OpenAI gives approximately 24 hours for search-bot changes to be reflected. This is not a universal deadline for every provider, cache or product.

Include these cases in the SEO migration checks during a broader change. A new domain, additional host or inherited default configuration can alter the intended rules. Recording the selected paths makes that difference inspectable.

Crawl Foundry's Site Audit organizes technical findings around affected URLs and verifiable fixes. For the narrower question of whether a supported AI crawler can fetch a website, use the AI Crawler Checker. Log analysis still needs your own provider, policy and response evidence. Neither tool establishes the identity of a bot in your logs or a later use of its content.

For generative engine optimization, visible sources and search results remain separate observations. The initial acceptance statement is narrower: decision recorded, rule checked, origin verified where possible and actual response understood. What another system selects afterwards needs its own evidence.

Build a keyword database you own

Import and organize keywords, audit technical issues with evidence, and export everything at any time.

Start Free
Matthias Ramahi

Matthias Ramahi

Founder, Crawl Foundry

SEOKeyword ResearchTechnical SEO

Matthias Ramahi is the founder of Crawl Foundry and leads the product, its data sources, and its editorial standards. Crawl Foundry is his central software and SEO project.

Prefer Crawl Foundry on Google

Add crawlfoundry.com as a preferred source in your Google account and our articles will appear higher up on Google. Optional, and you can undo it at any time.

Set as preferred source

You Might Also Like

Search this website

Find product information, FAQ and documentation with links to the original pages.