Googlebot at the firewall: Verify bots and diagnose blocks

Your website loads in a browser, the application reports no errors, and Google still cannot fetch the affected page. One possible cause sits in front of the application: the CDN's Web Application Firewall classifies the crawler as an unwanted bot and returns a block or a browser challenge.

A healthy server check will not prove that failure. Neither will a log entry containing the word Googlebot. Start with a specific request whose identity and response can be traced. That evidence tells you which protection to change and what a successful retest needs to establish.

Google discusses both hard blocks and verification interstitials in its guidance on CDNs and crawling. An error page returned with HTTP 200 is particularly misleading: the transport status says success while the intended content was not delivered. Always inspect the response body alongside its status.

Separate the questions before changing the firewall

Choose one affected URL and a bounded time window. Record when the failure was observed and which request or tool exposed it. A ticket saying only “Googlebot is blocked” leaves too many possible paths for an infrastructure team to investigate.

Then separate four questions. Did the request reach the CDN? Did the CDN forward it to the origin? What response reached the client? Does the request belong to the Google crawler category you intend to test?

Those observations are related but not interchangeable. An origin log with no matching entry may indicate an edge block. It may also mean the CDN served a valid cached response without contacting the origin. The edge event is needed to distinguish the two.

Crawling is also separate from later indexing. This investigation establishes whether a particular fetch could reach the intended content. It does not automatically explain every decline in organic visibility affecting the same site.

Keep the hypothesis narrow while collecting evidence. If one category path fails, do not assume the whole hostname is unavailable. If one bot category is affected, do not assume all automated requests receive the same treatment. Smaller, well-described cases are easier to reproduce and repair.

A fictional example: A country rule catches the crawler

Consider an invented shop at example.com. A country-based rule protects its public product pages. After a configuration change, some requests receive a browser challenge. Staff members test from their usual location and continue to see the product.

During the failure window, a URL inspection fetch instead shows the challenge. The CDN records a challenge action and a rule ID for the matching request. The request does not appear at the origin because the edge stopped it first. The application never had an opportunity to report an error.

The working hypothesis is now precise: this edge rule prevented this fetch. It is not a general claim that CDNs harm SEO. Other pages and clients may remain unaffected. Include a public delivery-path check after a website migration or infrastructure change, especially when security settings move with the hosting setup.

Before creating an exception, establish the request category. A tool fetching a URL on a user's behalf may have a different identity from an ordinary indexing crawler. Passing the tool's test and allowing the routine crawler are useful but distinct outcomes.

Verify Googlebot without trusting its name

The HTTP user-agent is a client-supplied statement. Another crawler can write Googlebot in that header. A rule based solely on the string would therefore be available to unrelated clients too.

Google's request verification documentation describes two approaches: match the source IP to the published ranges for the relevant crawler category, or perform reverse DNS followed by a confirming forward lookup. The documentation distinguishes common crawlers, special-case crawlers and user-triggered fetchers.

For a single event, the DNS method is easy to explain to a reviewer. Look up the source IP in reverse. Check the returned hostname against the domain documented for the expected category. Resolve that hostname forward and verify that the original IP is among the answers. A hostname merely containing the text google is insufficient.

You can run the two lookups with nslookup:

textnslookup IP-FROM-THE-EDGE-LOG
nslookup HOSTNAME-FROM-THE-FIRST-LOOKUP

Replace the placeholders with the values from the event you are investigating. For ongoing analysis, use the current official range list and record when it was retrieved. A copied list left unchanged for months is a weak basis for a permanent rule.

The source of the IP matters as well. Behind a reverse proxy, an application log may show the CDN's address. The original client address must come through a trusted proxy chain. A visitor-supplied forwarding header is not evidence of crawler identity. This is one reason to begin with the provider's matching edge event rather than an ambiguous application log.

Save what the public endpoint actually returns

Fetch the public URL with an HTTP client and retain both headers and body. This initial request does not automatically follow redirects:

bashcurl -sS -D edge-headers.txt -o edge-body.html "https://example.com/products/hiking-boot"

On Windows PowerShell, use curl.exe if necessary to call the executable. Inspect the status, redirect destination, content type and actual body. Does the body contain the product information or only a verification screen? A challenge with HTTP 200 is not a successful product-page fetch.

This local request comes from your machine. It describes the response to that request, not a verified Googlebot request. Changing its user-agent can help isolate a rule that matches text, but it does not turn your source IP into Google's or test how a verified-bot classifier behaves.

Avoid rapidly repeating requests during the first investigation. Doing so can trigger a rate limit that was not responsible for the original failure. A small set of timestamped checks is usually more useful than a burst of unlabelled requests that changes the conditions under test.

Do not treat a HEAD result as a complete substitute for a page fetch either. A header-only response cannot show whether the body is a challenge, and security systems may handle request methods differently. Use the request type that matches the evidence you are trying to compare.

Read edge, origin and rendering evidence together

Find the event by time, path and request identifier where available. Record the triggered rule, action and verified bot classification if the provider exposes them. If the plan or log export omits a field, record the gap rather than filling it with an assumption.

Compare the event with the origin logs. A request blocked at the edge may never appear there. A forwarded request can reach the application and fail for a different reason. Even when both layers report 200, inspect whether the returned document contains the intended page.

Align the timestamps before matching events. A dashboard may show local time while the server records UTC. Use the path and request ID too, so two similar requests are not mistaken for the same event. If an export contains only sampled traffic, a missing entry does not establish that no request occurred. Record its coverage and available time range.

If the alleged Googlebot request cannot be verified or connected to an edge event, the firewall hypothesis remains open. A suspicious rule and a visibility decline are not that connection. Prepare the URL, time window, observed response and missing fields for the responsible operator. Avoid changing several protections at once merely to obtain a passing test. That would leave the effective change and its wider consequences unclear.

For JavaScript applications, include critical resources. The main document can load while a necessary script or content API is blocked. The rendered page may then contain only an application shell. These rendering and edge failures require checking the dependent resource path as well as the page URL.

The following matrix helps prevent an observation from becoming a broader claim than it supports:

Google Search Console adds evidence from Google's fetch and rendering perspective. Record the test type and time. An older indexed observation and a current live test may describe different releases or security configurations.

Change the protection responsible for the failure

Once the evidence connects a request to a rule, adjust that rule deliberately. The objective is to make the intended public content reachable by the chosen verified crawler category. One false positive does not justify disabling every protection on the hostname.

Cloudflare, for example, documents a country-based challenge rule that excludes known bots. The condition uses the provider's bot classification rather than an arbitrary user-agent string. It is an example of a specific rule, not a promise that every other security phase will be bypassed.

Check rule order, the protection phase involved and the conditions available in the deployed product. An exception to one custom WAF rule may leave a separate rate limit or bot control unchanged. State exactly which mechanism was changed in the incident record.

Temporary overload also needs honest status handling. Google's HTTP status documentation explains that 429 and server errors can slow crawling. These responses are not a permanent SEO repair, and a prolonged failure can eventually affect existing index entries.

Keep technical access separate from permission in robots.txt. A URL allowed by crawler directives may still fail at the firewall. The same distinction matters when evaluating search and AI crawler access, where different bot purposes need their own decisions.

Retest the path that originally failed

Preserve the relevant pre-change configuration so the adjustment can be reversed precisely. After the correction, repeat the documented cases. Include the affected crawler category, an ordinary public request and a control request the rule is still intended to reject.

Group affected pages by template and delivery path. The Crawl Foundry site audit workflow helps connect technical findings to the examined URL scope. Add verified identity and the specific WAF event from infrastructure evidence. A successful platform crawl alone does not prove Googlebot succeeded.

Then observe comparable real events in production. Record which crawler categories were actually seen and the types of failures remaining. A change in crawl budget cannot be established from one successful request. It requires a relevant observation window and a clear baseline.

A defensible resolution is specific: the triggering rule was adjusted, the investigated verified request now receives the intended page, and the control case remains protected. Any later change in indexing or rankings is a separate observation. The repaired behavior is access to the content through the delivery path that previously failed.