Googlebot

Quick definition

Googlebot is the generic name for two Google Search web crawlers: Googlebot Smartphone and Googlebot Desktop. They fetch pages and resources so that later systems can render, process, and possibly index the content.

A fetch is only an early step. It does not mean the URL will be indexed, receive a snippet, or rank for a query. Google states three minimum technical requirements for indexing eligibility: Googlebot is not blocked, the page returns HTTP status 200, and it contains indexable content. Indexing is still not guaranteed.

Googlebot is not one robot

The name suggests one program methodically reading the entire web. Google's current description is different. Googlebot is a client of a centralized crawling platform that also serves other Google products under their own crawler names.

Google Search has two main variants:

• Googlebot Smartphone simulates a user on a mobile device.
• Googlebot Desktop simulates a user on a desktop device.

The variants send different HTTP user-agent strings. In robots.txt, however, both obey the product token Googlebot. A rule cannot target only the smartphone crawler or only the desktop crawler.

Why the smartphone crawler dominates

For most sites, Google uses the mobile version for indexing and ranking. Most Googlebot requests therefore come from the smartphone crawler, with a smaller share from desktop.

Responsive sites usually serve the same URL and HTML. Dynamic serving and separate mobile URLs create more room for mistakes. The mobile version should contain the same essential content, headings, metadata, robots directives, links, and structured data. Google may not see content that requires a user interaction such as clicking, swiping, or typing before it loads.

Mobile-first does not mean that a separate mobile index exists. It describes which version Google primarily uses to process and assess the page.

Googlebot, images, video, and news

Google says a robots.txt rule for Googlebot can affect Google Search, Discover, Google Images, Google Video, and Google News. It also documents specialized crawlers such as Googlebot-Image and Googlebot-Video, which have their own tokens and technical properties.

Google News has no separate crawler user agent anymore. Google fetches news content with the regular Googlebot user agents, but it still honors the Googlebot-News token in robots.txt for rules that only concern Google News. This is a specific legacy control, not a model for inventing arbitrary Googlebot subtypes.

If images or videos should appear in Search, the resources and their host pages must be accessible. Fetching a media file alone does not guarantee inclusion in image or video results.

How Googlebot discovers URLs

Googlebot discovers new URLs primarily through links in previously crawled pages. Sitemaps and URLs known from earlier crawls provide additional leads.

Internal links therefore do two jobs. They help people navigate and give crawlers real paths to new or deeper pages. A URL that exists only in a database and has no crawlable link may become known through a sitemap, but it remains poorly integrated into the site structure.

A sitemap is not a guaranteed indexing queue. It should contain canonical, indexable URLs genuinely intended for Search.

Separate robots.txt, noindex, and canonical

These three mechanisms serve different purposes.

robots.txt controls fetching

A Disallow rule tells compliant crawlers not to fetch the URL. The address can still appear in results if Google learns about it through other links. Without fetching the page, Google cannot read a noindex directive located on it.

noindex controls inclusion in the index

If a publicly accessible page should not be indexed, Google must be allowed to crawl it and find `noindex` in a robots meta tag or HTTP header. Blocking the same URL in robots.txt can prevent that processing.

Canonical helps with duplicates

A canonical is a hint about the preferred version of similar or duplicate pages. It does not block crawling, and Google is not required to select the specified URL.

Good diagnosis asks three separate questions: Can Googlebot fetch the URL? Is the content allowed to be indexed? Which URL does Google consider canonical?

Status codes and server responses

An indexable HTML page should return HTTP 200. Permanent redirects lead Googlebot to another URL; 404 and 410 identify missing content; 5xx indicates server failure. An error page returning 200 may be treated as a soft 404. The guide to HTTP status codes for SEO covers each response class and the fix that fits it.

The protocol matters even earlier. If http:// and https://, or www and non-www, all answer separately, Googlebot meets the same content several times. HTTP and HTTPS basics for SEO explains how one clean, secure host variant avoids that.

Slow or unstable infrastructure also affects crawling. Googlebot aims not to overload servers and can reduce activity when responses show problems. A lower crawl rate is therefore not automatically a quality judgment about the content.

The current fetch limits

Google's Googlebot documentation, last revised in February 2026, states the limits plainly: Googlebot crawls the first 2 MB of a supported file type and the first 64 MB of a PDF. At the cutoff, the downloaded portion is passed on and later bytes are ignored.

Referenced CSS and JavaScript resources are fetched separately and receive their own limits. Two megabytes is enormous for most HTML documents. Large inline images, scripts, or menus can still push the title, canonical, structured data, or main content past the boundary.

Treat these numbers as the current state of Google's documentation, not as permanent constants; Google can adjust them. They are technical guardrails, not a recommended target for normal page size.

Crawling, rendering, and indexing

A JavaScript application can be understood in three phases:

1. Googlebot takes a URL from the crawl queue, checks robots.txt, and fetches it.
2. Google may place a successful page in the rendering queue. The Web Rendering Service runs JavaScript with a current Chromium and produces rendered HTML.
3. Processing systems can use the rendered content and links for indexing and future crawling.

Rendering can be delayed. Google can execute JavaScript, but important information should not depend on fragile, user-triggered, or consistently failing requests. Server-side rendering or static output is not a Google requirement, though it can make a page faster and more robust for users and different crawlers.

When debugging a JavaScript page, compare the initial and rendered HTML. The important question is whether the main content, crawlable <a href> links, metadata, and structured data are present after rendering.

Verifying genuine Googlebot traffic

An HTTP user agent is easy to spoof. A request calling itself Googlebot does not automatically come from Google.

Google provides two reliable verification methods:

• Match the source IP against Google's published crawler IP ranges.
• Perform a reverse DNS lookup. The returned hostname must belong to googlebot.com, google.com, or googleusercontent.com, depending on the crawler type. Then perform a forward lookup to confirm that the hostname resolves back to the original IP.

Only after verification should a request be counted as genuine Googlebot traffic in access analysis or used to adjust blocking rules.

What server logs and Search Console each show

Server logs prove individual requests to your infrastructure. They show time, URL, user agent, status, transferred data, and often response time. After bot verification, they can reveal which URL patterns Googlebot fetches and which errors it encounters.

Search Console provides a different view. Crawl Stats aggregates crawl activity, Page indexing groups known URLs by index status, and URL Inspection helps investigate one page. None fully replaces raw logs. Raw logs, in turn, cannot tell you whether a crawled page was indexed or shown for relevant queries.

A sensible diagnostic order

1. Is the URL publicly reachable and linked through a crawlable internal path?
2. Does robots.txt allow the page and required resources to be fetched?
3. Does the URL return the intended status and a stable response?
4. Does critical information fall within the fetch boundary?
5. Are main content and links present in rendered HTML?
6. Do mobile content, metadata, and structured data match the intended version?
7. Does the page allow indexing, and is its canonical plausible?
8. What do Page indexing, URL Inspection, and Crawl Stats report?
9. Do verified logs confirm a Googlebot fetch?
10. Only then: does the content satisfy the search intent and offer enough independent value?

Example: a category page that only works in the browser

Take a hypothetical shop category page built as a client-side app. The server returns 200 with an almost empty HTML shell. Product tiles load later through an API call, and pagination uses buttons with click handlers instead of links.

In a browser, everything looks fine. For Googlebot, three things can go wrong. The fetch succeeds, but the main content only exists after rendering. If the API call fails or is blocked for crawlers, the rendered page stays thin. And because tiles and pagination are not <a href> links, Googlebot finds the product URLs only through the XML sitemap, if at all.

The fix is not a special Googlebot setting. Render the product list on the server or make sure it appears reliably in the rendered HTML, turn tiles and pagination into crawlable links, and confirm the result with URL Inspection in Search Console. Then check verified server logs to see whether Googlebot actually requests the product URLs.

Common mistakes

• treating every Googlebot fetch as proof of indexing
• trying to control smartphone and desktop separately in robots.txt
• using robots.txt as reliable deindexing
• expecting noindex to work on a simultaneously blocked URL
• treating a canonical as a crawl block
• trusting a user-agent string without verification
• equating JavaScript support with guaranteed rendering
• calling every low crawl frequency a crawl-budget problem
• inferring genuine Googlebot requests from Site Audit data

The Crawl Foundry perspective

Crawl Foundry's Site Audit crawls a website with its own crawler and keeps a crawl history. It checks the rendered HTML and records status codes, redirects, canonical and robots directives, and indexability. Internal links appear as the crawler found them in the DOM, with source, target, anchor text, and position. Because runs can be compared, you can confirm whether a fix actually changed the result.

That is reproducible evidence about the site's technical state, but it is not a Googlebot log or proof that Google fetched a URL in the same way. Search Console and verified server logs remain the sources for Google-specific evidence. Site Audit helps trace a cause across URL groups and verify a repair. It does not predict Google's indexing decision or crawl priority.

Related terms

• Crawling
• Indexing
• Crawl budget
• robots.txt
• XML sitemap
• Google Search Console
• Advanced technical SEO
• JavaScript rendering SEO
• Mobile-first indexing
• Server-side rendering
• Structured data
• Page speed

Review sources

• Google Search Central: What is Googlebot?
• Google Search Central: Technical requirements
• Google Search Central: JavaScript SEO basics
• Google Search Central: Mobile-first indexing best practices
• Google Crawling Infrastructure: Verify requests from Google crawlers
• Google Crawling Infrastructure: List of common crawlers
• Google Search Central: Inside Googlebot, demystifying crawling, fetching, and the bytes we process
• IETF: RFC 9309, Robots Exclusion Protocol