noindex

Quick definition

noindex is an instruction to search engines: do not show this page in your search results. You set it either in a robots meta tag in the HTML <head> or in an X-Robots-Tag HTTP response header. Google, Bing, and other major search engines support it.

The directive only works if the crawler can fetch the page and read it. When Google crawls a URL and finds noindex, it drops the page from its search results, even if other sites link to it. Until the next crawl, nothing changes.

Two ways to set noindex

The robots meta tag

For HTML pages, the most common form is a meta tag in the <head>:

html<meta name="robots" content="noindex">

The name robots addresses all crawlers. A specific crawler name addresses only that crawler, for example googlebot for Google Search. Values are not case-sensitive, and several values can be combined with commas, such as noindex, nofollow. The value none is shorthand for noindex, nofollow.

The tag has to be in the <head>. Google stops reading page metadata when it hits an element that does not belong in the head, so a robots meta tag placed after, for example, an <iframe> or invalid markup can be ignored.

The X-Robots-Tag header

For files that have no HTML head, such as PDFs, images, or video files, the server sends the directive as an HTTP header:

httpHTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

The header also works for HTML and is useful when you want to set rules for whole directories or file types in the server configuration. Like the meta tag, it can target a specific crawler, for example X-Robots-Tag: googlebot: noindex.

If a page carries several robots rules from different sources, Google combines them, and the more restrictive rule wins.

noindex needs crawling

This is the point where most noindex problems start. robots.txt controls whether a crawler may fetch a URL. noindex controls whether a fetched page may appear in search results. The two work at different stages.

If a URL is blocked in robots.txt, Google never fetches it and never sees the noindex on it. The URL can still show up in search results, typically without a description, because Google knows it from links. So the common combination "Disallow in robots.txt plus noindex on the page" does not remove a page. It hides the removal instruction.

The correct order for removing a page from Google is:

1. Leave the URL crawlable.
2. Add noindex in the meta tag or header.
3. Wait until Google has recrawled the URL, or request a recrawl through URL Inspection in Google Search Console.
4. Only after the page has dropped out, decide whether you also want to block crawling.

Google stopped supporting a noindex line inside robots.txt in September 2019. Rules like that are ignored.

How long removal takes

noindex takes effect when Google next crawls the page. For frequently crawled pages that can be days. For rarely crawled URLs deep in a site, it can be weeks or longer. A recrawl request through URL Inspection can speed this up for single URLs, but it is a request, not a guarantee.

If you need a page out of Google's results urgently, for example after accidentally publishing personal data, the Removals tool in Search Console hides it temporarily, for about six months. That buys time. The permanent fix still has to be noindex, deleting the page with a 404 or 410, or putting it behind a login.

noindex compared with the alternatives

noindex is one of several tools, and many problems need a different one.

The canonical row deserves attention. Google advises against using noindex to choose a canonical page within one site, because noindex removes the page from search completely rather than consolidating its signals. For URL variants, a canonical tag or a redirect is the right signal. A page that carries noindex and a canonical pointing to another URL sends contradictory instructions.

noindex and JavaScript

Google processes JavaScript sites in phases: it crawls the HTML, then renders the page later if needed. If the initial HTML contains a noindex, Google skips rendering and does not index the page. A script that removes the noindex afterward never gets the chance to run.

The reverse direction works differently. Google documents that JavaScript can add a robots meta tag during rendering, for example when a product API returns an error and the page should not be indexed. That is acceptable, but a noindex in the server response is more reliable and also works for crawlers that do not render JavaScript.

The practical consequence for audits: check both the raw HTML response and the rendered HTML. A page that looks indexable in the browser can be noindexed in the source.

Typical use cases

noindex makes sense for pages that need to be reachable for visitors but add nothing to search results:

• internal search result pages
• thank-you and confirmation pages after a form
• account, cart, and checkout steps
• filter combinations in shops that create near-identical listings
• tag or archive pages with very little unique content, a common form of thin content
• campaign landing pages meant only for paid traffic

Each case is a decision, not a default. A filter page for "red running shoes" may deserve to rank, while a page for "red running shoes, size 46, sorted by price descending" usually does not.

What noindex does not do

• It does not save crawl budget. Google has to fetch a page to see its noindex. Google's own crawl budget guidance recommends robots.txt, not noindex, for pages you do not want crawled at all.
• It does not protect content. Anyone with the URL can open the page. Private material needs authentication.
• It does not stop links from being followed immediately. noindex alone allows crawlers to follow the links on the page. Google's John Mueller has said publicly that pages kept on noindex for a long time are eventually crawled less and their links may be treated as nofollow. That is a statement, not formal documentation, but it is a reason not to hide important navigation hubs behind noindex.
• It does not control every AI system. For Google's AI features such as AI Overviews, a page has to be indexed and eligible for a snippet, so noindex excludes it there too. Other AI crawlers and assistants document their own controls, mostly through robots.txt user agents. Check each provider's documentation instead of assuming noindex covers them.

A worked example

An online shop relaunches on a new platform. Two weeks later, organic traffic to category pages has dropped sharply. Search Console's page indexing report shows a growing number of URLs under "Excluded by 'noindex' tag".

The source of the problem is a setting from the staging environment. The staging site used a global noindex in the HTML template to keep it out of Google. The setting was copied to production with the theme configuration. Because the meta tag sits in the server HTML, every category and product page now tells Google not to index it.

The team removes the tag, checks a sample of URLs with URL Inspection, requests recrawls for the most important categories, and resubmits the XML sitemap. Recovery takes several weeks, because Google has to recrawl each URL. The lasting fix is procedural: the staging site is now protected with a password instead of noindex, so there is nothing to copy by mistake.

A sensible audit order

1. List the URLs that should rank. Your XML sitemap should contain only those.
2. Check robots.txt: are any of these URLs blocked from crawling?
3. Check the raw HTML response and the HTTP headers for noindex.
4. Check the rendered HTML for robots meta tags added or changed by JavaScript.
5. Look for conflicts: noindex combined with a canonical to another URL, noindexed URLs in the sitemap, or noindexed pages in hreflang clusters.
6. Compare with Search Console: page indexing report, filtered to submitted pages, and URL Inspection for samples.
7. After fixes, request recrawls for key URLs and watch the report over the following weeks.

Common mistakes

• noindex from a staging environment or a CMS setting such as WordPress's "Discourage search engines" option going live
• blocking a URL in robots.txt and expecting the noindex on it to work
• noindex on paginated category pages, which can cut crawlers off from the products listed there
• using noindex instead of a canonical for URL variants
• listing noindexed URLs in the XML sitemap
• removing a noindex with JavaScript and expecting Google to see the change
• setting noindex on pages that are hreflang alternates of indexable pages
• forgetting PDFs and other files that need the X-Robots-Tag header

The Crawl Foundry perspective

Crawl Foundry's Site Audit uses our own crawler, keeps a crawl history, and checks the rendered HTML of each page. For every crawled URL it records status, metadata, indexability, and internal links, so a page that is excluded by noindex appears next to the links that point to it and the status code it returns. Because runs can be compared, you can check after a release or migration whether URLs that are meant to rank still allow indexing.

The Site Audit for technical SEO shows the state of your site from our crawler's point of view. It does not show which URLs Google has crawled or indexed. For that, Search Console remains the source. Crawl Foundry can connect to Search Console with read-only access, and the final check for a single URL is still URL Inspection, because only Google can tell you how Google processed it.

Related terms

• Indexing
• Crawling
• Googlebot
• Search Console reports
• Duplicate content

Review sources

• Google Search Central: Block search indexing with noindex
• Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
• Google Search Central: Introduction to robots.txt
• Google Search Central: JavaScript SEO basics
• Google Search Central: How to specify a canonical URL
• Google Search Central: AI features and your website
• Google Search Central Blog: A note on unsupported rules in robots.txt
• Bing Webmaster Guidelines
• MDN Web Docs: X-Robots-Tag header