The robots meta tag: every value, what each one does, and when to use the header instead

The robots meta tag is a line in the <head> of a page that tells search engines what they may do with that page once they have fetched it: keep it in the index or drop it, follow its links or ignore them, show a snippet or show nothing. It looks like <meta name="robots" content="noindex, follow">. The important distinction is this: robots.txt controls whether a crawler requests the page at all, and the robots meta tag controls what happens to the page after the crawler has read it. Because the directive lives inside the HTML, the crawler has to be allowed to fetch the page for the directive to count.

This page covers the robots meta tag and its HTTP header twin, the X-Robots-Tag. For crawl control and the file that governs it, see how to write a robots.txt.

Where the tag goes

It belongs in the head, before any body content, and there should be one robots meta tag per page per crawler name:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Thank you for your enquiry</title>
  <meta name="robots" content="noindex, follow">
</head>

The name attribute selects who is being addressed. robots speaks to every crawler that honours the standard. A crawler name speaks to one of them, and a named tag overrides the generic one for that crawler:

<meta name="robots" content="nosnippet">
<meta name="googlebot" content="max-snippet:160">

In that pair, Google reads its own line and other engines read the generic one. Values are comma separated and case insensitive.

The directives, and what each one produces

Value	What it does	Typical use
index / follow	The default behaviour, stated explicitly	Nothing. Omitting the tag says the same thing
noindex	Keeps the page out of search results	Thin pages, internal search results, thank you pages
nofollow	Stops the page passing signals through its links	Rare. Use link level attributes instead
none	Shorthand for noindex plus nofollow	Pages you want gone and sealed off
all	Shorthand for index plus follow	Overriding a site wide default in a template
noarchive	Suppresses a cached copy	Pages whose content changes by the hour
nosnippet	No text or video preview in results	Paywalled or licensed text
max-snippet:[n]	Caps the snippet at n characters	Publishers controlling how much is shown
max-image-preview:[setting]	none, standard or large image preview	Image heavy pages wanting the large treatment
max-video-preview:[n]	Caps a video preview at n seconds	Video libraries
noimageindex	Keeps images on the page out of image results	Licensed photography
notranslate	Suppresses the offer to translate	Pages whose wording is legally fixed
unavailable_after:[date]	Drops the page from results after a date	Event pages and time limited offers
indexifembedded	Allows indexing of noindex content when embedded in an indexable page	Media players loaded in an iframe

Google documents the current set and the exact behaviour of each one in its robots meta tag reference, which is the page to check when a client insists a directive exists.

The rule that catches almost everybody

A page that is blocked in robots.txt is never fetched, so its robots meta tag is never read. If you block a URL and also put noindex on it, the block wins and the directive is invisible. A URL in that state can still appear in search results as a bare link, because the engine knows it exists from links pointing at it without ever having seen the page.

So the sequence for removing a page from the index is: allow it to be crawled, serve noindex, wait for the recrawl, confirm it has dropped, and only then consider blocking the path. The same applies in reverse when you are trying to get something indexed: a stray noindex left in a template is one of the three or four reasons a brand new page never appears, which is where a crawl and index check pays for itself. The deeper mechanics are in how search engines work.

When to use the X-Robots-Tag header instead

A PDF, an image, a CSV or an XML feed has no HTML head, so there is nowhere to put a meta tag. Send the directive as a response header:

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, noarchive

In Apache, applying it to every PDF on the site:

<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex, noarchive"
</FilesMatch>

In Nginx:

location ~* \.(pdf|csv)$ {
  add_header X-Robots-Tag "noindex, noarchive";
}

The header accepts the same values as the tag, and it can also be addressed to one crawler with X-Robots-Tag: googlebot: noindex. Use the header for non HTML files and for whole directories, and the meta tag for individual pages where an editor needs to see the setting in the markup.

Patterns worth copying

• Internal search results. noindex, follow. They are infinite in number and useful to nobody arriving cold.
• Faceted filter URLs. noindex, follow on the combinations you do not want ranked, with a canonical on the ones you do.
• Thank you and confirmation pages. noindex, follow, so a goal page never shows up as an entry point.
• Tag and author archives on a small blog. noindex, follow when each one lists two posts. Index them once they are genuinely useful listings.
• Staging and preview hosts. HTTP authentication, with X-Robots-Tag: noindex as the second layer. A meta tag alone on a public staging site is one deploy away from going live by accident.
• Pagination. Index the paginated pages and let them keep follow. Removing them from the index removes the path to deep products.
• Expiring offers. unavailable_after with a real date, so the page leaves results on schedule rather than when somebody remembers.

How to verify what is actually being served

1. Fetch the page as a crawler would and read the head and the headers together: curl -sI https://example.com/page for the headers, and curl -s https://example.com/page | grep -i robots for the tag.
2. Check the rendered DOM as well as the source. A tag manager or a client side script can inject or remove a robots tag after load, and Google reads the rendered page.
3. Use the URL Inspection tool in Search Console and read the indexing section. It names the directive it found and the reason a page was excluded.
4. Crawl the whole site and list every URL carrying noindex. Review that list monthly, because the surprising entries are the point of the exercise.
5. After any template or plugin change, recheck one page of every type. Robots directives live in templates, so one edit moves hundreds of pages at once. The settings that do this on WordPress are covered in WordPress SEO.

Mistakes that cost pages

• Using noindex where a canonical belongs. Duplicates that you want consolidated should point at the original, not disappear. The difference is set out in canonical tags.
• Shipping two conflicting robots tags on one page. Engines take the most restrictive reading, so index plus noindex resolves to noindex.
• Putting the tag in the body. It has to be in the head to be honoured.
• Leaving nofollow site wide because somebody once read that it preserves crawl budget. It cuts the internal paths that distribute signals instead. Spend that effort on crawl budget properly.

What people ask about robots meta tag

Is there a difference between the robots meta tag and robots.txt?

Yes, and it is the whole point. robots.txt governs requests, so it decides what gets crawled. The robots meta tag governs what an engine does with a page it has already fetched, so it decides what gets indexed and how it is displayed.

Does noindex remove a page immediately?

It takes effect at the next crawl of that URL. Request indexing for the page in Search Console to bring the recrawl forward, and keep the page reachable until it has dropped.

Do I need to write index, follow on normal pages?

No. Those are the defaults. Write them only to override a site wide rule that a theme or a plugin has set on your behalf.

Does nofollow in the meta tag replace rel attributes on links?

Use the link level attributes for individual links: rel="sponsored" for paid placements, rel="ugc" for user generated content, rel="nofollow" where you do not want to pass an endorsement. The page level directive is a blunt instrument that applies to every link at once.

Do AI crawlers respect the robots meta tag?

Treat the robots meta tag and robots.txt as the two controls you set deliberately, then confirm the result for each named agent rather than assuming. Which agents to allow, and how to test the outcome, is covered in which AI crawlers to allow.

Can I noindex a page and still have it pass link signals?

Use noindex, follow. The page stays out of results and its outgoing links keep working, which is exactly what you want on filter pages and internal search results.

A directive aimed at crawlers, and the crawlers that now read for assistants

The robots meta tag tells a search engine whether to index a page and follow its links. Assistants fetch pages through their own crawlers, and each publishes its own user agent and rules. A noindex that works for Google answers a different question from whether a given assistant may read the page.

Check your robots.txt and headers for the agent names these systems publish, then ask each assistant about a page you meant to expose. AI Knows Us checks the answers from the buyer's side.