The “robots” meta tag is used to indicate to search engines whether or not they should index a webpage and/or follow its links. The values for the “robots” meta tag are as follows:
- “index” – Allows the page to be indexed
- “noindex” – Prevents the page from being indexed
- “follow” – Allows the search engine to follow the links on the page
- “nofollow” – Prevents the search engine from following the links on the page
- “none” – Equivalent to “noindex, nofollow”
- “all” – Equivalent to “index, follow” (this is the default value if the tag is not present)
To use the “robots” meta tag, you would add the following code to the head of your HTML document:
<meta name="robots" content="value">
Where “value” is one of the options listed above. For example, if you want to prevent a page from being indexed but allow search engines to follow its links, you would use:
<meta name="robots" content="noindex, follow">
You can also use multiple meta robot tags on one page.
<meta name="robots" content="noindex">
<meta name="robots" content="nofollow">
It’s important to note that while the “robots” meta tag is a widely supported standard, search engines may choose to ignore it or interpret it differently.
If using WordPress most themes have a selection at the bottom of the editor to select the type of meta tag you wish to use.
Why keep a blog post or page out of the search results?
-
The post is no longer relevant: Bloggers might choose to remove a post from the search results if it is no longer relevant or has become outdated.
-
The post contains sensitive information: Bloggers might remove a post from the search results if it contains sensitive information that they would prefer not to be easily accessible.
-
The post is a draft or test post: Bloggers might choose to remove a post from the search results if it is a draft or test post that is not ready for public consumption.
-
The post is duplicate content: Bloggers might choose to remove a post from the search results if it is duplicate content that has been republished elsewhere.
-
The post is a promotional post: Bloggers might remove a post from the search results if the post is a promotional post that they want to be only visible to people who are directed to the post through a link, and not via a search engine.
-
The post is behind a paywall: Bloggers might choose to remove a post from the search results if the post is behind a paywall, and is only available to subscribers.
In general, the “noindex” value of the “robots” meta tag is used to remove a specific post or page from the search results while “nofollow” is used to prevent search engines from following links on the page.