I’ve seen a whole lot of websites lately that are wasting space (and the time it took to write it) by implementing the Meta Robots tag like this:

1. <meta name="robots" content="all, index, follow" />

If you want search engine bots to crawl and index your website, then just leave it alone. By default, search engines will crawl, index, and archive every page on your website that they can find by following links. You also don’t have to say “all” — every bot that obeys the Robots Meta tag will obey it by default.

So not only is that example pointless because that is the default, it is repeating the same command twice!

Here are some handy uses for the Meta robots tag (these work for Yahoo!, MSN, and Google bots):

Normal talk: Just ignore this page – don’t keep a copy because I don’t want to see it in search engine results. I don’t want you to follow the links on this page, either.

Code:

1. <meta name="robots" content="none" />

(surprised that it’s not “noindex, nofollow”? – that’s just a long and unnecessary of telling search engines to ignore the page completely)

Normal talk: I don’t want to see this page in search results, so don’t keep a copy of this page. You can follow the links on this page, though – those websites deserve credit.

Code:

1. <meta name="robots" content="noindex, follow" />

Normal talk: Take as many copies of this page as you want and hook me up with traffic from your search engine results. Don’t follow the links on this page though, those websites aren’t that great anyways.

Code:

1. <meta name="robots" content="index, nofollow" />

Normal talk: You can make a copy of this page to plug into your algorithm, but don’t give anyone access to it with the “cache” link in search engine results.

Code:

1. <meta name="robots" content="noarchive" />

Normal talk: I don’t want you to pull any text from my website to use in search engine results. Just a link will do. You’re free to index this page and follow all links.

Code:

1. <meta name="robots" content="nosnippet" />

Normal talk: Stop using my Dmoz.org listing description in search results – I want you to use a snippet of text from the page to show users the contextual relevance of my site to their search query.

Code:

1. <meta name="robots" content="noodp" />

(NOODP is an acronym “No Open Directory Project”)

Some Useful Googlebot Meta Tags:

These Meta tags work exclusively in Google (all of these would work with the “robots” name, but since Googlebot is the only bot that officially follows these REP directives, I’ll use “googlebot” or “google”, in the case of the notranslate tag):

Normal talk: Ignore the images on this page – I don’t want to see them in Google Image Search.

Code:

1. <meta name="googlebot" content="noimageindex">

Normal talk: Heads up – this page will be out of commission (deleted, expired, moved, etc.) on this day.

Code:

1. <meta name="googlebot" content="unavailable_after: 13-DEC-2011 11:45:00 PST">

Normal talk: Don’t translate the content on this page into other languages.

Code:

1. <meta name="google" value="notranslate">

! Notice that in this case, the Meta name is “google”, not “googlebot”.