If you're new here, you may want to subscribe to my RSS feed. My blog is still pretty new, and is mainly for my own SEO web design reference but feel free to bookmark and comment. Thanks for visiting!
Tutorial: How to use the Meta Robots tag — code examples, meanings, and all the major REP directives.
I’ve seen a whole lot of websites lately that are wasting space (and the time it took to write it) by implementing the Meta Robots tag like this:
<meta name="robots" content="all, index, follow" />
If you want search engine bots to crawl and index your website, then just leave it alone. By default, search engines will crawl, index, and archive every page on your website that they can find by following links. You also don’t have to say “all” — every bot that obeys the Robots Meta tag will obey it by default.
So not only is that example pointless because that is the default, it is repeating the same command twice!
Here are some handy uses for the Meta robots tag (these work for Yahoo!, MSN, and Google bots):
Normal talk: Just ignore this page – don’t keep a copy because I don’t want to see it in search engine results. I don’t want you to follow the links on this page, either.
Code:
<meta name="robots" content="none" />
(surprised that it’s not “noindex, nofollow”? – that’s just a long and unnecessary of telling search engines to ignore the page completely)
Normal talk: I don’t want to see this page in search results, so don’t keep a copy of this page. You can follow the links on this page, though – those websites deserve credit.
Code:
<meta name="robots" content="noindex, follow" />
Normal talk: Take as many copies of this page as you want and hook me up with traffic from your search engine results. Don’t follow the links on this page though, those websites aren’t that great anyways.
Code:
<meta name="robots" content="index, nofollow" />
Normal talk: You can make a copy of this page to plug into your algorithm, but don’t give anyone access to it with the “cache” link in search engine results.
Code:
<meta name="robots" content="noarchive" />
Normal talk: I don’t want you to pull any text from my website to use in search engine results. Just a link will do. You’re free to index this page and follow all links.
Code:
<meta name="robots" content="nosnippet" />
Normal talk: Stop using my Dmoz.org listing description in search results – I want you to use a snippet of text from the page to show users the contextual relevance of my site to their search query.
Code:
<meta name="robots" content="noodp" />
(NOODP is an acronym “No Open Directory Project”)
Some Useful Googlebot Meta Tags:
These Meta tags work exclusively in Google (all of these would work with the “robots” name, but since Googlebot is the only bot that officially follows these REP directives, I’ll use “googlebot” or “google”, in the case of the notranslate tag):
Normal talk: Ignore the images on this page – I don’t want to see them in Google Image Search.
Code:
<meta name="googlebot" content="noimageindex">
Normal talk: Heads up – this page will be out of commission (deleted, expired, moved, etc.) on this day.
Code:
<meta name="googlebot" content="unavailable_after: 13-DEC-2011 11:45:00 PST">
Normal talk: Don’t translate the content on this page into other languages.
Code:
<meta name="google" value="notranslate">
! Notice that in this case, the Meta name is “google”, not “googlebot”.
I just inherited a site that I did not develop. For whatever reason, although it is cleanly coded, and optimized, Google is not picking it up whatsoever. I was wondering if having two meta robot tags would cause the site to just be ignored altogether… here is how they appear:
I will delete the second one anyway, to be proper, but i’m wondering if that is indeed the main problem they are experiencing.
Sorry for the long delay @Christian, I didn’t get that code – if you want to email it to me in my contact form, I’ll be more than happy to help, although I think I’m a bit late in responding so you probably have it figured it out by now.