Tutorial: How to use the Meta Robots tag — code examples, meanings, and all the major REP directives.
I’ve seen a whole lot of websites lately that are wasting space (and the time it took to write it) by implementing the Meta Robots tag like this:
<meta name="robots" content="all, index, follow" />
If you want search engine bots to crawl and index your website, then just leave it alone. By default, search engines will crawl, index, and archive every page on your website that they can find by following links. You also don’t have to say “all” — every bot that obeys the Robots Meta tag will obey it by default.
So not only is that example pointless because that is the default, it is repeating the same command twice!
