It is always good to use it though many seedy search engines ignore it. The main searches though like Google and Bing honor it. Those are the ones you care about really. Others you can use htaccess to block if you are concerned with them.
You can use it to block pages you do not want indexed in the Google, etc. For example if you have a really spammy advertising section or something. You also add your sitemap there to keep google crawling it though you should have your sitemap submitted using webmaster tools anyways.
On my sites I use it to keep the focus on my topic content. SEO is about quality, not quantity so I do not index my off topic forums.
Also, as done in yours you can use robots.txt to put a delay on search bots if your resources are limited.
As for Attractaca.. I would not waste time using that.
|