Where do you locate one and is this essential to have for your website? When I first got started on this I had a tech support and they told me to have one like this.
User-Agent: *
Disallow:
Now I have gotten submitted my site to a few places and one of them says that this isn't good to have and is not readable. So I guess my question is where do I find one that I should be using for vbulletin and my sites. Very confusing I have done a search in all forums but only come up with hacks to use. Any suggestions on how to get the right robot.text ? Also have been looking should it be robot.text or robots.text? Is there a difference.
11-20-2007, 02:01 PM
Robot Text File
Threaded Mode