Robot Text File

[Chance4Today]

Where do you locate one and is this essential to have for your website? When I first got started on this I had a tech support and they told me to have one like this.

User-Agent: *
Disallow:

Now I have gotten submitted my site to a few places and one of them says that this isn't good to have and is not readable. So I guess my question is where do I find one that I should be using for vbulletin and my sites. Very confusing I have done a search in all forums but only come up with hacks to use. Any suggestions on how to get the right robot.text ? Also have been looking should it be robot.text or robots.text? Is there a difference.

[Lynne]

You would put it in your root folder - so it would be here - /home/yoursitename/public_html/robots.txt Here's the top portion of mine:

User-agent: *
Crawl-delay: 40
Disallow: /forums/announcement.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/editpost.php
Disallow: /forums/member.php

So, you need to put the names of the files you don't want the spiders to crawl after the disallow. The crawl-delay works to only allow so many spiders of a certain type (like those yahoo slurp spiders!) onto the site at a time.

[Chance4Today]

OK so is that all I need to put in mine? I guess I am really young at all of this and have been searching the internet oh I might remember where one was let me look LOL

[Lynne]

Originally Posted by Chance4Today

OK so is that all I need to put in mine? I guess I am really young at all of this and have been searching the internet oh I might remember where one was let me look LOL

Well, you want to edit yours to list only the files you do not want the spiders to get into. I have several files I don't need them to index, and so I list them in the robots text. I also have problems with the yahoo slurp spiders which is why I have the crawl-delay. So, you just need to taylor yours to fit your needs.

If you google robots.txt, you will come up with several sites that explain all about way the file does and how to use it.

[snakes1100]

yahoo slurp bots dont listen anyways, only way to kill the majority of them is to ban all but one of the ips they come from, if i recall they spam from about 12 ips, but yahoo could of added more

[Lynne]

Originally Posted by snakes1100

yahoo slurp bots dont listen anyways, only way to kill the majority of them is to ban all but one of the ips they come from, if i recall they spam from about 12 ips, but yahoo could of added more

It works for me now. It took about three or four weeks, but the robots.txt finally worked.