robots.txt help

[Chachacha]

Hello all. I have been doing a bit of research here on vb.org about robots.txt and have added it to my site... but its been 3 days and nothing has changed. My biggest problem is Yahoo! Slurp Spider. There are constantly multiple instances of it on my Who's Online page and they are constantly spidering my newthread.php and other pages that I don't want them wasting their time on. Google is a good boy, it only appears once at any given time in my WOL and it seems to crawl pages that I'd like it to crawl.

I've read that the robots.txt will fix this problem... but nothing has changed in 3 days. Isn't it supposed to take effect within 24 hrs?

I chmodded it to 755 is that correct? If not, what should I chmod it to?

Heres an exact copy of my robots.txt. Tell me if its correct...

User-agent: Slurp
Crawl-delay: 60
User-agent: Fasterfox
Disallow: /
User-agent: *
Disallow: /forums/admincp/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/images/
Disallow: /forums/modcp/
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showpost.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/credits.php
Disallow: /forums/arcade.php
Disallow: /forums/vbimagehost.php

[Lynne]

The yahoo slurp! spider will take several days, if not a couple of weeks, until it starts obeying your robots.txt file. But it will finally do so, you just need to be patient.

[Chachacha]

Originally Posted by Lynne

The yahoo slurp! spider will take several days, if not a couple of weeks, until it starts obeying your robots.txt file. But it will finally do so, you just need to be patient.

Thanks for your reply. One more question. In the part of the robots.txt file that says--->

User-agent: Slurp
Crawl-delay: 60

...Should I change "Slurp" to "Yahoo! Slurp Spider"?

[Lynne]

According to this link, you just use Slurp. http://help.yahoo.com/l/us/yahoo/sea.../slurp-03.html

[Chachacha]

Originally Posted by Lynne

According to this link, you just use Slurp. http://help.yahoo.com/l/us/yahoo/sea.../slurp-03.html

You've been most helpful. Thank you very much.

[Chachacha]

Its been 2 weeks and Yahoo! Slurp Spider is still visiting pages that I've disallowed in my robots.txt.

[SEOvB]

Originally Posted by Chachacha

Its been 2 weeks and Yahoo! Slurp Spider is still visiting pages that I've disallowed in my robots.txt.

make sure the robots.txt is in your web root and not the forum root.

Make sure the paths are correct, ex if your forums are in domain.com/forums/ folder then your posted robots.txt is correct. If your forums are not in the domain.com/forums/ folder, then your robots.txt is incorrect.

[Chachacha]

Originally Posted by FRDS

make sure the robots.txt is in your web root and not the forum root.

Make sure the paths are correct, ex if your forums are in domain.com/forums/ folder then your posted robots.txt is correct. If your forums are not in the domain.com/forums/ folder, then your robots.txt is incorrect.

I have 3 copies of my robots.txt. One is in the MAIN folder when (if using c/panel) you go to "File Manager"... then I have one in the "Public_html" folder... and one in the "forums" folder. Yes, the path is correct.

[SEOvB]

you dont need any other than the public_html folder, they aren't hurting anything being in different locations to my knowledge, but they are just useless.

If the paths are correct and the yahoo spider is still going insane, try to run your robots.txt thru a robots.txt generator or checker (many available free online) to make sure there are no errors.

After that, i dont have a clue of why Yahoo isn't picking it up.

[Chachacha]

Originally Posted by FRDS

you dont need any other than the public_html folder, they aren't hurting anything being in different locations to my knowledge, but they are just useless.

If the paths are correct and the yahoo spider is still going insane, try to run your robots.txt thru a robots.txt generator or checker (many available free online) to make sure there are no errors.

After that, i dont have a clue of why Yahoo isn't picking it up.

Thank you. I will try that.

EDIT: There were errors. I should have had an empty line between each block of code. I fixed it. Hope that was the problem.