The Arcive of Official vBulletin Modifications Site.It is not a VB3 engine, just a parsed copy! |
|
#1
|
||||
|
||||
robots.txt help
Hello all. I have been doing a bit of research here on vb.org about robots.txt and have added it to my site... but its been 3 days and nothing has changed. My biggest problem is Yahoo! Slurp Spider. There are constantly multiple instances of it on my Who's Online page and they are constantly spidering my newthread.php and other pages that I don't want them wasting their time on. Google is a good boy, it only appears once at any given time in my WOL and it seems to crawl pages that I'd like it to crawl.
I've read that the robots.txt will fix this problem... but nothing has changed in 3 days. Isn't it supposed to take effect within 24 hrs? I chmodded it to 755 is that correct? If not, what should I chmod it to? Heres an exact copy of my robots.txt. Tell me if its correct... Code:
User-agent: Slurp Crawl-delay: 60 User-agent: Fasterfox Disallow: / User-agent: * Disallow: /forums/admincp/ Disallow: /forums/clientscript/ Disallow: /forums/cpstyles/ Disallow: /forums/customavatars/ Disallow: /forums/customprofilepics/ Disallow: /forums/images/ Disallow: /forums/modcp/ Disallow: /forums/ajax.php Disallow: /forums/attachment.php Disallow: /forums/calendar.php Disallow: /forums/cron.php Disallow: /forums/editpost.php Disallow: /forums/global.php Disallow: /forums/image.php Disallow: /forums/inlinemod.php Disallow: /forums/joinrequests.php Disallow: /forums/login.php Disallow: /forums/misc.php Disallow: /forums/moderator.php Disallow: /forums/newattachment.php Disallow: /forums/newreply.php Disallow: /forums/newthread.php Disallow: /forums/online.php Disallow: /forums/poll.php Disallow: /forums/postings.php Disallow: /forums/printthread.php Disallow: /forums/private.php Disallow: /forums/profile.php Disallow: /forums/register.php Disallow: /forums/report.php Disallow: /forums/reputation.php Disallow: /forums/search.php Disallow: /forums/sendmessage.php Disallow: /forums/showpost.php Disallow: /forums/subscription.php Disallow: /forums/threadrate.php Disallow: /forums/usercp.php Disallow: /forums/usernote.php Disallow: /forums/credits.php Disallow: /forums/arcade.php Disallow: /forums/vbimagehost.php |
#2
|
||||
|
||||
The yahoo slurp! spider will take several days, if not a couple of weeks, until it starts obeying your robots.txt file. But it will finally do so, you just need to be patient.
|
#3
|
||||
|
||||
Quote:
User-agent: Slurp Crawl-delay: 60 ...Should I change "Slurp" to "Yahoo! Slurp Spider"? |
#4
|
||||
|
||||
According to this link, you just use Slurp. http://help.yahoo.com/l/us/yahoo/sea.../slurp-03.html
|
#5
|
||||
|
||||
Quote:
|
#6
|
||||
|
||||
Its been 2 weeks and Yahoo! Slurp Spider is still visiting pages that I've disallowed in my robots.txt.
|
#7
|
||||
|
||||
Quote:
Make sure the paths are correct, ex if your forums are in domain.com/forums/ folder then your posted robots.txt is correct. If your forums are not in the domain.com/forums/ folder, then your robots.txt is incorrect. |
#8
|
||||
|
||||
Quote:
|
#9
|
||||
|
||||
you dont need any other than the public_html folder, they aren't hurting anything being in different locations to my knowledge, but they are just useless.
If the paths are correct and the yahoo spider is still going insane, try to run your robots.txt thru a robots.txt generator or checker (many available free online) to make sure there are no errors. After that, i dont have a clue of why Yahoo isn't picking it up. |
#10
|
||||
|
||||
Quote:
EDIT: There were errors. I should have had an empty line between each block of code. I fixed it. Hope that was the problem. |
|
|
X vBulletin 3.8.12 by vBS Debug Information | |
---|---|
|
|
More Information | |
Template Usage:
Phrase Groups Available:
|
Included Files:
Hooks Called:
|