The Arcive of Official vBulletin Modifications Site.It is not a VB3 engine, just a parsed copy! |
|
|
#1
|
||||
|
||||
robots.txt help
Hello all. I have been doing a bit of research here on vb.org about robots.txt and have added it to my site... but its been 3 days and nothing has changed. My biggest problem is Yahoo! Slurp Spider. There are constantly multiple instances of it on my Who's Online page and they are constantly spidering my newthread.php and other pages that I don't want them wasting their time on. Google is a good boy, it only appears once at any given time in my WOL and it seems to crawl pages that I'd like it to crawl.
I've read that the robots.txt will fix this problem... but nothing has changed in 3 days. Isn't it supposed to take effect within 24 hrs? I chmodded it to 755 is that correct? If not, what should I chmod it to? Heres an exact copy of my robots.txt. Tell me if its correct... Code:
User-agent: Slurp Crawl-delay: 60 User-agent: Fasterfox Disallow: / User-agent: * Disallow: /forums/admincp/ Disallow: /forums/clientscript/ Disallow: /forums/cpstyles/ Disallow: /forums/customavatars/ Disallow: /forums/customprofilepics/ Disallow: /forums/images/ Disallow: /forums/modcp/ Disallow: /forums/ajax.php Disallow: /forums/attachment.php Disallow: /forums/calendar.php Disallow: /forums/cron.php Disallow: /forums/editpost.php Disallow: /forums/global.php Disallow: /forums/image.php Disallow: /forums/inlinemod.php Disallow: /forums/joinrequests.php Disallow: /forums/login.php Disallow: /forums/misc.php Disallow: /forums/moderator.php Disallow: /forums/newattachment.php Disallow: /forums/newreply.php Disallow: /forums/newthread.php Disallow: /forums/online.php Disallow: /forums/poll.php Disallow: /forums/postings.php Disallow: /forums/printthread.php Disallow: /forums/private.php Disallow: /forums/profile.php Disallow: /forums/register.php Disallow: /forums/report.php Disallow: /forums/reputation.php Disallow: /forums/search.php Disallow: /forums/sendmessage.php Disallow: /forums/showpost.php Disallow: /forums/subscription.php Disallow: /forums/threadrate.php Disallow: /forums/usercp.php Disallow: /forums/usernote.php Disallow: /forums/credits.php Disallow: /forums/arcade.php Disallow: /forums/vbimagehost.php |
|
|
X vBulletin 3.8.12 by vBS Debug Information | |
---|---|
|
|
More Information | |
Template Usage:
Phrase Groups Available:
|
Included Files:
Hooks Called:
|