Great question!! But how do you do this
exactly? I usually have 100 of these spiders on my site at all times of the day!! I just want to make sure I don't block the other bots as well.
Here is my current robots.txt I have just made this recently so let me know if there is somethings you would change in it....
PHP Code:
User-agent: *
Allow: /
User-agent: *
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/images/
Disallow: /forums/ajax.php
Disallow: /forums/arcade.php
Disallow: /forums/attachment.php
Disallow: /forums/awards.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/faq.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/market.php
Disallow: /forums/market_bank.php
Disallow: /forums/market_convert.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/vbactivity.php
Disallow: *.js
Disallow: /forums/includes/
Disallow: /forums/install/
Disallow: /forums/customavatars/
Disallow: /css/
12-23-2010, 09:34 PM
Threaded Mode