JIMRANGERS
12-20-2011, 09:36 AM
User-agent: Baiduspider
Disallow: /
User-agent: BoardTracker
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Twiceler
Disallow: /
User-agent: Slurp
Crawl-delay: 2
User-agent: msnbot
Crawl-delay: 2
User-agent: *
Disallow: *.js
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /post.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Sitemap: http://yoursitessitemap.php
thanks in advance, was given this robot.txt from other vbulletin site but im not sure its correct as lots of URLs are being locked out according to google. Is this good or bad?
Disallow: /
User-agent: BoardTracker
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Twiceler
Disallow: /
User-agent: Slurp
Crawl-delay: 2
User-agent: msnbot
Crawl-delay: 2
User-agent: *
Disallow: *.js
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /post.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Sitemap: http://yoursitessitemap.php
thanks in advance, was given this robot.txt from other vbulletin site but im not sure its correct as lots of URLs are being locked out according to google. Is this good or bad?