Quote:
User-agent: Baiduspider
Disallow: /
User-agent: BoardTracker
Disallow: /
User-agent: Gigabot
Disallow: /
User-agent: Twiceler
Disallow: /
User-agent: Slurp
Crawl-delay: 2
User-agent: msnbot
Crawl-delay: 2
User-agent: *
Disallow: *.js
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /post.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Sitemap: http://yoursitessitemap.php
|
thanks in advance, was given this robot.txt from other vbulletin site but im not sure its correct as lots of URLs are being locked out according to google. Is this good or bad?
12-20-2011, 09:36 AM
please verify robots.txt - much appreciated
Hybrid Mode
