Updated robot.txt file?

[DemOnstar]

Ok, from the example above, I am currently using this.

Sitemap: https://www.mysite.com/sitemap_index.xml.gz


User-agent: Mediapartners-Google
Disallow:


User-agent: Baiduspider
Disallow: /



User-agent: *
Disallow: clientscript/
Disallow: cpstyles/
Disallow: customavatars/
Disallow: customgroupicons/
Disallow: customprofilepics/
Disallow: customsignaturepics/
Disallow: forumrunner/
Disallow: images/
Disallow: includes/
Disallow: members/
Disallow: sitemap/
Disallow: ajax.php
Disallow: attachment.php
Disallow: calendar.php
Disallow: cron.php
Disallow: editpost.php
Disallow: global.php
Disallow: image.php
Disallow: inlinemod.php
Disallow: joinrequests.php
Disallow: login.php
Disallow: member.php
Disallow: memberlist.php
Disallow: misc.php
Disallow: moderator.php
Disallow: newattachment.php
Disallow: newreply.php
Disallow: newthread.php
Disallow: online.php
Disallow: poll.php
Disallow: postings.php
Disallow: printthread.php
Disallow: private.php
Disallow: profile.php
Disallow: register.php
Disallow: report.php
Disallow: reputation.php
Disallow: search.php
Disallow: sendmessage.php
Disallow: subscription.php
Disallow: threadrate.php
Disallow: thumbs.php
Disallow: thumb_center.php
Disallow: usercp.php
Disallow: usernote.php
Disallow: visitormessage.php

Does that seem in order?

I have removed some stuff..and added other things..

Thanks...

[gsmlover4u]

what is benifit to disallow every thing
i think then search engine will not crawl website ?

[Max Taxable]

There is no reason to have modcp or admincp in robots.txt - first of all this file is public, so you are telling the script kiddies out there where these pages are, no matter what you have named them.

Second and most important, bad bots ignore robots.txt anyway. Yes that's right - robots.txt isn't a law, or a automated restriction, it's a "please don't index this" page with no teeth at all.

Robots.txt is alot like gun laws - only the law abiding obey them or even pay any attention to them.

If google and other good bots index these pages, they generally won't show in search results because they have no content relevance.

If you want the bad bots out of your life installing THIS GREAT MOD is the way to go.

[webmastersun]

Originally Posted by pityocamptes

Does VB4 or this site provide an up-to-date robot.txt file? Any help appreciated.

You should do robots.txt depend on your forum, which sections/parts you want to block or allow robots then add
Disallow or Allow in your robots.txt file

Since vBulletin community is a forum to discuss all things about vBulletin, not SEO then I would suggest some SEO experts on this SEO forum
http://www.webmastersun.com/forums/1...e-Optimization

They can support solve more SEO problems and give you best solutions for your SEO questions.

Originally Posted by blake247

Disallow: /images/
Disallow: /register.php

I am wondering why you want to block this folder and file? it's still useful for search engine crawl and index your images file and register page.

Hope it helps!