robots question

[Stormraven]

I'm trying to optimize my vbulletin for seo with robots.txt. But I just have one small question. I saw some robots.txt file contents that Redlinemotorsports posted in another thread, this is the contents of my robots.txt file:

User-agent: *
Disallow: /archive/
Disallow: /lo-fi/
Disallow: /general-announcements/
Disallow: /cgi-bin/
Disallow: /clientscript/
Disallow: /includes/
Disallow: /install/
Disallow: /sitemap/
Disallow: /tags/
Disallow: /ads/
Disallow: /video/
Disallow: /games/
Disallow: /partners/
Disallow: /printthread.php
Disallow: /subscription.php
Disallow: /payments.php
Disallow: /profile.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /online.php
Disallow: /faq.php
Disallow: /calendar.php
Disallow: /private.php
Disallow: /sendmessage.php
Disallow: /sendmessage.php?do=
Disallow: /showgroups.php
Disallow: /reputation.php
Disallow: /report.php
Disallow: /threadrate.php
Disallow: /postings.php
Disallow: /newthread.php
Disallow: /search.php
Disallow: /newreply.php
Disallow: /register.php
Disallow: /login.php
Disallow: /faq.php
Disallow: /image.php
Disallow: /cron.php
Disallow: /joinrequests.php
Disallow: /usercp.php
Disallow: /vbseocp.php

but this is what Redlinemotorsports posted:

User-agent: *
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php

But my question is, what is the difference between the two? do they both do the same thing? im trying to make google focus on main fresh content but am i duplicating things?

[HMBeaty]

Answered via Instant Messenger.

If anyone else needs the answer to this, its yes. Basically both that are listed, will do the same thing, but the 2nd one (the one I posted) is more for people who have their forums installed to yoursite.com/forums/. The FIRST one, is for people who have their forums installed to their root directory (yoursite.com/).

However, you'll see alot of people using the 2nd one (the one I posted) since its the more common one. Just paste that into your robots.txt file and remove what you want crawled, and add what you don't want crawled. It's that easy

[BSMedia]

Nice robots.txt, though it seems a little out of date and missing some files like tags.php albums.php and groups.php you can even take it a little further and block off your entire includes folder (I've seen vBulletin javascript get indexed :roll and the clientscript folders if you don't want your images cralwed, also block that off


Things to remember:

• Robots.txt can only be served from your public_html (or www) folder. It can't not be placed in a sub-folder on your domain.
• If your vBulletin isn't in /forums be sure to update the path to the correct location or else it won't do any good.

[Stormraven]

Which one do you mean is outdated MSMedia?

Kind Regards
Stormraven.

[BSMedia]

Both of them are.

There are simply missing 3 .php files thats all

Add

Disallow: tags.php
Disallow: groups.php
Disallow: albums.php
Disallow: clientscript/
Disallow: includes/

And that should pretty much fix you right up