Robots.txt file preventing search indexing?

[noppid]

Originally Posted by 007

I'm not sure if this helped. Is there anything else in there that would be stopping spiders? They don't seem to be getting past the index.

First off, that file assumes a ROOT install. Secondly, there are a bunch of files blocked that are beneficial to you in the SERPs in that list.

The fact that index.php would be there just absoluty blows me away! I'd throw it way and start over if I were you.

Here's a good base to start with. This file assumes vBulletin is in the folder named "forums".

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /forums/admincp/
Disallow: /forums/modcp/
Disallow: /forums/install/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/images/
Disallow: /forums/includes/

The first two lines assure that the adsense spider can go where ever it wants. The next lines exclude spiders from most of the places that the spiders don't need to waste time on.

Not excluding everything useless to the spiders has never inflicted any penalty in the SERPs for any site that I am aware of. Keep it simple.