Robots.txt file preventing search indexing?

[007]

My sites no longer seem to be getting indexed. I'm fairly certain it has to do with my robots.txt file that I got from a thread here. The problem started shortly after I added this. Search engine bots never seem to get past the main page now. :-(

#ALL BOTS
User-agent: *
Disallow: /admincp/
Disallow: /ajax.php
Disallow: /arcade.php
Disallow: /attachments/
Disallow: /chat.php
Disallow: /chat/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /images/
Disallow: /includes/
Disallow: /install/
Disallow: /modcp/
Disallow: /subscriptions/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /announcement.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /external.php
Disallow: /faq.php
Disallow: /frm_attach
Disallow: /image.php
Disallow: /index.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php?
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /usercp.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

[Marco van Herwaarden]

Disallow: /index.php

Remove that one.

[007]

The thread I got this from said that was ok to leave in.. I will remove it and see how it works. Hopefully that's the problem. :-)

[007]

I'm not sure if this helped. Is there anything else in there that would be stopping spiders? They don't seem to be getting past the index.

[noppid]

I'm not sure if this helped. Is there anything else in there that would be stopping spiders? They don't seem to be getting past the index.


First off, that file assumes a ROOT install. Secondly, there are a bunch of files blocked that are beneficial to you in the SERPs in that list.

The fact that index.php would be there just absoluty blows me away! I'd throw it way and start over if I were you.

Here's a good base to start with. This file assumes vBulletin is in the folder named "forums".


User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /forums/admincp/
Disallow: /forums/modcp/
Disallow: /forums/install/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/images/
Disallow: /forums/includes/


The first two lines assure that the adsense spider can go where ever it wants. The next lines exclude spiders from most of the places that the spiders don't need to waste time on.

Not excluding everything useless to the spiders has never inflicted any penalty in the SERPs for any site that I am aware of. Keep it simple.