robots.txt for 3.8.2 - Any Ideas?

[vbplusme]

Hello and Greetings,

I have just noticed that Google Webmaster Tools is complaining about a LOT of URLs being restricted by my robots.txt file.

Is anyone else having this problem? If not, can I get an example of a robots.txt written for 3.8.2?

I tweaked mine thinking that I was fixing a duplicate content problem but I apparently crossed the line on it

Any ideas, suggestions greatly appreciated.

TIA

[Dismounted]

What is it currently?

[vbplusme]

Sorry, should have thought to post it:

User-agent: *

#Crawl-Delay: 10

Disallow: /admincp/
Disallow: /ajax.php
Disallow: /announcement.php
Disallow: /archive/
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cgi-bin/
Disallow: /chat/
Disallow: /clientscript/
Disallow: /converse.php
Disallow: /cpstyles/
Disallow: /cron.php
Disallow: /customavatars/
Disallow: /customgroupicons/
Disallow: /customprofilepics/
Disallow: /editpost.php
Disallow: /faq.php
Disallow: /forumdisplay.php?daysprune
Disallow: /forumdisplay.php?do
Disallow: /forumdisplay.php?order
Disallow: /forumdisplay.php?page
Disallow: /forumdisplay.php?pp
Disallow: /forumdisplay.php?sort
Disallow: /gallery/
Disallow: /global.php
Disallow: /group_inlinemod.php
Disallow: /groupsubscription.php
Disallow: /images/
Disallow: /includes/
Disallow: /infraction.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /member_inlinemod.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderation.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /personal/
Disallow: /printthread.php
Disallow: /profile.php?do
Disallow: /register.php
Disallow: /report.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showpost.php
Disallow: /showthread.php?goto
Disallow: /showthread.php?mode
Disallow: /showthread.php?p
Disallow: /showthread.php?page
Disallow: /showthread.php?post
Disallow: /showthread.php?pp
Disallow: /signaturepics/
Disallow: /subscription.php

User-Agent: msnbot
Crawl-Delay: 10

User-Agent: Slurp
Crawl-Delay: 10

[veenuisthebest]

Two points in addition to above robots.txt:-

1. We should not give out our admincp directory in robots.txt as it makes the location displayable to the world. What is the use of renaming admincp feature then?

2. Also its good to give a referance to our sitemap at the end of robots.txt

Sitemap: http://site.com/sitemap_index.xml.gz

[vbplusme]

Thanks for the comments, had not thought about the sitemap reference in there. thanks for that. I double password protect my admincp folder though I could easily take it out of the list altogether as the bots can not access it anyway so thanks for that comment as well.

[vbplusme]

Anyone see any problem with the content of this robots.txt or have any idea how to fix the "google" complaining about the restrictions? TIA

[hambil]

Originally Posted by vbplusme

Thanks for the comments, had not thought about the sitemap reference in there. thanks for that. I double password protect my admincp folder though I could easily take it out of the list altogether as the bots can not access it anyway so thanks for that comment as well.

It's not an issue of whether they access it or not, it is whether they try. Everything they try and fail at is wasted bandwidth and resources. If you password protect your admincp and modcp directories there is no reason to leave them out of robots.txt.

I pretty much followed the advice in this article, and have had not complaints from google: http://www.theadminzone.com/forums/s...ad.php?t=19872

[vbplusme]

As a matter of fact I did use those guidelines to construct my robots.txt (and the follow on suggestions in that thread). I forgot about that, thanks for reminding me about it.

[vbplusme]

I do have a follow on question on the robots.txt file that I am currently using. I have the vbulletin blog software installed on this site as well as wordpress. I have not disallows in this robots.txt for any blog files. I would not have thought anything about it except that I just looked at my sitemap and see a huge number of URLs for blog stuff that doesn't really exist like archives from 1983?

Anyone have a suggestion about a sensible robots.txt entry for both the vbulletin blog and wordpress?

TIA for any ideas.

[hambil]

Sounds like a sitemap issue not a robot.txt issue. I know that's not an answer per say, but I'd be looking at why your sitemap contains links that don't exist, instead.