DO NOT place your admincp and modcp in the robots.txt file! This will only alert hackers that they exist in that exact position. Leave them out completely, this also goes if you have renamed them.
Robots can ignore your robots.txt file. Especially malware robots that scan the web for security vulnerabilities and email address harvesters used by spammers. The robots.txt file is a publicly available file. Anyone can see what sections of your site you don't want robots to got to, so don't go broadcasting directions to secure areas of your site that you don't want anyone to go to (either with good intentions or bad).
The purpose of robots.txt file is to inform the "good" robots of your site layout and what you do and don't want to be indexed.
Each person's robots.txt file should be a little different. Depending on how your forum was installed (in the root or not) and what add-ons and changes have been made since the initial install.
You should always include a reference to your sitemap. There are robots.txt "validators" that will not validate unless the sitemap is included within the file. Both Google and MSNbot (Bing / Yahoo) use validators and will look for a sitemap reference. Your sitemap needs to be the full url. Here is a sitemap reference example:
Code:
Sitemap: http://www.yoursite.com/sitemap.xml
Your sitemap name and location may be different. I recommend placing the sitemap in the first line of your robots.txt file.
The robots.txt file below is a little more in-depth than the first post above and will yield you better results.
This example that can be used for an initial forum install:
Code:
Sitemap: http://www.yoursite.com/sitemap_index.xml.gz
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: clientscript/
Disallow: cpstyles/
Disallow: customavatars/
Disallow: customgroupicons/
Disallow: customprofilepics/
Disallow: customsignaturepics/
Disallow: forumrunner/
Disallow: images/
Disallow: includes/
Disallow: install/
Disallow: members/
Disallow: mobiquo/
Disallow: sitemap/
Disallow: ajax.php
Disallow: attachment.php
Disallow: calendar.php
Disallow: cron.php
Disallow: editpost.php
Disallow: global.php
Disallow: image.php
Disallow: inlinemod.php
Disallow: joinrequests.php
Disallow: login.php
Disallow: member.php
Disallow: memberlist.php
Disallow: misc.php
Disallow: moderator.php
Disallow: newattachment.php
Disallow: newreply.php
Disallow: newthread.php
Disallow: online.php
Disallow: poll.php
Disallow: postings.php
Disallow: printthread.php
Disallow: private.php
Disallow: profile.php
Disallow: register.php
Disallow: report.php
Disallow: reputation.php
Disallow: search.php
Disallow: sendmessage.php
Disallow: subscription.php
Disallow: threadrate.php
Disallow: usercp.php
Disallow: usernote.php
This is a modified example of the above:
This example shows how a robots.txt could look for a forum install outside the root with some added mods, some additional image directory changes and the complete blocking of user-agent Baiduspider.
I recommend if you don't cater to an audience in China to block all Baiduspiders otherwise they will hammer your site hundreds of times a day. For additional details on Baiduspider:
http://chineseseoshifu.com/blog/what...iduspider.html.
There are other mods for vB available for blocking user-agents I recommend this one:
https://vborg.vbsupport.ru/showthread.php?t=268208. In the meantime Baiduspider is the only user-agent I block that I know does not ignore the robots.txt block.
Code:
Sitemap: http://www.yoursite.com/sitemap_index.xml.gz
User-agent: Mediapartners-Google
Disallow:
User-agent: Baiduspider
Disallow: /
User-agent: *
Disallow: /doubleclick/
Disallow: /eyeblaster/
Disallow: /forum/archive/
Disallow: /forum/clientscript/
Disallow: /forum/cpstyles/
Disallow: /forum/customavatars/
Disallow: /forum/customgroupicons/
Disallow: /forum/customprofilepics/
Disallow: /forum/customsignaturepics/
Disallow: /forum/dbtech/
Disallow: /forum/forumrunner/
Disallow: /forum/images/
Disallow: /forum/includes/
Disallow: /forum/install/
Disallow: /forum/members/
Disallow: /forum/mobiquo/
Disallow: /forum/sitemap/
Disallow: /forum/vbseo/
Disallow: /forum/ajax.php
Disallow: /forum/attachment.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/global.php
Disallow: /forum/image.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/member.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php
Disallow: /javascript/
Disallow: /misc/
Disallow: /styles/
Disallow: /xcache/
More Information