Updated robot.txt file?

[pityocamptes]

Does VB4 or this site provide an up-to-date robot.txt file? Any help appreciated. I have no mods at the moment, basically a vanilla copy with custom template. However, when I do install mods should those be added to the robot.txt file, and if so, which file generally is added? Thanks again.

On a side note, I have read some posts where people are changing the name fo their admincp file, which I have done and password protected. I have also seen where others are password protecting the modcp (?) file as well? I was wondering the reason why, and should the name be changed as well, or just PW protected? Any other files? Thanks.

[crazyboy1661]

me too looking for dat.

[blake247]

# Robots.txt File for the domain http://www.myfreemasonry.com
# All robots will spider the domain

User-agent: myfreemasonry-Google
Disallow:

User-agent: *
Disallow: /go/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /admincp/
Disallow: /modcp/
Disallow: /attachment.php
Disallow: /search.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /editpost.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /login.php
Disallow: /subscription.php
Disallow: /private.php
Disallow: /report.php
Disallow: /sendmessage.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /postings.php
Disallow: /sendtofriend.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /showgroups.php
Disallow: /announcement.php

Sitemap: http://www.myfreemasonry.com/vbulletin_sitemap_index.xml.gz


what I use

[pityocamptes]

Thanks! I assume this list does not include mods? If I have mods do I include those as well? Thanks for the feedback!

[DF031]

Thanks, that list is real helpfull !

BTW: One should not list the name of the admin en moderator panel in case they are renamed (as recommended by VB)

[pityocamptes]

Thanks, that list is real helpfull !

BTW: One should not list the name of the admin en moderator panel in case they are renamed (as recommended by VB)

How about the modcp? Is that ok to name?

[ForceHSS]

How about the modcp? Is that ok to name?

Best to remove the admin and modcp from that list above. Not a good idea to have that there

[pityocamptes]

Best to remove the admin and modcp from that list above. Not a good idea to have that there

ok, can modcp be renamed, and where do you need to make changes? Or is it not necessary?

[ozzy47]

Rename Vbulletin admincp and modcp Directories for Better Security.

1) Rename the Admincp and Modcp folders to your chosen name

2) Rename the folder in your ftp client

3) Change vBulletin’s config.php file to match the new folders.

Important: after renaming the admincp and modcp folders, YOU MUST change their names in the config.php file to match the new names. Also when upgrading, make sure to rename the directories.

// ****** PATH TO ADMIN & MODERATOR CONTROL PANELS ******
// This setting allows you to change the name of the folders that the admin and
// moderator control panels reside in. You may wish to do this for security purposes.
// Please note that if you change the name of the directory here, you will still need
// to manually change the name of the directory on the server.

$config['Misc']['admincpdir'] = 'admincp';

$config['Misc']['modcpdir'] = 'modcp';

[crazyboy1661]

ok, can modcp be renamed, and where do you need to make changes? Or is it not necessary?

for security reasons it is must to change/rename the admin and modcp names as explained by ozzy47

[DemOnstar]

Ok, from the example above, I am currently using this.

Sitemap: https://www.mysite.com/sitemap_index.xml.gz


User-agent: Mediapartners-Google
Disallow:


User-agent: Baiduspider
Disallow: /



User-agent: *
Disallow: clientscript/
Disallow: cpstyles/
Disallow: customavatars/
Disallow: customgroupicons/
Disallow: customprofilepics/
Disallow: customsignaturepics/
Disallow: forumrunner/
Disallow: images/
Disallow: includes/
Disallow: members/
Disallow: sitemap/
Disallow: ajax.php
Disallow: attachment.php
Disallow: calendar.php
Disallow: cron.php
Disallow: editpost.php
Disallow: global.php
Disallow: image.php
Disallow: inlinemod.php
Disallow: joinrequests.php
Disallow: login.php
Disallow: member.php
Disallow: memberlist.php
Disallow: misc.php
Disallow: moderator.php
Disallow: newattachment.php
Disallow: newreply.php
Disallow: newthread.php
Disallow: online.php
Disallow: poll.php
Disallow: postings.php
Disallow: printthread.php
Disallow: private.php
Disallow: profile.php
Disallow: register.php
Disallow: report.php
Disallow: reputation.php
Disallow: search.php
Disallow: sendmessage.php
Disallow: subscription.php
Disallow: threadrate.php
Disallow: thumbs.php
Disallow: thumb_center.php
Disallow: usercp.php
Disallow: usernote.php
Disallow: visitormessage.php


Does that seem in order?

I have removed some stuff..and added other things..

Thanks...

[gsmlover4u]

what is benifit to disallow every thing
i think then search engine will not crawl website ?

[Max Taxable]

There is no reason to have modcp or admincp in robots.txt - first of all this file is public, so you are telling the script kiddies out there where these pages are, no matter what you have named them.

Second and most important, bad bots ignore robots.txt anyway. Yes that's right - robots.txt isn't a law, or a automated restriction, it's a "please don't index this" page with no teeth at all.

Robots.txt is alot like gun laws - only the law abiding obey them or even pay any attention to them.

If google and other good bots index these pages, they generally won't show in search results because they have no content relevance.

If you want the bad bots out of your life installing THIS GREAT MOD (https://vborg.vbsupport.ru/showthread.php?t=264932)is the way to go.

[webmastersun]

Does VB4 or this site provide an up-to-date robot.txt file? Any help appreciated.
You should do robots.txt depend on your forum, which sections/parts you want to block or allow robots then add
Disallow or Allow in your robots.txt file

Since vBulletin community is a forum to discuss all things about vBulletin, not SEO then I would suggest some SEO experts on this SEO forum
http://www.webmastersun.com/forums/10-Search-Engine-Optimization

They can support solve more SEO problems and give you best solutions for your SEO questions.

Disallow: /images/
Disallow: /register.php

I am wondering why you want to block this folder and file? it's still useful for search engine crawl and index your images file and register page.

Hope it helps!