perfect robots.txt for vb 3.7?

[logicuk]

Does anyone care to share the perfect robots.txt for vb 3.7

here is what i have


User-agent: *
Disallow: /admincp/
Disallow: /archive/
Disallow: /backup/
Disallow: /announcement.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /faq.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /misc.php
Disallow: /modcp/
Disallow: /moderator.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /subscriptions.php
Disallow: /threadrate.php
Disallow: /usercp.php

[Boofo]

You should have that in the domain root.

[Bradley_Wint]

don't mind me sounding like a newbie here but i added the content to a robots.txt file, upped it to my server html root folder like 8 hrs ago..now when i check my who's online list, i still see google bots murdering my calender.php, login.php...etc...

anything i did wrong?

[blike2]

Google only checks robots.txt every day or two, I think. Give it a little time.

[Bradley_Wint]

Google only checks robots.txt every day or two, I think. Give it a little time.
thanks mate :)

[SEOvB]

once every 24hrs :)

[Bradley_Wint]

yes apparently it worked like 5 minutes after i posted... only the important files like forumdisplay, showpost, showthread etc we being scanned. thanks for this mate. cheers. :)

[jambo_1969]

Make sure you add in

User-agent: Fasterfox
Disallow: /

I managed to catch a thread on my board where people were discussing have Fasterfox set as turbo charged...my loads went through the roof !!

[Boofo]

Can we combine these 2?


User-Agent: Googlebot-Image
Disallow: /

User-agent: Fasterfox
Disallow: /


Like this?


User-Agent: Googlebot-Image
User-agent: Fasterfox
Disallow: /

[jambo_1969]

Can we combine these 2?



Like this?
Not sure...

This is how mine looks (part of it anyway)

User-agent: Fasterfox
Disallow: /
User-agent: *
Disallow: /.k.k.k./

?

[SEOvB]

Can we combine these 2?



Like this?


yes you can

[Boofo]

Thank you, sir. ;)

[jambo_1969]

Was there ever anything to stop Google Web Accelerator ?

[logicuk]

yeah i would like to know that also.

[punchbowl]

is it a bit of a security issue telling everyone where your admincp is?

[Boofo]

I don't see how as it is the default directory name for vb.

[punchbowl]

I don't see how as it is the default directory name for vb.

It's been suggested you don't use the default directory though just to provide an extra layer of security. I have mine in a non-default location, htaccessed and with no links anywhere to it on my site. How would the googlebot ever find the location?

Considering all the brute force tools available to script kiddies I say out of sight, out of mind.

[Boofo]

All they would have to do was know any names of the default files in the admincp directory and they could find the directory that way. So even that isn't failsafe.

[punchbowl]

All they would have to do was know any names of the default files in the admincp directory and they could find the directory that way. So even that isn't failsafe.


how?

http://www.domain.com/forum/yabbadabbado/index.php

[Boofo]

If they can get on the server, they can find the directory. There is no foolproof way to stop someone who wants to get to it bad enough. All you can do is slow them down. But good luck with your theory. ;)

[punchbowl]

If they can get on the server, they can find the directory. There is no foolproof way to stop someone who wants to get to it bad enough. All you can do is slow them down. But good luck with your theory. ;)

It's not my theory. I think it may have been suggested on here. (Edit: Maybe not. Mentioned here http://www.vbulletin.com/forum/showthread.php?t=194701 also remember this?

// ****** PATH TO ADMIN & MODERATOR CONTROL PANELS ******
// This setting allows you to change the name of the folders that the admin and
// moderator control panels reside in. You may wish to do this for security purposes.
// Please note that if you change the name of the directory here, you will still need

)

I only allow people http access to my server. Apart from guessing I'm a big fan of the flintstones it does provide extra hoops that any hacker will have to jump through. Not theoretical hoops either : They have to guess another defacto password. If someone has access to my server through ssh or whatever then it's already too late.

From reading on here 90% of board hacks are through a browser simply putting admincp after the board url and guessing. A large percentage of these 'hacks' are inside jobs by disgruntled mods who presumably know their way around a standard installation.

Why exclude something by name when googlebot only follows links and the admin link will never appear for guests? Are you putting your admin cp in a sitemap too?! (j/k re sitemap!)

Either way it's not a major issue but I just don't see the point in listing it.

[Brandon Sheley]

Make sure you add in

User-agent: Fasterfox
Disallow: /

I managed to catch a thread on my board where people were discussing have Fasterfox set as turbo charged...my loads went through the roof !!

Does anyone have any info on this? Does it really work? What's the point?

I've never heard of this before, I'll add it if it helps, but I really hate messing with my robots.txt file, I don't want to stop the bots from me site :D

thanks
-Brandon

[punchbowl]

Does anyone have any info on this? I've never heard of this before

thanks
-Brandon

Yeah I use it. Worth it.

There's a few other download accelerators too but some allow surfers spoof their useragent (http://www.speedbit.com/Help/UserAgent.asp) which is unfortunate.

[Phooey]

Does the fasterfox addon in your robots.txt completely block anyone using fasterfox or just blocks the functionality of the add-on on the site?

[s3nt3nc3d]

@Phooey
I'd assume it just blocks the functionality of the add-on

I run multiple websites on my server...only one is semi-large and having issues with spiders hogging all the resources (crashing my database multiple times a day due to having too many connections). Should I upload that file into the root folder of that one site or should I modify it and upload it into my server root folder where my xxxx.com, yyyyy.com, zzzzz.com, etc folders are located for my individual websites? I'm using 3.6.9 (soon to upgrade to 3.6.10--not quite ready for the jump to 3.7 but eventually will do so), so I'm gonna have to modify this a little to suit my needs prolly.

[Jase2]

don't mind me sounding like a newbie here but i added the content to a robots.txt file, upped it to my server html root folder like 8 hrs ago..now when i check my who's online list, i still see google bots murdering my calender.php, login.php...etc...

anything i did wrong?

Give it a while. But are you sure they don't have the red error icon next to them?

Here's what I use :

User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/album.php
Disallow: /forum/announcement.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/faq.php
Disallow: /forum/showgroups.php
Disallow: /forum/images/
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/memberslist.php
Disallow: /forum/misc.php
Disallow: /forum/modcp/
Disallow: /forum/moderator.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/poll.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showpost.php
Disallow: /forum/subscription.php
Disallow: /forum/subscriptions.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php

[SEOvB]

Heres more information on Robots.txt for vBulletin (http://seovb.com/vbulletin-robotstxt-11/index.html) including some other files and folders you should block out.

After you upload it, it could take up to 24hrs for spiders to find and start to follow the rules of robots.txt most only download it once a day.

[Boofo]

Too bad that link doesn't work.

[Jase2]

It does work....