Kill Google Spider!

[carmichael1973]

I've read tons of posts and founds hacks that increase spider activity. Unfortunately I need to ABSOLUTELY KILL all spider activity on my site. We created our new site about 1 month ago. It is a PRIVATE site that it is used for company employees only. None of the information contained within should ever be viewed by the general public. The problem we face now is that in the short time this site has been active, google spiders have been working their A** off. Our content is all over the search engines. The good news is that we have not populated the site with any confidential info just yet.

Does anyone know of a way to completely block spiders from exploring our site?

[Dismounted]

Search Google for a robots.txt "guide".

If it is a private site, why not just disable all guest viewing, or add a .htaccess/.htpasswd protection and send the username/password to all the employees?

[Jase2]

You could add an authentication to the site, however, that would require everyone to know the username and password to access the site. You're best just closing the forums to guests, which means bots cannot index your content.

[Dismounted]

You could even put the username/password combo on the description for the password prompt, that would stop bots too .

[carmichael1973]

Originally Posted by Dismounted

Search Google for a robots.txt "guide".

If it is a private site, why not just disable all guest viewing, or add a .htaccess/.htpasswd protection and send the username/password to all the employees?

I was actually thinking of something just like that. We create the user accounts and then send them their login info via email. It's been a few years since I've used VB and the code has changed alot. For the better that is.

Originally Posted by Jase2

You could add an authentication to the site, however, that would require everyone to know the username and password to access the site. You're best just closing the forums to guests, which means bots cannot index your content.

I've considered that option as well.

Originally Posted by Dismounted

You could even put the username/password combo on the description for the password prompt, that would stop bots too .

I'm also running VBPORTAL which is installed in my root directory. My forums is installed in ROOT/FORUMS. What I'd like to do is have an .htaccess/.htpaasword script run and require a VALID MEMBER login before displaying anything. If login fails redirect to a custom error page.

My propblem is I'm not familiar with the current VB code to do that. Then making it work on a non-vb page as well.

Basically, I need to establish a login verification that will be global to my site before displaying any requested pages.

[Jase2]

A .htaccess/.htpasswd protection would be global and is not strictly pointed to vBulletin itself.

[carmichael1973]

Originally Posted by Jase2

A .htaccess/.htpasswd protection would be global and is not strictly pointed to vBulletin itself.

Is there anything already available to use the users VB information for authentication?

[Jase2]

Not that I'm aware of. It's just a global username and password for accessing the site.

[carmichael1973]

Yea, I've been reading a few hundred posts and I don't find any existing solution. I brought myself up to speed on the login code for 3.7.0. I've managed to get a successful login using htaccess and redirect on failed login working.

Upon successful login it displays the requested page fine. However you are still required to login using the forum login. I'm working on making it log you in to the forums at the same time. I know that I should be happy with what I have now, but unfortunately the employees who will be using this complain about every little thing.

[badboyz]

bro just ban google.com ips in .htaccess