How do I hide my site from crawlers and bots?

[K4GAP]

How do I hide my site from crawlers and bots?

[War.Frog]

Originally Posted by GaryT

How do I hide my site from crawlers and bots?

Just create a text file in Notepad with the following:

User-agent: *
Disallow: /

Save it as robots.txt and upload it to your site's top level directory.

[K4GAP]

I already have robot.txt in every top level folder just to make sure. I'm still showing an average of 8 to 10 crawlers from baidu.com the chinese crawler.

[firefoxextreme]

Block Ips: 58, 124, 125 etc in cpanel

Or add to .htaccass

<Limit GET HEAD POST>
order allow,deny
deny from 58.
deny from 123.
deny from 124.
deny from 125.
allow from all
</LIMIT>

[Brandon Sheley]

why not just hide everything with usergroups?

[K4GAP]

Originally Posted by Loco.M

why not just hide everything with usergroups?

Because I want my forums to be readable by those not registered.

[LorinPotter]

The purpose of these crawlers is to allow for valid placement on the search engines. If you want new 'guests', it's important to allow the crawlers, and very easy to keep them out of areas they you do not want them to see, with forums.

Just in case, you can easily not allow them to post, view members lists, etc.