no index by google

[booble]

I'm searching some ways for NO index my forum (Google, Bing, Yahoo...).

I put hidden all subforum. Any ideas?

[Max Taxable]

Spiders are going to archive everything. Even with a extensive robots.txt file. Even if it's just a login page.

[Max Taxable]

Of course, I suppose you could install Simon's "Ban Spiders by User Agent" and put all known spiders on the list..... A very large spider definition list is available at OzzModz and is updated frequently.

[ozzy47]

Add this to your robots.txt

User-agent: *
Disallow: /

[Max Taxable]

Originally Posted by ozzy47

Add this to your robots.txt

User-agent: *
Disallow: /

Right but... That's not a block or even a rule, it is a polite request a good many spiders just ignore. Kind of like gun laws - the criminals don't pay any attention to it.

[booble]

and NoIndex, NoFollow in robots.txt?

Ban Spiders by User Agent is not a good option, ozzy?

[Max Taxable]

If you want to block them, use the mod and the list I suggested. If you want to ask them nicely and politely to please not index your site, use the robots.txt. But there's no harm at all in doing both.

[ozzy47]

Ban Spiders by User Agent is fine, with the list Max linked you to, and also do the robots.txt

[Max Taxable]

Baidu is one of the worst offenders for ignoring robots.txt, one of 100s who do ignore it. That's just one major example.

If your goal is to NOT have your site indexed anywhere at all, you want to ban them and keep the list updated.

[booble]

Aha ok ok, but if I dont install it and only hidden all subforums for unregistered... google show threads etc?:erm: