no index by google

[booble]

I'm searching some ways for NO index my forum (Google, Bing, Yahoo...).

I put hidden all subforum. Any ideas?

[Max Taxable]

Spiders are going to archive everything. Even with a extensive robots.txt file. Even if it's just a login page.

[Max Taxable]

Of course, I suppose you could install Simon's "Ban Spiders by User Agent (https://vborg.vbsupport.ru/showthread.php?t=268208)" and put all known spiders on the list..... A very large spider definition list is available at OzzModz (http://ozzmodz.com/showthread.php/943-vBulletin-Spiders-List-Updated) and is updated frequently.

[ozzy47]

Add this to your robots.txt

User-agent: *
Disallow: /

[Max Taxable]

Add this to your robots.txt

User-agent: *
Disallow: / Right but... That's not a block or even a rule, it is a polite request a good many spiders just ignore. Kind of like gun laws - the criminals don't pay any attention to it.:D

[booble]

and NoIndex, NoFollow in robots.txt?

Ban Spiders by User Agent is not a good option, ozzy?

[Max Taxable]

If you want to block them, use the mod and the list I suggested. If you want to ask them nicely and politely to please not index your site, use the robots.txt. But there's no harm at all in doing both.

[ozzy47]

Ban Spiders by User Agent is fine, with the list Max linked you to, and also do the robots.txt

[Max Taxable]

Baidu is one of the worst offenders for ignoring robots.txt, one of 100s who do ignore it. That's just one major example.

If your goal is to NOT have your site indexed anywhere at all, you want to ban them and keep the list updated.

[booble]

Aha ok ok, but if I dont install it and only hidden all subforums for unregistered... google show threads etc?:erm:

[Max Taxable]

Aha ok ok, but if I dont install it and only hidden all subforums for unregistered... google show threads etc?:erm:Google typically obeys robots.txt and will not index you at all if you follow Ozzy's instructions on that.

[ozzy47]

I would be willing to bet you could add this to the very top of your .htaccess file, and it should do it also,
Header set X-Robots-Tag "noindex, nofollow"

[Max Taxable]

Aha ok ok, but if I dont install it and only hidden all subforums for unregistered... google show threads etc?:erm:What actually are you wanting to do? Because if you have hidden forums, threads, or any other areas of your site that require login to view, google and all other spiders, cannot index them. Because they cannot see them.

If you block spiders completely or command 'no index nofollow' you will not show up in searches and makes it difficult for people to ever find your site.

[Max Taxable]

With Bop5's mod here (https://vborg.vbsupport.ru/showthread.php?t=269201), you can also hide individual threads from the unregistered/not logged in usergroup and this includes all spiders. This hides threads you designate regardless of what forum they are in, or of existing permissions.

[booble]

What actually are you wanting to do? Because if you have hidden forums, threads, or any other areas of your site that require login to view, google and all other spiders, cannot index them. Because they cannot see them.

aahh ok, Its perfect then. This is my doubt. Further I will install the mod...

On the future,within a time If I unninstall this mod and I want index, there is any problem?

[Max Taxable]

aahh ok, Its perfect then. This is my doubt. Further I will install the mod...

On the future,within a time If I unninstall this mod and I want index, there is any problem?If you install the mod you will likely be blacklisted by the legitimate search engines. Install it ONLY if this is what you want.

[booble]

With Bop5's mod here (https://vborg.vbsupport.ru/showthread.php?t=269201), you can also hide individual threads from the unregistered/not logged in usergroup and this includes all spiders. This hides threads you designate regardless of what forum they are in, or of existing permissions.

very interesting...

--------------- Added 1396230562 at 1396230562 ---------------

If you install the mod you will likely be blacklisted by the legitimate search engines. Install it ONLY if this is what you want.

ok, thank you ozzy and Maxx. GOOD HELP! :)

[ozzy47]

Not a problem, glad to help. :)

[Max Taxable]

If you install the mod you will likely be blacklisted by the legitimate search engines. Install it ONLY if this is what you want.I should rephrase this. The mod alone won't get you blacklisted and in fact it is one of the better anti-spam tools around. It's the LIST you use with it - the one I linked to is for legitimate spiders for identification in "Who's Online" only. Using that LIST with this mod might lead to search engine blacklists. Follow the link in my signature for the right definitions list, to use with the "Ban Spiders" Mod.

[ozzy47]

Also, if you want to know if something will be indexed by a robot, view your site as a guest, and what ever you see while not logged in, is what the robots will see.

[Seven Skins]

Add the code below to your headinclude or headinclude_bottom template along with other suggestions in this thread.

<meta name="robots" content="noindex,nofollow" />

[booble]

ok thank you...good info friends

[RichieBoy67]

You can also go into webmaster tools and request to have your current indexed data removed. As long as you have the robots.txt set up to block crawling they will pull your data.

I know this works in Google but I am not sure about Yahoo and Bing. I believe though you can also request them to remove your indexed data.