vb.org Archive - How do I hide my site from crawlers and bots?

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)

- - How do I hide my site from crawlers and bots? (https://vborg.vbsupport.ru/showthread.php?t=239405)

K4GAP

03-30-2010 10:55 AM

How do I hide my site from crawlers and bots?

How do I hide my site from crawlers and bots?

War.Frog

03-30-2010 11:36 AM

Quote:

Originally Posted by GaryT (Post 2012794)

How do I hide my site from crawlers and bots?

Just create a text file in Notepad with the following:

Code:

User-agent: *

Disallow: /

Save it as robots.txt and upload it to your site's top level directory.

K4GAP

03-31-2010 03:23 PM

I already have robot.txt in every top level folder just to make sure. I'm still showing an average of 8 to 10 crawlers from baidu.com the chinese crawler.

firefoxextreme

03-31-2010 03:28 PM

Block Ips: 58, 124, 125 etc in cpanel

Or add to .htaccass

Code:

<Limit GET HEAD POST>

order allow,deny

deny from 58.

deny from 123.

deny from 124.

deny from 125.

allow from all

</LIMIT>

Brandon Sheley

03-31-2010 04:42 PM

why not just hide everything with usergroups?

K4GAP

04-01-2010 03:32 PM

Quote:

Originally Posted by Loco.M (Post 2013561)

why not just hide everything with usergroups?

Because I want my forums to be readable by those not registered.

LorinPotter

04-01-2010 05:17 PM

The purpose of these crawlers is to allow for valid placement on the search engines. If you want new 'guests', it's important to allow the crawlers, and very easy to keep them out of areas they you do not want them to see, with forums.

Just in case, you can easily not allow them to post, view members lists, etc.

All times are GMT. The time now is 06:32 AM.

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01103 seconds Memory Usage 1,720KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (2)bbcode_code_printable (2)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)post_thanks_navbar_search (1)printthread (7)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: