Go Back   vb.org Archive > vBulletin 3 Discussion > vB3 General Discussions
Reload this Page How do I hide my site from crawlers and bots?
FAQ Members List Calendar Search Today's Posts Mark Forums Read
  #1  
Old 03-30-2010, 10:55 AM
K4GAP K4GAP is offline
 
Join Date: Mar 2008
Posts: 1,255
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default How do I hide my site from crawlers and bots?
How do I hide my site from crawlers and bots?
Reply With Quote
  #2  
Old 03-30-2010, 11:36 AM
War.Frog's Avatar
War.Frog War.Frog is offline
 
Join Date: Aug 2009
Location: Ontario, Canada
Posts: 262
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Quote:
Originally Posted by GaryT View Post
How do I hide my site from crawlers and bots?
Just create a text file in Notepad with the following:

Code:
User-agent: *
Disallow: /
Save it as robots.txt and upload it to your site's top level directory.
Reply With Quote
  #3  
Old 03-31-2010, 03:23 PM
K4GAP K4GAP is offline
 
Join Date: Mar 2008
Posts: 1,255
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I already have robot.txt in every top level folder just to make sure. I'm still showing an average of 8 to 10 crawlers from baidu.com the chinese crawler.
Reply With Quote
  #4  
Old 03-31-2010, 03:28 PM
firefoxextreme firefoxextreme is offline
 
Join Date: Mar 2010
Posts: 52
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Block Ips: 58, 124, 125 etc in cpanel

Or add to .htaccass

Code:
<Limit GET HEAD POST>
order allow,deny
deny from 58.
deny from 123.
deny from 124.
deny from 125.
allow from all
</LIMIT>
Reply With Quote
  #5  
Old 03-31-2010, 04:42 PM
Brandon Sheley's Avatar
Brandon Sheley Brandon Sheley is offline
 
Join Date: Mar 2005
Location: Google Kansas
Posts: 4,678
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
why not just hide everything with usergroups?
Reply With Quote
  #6  
Old 04-01-2010, 03:32 PM
K4GAP K4GAP is offline
 
Join Date: Mar 2008
Posts: 1,255
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Quote:
Originally Posted by Loco.M View Post
why not just hide everything with usergroups?
Because I want my forums to be readable by those not registered.
Reply With Quote
  #7  
Old 04-01-2010, 05:17 PM
LorinPotter LorinPotter is offline
 
Join Date: Jan 2008
Location: Luzerne, Iowa
Posts: 80
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
The purpose of these crawlers is to allow for valid placement on the search engines. If you want new 'guests', it's important to allow the crawlers, and very easy to keep them out of areas they you do not want them to see, with forums.

Just in case, you can easily not allow them to post, view members lists, etc.
Reply With Quote
Reply

« Previous Thread | Next Thread »
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 01:10 AM.

Contact Us - Archive - Top

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.04103 seconds
  • Memory Usage 2,209KB
  • Queries Executed 13 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (2)bbcode_code
  • (2)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (7)post_thanks_box
  • (7)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (7)post_thanks_postbit_info
  • (7)postbit
  • (7)postbit_onlinestatus
  • (7)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 
Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete