Go Back   vb.org Archive > vBulletin 5 Connect Discussion > vB5 General Discussions
Reload this Page Blocking crawlers from certain pages (htaccess)?
FAQ Community Calendar Today's Posts Search

Reply
 
Thread Tools Display Modes
  #1  
Old 11-17-2014, 03:50 PM
DisasterDotCom DisasterDotCom is offline
 
Join Date: Feb 2013
Posts: 37
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default Blocking crawlers from certain pages (htaccess)?
Hi All -

Has anyone put an htaccess together to block crawlers from indexing the following aspects of vb5?
  • Profiles
  • Signature invisible for guests/google bot
  • Print pages
  • Stats
  • Help
  • Attach
  • Subscribe
  • Search
Reply With Quote
  #2  
Old 11-17-2014, 11:38 PM
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Jul 2009
Location: USA
Posts: 10,929
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
You would want to use a robots.txt file.
https://support.google.com/webmaster.../6062596?hl=en
https://support.google.com/webmaster.../6062608?hl=en

All good spiders will follow this, but some of the bad ones might not.
Reply With Quote
Благодарность от:
RichieBoy67
  #3  
Old 11-18-2014, 12:19 AM
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Posts: 3,134
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Keep in mind that doesn't block anything, it's just a "please don't index these" kind of deal and it is a PUBLICLY READABLE document. Meaning your admincp and modcp locations will be PUBLISHED.

Use "ban Spiders by User Agent" mod instead, for the bad actors.

Quote:
Originally Posted by DisasterDotCom View Post
Hi All -

Has anyone put an htaccess together to block crawlers from indexing the following aspects of vb5?
  • Profiles
  • Signature invisible for guests/google bot
  • Print pages
  • Stats
  • Help
  • Attach
  • Subscribe
  • Search
Plus these are all blockable from public view, in Usergroup Options.
Reply With Quote
Благодарность от:
CAG CheechDogg
  #4  
Old 11-18-2014, 12:22 AM
ozzy47's Avatar
ozzy47 ozzy47 is offline
 
Join Date: Jul 2009
Location: USA
Posts: 10,929
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
He can not use those mods, as he has a vB5 site.
Reply With Quote
  #5  
Old 11-18-2014, 12:37 AM
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Posts: 3,134
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Quote:
Originally Posted by ozzy47 View Post
He can not use those mods, as he has a vB5 site.
Yeah I know - wanna show him what he's missing without hooks!
Reply With Quote
Благодарность от:
RichieBoy67
Reply

« Previous Thread | Next Thread »

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 12:56 PM.

Contact Us - Archive - Top

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2025, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.08201 seconds
  • Memory Usage 2,206KB
  • Queries Executed 13 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (2)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (5)post_thanks_box
  • (3)post_thanks_box_bit
  • (5)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (3)post_thanks_postbit
  • (5)post_thanks_postbit_info
  • (5)postbit
  • (5)postbit_onlinestatus
  • (5)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 
Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • fetch_musername
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • post_thanks_function_fetch_thanks_bit_start
  • post_thanks_function_show_thanks_date_start
  • post_thanks_function_show_thanks_date_end
  • post_thanks_function_fetch_thanks_bit_end
  • post_thanks_function_fetch_post_thanks_template_start
  • post_thanks_function_fetch_post_thanks_template_end
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete