Go Back   vb.org Archive > vBulletin 3 Discussion > vB3 General Discussions
  #1  
Old 04-05-2018, 08:36 AM
kai_sey kai_sey is offline
 
Join Date: Nov 2007
Posts: 10
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default How to block crawl bots without blocking users?

I need to block google & co from searching particular forums on my site, whilst still allowing access to guests & members.

I know this can be done via the robots.txt file - I'm just wondering what to put there.

Lets say the particular forum I want to block is: www.domain.com/forums/audi-versus-bmw/ (Note: I need to disallow all threads/posts in that particular forum).

So, is the following correct?

User-agent: *
Disallow: /forums/audi-versus-bmw/

Would the above disallow also include blocking of the actual threads & posts in that forum? Or would it only block the actual url www.domain.com/forums/audi-versus-bmw?

Thanks.
Reply With Quote
  #2  
Old 04-05-2018, 02:49 PM
Max Taxable's Avatar
Max Taxable Max Taxable is offline
 
Join Date: Feb 2011
Posts: 3,134
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by kai_sey View Post

Would the above disallow also include blocking of the actual threads & posts in that forum?.
Robots.txt doesn't disallow or block anything. Think of it as a sign on the door saying "please don't look in the drawers in the rooms." The search engine spiders don't get blocked, and don't have to comply in any way. It's just a way to ask the nice ones "please don't index this."

If you want human guests to see something while not logged in, then google and every other spider can see it as well.
Reply With Quote
  #3  
Old 04-05-2018, 03:10 PM
kai_sey kai_sey is offline
 
Join Date: Nov 2007
Posts: 10
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Quote:
Originally Posted by Max Taxable View Post
Robots.txt doesn't disallow or block anything. Think of it as a sign on the door saying "please don't look in the drawers in the rooms." The search engine spiders don't get blocked, and don't have to comply in any way. It's just a way to ask the nice ones "please don't index this."

If you want human guests to see something while not logged in, then google and every other spider can see it as well.
Alright thanks.

Then is it possible to do this via .htaccess?
Reply With Quote
  #4  
Old 04-05-2018, 07:50 PM
snakes1100 snakes1100 is offline
 
Join Date: Dec 2001
Location: Michigan
Posts: 3,733
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default

Yes, you can use this list for example.

http://tab-studio.com/en/blocking-robots-on-your-page/
Reply With Quote
Благодарность от:
Max Taxable
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 12:35 AM.


Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.05168 seconds
  • Memory Usage 2,183KB
  • Queries Executed 11 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)ad_showthread_firstpost
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (2)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)navbar
  • (3)navbar_link
  • (120)option
  • (4)post_thanks_box
  • (1)post_thanks_box_bit
  • (4)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (1)post_thanks_postbit
  • (4)post_thanks_postbit_info
  • (4)postbit
  • (4)postbit_onlinestatus
  • (4)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 

Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • fetch_musername
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • post_thanks_function_fetch_thanks_bit_start
  • post_thanks_function_show_thanks_date_start
  • post_thanks_function_show_thanks_date_end
  • post_thanks_function_fetch_thanks_bit_end
  • post_thanks_function_fetch_post_thanks_template_start
  • post_thanks_function_fetch_post_thanks_template_end
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete