vb.org Archive - Blocking crawlers from certain pages (htaccess)?

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vB5 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=262)

- - Blocking crawlers from certain pages (htaccess)? (https://vborg.vbsupport.ru/showthread.php?t=315474)

DisasterDotCom

11-17-2014 03:50 PM

Blocking crawlers from certain pages (htaccess)?

Hi All -

Has anyone put an htaccess together to block crawlers from indexing the following aspects of vb5?

Profiles
Signature invisible for guests/google bot
Print pages
Stats
Help
Attach
Subscribe
Search

ozzy47

11-17-2014 11:38 PM

You would want to use a robots.txt file.
https://support.google.com/webmaster.../6062596?hl=en
https://support.google.com/webmaster.../6062608?hl=en

All good spiders will follow this, but some of the bad ones might not.

Max Taxable

11-18-2014 12:19 AM

Keep in mind that doesn't block anything, it's just a "please don't index these" kind of deal and it is a PUBLICLY READABLE document. Meaning your admincp and modcp locations will be PUBLISHED.

Use "ban Spiders by User Agent" mod instead, for the bad actors.

Quote:

Originally Posted by DisasterDotCom (Post 2523143)

Hi All -

Has anyone put an htaccess together to block crawlers from indexing the following aspects of vb5?

Profiles
Signature invisible for guests/google bot
Print pages
Stats
Help
Attach
Subscribe
Search

Plus these are all blockable from public view, in Usergroup Options.

ozzy47

11-18-2014 12:22 AM

He can not use those mods, as he has a vB5 site.

Max Taxable

11-18-2014 12:37 AM

Quote:

Originally Posted by ozzy47 (Post 2523221)

He can not use those mods, as he has a vB5 site.

Yeah I know - wanna show him what he's missing without hooks!:D

All times are GMT. The time now is 09:42 AM.

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.00989 seconds Memory Usage 1,719KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (2)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)post_thanks_navbar_search (1)printthread (5)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: