vb.org Archive - bot to crawl private sections

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vB3 General Discussions (https://vborg.vbsupport.ru/forumdisplay.php?f=111)

- - bot to crawl private sections (https://vborg.vbsupport.ru/showthread.php?t=216588)

njoy	06-19-2009 07:32 PM

bot to crawl private sections

Ok before you say why do you want to do this let me explain. I do not want to do this for search engines, there is a site that needs to crawl your site to check links. Now I have installed kx permissions mod but it does not seem to work for me. Gave all permissions in options for the mod and edited the vbulletin spiders file to include the bot I need to crawl the private sections of my site. However google can see it while testing and the site that needs to crawl sees login page. Anyway to fix this? Once I get it to work is there also a way to only give permission to this site? and not all search engines?

njoy	06-22-2009 04:03 AM

no one?

JamesC70

06-22-2009 04:20 AM

Quote:

Originally Posted by njoy (Post 1832898)

there is a site that needs to crawl your site to check links.... However google can see it while testing and the site that needs to crawl sees login page. Anyway to fix this? Once I get it to work is there also a way to only give permission to this site? and not all search engines?

Is Google the site you want to include, or the one to exclude?

Create a robots.txt file, and place it in your sever's web root (wherever www.yoursite.com/index.html is).

In your robots.txt, place the following code:

Code:

User-agent: *

Disallow: /



User-agent: Mediapartners-Google

Allow: /

This allowes the AdSense crawler in, but blocks the Googlebot. Replace the "Mediapartners-Google" agent with the agent identifier of whatever other bot you want to crawl your site.

Source: http://www.google.com/support/webmas...y?answer=40360

All times are GMT. The time now is 11:45 AM.

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.00926 seconds Memory Usage 1,714KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)bbcode_code_printable (1)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)post_thanks_navbar_search (1)printthread (3)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: