bot to crawl private sections
Ok before you say why do you want to do this let me explain. I do not want to do this for search engines, there is a site that needs to crawl your site to check links. Now I have installed kx permissions mod but it does not seem to work for me. Gave all permissions in options for the mod and edited the vbulletin spiders file to include the bot I need to crawl the private sections of my site. However google can see it while testing and the site that needs to crawl sees login page. Anyway to fix this? Once I get it to work is there also a way to only give permission to this site? and not all search engines?
|
06-19-2009, 07:32 PM
Hybrid Mode
