vb.org Archive - Board Optimization - KX

vb.org Archive (https://vborg.vbsupport.ru/index.php)

- vBulletin 3.8 Add-ons (https://vborg.vbsupport.ru/forumdisplay.php?f=235)

- - Board Optimization - KX - Spider Permissions (https://vborg.vbsupport.ru/showthread.php?t=196420)

So is there an answer to my question. The problem I am having.

I am trying to understand what this does and how adding:

Code:

Disallow: /member.php

Disallow: /memberlist.php

Disallow: /misc.php

Disallow: /moderator.php

Disallow: /newattachment.php

Disallow: /newreply.php

Disallow: /newthread.php

Etc...

in a robots.txt file is any different outside of assigning a specific user group within VB to do the same?

Try allowing them to access usercp :)

Quote:

Originally Posted by Ziki (Post 1668104)

Try allowing them to access usercp :)

Then just use:

Code:

Disallow: /usercp.php

Your reply was less then helpful. I still do not understand.

In fact a robotos.txt file for VB should encompass:

Code:

User-agent: *

Disallow: /ajax.php

Disallow: /attachment.php

Disallow: /calendar.php

Disallow: /cron.php

Disallow: /editpost.php

Disallow: /global.php

Disallow: /image.php

Disallow: /inlinemod.php

Disallow: /joinrequests.php

Disallow: /login.php

Disallow: /member.php

Disallow: /memberlist.php

Disallow: /misc.php

Disallow: /moderator.php

Disallow: /newattachment.php

Disallow: /newreply.php

Disallow: /newthread.php

Disallow: /online.php

Disallow: /poll.php

Disallow: /postings.php

Disallow: /printthread.php

Disallow: /private.php

Disallow: /profile.php

Disallow: /register.php

Disallow: /report.php

Disallow: /reputation.php

Disallow: /search.php

Disallow: /sendmessage.php

Disallow: /showgroups.php

Disallow: /subscription.php

Disallow: /threadrate.php

Disallow: /usercp.php

Disallow: /usernote.php

No being a hater, I just do not understand what differences your plugin accomplishes that a robots.txt file can't do.

I said allow bots to access usercp not to disallow ;)

Devils advocate...

I thought unless you dis-allow that robots will auto spider your site. Also why allow usercp.php? I would think this would be duplicate information. Duplicate information is not seo friendly plus a bandwidth waster?

Well I don't care what people do with this modification,I released this because of a request.

Quote:

Originally Posted by joh (Post 1667009)

Lizard King, it is people like you that may cause a broad to get ban, stop hating on the guy mod and make one that is better.

Don't bash people who are trying to help us understand the ramifications of using a particular modification. I certainly don't want to break the Google rules and if there is a chance that could happen and the mod author forgot to tell us about that issue, then I appreciate someone like Lizard King letting me know. It sounds like if you knew a law or rule was being broken that might harm someone's website, you wouldn't say a word. Someone like that would be far more problematic than Lizard King.

Quote:

Originally Posted by Lizard King (Post 1667429)

It can be done via robot.txt Ziki.

Not always. I am checking this mod out for a specific purpose that can't be handled by robots.txt.

Quote:

Originally Posted by Lizard King (Post 1667432)

vBSEO is using all white hat seo tactics and there is no feature that will get you banned on google. I donot have hate at all. I just warn people with the truth.

If they use this mod and they choose this mod to display bots different content than guest. It will get their site dropped out of SE 's index.

Keep doing what you're doing, Lizard King. I appreciate your efforts. However, this mod may just be the thing some of us might need to allow the Adsense spiders to crawl private forums that also have ads.

Quote:

Originally Posted by Ziki (Post 1668454)

Well I don't care what people do with this modification,I released this because of a request.

I do appreciate your hard work on this, but I always appreciate these kinds of efforts more when the author of a mod also cares about how the modification is used. If it might cause a problem with a search engine's rules about accessing content that an unregistered visitor couldn't also access, you might put a disclaimer in the modification's description. Give the user the information they require to make a wise decision.

What I am interested in doing, and I think there might be a wide use for this, is to allow the Mediapartners-Google* spiders to crawl private forums that also contain Adsense ads. I run a forum for flooring professionals and consumers. There are a number of forums on my site that only pros can view and participate in. Consumers and, of course, unregistered or unlogged in members can't view the content there. I need a way to allow the Adsense only spiders to access the private forums to help them place content relevant advertisements.

This is NOT against the Google rules. There is a place you can set up authentication for the Adsense spiders (https://www.google.com/adsense/edit-auth), but it doesn't appear to work with vBulletin private forums. If this modification works for that particular bot, then you truly do have a very valuable tool to offer vBulletin webmasters.

My hope is that the modification has the ability to select which bots you want to give access - and that the Mediapartners-Google* bot is one of them. Maybe you could edit your description to let us know this and post some screen shots of the options panel. As it is, I have to download and install to know the answers to these questions.

Anyway, thanks for offering a solution - if it is a solution to what I need.

Best R'gards,

Jim McClain

Well, apparently this mod does not give you very many choices. There are no choices as to which spiders can have access and which ones cannot. Although spiders can be added to specific usergroups, there doesn't seem to be a way a spider could log in or utilize a password. If there is some way that any ol' spider can have access to private areas of my forum, then what is there to prevent a malicious user from impersonating a search spider to gain access to secure areas of my forum?

Sorry, this doesn't seem to be a secure modification with enough options to make it useful to me.

Jim

This modification places the spiders defined in your spiders.xml into the usergroup specified.So you cannot fool it ;)

X vBulletin 3.8.12 by vBS Debug Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (3)bbcode_code_printable (5)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (4)pagenav_pagelink (1)post_thanks_navbar_search (1)printthread (10)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages:

Phrase Groups Available:

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01919 seconds Memory Usage 1,751KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (3)bbcode_code_printable (5)bbcode_quote_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (4)pagenav_pagelink (1)post_thanks_navbar_search (1)printthread (10)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: