Board Optimization - KX - Spider Permissions

This modification was brought to you by
KXDesign

http://www.kxdesign.com/

~Modification name
Spider Permissions

~Modification description
This modification allows you to set view permissions for the spiders/bots on your website or you can place them in your desired usergroup.


~Modification options

• Enable?
• Place into usergroup?
• Usergroup ID
• Can view forums?
• Can view threads?
• Can view who's online?
• Can view profiles?
• Can view albums?

~Modification info
File uploads: 0
File edits: 0
Templates: 0
Template edits: 0
Plugins: 1
SQL Queries: 0
Phrases: 17
Settings: 8
Hooks: 0

Install time: 3 sec
Install level: Easy

~Modification installation

Step 1: Import Product


~Modification changelog

• 1.0.0 - First Release

~Modification copyright
This may not be distributed,released or claimed as your work without author's permission.

[dirtyfeast]

So is there an answer to my question. The problem I am having.

[Hostboard]

I am trying to understand what this does and how adding:

Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Etc...

in a robots.txt file is any different outside of assigning a specific user group within VB to do the same?

[Ziki]

Try allowing them to access usercp

[Hostboard]

Originally Posted by Ziki

Try allowing them to access usercp

Then just use:

Disallow: /usercp.php

Your reply was less then helpful. I still do not understand.

In fact a robotos.txt file for VB should encompass:

User-agent: *
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

No being a hater, I just do not understand what differences your plugin accomplishes that a robots.txt file can't do.

[Ziki]

I said allow bots to access usercp not to disallow

[Hostboard]

Devils advocate...

I thought unless you dis-allow that robots will auto spider your site. Also why allow usercp.php? I would think this would be duplicate information. Duplicate information is not seo friendly plus a bandwidth waster?

[Ziki]

Well I don't care what people do with this modification,I released this because of a request.

[eJM]

Originally Posted by joh

Lizard King, it is people like you that may cause a broad to get ban, stop hating on the guy mod and make one that is better.

Don't bash people who are trying to help us understand the ramifications of using a particular modification. I certainly don't want to break the Google rules and if there is a chance that could happen and the mod author forgot to tell us about that issue, then I appreciate someone like Lizard King letting me know. It sounds like if you knew a law or rule was being broken that might harm someone's website, you wouldn't say a word. Someone like that would be far more problematic than Lizard King.

Originally Posted by Lizard King

It can be done via robot.txt Ziki.

Not always. I am checking this mod out for a specific purpose that can't be handled by robots.txt.

Originally Posted by Lizard King

vBSEO is using all white hat seo tactics and there is no feature that will get you banned on google. I donot have hate at all. I just warn people with the truth.

If they use this mod and they choose this mod to display bots different content than guest. It will get their site dropped out of SE 's index.

Keep doing what you're doing, Lizard King. I appreciate your efforts. However, this mod may just be the thing some of us might need to allow the Adsense spiders to crawl private forums that also have ads.

Originally Posted by Ziki

Well I don't care what people do with this modification,I released this because of a request.

I do appreciate your hard work on this, but I always appreciate these kinds of efforts more when the author of a mod also cares about how the modification is used. If it might cause a problem with a search engine's rules about accessing content that an unregistered visitor couldn't also access, you might put a disclaimer in the modification's description. Give the user the information they require to make a wise decision.

What I am interested in doing, and I think there might be a wide use for this, is to allow the Mediapartners-Google* spiders to crawl private forums that also contain Adsense ads. I run a forum for flooring professionals and consumers. There are a number of forums on my site that only pros can view and participate in. Consumers and, of course, unregistered or unlogged in members can't view the content there. I need a way to allow the Adsense only spiders to access the private forums to help them place content relevant advertisements.

This is NOT against the Google rules. There is a place you can set up authentication for the Adsense spiders (https://www.google.com/adsense/edit-auth), but it doesn't appear to work with vBulletin private forums. If this modification works for that particular bot, then you truly do have a very valuable tool to offer vBulletin webmasters.

My hope is that the modification has the ability to select which bots you want to give access - and that the Mediapartners-Google* bot is one of them. Maybe you could edit your description to let us know this and post some screen shots of the options panel. As it is, I have to download and install to know the answers to these questions.

Anyway, thanks for offering a solution - if it is a solution to what I need.

Best R'gards,

Jim McClain

[eJM]

Well, apparently this mod does not give you very many choices. There are no choices as to which spiders can have access and which ones cannot. Although spiders can be added to specific usergroups, there doesn't seem to be a way a spider could log in or utilize a password. If there is some way that any ol' spider can have access to private areas of my forum, then what is there to prevent a malicious user from impersonating a search spider to gain access to secure areas of my forum?

Sorry, this doesn't seem to be a secure modification with enough options to make it useful to me.

Jim

[Ziki]

This modification places the spiders defined in your spiders.xml into the usergroup specified.So you cannot fool it