Member Bots 1.0 - Allow bots to act as members

--------- Member Bots 1.0 ---------
Written by Trilljester - http://www.trilliummud.com

This small mod allows you to treat search engine robots (i.e. GoogleBot, Yahoo Slurp) as registered users instead of guests, as they are treated by default in VBulletin.

This is handy if your site allows limited access to guests, but full access to registered users. By default, a search engine robot would only be allowed to index what a guest could see, but with this mod, it would allow them to fully index your site as a registered user.

This mod uses the spiders_vbulletin.xml file for determining if a visitor is a bot or not, so make sure you have that file up to date. The one that comes with VB is pretty sparse, but it works. If you're only interested in GoogleBot, then the default file works great.

WARNING! This mod requires a very small edit to a VBulletin core file. This means whenever you upgrade to a new version of VBulletin, you'll need to re-do this edit, if you overwrite this file. If the idea of editing a VBulletin core file bothers you, don't install this mod! I won't be held responsible for your forum being fouled up. This mod is very safe however, so don't fret.

Install Instructions:

1. Unzip the product-memberbot.xml file on your system.

2. Go to your VBulletin AdminCP and click on Manage Products under the Plugin Manager section.

3. Click on Add/Import Product.

4. Click on Browse on the first line (Upload the XML from your computer) and locate the product-memberbot.xml file. Click on Import to upload it to the system.

5. Click on Plugin Manager, and ensure that the new plugin named Bot Checker, which is found under the init_startup hook, is enabled.

6. Now the fun part, you'll need to edit the core VBulletin file init.php which is located in the includes directory under your forum home.

For example, if your forum home is located in /home/blah/forum, then init.php is in /home/blah/forum/includes

If you can edit the file on the server without having to download it, more power to you.

In init.php, find this line (should be near the very bottom of the file, it was line 403 in mine):

if (!empty($db->explain))

Add ABOVE this line:

if ($is_bot == 1) {
        $vbulletin->userinfo['usergroupid'] = 2;
}

NOTE: My registered users group is ID #2. Yours may be different, you may change this to any group ID you like.

7. Save init.php and upload back to your server if you had to download the file to edit it.

If you're editing on the server, save it, and that's it!

================================================== ========
That's it! Search Engine bots should now "see" your forum as a regular user, not a guest.

Questions, comments, improvements are more than welcome. Please use this thread as a cental support site. It makes it easier for me to help out. Also, you may visit my site and fire me off a PM for help, or just send a quick thanks or shout out.

Enjoy!
// trilljester

[denby]

Originally Posted by denby

Nice mod but it's operating in the grey zone and until Google definitively clarifies whether it's OK for a spider to see my forum content while an unlogged-in visitor can not, I will not be installing it. I can't afford the risk of losing my significant monthly GoogleAds income or my healthy SE ranking.

I will write to Google and report here if I get a clearer answer than trilljester received.

Well, they will not commit to any further elaboration so I say act on the side of caution and beware.

I have settled on a compromise: I will allow all guests (including spiders) to view the forums and thread titles but stop short of showing thread content. Hopefully most thread titles will be informative (and not of the "Help!", "One more thing" and "Well what do you know..." variety) and so entice Google search engine users to come and register.

You can achieve this with the built-in permissions options by setting the forum permissions for each forum for the Unregistered/Not Logged In usergroup as follows:
Can View Forum Yes
Can View Thread Content No
Can View Others' Threads Yes

vB can't tell who the guest is (whether or not they have posted in the forum before) and so all threads are by someone else. But they are still not allowed to view the content.

The quickest way to implement this for all your forums is to do it once manually for one forum and then use the Forum-Based Permission Duplicator . You can find a link to that duplicator at the top of the page:
Admin Control Panel -> Forums and Moderators -> Forum Permissions

I hope this helps.

[kall]

Originally Posted by Lammypie

Corbynt, I'm not sure mine is working either. I've adjusted my useragent to "Googlebot/2.X (http://www.googlebot.com/bot.html)" and gone to my site (not logged in as a user, the access I get is contolled by the user group setting, so thats working, its just that I don't appear in the whos online list as googlebot, only as guest. ? any suggestions whats wrong.


Reg Car, I also need to stop spiders from seeing the welcome headers, as they are ruining my search results.

All suggestions welcome

Do you have Googlebot defined as such in your AdminCP - Spider Identification?

ie: Does googlebot show up in your Who's Online ever?

[venkyrahul]

seems a good hack , thanks

[Smiry Kin's]

donno if its working for me..??

but i see that it has no permissions :S

now guests have permission to view profiles n stuff :s.... so do memebers...?

[moonclamp]

Just wondering, has anyone actually seen their rank improved as a result of this hack?

[Lammypie]

Originally Posted by kall

Do you have Googlebot defined as such in your AdminCP - Spider Identification?

ie: Does googlebot show up in your Who's Online ever?

I have something like 11 members and 2 guests show up quite a bit. When I enter the whos online page, it will say one of the guests - I can tell its googlebot by the browser id only.

googlebot is listed in the spider id xml file if thats what you mean?

[WebsiteDonation]

Dear trilljester & everybody,

I didn't try this mod yet but after reviewing all of your discussions here, I have some ideas :

1. This mod is alright with Google, you guys don't need to worry. Because, very simple, you can set for Guests permission to view all of pages on your forums. In this mod, you set for Spiders permission to view all of pages, there is no problem from Google or whether.

2. I recommend that you guys should create a separate usergroup for Bots, with viewing permissions only. This is good in case if someone try to be .. a bot. In this moment, I am willing to be a spider to make some posts annoy your forums because Bots are in Registered usergroup.

Just small ideas.

[WebsiteDonation]

Additional, a short question : Is there any accident if we use this mod and vbBOL ?

[boeserwolf]

boeserwolf klicks install

It´s very useful for my community as a lot of members would complain if we would allow guests to read the threads. We have a legal topic in our board, but we also want to improve our search engine ranking, so I think this is one of the best hacks to fullfil both needs.

Thanks a lot for this

Markus

[kall]

Originally Posted by boeserwolf

boeserwolf klicks install

It?s very useful for my community as a lot of members would complain if we would allow guests to read the threads. We have a legal topic in our board, but we also want to improve our search engine ranking, so I think this is one of the best hacks to fullfil both needs.

Thanks a lot for this


Markus

You don't appear to understand.

By allowing SE bots to read these 'member-only' threads, you ARE allowing Guests to read them.

a) by viewing the cached page in the SE results.
b) by changing their Useragent to match that of a known bot, thereby allowing an actual Guest to read the thread.