Member Bots 1.0 - Allow bots to act as members

--------- Member Bots 1.0 ---------
Written by Trilljester - http://www.trilliummud.com

This small mod allows you to treat search engine robots (i.e. GoogleBot, Yahoo Slurp) as registered users instead of guests, as they are treated by default in VBulletin.

This is handy if your site allows limited access to guests, but full access to registered users. By default, a search engine robot would only be allowed to index what a guest could see, but with this mod, it would allow them to fully index your site as a registered user.

This mod uses the spiders_vbulletin.xml file for determining if a visitor is a bot or not, so make sure you have that file up to date. The one that comes with VB is pretty sparse, but it works. If you're only interested in GoogleBot, then the default file works great.

WARNING! This mod requires a very small edit to a VBulletin core file. This means whenever you upgrade to a new version of VBulletin, you'll need to re-do this edit, if you overwrite this file. If the idea of editing a VBulletin core file bothers you, don't install this mod! I won't be held responsible for your forum being fouled up. This mod is very safe however, so don't fret.

Install Instructions:

1. Unzip the product-memberbot.xml file on your system.

2. Go to your VBulletin AdminCP and click on Manage Products under the Plugin Manager section.

3. Click on Add/Import Product.

4. Click on Browse on the first line (Upload the XML from your computer) and locate the product-memberbot.xml file. Click on Import to upload it to the system.

5. Click on Plugin Manager, and ensure that the new plugin named Bot Checker, which is found under the init_startup hook, is enabled.

6. Now the fun part, you'll need to edit the core VBulletin file init.php which is located in the includes directory under your forum home.

For example, if your forum home is located in /home/blah/forum, then init.php is in /home/blah/forum/includes

If you can edit the file on the server without having to download it, more power to you.

In init.php, find this line (should be near the very bottom of the file, it was line 403 in mine):

if (!empty($db->explain))

Add ABOVE this line:

if ($is_bot == 1) {
        $vbulletin->userinfo['usergroupid'] = 2;
}

NOTE: My registered users group is ID #2. Yours may be different, you may change this to any group ID you like.

7. Save init.php and upload back to your server if you had to download the file to edit it.

If you're editing on the server, save it, and that's it!

================================================== ========
That's it! Search Engine bots should now "see" your forum as a regular user, not a guest.

Questions, comments, improvements are more than welcome. Please use this thread as a cental support site. It makes it easier for me to help out. Also, you may visit my site and fire me off a PM for help, or just send a quick thanks or shout out.

Enjoy!
// trilljester

[amykhar]

Just a comment. You may want to consider changing the title of this mod to use the word spiders instead of bots. Most people here consider a bot a script that automatically posts something to the forums, while a search engine agent is a spider. More people will search for spider when they want this mod.

[DJ J@M?S]

this mod not work guys

[Quarterbore]

All of you saying this is not working had me worried so I made some changes to mine...

I made a new group that works like my registered group but it is named
"Search Bots" plus I updated my spider list... Next I assigned my bots to be in this new group and saw the bots were in the forums...

I shut their access down to access forums, threads, etc... and the bots all got no permissions errors....

So, I then went back in and updated my settings so they can see forums and threads BUT I made it so they can't see profiles (Can't harvest E-mails that way) and locked them out of forums I didn't want them to see based on the user group...

Works great for me!

Thanks!

[gavinzac]

Originally Posted by amykhar

Just a comment. You may want to consider changing the title of this mod to use the word spiders instead of bots. Most people here consider a bot a script that automatically posts something to the forums, while a search engine agent is a spider. More people will search for spider when they want this mod.

yeah, pretty much. it seems strange to me that there is an auto-response bot for phpBB & not for vBulletin.

[clasione]

This is too risky for my blood..... Showing one page to a spider and one page to a guest is extreamly dangerous.... Often search engines will send "secret spiders" to test and see if you are serving different pages.....

[denby]

Originally Posted by trilljester

This is not a definitive clarification of the issue in any way. Google has basically said they don't discuss individual cases or divilge their indexing or verfication processes (apart from saying it's done by spiders not humans), and that you'll find the answers to all your questions in their webmasters' FAQ. It's left completely to the individual to interpret the TOS and decide what's black hat.

Originally Posted by Andreas

Attention
Using this Hack is to be considered cloaking your site, and you risk being banned from search engine indexes - like it recently happened to BMW.

To be fair, the BMW case involves very different issues. BMW.de was using quite blatant and crude cloaking: spiders were redirected to pages chocked full of car related keywords at a much greater density than on the pages shown to ordinary visitors in an attempt to boost page rank. Very old, tired strategies that Google can spot at 100 MegaPaces.


I face the usual competing goals of retaining control of my site's privacy and member priviledges while ensuring good SE ranking and the delivery of relevant GoogleAds (and hence healthy GoogleAds revenues). One added problem in all this is that if you have restricted the forum viewing access to logged-in visitors, installed the mod and had your forum pages indexed by Google (essentially the archive URL versions of them), the Google search results will reveal the content via the "Cached" link for each result. If unlogged-in visitors from the Google results click the ordinary link, they will be confronted by a login screen but not if they click the "Cached" link. To avoid this hole in your security you need to add a robots no-cache meta tag to your headerinclude template:

<meta name="robots" content="index,follow,noarchive">

This says to all spiders that visit this page "you may index this page, and any pages linked to it via hyperlinks but you may not store the content of any of them in the Googlesearch engine cache".

You should also edit/install a robots.txt file in your root or forum folder with the following content:

User-agent: Googlebot
Disallow:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /

These settings say to the 2 spiders Googlebot and Mediapartners-Google (the GoogleAds spider) "roam my site at will" but tells all others to not crawl any part of my site. It should be noted that only spiders that play by the rules read the robots.txt file. All the big names obey the robots.txt directives but there are lots of smaller rogue spiders who ignore them. Another good reason to edit your spiders_vbulletin.xml file to define which spiders will be accepted into your forum by this mod.


Nice mod but it's operating in the grey zone and until Google definitively clarifies whether it's OK for a spider to see my forum content while an unlogged-in visitor can not, I will not be installing it. I can't afford the risk of losing my significant monthly GoogleAds income or my healthy SE ranking.

I will write to Google and report here if I get a clearer answer than trilljester received.

[MikeF]

Nice mod but it's operating in the grey zone and until Google definitively clarifies whether it's OK for a spider to see my forum content while an unlogged-in visitor can not, I will not be installing it.

I don't understand how the point hasn't been clarified by the major search engines. Cloaking is when the spiders see content a person casually going to the site doesn't see for the purposes of gaining ranking. If the default usergroup is Unregistered with ability to read posts restricted, and the bots see much more its cloaking...

The bottom line is that its not OK.

Imagine you are Google or Yahoo trying to get the most relevant search results. The archive is a duplicate content / optimization issue... they may rank you lower or higher for duplicate content depending on duplication but theyre not going to terminate you. Now if you added 100 content relevant keywords at the end of every archive page to gain better rank, they probably would.

The same goes for this - if there are some general discrepencies between what registered/unregistered people see its not a big deal. But turning spiders into forum members... Combine this with the vBPopup mod and someone might as well start cheesyforum.org

[AzzidReign]

Hmm...seems like this isnt working for me either. I'm lookin at the bots and its saying that they are getting "no permissions" error. Any definite fix to this yet?

[Raydar]

My guests can see everything that my members can see. So I'm thinking that I don't need this hack. Unless there is some other advantage to using it?

I also have welcome headers installed. From reading parts of this thread, it seems that welcome headers may be what the google bot is reading when crawling my pages. If so, is there a way to make google ignore the welcome header?

Thanks for any advice.

Reg

[Lammypie]

Corbynt, I'm not sure mine is working either. I've adjusted my useragent to "Googlebot/2.X (http://www.googlebot.com/bot.html)" and gone to my site (not logged in as a user, the access I get is contolled by the user group setting, so thats working, its just that I don't appear in the whos online list as googlebot, only as guest. ? any suggestions whats wrong.

Reg Car, I also need to stop spiders from seeing the welcome headers, as they are ruining my search results.

All suggestions welcome