Member Bots 1.0 - Allow bots to act as members

--------- Member Bots 1.0 ---------
Written by Trilljester - http://www.trilliummud.com

This small mod allows you to treat search engine robots (i.e. GoogleBot, Yahoo Slurp) as registered users instead of guests, as they are treated by default in VBulletin.

This is handy if your site allows limited access to guests, but full access to registered users. By default, a search engine robot would only be allowed to index what a guest could see, but with this mod, it would allow them to fully index your site as a registered user.

This mod uses the spiders_vbulletin.xml file for determining if a visitor is a bot or not, so make sure you have that file up to date. The one that comes with VB is pretty sparse, but it works. If you're only interested in GoogleBot, then the default file works great.

WARNING! This mod requires a very small edit to a VBulletin core file. This means whenever you upgrade to a new version of VBulletin, you'll need to re-do this edit, if you overwrite this file. If the idea of editing a VBulletin core file bothers you, don't install this mod! I won't be held responsible for your forum being fouled up. This mod is very safe however, so don't fret.

Install Instructions:

1. Unzip the product-memberbot.xml file on your system.

2. Go to your VBulletin AdminCP and click on Manage Products under the Plugin Manager section.

3. Click on Add/Import Product.

4. Click on Browse on the first line (Upload the XML from your computer) and locate the product-memberbot.xml file. Click on Import to upload it to the system.

5. Click on Plugin Manager, and ensure that the new plugin named Bot Checker, which is found under the init_startup hook, is enabled.

6. Now the fun part, you'll need to edit the core VBulletin file init.php which is located in the includes directory under your forum home.

For example, if your forum home is located in /home/blah/forum, then init.php is in /home/blah/forum/includes

If you can edit the file on the server without having to download it, more power to you.

In init.php, find this line (should be near the very bottom of the file, it was line 403 in mine):

if (!empty($db->explain))

Add ABOVE this line:

if ($is_bot == 1) {
        $vbulletin->userinfo['usergroupid'] = 2;
}

NOTE: My registered users group is ID #2. Yours may be different, you may change this to any group ID you like.

7. Save init.php and upload back to your server if you had to download the file to edit it.

If you're editing on the server, save it, and that's it!

================================================== ========
That's it! Search Engine bots should now "see" your forum as a regular user, not a guest.

Questions, comments, improvements are more than welcome. Please use this thread as a cental support site. It makes it easier for me to help out. Also, you may visit my site and fire me off a PM for help, or just send a quick thanks or shout out.

Enjoy!
// trilljester

[trilljester]

Originally Posted by bashy

Hi

I still have bots getting the No Permissions when trying to view a forum that users are allowed to view?

Are the bots in your spiders_vbulletin.xml file (located in includes/xml)? This mod uses this file to determine if the visitor is a bot or not.

[ScienceofSpock]

Just a quick question/security concern:
How are you identifying SE bots? Are you using the user-agent?
If so, are bots allowed to post (considering they're regarded as regular members) ?
If so, What's to stop me loading up Opera, changing my user-agent to googlebot and posting on your forums?

I'm not trying to be sarcastic here, I'd really like to add this to my forum. I'm just trying to play devil's advocate and perform due diligence.

[trilljester]

Originally Posted by ScienceofSpock

Just a quick question/security concern:
How are you identifying SE bots? Are you using the user-agent?
If so, are bots allowed to post (considering they're regarded as regular members) ?
If so, What's to stop me loading up Opera, changing my user-agent to googlebot and posting on your forums?

I'm not trying to be sarcastic here, I'd really like to add this to my forum. I'm just trying to play devil's advocate and perform due diligence.

Yes, the modification uses the spiders_vbulletin.xml and checks the user-agent to determine if you're a bot or not. And if you follow the default installation, you'll be allowing bots or humans who are clever enough to change their user-agent to post on your site.

HOWEVER, you can set whatever group you want the bots/humans to be in. If you want the bots to be able to see your site but not post, create a new usergroup that doesn't allow posting, just viewing. Which is what I would recommend in any case, so nobody can spoof a bot and post on your forums.

To change to that group, just modify the code you added in init.php and change that group ID to whatever ID you have setup for that special group.

[lionheart53]

I want only the spider for google adsense to get through and not the others. So I tried changing the spiders_vbulletin.xml to just what's below but then my pages in the forum don't come up at all. Any ideas why? Is there a better way to do this?

<?xml version="1.0" encoding="ISO-8859-1"?>

<searchspiders>
<spider ident="Mediapartners-Google">
<name>Google AdSense</name>
<info>https://www.google.com/adsense/faq</info>
<email>adsense-support@google.com</email>
</spider>
</searchspiders>

<!-- CVS: $RCSfile: spiders_vbulletin.xml,v $ - $Revision: 1.2 $ -->

[trilljester]

Originally Posted by lionheart53

I want only the spider for google adsense to get through and not the others. So I tried changing the spiders_vbulletin.xml to just what's below but then my pages in the forum don't come up at all. Any ideas why? Is there a better way to do this?

For some reason not entirely clear to me, VBulletin doesn't handle a single entry in the spiders_vbulletin.xml file very well. So, just add another Spider in the file. For your example, I added GoogleBot as well as Google Adsense.

Try this spiders_vbulletin.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>

<spiderlist version="1.0">
        <spider ident="Mediapartners-Google">
                <name>Google AdSense</name>
                <type>searchspider</type>
                <info>https://www.google.com/adsense/faq</info>
                <email>adsense-support@google.com</email>
        </spider>
        <spider ident="Googlebot/">
                <name>Google</name>
                <type>searchspider</type>
                <info>http://www.google.com/bot.html</info>
                <email>googlebot@google.com</email>
                <addresses>
                        <address type="range">64.233.160.0-64.233.191.255</address>
                        <address type="range">66.249.64.0-66.249.95.255</address>
                        <address type="range">72.14.192.0-72.14.207.255</address>
                        <address type="range">216.239.32.0-216.239.63.255</address>
                </addresses>
        </spider>
</spiderlist>

[lionheart53]

Thanks. That resolved it. Very weird but I can work with that.

[trilljester]

Originally Posted by lionheart53

Thanks. That resolved it. Very weird but I can work with that.

Yeah, it is weird, I'll have to look into it a little more. I might have to update the mod to use it's own XML file for what bots you want in or not, so as not to mess with the original file.

Maybe if I have time this weekend...

[Zia]

Originally Posted by trilljester

For some reason not entirely clear to me, VBulletin doesn't handle a single entry in the spiders_vbulletin.xml file very well. So, just add another Spider in the file. For your example, I added GoogleBot as well as Google Adsense.

Try this spiders_vbulletin.xml:

<?xml version="1.0" encoding="ISO-8859-1"?>

<spiderlist version="1.0">
        <spider ident="Mediapartners-Google">
                <name>Google AdSense</name>
                <type>searchspider</type>
                <info>https://www.google.com/adsense/faq</info>
                <email>adsense-support@google.com</email>
        </spider>
        <spider ident="Googlebot/">
                <name>Google</name>
                <type>searchspider</type>
                <info>http://www.google.com/bot.html</info>
                <email>googlebot@google.com</email>
                <addresses>
                        <address type="range">64.233.160.0-64.233.191.255</address>
                        <address type="range">66.249.64.0-66.249.95.255</address>
                        <address type="range">72.14.192.0-72.14.207.255</address>
                        <address type="range">216.239.32.0-216.239.63.255</address>
                </addresses>
        </spider>
</spiderlist>

umm nice..support......do u mind to rrelase an update spiderlist.xml

i trust,u tweak/customize ur spider list......can u relase it....
in fact now a days no one taking care of spiders list...

[dutchbb]

it's a black hat technique

google says don't optimize your pages for spiders, if you do, you are taking a high risk...

[trilljester]

Originally Posted by Zia

umm nice..support......do u mind to rrelase an update spiderlist.xml

i trust,u tweak/customize ur spider list......can u relase it....
in fact now a days no one taking care of spiders list...

I downloaded the one offered at vbulletin.com - here's the link:

http://www.ragnarokonline.de/spiderlist/spiderlist.xml

Upload it into includes/xml, name it spiders_vbulletin.xml

You could also name it that before uploading and overwrite the one in that directory. I keep one called spiders_vbulletin.xml.new in includes/xml so when upgrading VB versions, I can just copy that one over the default one given in VB.