Forum Display Enhancements - Remove Spiders from Who's Online

*** Staff note: The author of this modification has passed away in a diving accident. We wish his family all strength in dealing with this traggic issue. ***

Remove Spiders from Who's Online

The problem

The majority of the the "guests" on your forum are probably spiders. While you might think that it is impressive to show so many guests using your forum, this does not reflect the true number of people online. The who's online page does offer visitors the chance to change the display based on members, guests, or spiders, however the totals (and the record number of users online at one time) do not exclude spiders.

This mod does not remove spiders from your forum, it simply stops them being counted in the totals.

Some search engine spiders such as Yahoo's Slurp can initiate hundreds of connections at the same time (from different IP addresses). Each one of these connections will be included in your totals as a "guest".



What does this modification do?

• Completely remove all known spiders (spiders found in your includes/xml/spiders_vbulletin.xml file) from the who's online display
• Updates the true guest totals displayed on the main forum page
• Resets your maximum users online record on installation
• Includes changes for vBadvanced's online users module
• Doesn't offer separate spider statistics or configuration, it just makes it appear like those spiders aren't on your forum (even though they are)
• Easy access through template conditionals and variables to display different behaviours based on whether the connection is from a spider or not.
  

Who should use this modification?

Anyone who wants to their forum to show the true number of users online in the statistics

I run a big system, will this slow my board down?

Absolutely not. It's been written with big boards in mind. Even if you use a very large spiders_vbulletin.xml file, this file will only be checked once for each new session created. After that, whether the session is used by a spider is recorded in the session table. For users running vBulletin 3.6.6 or later, it will actually increase the speed of the who's online page, since known spiders are no longer compared against the list of known spider user agents. No new database queries are added, as existing queries are used or modified instead.

Hasn't this been done before?

There are a couple, but I've taken a slightly different approach with this mod - my focus was on simplicity and performance. You might also want to check out some other mods here and here, to see if they suit your purposes better.



Installation instructions

• If using vBadvanced, see the instructions below for a manual edit of your onlineusers.php file
• Install the enclosed .xml file via AdminCP -> Plugins & Products -> Manage Products -> Add/Import Product. Remember to click overwrite if upgrading
• Wait a few minutes - existing sessions must have some activity before they are reclassifed as a genuine member or guest
• (optional) Edit your admincp quick statistics file (instructions below)
• (optional) Reset your maximum online statistics again (if you feel the need, but it is done automatically during installation)

Configuration instructions

Nothing to configure! Just see the note below about spiders_vbulletin.xml

How can I make this more effective?

Spiders are identified via the includes/xml/spiders_vbulletin.xml file. However, the version that ships with vBulletin is quite small, and will recognise only the most popular spiders. Anything else will still show up in your statistics as a guest, making it hard to tell real guests from spiders.

Therefore it is recommended to update your spiders_vbulletin.xml if using this mod. You can obtain a much better version of this file from Christian Stadler here. Please contribute new user agent strings directly to him and encourage him to keep providing new updates of this file.

Will using a big spiders_vbulletin.xml slow things down? Theoretically yes, but in practical terms now. Lookups are only one once for each session when it is created. If you are running a version earlier than vBulletin 3.6.6 it will mean that more work must be done on the who's online page, but nowhere else.

Which versions of vBulletin will this work on?

This mod is designed to work for 3.6.6 and above, but there is some additional code included to handle 3.6.0 - 3.6.5 as well.

Optional extras

If you are using vBAdvanced then you'll want to modify your online users module. The following change should work for but versions 2.x and 3.x. Go to your modules/onlineusers.php file and locate this line:

WHERE session.lastactivity > " . (TIMENOW - $vbulletin->options['cookietimeout']) . "

Directly under this line, insert the following line (on a line by itself):

" . ($killspiders ? " AND " . TABLE_PREFIX . "session.spider = -1 " : null) . "

Please don't forget to include the quotes exactly as they appear above.

If you'd like to also include the quick stats in your admincp with this modification open your admincp/index.php file and locate this line:

$guestsarry = $db->query_first("SELECT COUNT(host) AS sessions FROM " . TABLE_PREFIX . "session WHERE userid = 0 AND lastactivity > $datecut");

replace it with:

$guestsarry = $db->query_first("SELECT COUNT(host) AS sessions FROM " . TABLE_PREFIX . "session WHERE userid = 0 " . (($vbulletin->products['whosonline_kill_spiders'] AND $vbulletin->options['enablespiders']) ? ' AND spider = -1 ' : null) . "AND lastactivity > $datecut");

Would you like to reset your maximum online users statistic (the one on the front page or who's online page that tells you the most number of users ever online)? go to the AdminCP -> Maintenance -> Execute SQL Query and run the following query:

DELETE FROM datastore WHERE title = 'maxloggedin' LIMIT 1

There is an additional file, spider_test.php that can be used to test the operation of this mod, as well as giving you a handy overview of which guests have not been classified as spiders, but probably should be. You can then use the user agent strings to expand your spiders_vbulletin.xml file. You can run spider_test.php from any forum directory, just upload it and then type the path to the file into your browser.

Support

Will be provided to those who click install

Version history

1.0 (15.07.2007) Initial version

Disclaimer: No actual spiders where harmed during the development of this modification.

[AndyP]

There wasn't a where statement.

However, I got it to fix the most guests online record from PaulM's mod by amending the execution order of your mod at the forumhome_complete hook to be earlier than Paul's. This way your mod was amending the max number of guests before Paul's mod used it.

Unfortunately the max users ever online is still including spiders and I don't believe PaulM's mod affected this.

Example:
The most ever members online was 16, Today at 09:06 PM. The most ever guests online was 4, Today at 09:08 PM.
=> Most users ever online was 35, Today at 09:10 PM.

[Ronseal]

This Mod appears to be working fine on my Forum Index, but still shows far too many guests on the VBadvanced portal page....... is it another case where I just have to wait?

I've added the code as advised to onlineusers.php but it doesn't seem to have made any diffence.

[Ronseal]

Sorry to bump the thread but has anybody any ideas why this Mod doesn't work on my portal page?

[mfyvie]

Originally Posted by Ronseal

This Mod appears to be working fine on my Forum Index, but still shows far too many guests on the VBadvanced portal page....... is it another case where I just have to wait?

I've added the code as advised to onlineusers.php but it doesn't seem to have made any diffence.

Hi Ron,

Sorry I can't really help with that. This mod doesn't officially support vbadvanced mod. I just added that code as a courtesy. However, others have got it to work. If you see no difference at all you may have inserted the code into the wrong file, or the wrong section of the file. I suggest you go back and start from the beginning and try and debug it yourself. Sorry.

[Ronseal]

Thanks for the reply mfyvie but I have to admit I'm stumped, I've added the code as advised but it doesn't appear to correct the count on the vbadvanced portal page. Can anybody else spot anything I've done wrong, here's where I've added the code and I've also tried it without the line spaces above and below?

FROM " . TABLE_PREFIX . "session AS session
LEFT JOIN " . TABLE_PREFIX . "user AS user USING (userid)
WHERE session.lastactivity > " . (TIMENOW - $vbulletin->options['cookietimeout']) . "

" . ($killspiders ? " AND " . TABLE_PREFIX . "session.spider = -1 " : null) . "

ORDER BY username ASC
");

Many thanks
Ron

[mfyvie]

Hi Ron,

I can only see two possibilities -

1. You may be editing the wrong file. Try to deliberately insert a syntax error and see what happens. If nothing changes, you may be editing the wrong file. Try renaming the file, just to make sure that everything crashes (then rename it back)
2. The variable $killspiders must be set to true to see any difference. This variable is set by the mod itself earlier on, so should be true (unless you've disabled it). I could not tell you why it is not set to true, but you could try to force is by inserting the following line earlier in the file:

$killspiders = true;

Apart from those tips I really have no idea without looking at and playing with your system directly.

[Ronseal]

mfyvie, you've done it!
I simply added the killspiders code you've mentioned at the top of the 'onlineusers.php' file and hey-presto it works perfect. I'm not sure where this setting is supposed to be set, but for the time being untill I find out where, everything works perfect. Although just to clarify I hope this doesn't prevent spiders from crawling my site and indexing it, I just want to remove spiders from the online users count.

Many thanks mfyvie!

[mfyvie]

Ron, I'm glad you used this trick to at least verify that your edits are working.

Here is the code that sets this variable - it runs from the hook global_start, and therefore should be included on all vbadvanced pages as well.

$killspiders = $vbulletin->options['enablespiders']; 

I suspect that you may have disabled the vbulletin setting "Enable spider display". If vbulletin does not identify connections as spiders or not, this mod cannot work correctly.

[Ronseal]

'Enable Spider Display' is set to 'Yes' and always has been as far as I recall, and I'm not sure what you mean by the hook 'global_start'. I've searched for a reference to this code in a few of my vBadvanced CMPS templates and I can't find any sign of it so I assume I'm looking in the wrong place.

[mfyvie]

Hi Ron, if you look in your vbadvanced files you'll see a line that reads like this:

require_once('./global.php');

If you then look in global.php you'll find a line that looks like this:

($hook = vBulletinHook::fetch_hook('global_start')) ? eval($hook) : false;

The code in my previous post fires at global_start, so in other words, if that setting is set to on as you say, then the variable should be set, and you shouldn't have to set it manually. Why it doesn't work is a mystery to me