Forum Display Enhancements - Remove Spiders from Who's Online

*** Staff note: The author of this modification has passed away in a diving accident. We wish his family all strength in dealing with this traggic issue. ***

Remove Spiders from Who's Online

The problem

The majority of the the "guests" on your forum are probably spiders. While you might think that it is impressive to show so many guests using your forum, this does not reflect the true number of people online. The who's online page does offer visitors the chance to change the display based on members, guests, or spiders, however the totals (and the record number of users online at one time) do not exclude spiders.

This mod does not remove spiders from your forum, it simply stops them being counted in the totals.

Some search engine spiders such as Yahoo's Slurp can initiate hundreds of connections at the same time (from different IP addresses). Each one of these connections will be included in your totals as a "guest".



What does this modification do?

• Completely remove all known spiders (spiders found in your includes/xml/spiders_vbulletin.xml file) from the who's online display
• Updates the true guest totals displayed on the main forum page
• Resets your maximum users online record on installation
• Includes changes for vBadvanced's online users module
• Doesn't offer separate spider statistics or configuration, it just makes it appear like those spiders aren't on your forum (even though they are)
• Easy access through template conditionals and variables to display different behaviours based on whether the connection is from a spider or not.
  

Who should use this modification?

Anyone who wants to their forum to show the true number of users online in the statistics

I run a big system, will this slow my board down?

Absolutely not. It's been written with big boards in mind. Even if you use a very large spiders_vbulletin.xml file, this file will only be checked once for each new session created. After that, whether the session is used by a spider is recorded in the session table. For users running vBulletin 3.6.6 or later, it will actually increase the speed of the who's online page, since known spiders are no longer compared against the list of known spider user agents. No new database queries are added, as existing queries are used or modified instead.

Hasn't this been done before?

There are a couple, but I've taken a slightly different approach with this mod - my focus was on simplicity and performance. You might also want to check out some other mods here and here, to see if they suit your purposes better.



Installation instructions

• If using vBadvanced, see the instructions below for a manual edit of your onlineusers.php file
• Install the enclosed .xml file via AdminCP -> Plugins & Products -> Manage Products -> Add/Import Product. Remember to click overwrite if upgrading
• Wait a few minutes - existing sessions must have some activity before they are reclassifed as a genuine member or guest
• (optional) Edit your admincp quick statistics file (instructions below)
• (optional) Reset your maximum online statistics again (if you feel the need, but it is done automatically during installation)

Configuration instructions

Nothing to configure! Just see the note below about spiders_vbulletin.xml

How can I make this more effective?

Spiders are identified via the includes/xml/spiders_vbulletin.xml file. However, the version that ships with vBulletin is quite small, and will recognise only the most popular spiders. Anything else will still show up in your statistics as a guest, making it hard to tell real guests from spiders.

Therefore it is recommended to update your spiders_vbulletin.xml if using this mod. You can obtain a much better version of this file from Christian Stadler here. Please contribute new user agent strings directly to him and encourage him to keep providing new updates of this file.

Will using a big spiders_vbulletin.xml slow things down? Theoretically yes, but in practical terms now. Lookups are only one once for each session when it is created. If you are running a version earlier than vBulletin 3.6.6 it will mean that more work must be done on the who's online page, but nowhere else.

Which versions of vBulletin will this work on?

This mod is designed to work for 3.6.6 and above, but there is some additional code included to handle 3.6.0 - 3.6.5 as well.

Optional extras

If you are using vBAdvanced then you'll want to modify your online users module. The following change should work for but versions 2.x and 3.x. Go to your modules/onlineusers.php file and locate this line:

WHERE session.lastactivity > " . (TIMENOW - $vbulletin->options['cookietimeout']) . "

Directly under this line, insert the following line (on a line by itself):

" . ($killspiders ? " AND " . TABLE_PREFIX . "session.spider = -1 " : null) . "

Please don't forget to include the quotes exactly as they appear above.

If you'd like to also include the quick stats in your admincp with this modification open your admincp/index.php file and locate this line:

$guestsarry = $db->query_first("SELECT COUNT(host) AS sessions FROM " . TABLE_PREFIX . "session WHERE userid = 0 AND lastactivity > $datecut");

replace it with:

$guestsarry = $db->query_first("SELECT COUNT(host) AS sessions FROM " . TABLE_PREFIX . "session WHERE userid = 0 " . (($vbulletin->products['whosonline_kill_spiders'] AND $vbulletin->options['enablespiders']) ? ' AND spider = -1 ' : null) . "AND lastactivity > $datecut");

Would you like to reset your maximum online users statistic (the one on the front page or who's online page that tells you the most number of users ever online)? go to the AdminCP -> Maintenance -> Execute SQL Query and run the following query:

DELETE FROM datastore WHERE title = 'maxloggedin' LIMIT 1

There is an additional file, spider_test.php that can be used to test the operation of this mod, as well as giving you a handy overview of which guests have not been classified as spiders, but probably should be. You can then use the user agent strings to expand your spiders_vbulletin.xml file. You can run spider_test.php from any forum directory, just upload it and then type the path to the file into your browser.

Support

Will be provided to those who click install

Version history

1.0 (15.07.2007) Initial version

Disclaimer: No actual spiders where harmed during the development of this modification.

[Charlie98902]

When it is running it may be conflicting with one of my other mods, it was just a want not a I need this really, so I can live without it.

[Kohhal]

Originally Posted by mfyvie

Ok, well there are only 3 places I know of where this can happen. The admincp (which my mod stops from updating the total), the who's online display, the main forum display, and the vbadvanced module (if you are running vbadvanced). If for example you are running vbadvanced and forgot to edit this module, it will be reseting the totals for you.

Can you confirm that you aren't running vbadvanced, and that you don't have any other areas of your forum where the maxonline is displayed (or updated)? Also, which version of vb are you running? If you are running pre 3.6.6 there is different code, but I wasn't able to test this because I'm running 3.6.7.

I'm not running vbadvanced and I've no hacks or anything installed that should affect this. Only couple of minor hacks installed.

I don't have any non-default displays of max users online anywhere so dunno if that could be it.

I'm running 3.6.7 PL1

[SnitchSeeker]

I've never understood why people want to know the number of spiders on their board - it's a totally meaningless number - you could have 500 yahoo slurp spiders crawling one page per session, or you could have a single spider crawl 500 pages in a single session. What's more important to check is the spider activity in your weblogs - it really doesn't have any place in your online users in vbulletin.

I suppose you're right. I was thinking mainly for my own benefit so that if the site is slow I can see if it's due to a high number of spiders on. I've never been good at checking logs. They're complicated to understand for me at least. I'm not very technical savvy.

[rungok]

Thanx for this mod. But I have a problem wich you have talked about before. When I insert

" . ($killspiders ? " AND " . TABLE_PREFIX . "session.spider = -1 " : null) . "

in the onlineusers.php I get this fault:

Database error in vBulletin 3.6.5:

Invalid SQL:

SELECT session.userid, username, usergroupid, (user.options & 512) AS invisible, IF(displaygroupid=0, user.usergroupid, displaygroupid) AS displaygroupid
FROM vb_session AS session
LEFT JOIN vb_user AS user USING (userid)
WHERE session.lastactivity > 1185130825
AND vb_session.spider = -1
ORDER BY invisible ASC, username ASC;

MySQL Error : Unknown table 'vb_session' in where clause
Error Number : 1109
Date : Sunday, July 22nd 2007 @ 09:30:25 PM
Script : http://www.nettkafeen.no/forums/portal.php
Referrer : http://www.nettkafeen.no/forums/index.php
IP Address : 81.167.190.76
Username : Mr.Man
Classname : vb_database

and the vb_session table exist.

[rungok]

Solved it. I just pasted this line without the table prefix, and it worked.

" . ($killspiders ? " AND session.spider = -1 " : null) . "

All of my tables have table-prefix _vb, but the rest of the original query refers to this table without it.

[AndyP]

Is anyone else trying to use this mod with PaulM's most ever mod?

I'm currently getting low counts for active users, that is, spiders aren't included.
But the most ever guest count is a lot higher than I'd expect. It must be including spiders, and in turn affecting the most ever users online stat.

Are they compatable? To me it doesn't look like it.

[lebanon]

I just added ur mod, however whats weird is my record of users went down from 1600 to 172 members online and today as the record.
My who is online shows guests but the main page shows 0 guests,
Am only confused cause i cant see in the code what could mess up the record query !

[mfyvie]

Originally Posted by rungok

Solved it. I just pasted this line without the table prefix, and it worked.

" . ($killspiders ? " AND session.spider = -1 " : null) . "

All of my tables have table-prefix _vb, but the rest of the original query refers to this table without it.

Thanks for that. Very strange though - I have to check more into that syntax. I don't use table prefixes, or mysql 5 - are you using mysql 5 and if so, is strict mode set?

Originally Posted by AndyP

Is anyone else trying to use this mod with PaulM's most ever mod?

I'm currently getting low counts for active users, that is, spiders aren't included.
But the most ever guest count is a lot higher than I'd expect. It must be including spiders, and in turn affecting the most ever users online stat.

Are they compatable? To me it doesn't look like it.

From the looks of it they wouldn't be compatible. The reason is that Paul's mod will not be differentiating spiders in the totals, and is probably re-writing the max online statistic.

Originally Posted by lebanon

I just added ur mod, however whats weird is my record of users went down from 1600 to 172 members online and today as the record.
My who is online shows guests but the main page shows 0 guests,
Am only confused cause i cant see in the code what could mess up the record query !

Sounds like the mod is functioning exactly as designed. I suggest you give it time (see other posts on this thread which cover the same subject) and the number of guests will increase to the correct number. If in doubt -use the test script I included to verify that everything is working as you expect.

[AndyP]

Thanks mfyvie, that's what I thought. It would be great if they did work together though, as now it looks like I might have to drop one of them, unless I can work out a way to edit it myself.

[mfyvie]

Originally Posted by AndyP

Thanks mfyvie, that's what I thought. It would be great if they did work together though, as now it looks like I might have to drop one of them, unless I can work out a way to edit it myself.

Take a look into Paul's mod and see if you can see the part where it does a query to get the number of guests. It should just be a matter of adding "AND spider = -1" to the "WHERE" section of the query. After that you'll probably want to manually reset the max online totals again.