Forum Display Enhancements - Remove Spiders from Who's Online

*** Staff note: The author of this modification has passed away in a diving accident. We wish his family all strength in dealing with this traggic issue. ***

Remove Spiders from Who's Online

The problem

The majority of the the "guests" on your forum are probably spiders. While you might think that it is impressive to show so many guests using your forum, this does not reflect the true number of people online. The who's online page does offer visitors the chance to change the display based on members, guests, or spiders, however the totals (and the record number of users online at one time) do not exclude spiders.

This mod does not remove spiders from your forum, it simply stops them being counted in the totals.

Some search engine spiders such as Yahoo's Slurp can initiate hundreds of connections at the same time (from different IP addresses). Each one of these connections will be included in your totals as a "guest".



What does this modification do?

• Completely remove all known spiders (spiders found in your includes/xml/spiders_vbulletin.xml file) from the who's online display
• Updates the true guest totals displayed on the main forum page
• Resets your maximum users online record on installation
• Includes changes for vBadvanced's online users module
• Doesn't offer separate spider statistics or configuration, it just makes it appear like those spiders aren't on your forum (even though they are)
• Easy access through template conditionals and variables to display different behaviours based on whether the connection is from a spider or not.
  

Who should use this modification?

Anyone who wants to their forum to show the true number of users online in the statistics

I run a big system, will this slow my board down?

Absolutely not. It's been written with big boards in mind. Even if you use a very large spiders_vbulletin.xml file, this file will only be checked once for each new session created. After that, whether the session is used by a spider is recorded in the session table. For users running vBulletin 3.6.6 or later, it will actually increase the speed of the who's online page, since known spiders are no longer compared against the list of known spider user agents. No new database queries are added, as existing queries are used or modified instead.

Hasn't this been done before?

There are a couple, but I've taken a slightly different approach with this mod - my focus was on simplicity and performance. You might also want to check out some other mods here and here, to see if they suit your purposes better.



Installation instructions

• If using vBadvanced, see the instructions below for a manual edit of your onlineusers.php file
• Install the enclosed .xml file via AdminCP -> Plugins & Products -> Manage Products -> Add/Import Product. Remember to click overwrite if upgrading
• Wait a few minutes - existing sessions must have some activity before they are reclassifed as a genuine member or guest
• (optional) Edit your admincp quick statistics file (instructions below)
• (optional) Reset your maximum online statistics again (if you feel the need, but it is done automatically during installation)

Configuration instructions

Nothing to configure! Just see the note below about spiders_vbulletin.xml

How can I make this more effective?

Spiders are identified via the includes/xml/spiders_vbulletin.xml file. However, the version that ships with vBulletin is quite small, and will recognise only the most popular spiders. Anything else will still show up in your statistics as a guest, making it hard to tell real guests from spiders.

Therefore it is recommended to update your spiders_vbulletin.xml if using this mod. You can obtain a much better version of this file from Christian Stadler here. Please contribute new user agent strings directly to him and encourage him to keep providing new updates of this file.

Will using a big spiders_vbulletin.xml slow things down? Theoretically yes, but in practical terms now. Lookups are only one once for each session when it is created. If you are running a version earlier than vBulletin 3.6.6 it will mean that more work must be done on the who's online page, but nowhere else.

Which versions of vBulletin will this work on?

This mod is designed to work for 3.6.6 and above, but there is some additional code included to handle 3.6.0 - 3.6.5 as well.

Optional extras

If you are using vBAdvanced then you'll want to modify your online users module. The following change should work for but versions 2.x and 3.x. Go to your modules/onlineusers.php file and locate this line:

WHERE session.lastactivity > " . (TIMENOW - $vbulletin->options['cookietimeout']) . "

Directly under this line, insert the following line (on a line by itself):

" . ($killspiders ? " AND " . TABLE_PREFIX . "session.spider = -1 " : null) . "

Please don't forget to include the quotes exactly as they appear above.

If you'd like to also include the quick stats in your admincp with this modification open your admincp/index.php file and locate this line:

$guestsarry = $db->query_first("SELECT COUNT(host) AS sessions FROM " . TABLE_PREFIX . "session WHERE userid = 0 AND lastactivity > $datecut");

replace it with:

$guestsarry = $db->query_first("SELECT COUNT(host) AS sessions FROM " . TABLE_PREFIX . "session WHERE userid = 0 " . (($vbulletin->products['whosonline_kill_spiders'] AND $vbulletin->options['enablespiders']) ? ' AND spider = -1 ' : null) . "AND lastactivity > $datecut");

Would you like to reset your maximum online users statistic (the one on the front page or who's online page that tells you the most number of users ever online)? go to the AdminCP -> Maintenance -> Execute SQL Query and run the following query:

DELETE FROM datastore WHERE title = 'maxloggedin' LIMIT 1

There is an additional file, spider_test.php that can be used to test the operation of this mod, as well as giving you a handy overview of which guests have not been classified as spiders, but probably should be. You can then use the user agent strings to expand your spiders_vbulletin.xml file. You can run spider_test.php from any forum directory, just upload it and then type the path to the file into your browser.

Support

Will be provided to those who click install

Version history

1.0 (15.07.2007) Initial version

Disclaimer: No actual spiders where harmed during the development of this modification.

[keyness]

Originally Posted by mfyvie

How are you sure that there are 10 active users? The other 4 may be spiders, or they may be inactive. Which view was this? Check both the front page of your forum, and the who's online display - in the first 15-30 minutes these may report different numbers. If you still don't believe it, you can manually check the session table in the database (however, this also shows sessions that are inactive). In the column "spider" you'll see -1, 0 or 1. 1 means spider, -1 means non-spider and 0 means that it hasn't decided yet (happens on new installs for inactive sessions). I wouldn't be so quick to pronounce something wrong with the mod, it could simply be that you have less "real users" than you thought

Come on mfyvie, I'm not a newbie at all

Well, I also have "who has wisited today" add-on, it shows the user's last activity time when you hover on nickname. I've counted more than 10 while who's online page shows only 5 members and 1 guest.

[mfyvie]

Originally Posted by keyness

Come on mfyvie, I'm not a newbie at all

Well, I also have "who has wisited today" add-on, it shows the user's last activity time when you hover on nickname. I've counted more than 10 while who's online page shows only 5 members and 1 guest.

I'm not suggesting you are a newbie. I thought myself that I wasn't showing enough guests, but when I checked it was correct.

Are you saying the display of the members or the guests is incorrect? At first you said it was guests, then you bring up the point about your who has visited today mod which gives you information on members. My mod does nothing at all to the display of members.

If you still doubt the mod, best to look into the session table. Which version of vb are you running? It makes a difference because two different sets of code run depending on the version.

[efil]

Hi,
Now there is no database error , but despite there is 6 members , i can see in vBadvanced just 1, me.
It's working good at forumhome.

[mfyvie]

Sorry, I thought I'd changed the example to -1, it was still on 0 (now fixed). In your vbadanced module the line should read spider = -1, not spider = 0. Change that and it should work the way you want it to.

[efil]

Originally Posted by mfyvie

Sorry, I thought I'd changed the example to -1, it was still on 0 (now fixed). In your vbadanced module the line should read spider = -1, not spider = 0. Change that and it should work the way you want it to.

Thank you!
Now it's working.

[keyness]

Originally Posted by mfyvie

Are you saying the display of the members or the guests is incorrect? At first you said it was guests, then you bring up the point about your who has visited today mod which gives you information on members. My mod does nothing at all to the display of members.

If you still doubt the mod, best to look into the session table. Which version of vb are you running? It makes a difference because two different sets of code run depending on the version.

Well both are incorrect. For more than 30 minutes, it just showed maximum 5 members and only 1 guest (and another proof, more than 20 guests enter forum just via google in an hour)

Maybe it's because i use too many mods. By the way I'm using classic vbulletin 3.6.7 version.

[mfyvie]

Keyness, the only way for you to verify this is to look into the session table. There you will see the guests and how they are classified (by checking the spider column). -1 means a real guest, 0 means not decided, and 1 means a spider.

I have many guests who arrive on my forum by google, but in reality they don't do anything and they don't move around the site. In this case their status may stay on "0" and they won't get shown in the totals. For example, I have a lot of hits where google has brought someone directly to an attachment or an image has been sideloaded because an image in one of my threads has been directly linked in another forum's thread. This type of thing creates sessions in your table, but these users show no activity on your forum in reality.

I don't think any other mods are affecting it, because those totals are nothing more than the number of rows returned from the mysql query that checks the spider status of each connection.

For example, on my session table right now I have 89 guests with spider status -1 (real guest), 4 with status 0 (unknown) and 81 with status 1 (spider). Of course different forums will vary in how many real guests they have versus spiders.

Just bear in mind that a visitor coming in via google could have come in for any reason - it could even be that they found an image via google images and that image was displayed on the google images page - this will appear like a visit, but it isn't really.

If you want to be really sure you could do something like a grep on your web server logs for each IP address you want to test. I did this when I was testing the mod, because like you, I thought it might not be working. During this process I actually discovered how many other forums were sideloading my attachments and I reconfigured my web server to block them!

[testebr]

What software did you use to generate those graphs and how to?

[mfyvie]

The software is called MRTG. It's free, but to be honest it isn't that simple to setup and get running. Once you get it running then you need some extra code to provide the vbulletin specific stuff, I was thinking of releasing that as a mod, but I didn't want to deal with the million requests about how to get MRTG running

[Kohhal]

This is the who's online hack I've been waiting for, much simplier and truer than others out there. Like you, I just want to hide the spiders, my users won't care to see what spiders are on the site so no reason to display them IMO, great hack, installing now