Forum Display Enhancements - Remove Spiders from Who's Online

*** Staff note: The author of this modification has passed away in a diving accident. We wish his family all strength in dealing with this traggic issue. ***

Remove Spiders from Who's Online

The problem

The majority of the the "guests" on your forum are probably spiders. While you might think that it is impressive to show so many guests using your forum, this does not reflect the true number of people online. The who's online page does offer visitors the chance to change the display based on members, guests, or spiders, however the totals (and the record number of users online at one time) do not exclude spiders.

This mod does not remove spiders from your forum, it simply stops them being counted in the totals.

Some search engine spiders such as Yahoo's Slurp can initiate hundreds of connections at the same time (from different IP addresses). Each one of these connections will be included in your totals as a "guest".



What does this modification do?

• Completely remove all known spiders (spiders found in your includes/xml/spiders_vbulletin.xml file) from the who's online display
• Updates the true guest totals displayed on the main forum page
• Resets your maximum users online record on installation
• Includes changes for vBadvanced's online users module
• Doesn't offer separate spider statistics or configuration, it just makes it appear like those spiders aren't on your forum (even though they are)
• Easy access through template conditionals and variables to display different behaviours based on whether the connection is from a spider or not.
  

Who should use this modification?

Anyone who wants to their forum to show the true number of users online in the statistics

I run a big system, will this slow my board down?

Absolutely not. It's been written with big boards in mind. Even if you use a very large spiders_vbulletin.xml file, this file will only be checked once for each new session created. After that, whether the session is used by a spider is recorded in the session table. For users running vBulletin 3.6.6 or later, it will actually increase the speed of the who's online page, since known spiders are no longer compared against the list of known spider user agents. No new database queries are added, as existing queries are used or modified instead.

Hasn't this been done before?

There are a couple, but I've taken a slightly different approach with this mod - my focus was on simplicity and performance. You might also want to check out some other mods here and here, to see if they suit your purposes better.



Installation instructions

• If using vBadvanced, see the instructions below for a manual edit of your onlineusers.php file
• Install the enclosed .xml file via AdminCP -> Plugins & Products -> Manage Products -> Add/Import Product. Remember to click overwrite if upgrading
• Wait a few minutes - existing sessions must have some activity before they are reclassifed as a genuine member or guest
• (optional) Edit your admincp quick statistics file (instructions below)
• (optional) Reset your maximum online statistics again (if you feel the need, but it is done automatically during installation)

Configuration instructions

Nothing to configure! Just see the note below about spiders_vbulletin.xml

How can I make this more effective?

Spiders are identified via the includes/xml/spiders_vbulletin.xml file. However, the version that ships with vBulletin is quite small, and will recognise only the most popular spiders. Anything else will still show up in your statistics as a guest, making it hard to tell real guests from spiders.

Therefore it is recommended to update your spiders_vbulletin.xml if using this mod. You can obtain a much better version of this file from Christian Stadler here. Please contribute new user agent strings directly to him and encourage him to keep providing new updates of this file.

Will using a big spiders_vbulletin.xml slow things down? Theoretically yes, but in practical terms now. Lookups are only one once for each session when it is created. If you are running a version earlier than vBulletin 3.6.6 it will mean that more work must be done on the who's online page, but nowhere else.

Which versions of vBulletin will this work on?

This mod is designed to work for 3.6.6 and above, but there is some additional code included to handle 3.6.0 - 3.6.5 as well.

Optional extras

If you are using vBAdvanced then you'll want to modify your online users module. The following change should work for but versions 2.x and 3.x. Go to your modules/onlineusers.php file and locate this line:

WHERE session.lastactivity > " . (TIMENOW - $vbulletin->options['cookietimeout']) . "

Directly under this line, insert the following line (on a line by itself):

" . ($killspiders ? " AND " . TABLE_PREFIX . "session.spider = -1 " : null) . "

Please don't forget to include the quotes exactly as they appear above.

If you'd like to also include the quick stats in your admincp with this modification open your admincp/index.php file and locate this line:

$guestsarry = $db->query_first("SELECT COUNT(host) AS sessions FROM " . TABLE_PREFIX . "session WHERE userid = 0 AND lastactivity > $datecut");

replace it with:

$guestsarry = $db->query_first("SELECT COUNT(host) AS sessions FROM " . TABLE_PREFIX . "session WHERE userid = 0 " . (($vbulletin->products['whosonline_kill_spiders'] AND $vbulletin->options['enablespiders']) ? ' AND spider = -1 ' : null) . "AND lastactivity > $datecut");

Would you like to reset your maximum online users statistic (the one on the front page or who's online page that tells you the most number of users ever online)? go to the AdminCP -> Maintenance -> Execute SQL Query and run the following query:

DELETE FROM datastore WHERE title = 'maxloggedin' LIMIT 1

There is an additional file, spider_test.php that can be used to test the operation of this mod, as well as giving you a handy overview of which guests have not been classified as spiders, but probably should be. You can then use the user agent strings to expand your spiders_vbulletin.xml file. You can run spider_test.php from any forum directory, just upload it and then type the path to the file into your browser.

Support

Will be provided to those who click install

Version history

1.0 (15.07.2007) Initial version

Disclaimer: No actual spiders where harmed during the development of this modification.

[SnitchSeeker]

This looks like an EXCELLENT mod!

But honestly, it's more than I want. For example, I don't want to reset my max users online stats and I already have a mod (vbBOL) that shows spiders as spiders in the "who's online" page.

Mainly, I am looking for some way of hiding certain html code in my templates from spiders (or showing them custom code), as you show in this example:

<if condition="$session[spider] == 1">Come into my parlour, said the spider to the fly</if>

Can I install this without resetting stats? Or do you know of another way that I can change code in templates for spiders?

I don't mind not showing spiders as guests in the "Currently Active Users" stats, I just don't want to ruin anything else.

I think it would be cool to show how many bots/spiders are online in the "Currently Active Users" stats. As an addition after "__guests" like (42 members, 192 guests and 63 spiders)

[mfyvie]

Originally Posted by keyness

Well both are incorrect. For more than 30 minutes, it just showed maximum 5 members and only 1 guest (and another proof, more than 20 guests enter forum just via google in an hour)

Ok, I've now attached an additional file called spider_test.php, you can run this from any forum directory. It will test your installation and basically show you the things that I have been describing to you about this issue (regarding when I asked you to verify your doubts by looking into the session table). Since this file shows you directly what is in your session table, but classified according the spider type, you should be able to see that the statistics being reported are the true statistics.

This file is also useful for anyone else who doubts that this mod is working, or for anyone who wants to find new spider strings to add to their spiders_vbulletin.xml.

Originally Posted by TheMasterG

This is the who's online hack I've been waiting for, much simplier and truer than others out there. Like you, I just want to hide the spiders, my users won't care to see what spiders are on the site so no reason to display them IMO, great hack, installing now

Thanks for your encouraging words. Within an hour of this mod being released 3 people rated it "Terrible" via the rating system, but nobody had actually downloaded it. I for one use the rating system when I'm deciding whether to install a mod, so I was puzzled as to why 3 people would have done this (without even looking at the mod). But anyway, I'm glad there are people out there that do understand why this mod is a little different and like it for exactly what it is.

Originally Posted by SnitchSeeker

This looks like an EXCELLENT mod!

But honestly, it's more than I want. For example, I don't want to reset my max users online stats and I believe I already have a mod to not show spiders as guests.

Mainly, I am looking for some way of hiding certain html code in my templates from spiders (or showing them custom code), as you show in this example:

<if condition="$session[spider] == 1">Come into my parlour, said the spider to the fly</if>

How can I do that without resetting or changing the other stuff?

Well actually, it sounds like this mod is not more than you want, it's exactly what you've been looking for, and it does exactly what you need. If you believe that you already have a mod that excludes spiders, then disable it - because this mod does that as well, but probably with less overhead to your forum.

It seems like your only beef with this mod is that it will reset your max online statistics when it installs? The reason it does this is because your max online statistics are probably false! Even if other mods remove the spiders from your display, are they stopping the artificially high number from inflating your max online statistics? Even if they stop it at the front end, have they also thought to stop it in the back end as well? Because as soon as you login to your admincp the wrong statistics will be recorded! My mod stops the admincp from updating this number, even if you haven't done the optional manual edit in the admincp.

But if you still want to disable the reset of the maxonline, simply download the .xml file and remove the entire block that looks like this:

        <code version="*">
            <installcode><![CDATA[// Each time we install we'll zero out the max online counter.

$vbulletin->maxloggedin['maxonline'] = $totalonline;
$vbulletin->maxloggedin['maxonlinedate'] = TIMENOW;
build_datastore('maxloggedin', serialize($vbulletin->maxloggedin), 1);]]></installcode>
            <uninstallcode />
        </code>

Delete it, then save the .xml file. Now when you install it, it won't touch your (falsely inflated) max online stats.

[Kohhal]

One thing I've noticed, I reset the max users online, used to be ridiculously high and inaccurate, immediatly I get a new figure for the time that I did the reset , BUT, it still seems to be including the spiders in this figure as the forum shows 10 members and maybe 10 guests, but most users online after reset shows 100+ ?

[SnitchSeeker]

Delete it, then save the .xml file. Now when you install it, it won't touch your (falsely inflated) max online stats.

Thanks!

My stats are falsely inflated, but not by more than a couple hundred at most. If I reset it, the figure would be incredibly low and inaccurate, so I don't know which is worse. My boards were WAY more popular 2 years ago than today. The reason I know this (other than the inflated "most users" stat) is we had over 100 new users registering each day. Now we have maybe 30 new users on average per day. "Most users online" currently shows 1,786 (dated 2 years ago) and most of late is around 1,000 less.

I just noticed I am running 3.6.5 and not 3.6.6. Do you think it will be ok for me to install it?

PS. I think it would be cool to show how many bots/spiders are online in the "Currently Active Users" stats. As an addition after "__guests" like (42 members, 192 guests and 63 spiders)

[mfyvie]

Originally Posted by TheMasterG

One thing I've noticed, I reset the max users online, used to be ridiculously high and inaccurate, immediatly I get a new figure for the time that I did the reset , BUT, it still seems to be including the spiders in this figure as the forum shows 10 members and maybe 10 guests, but most users online after reset shows 100+ ?

Ok, well there are only 3 places I know of where this can happen. The admincp (which my mod stops from updating the total), the who's online display, the main forum display, and the vbadvanced module (if you are running vbadvanced). If for example you are running vbadvanced and forgot to edit this module, it will be reseting the totals for you.

Can you confirm that you aren't running vbadvanced, and that you don't have any other areas of your forum where the maxonline is displayed (or updated)? Also, which version of vb are you running? If you are running pre 3.6.6 there is different code, but I wasn't able to test this because I'm running 3.6.7.

Originally Posted by SnitchSeeker

My stats are falsely inflated, but not by more than a couple hundred at most. If I reset it, the figure would be incredibly low and inaccurate, so I don't know which is worse. My boards were WAY more popular 2-3 years ago than today. The reason I know this (other than the inflated "most users" stat) is we had over 100 new users registering each day. Now we have maybe 30-50 on average. "Most users online" currently shows 1,786 (dated 2 years ago) and most of late is around 1,000 less.

Originally Posted by SnitchSeeker

I just noticed I am running 3.6.5 and not 3.6.6. Do you think it will be ok for me to install it?

It should work, though I'm not able to test it - there's a note in the installation instructions about pre 3.6.6 sites.

Originally Posted by SnitchSeeker

PS. I think it would be cool to show how many bots/spiders are online in the "Currently Active Users" stats. As an addition after "__guests" like (42 members, 192 guests and 63 spiders)

Actually this mod was designed not to even offer that as an option. I'm reusing the same query, so there's no additional overhead. To start getting fancy means to depart from the original design goal of this mod - simplicity and speed. I've never understood why people want to know the number of spiders on their board - it's a totally meaningless number - you could have 500 yahoo slurp spiders crawling one page per session, or you could have a single spider crawl 500 pages in a single session. What's more important to check is the spider activity in your weblogs - it really doesn't have any place in your online users in vbulletin.

[R@V3N]

Very nice mod. Just what I needed. Someone decided it would be fun to spam he online count and I like to report real numbers. Keep up the good work.

[Charlie98902]

I tried this out on my forum and there seems to be a error on IE 7 users the error # is 163 I think. There is more text and I'll gladly install it again to be verbatim if you need it.

[mfyvie]

Originally Posted by Charlie98902

I tried this out on my forum and there seems to be a error on IE 7 users the error # is 163 I think. There is more text and I'll gladly install it again to be verbatim if you need it.

Do you mean you saw this error when you tried to install, or when it was running? This mod doesn't actually directly change anything that is output to HTML, it simply changes things in the background. I've no idea what might have caused this? File permissions?

[koxito]

Wow .. a very good mod, installed on my board!

[TheBlackPoet]

hey... i love it.... i hate the deceptive countup on my site.... real members and real visitors work better for me to guage what's what!! so thanks..