psiStats 2006 (The New vBStats for vBulletin 3.5.x - 3.8.x)

I no longer support my hacks. Please feel free to update them and release new versions elsewhere as long as I get the credit for the original modification.

/*===================================*\
|| psiStats 2009
||
|| Author : Anton Kanevsky
|| Works on : vBulletin 3.5.x - 3.8.x
|| Released : Feb 28, 2010
||
|| Time required to install: 1 minute
|| Difficulty: easy
\*===================================*/

DESCRIPTION

A sophisticated statistics module for your vBulletin board.

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

STATISTICS

File Uploads: 1
Products to Install: 1


FEATURES

Statistics Tracking:

• General Board Statistics
• Style Usage Statistics
• Top Threads & Forums Statistics
• Display Colors + Resolutions (Requires JavaScript) Statistics
• Countries + Regions Statistics
• Browsers + Operating Systems Statistics
• Referring Sites Statistics
• Referrer Statistics
• Age & Gender Demographics

Features:

• No code or template modifications required.
• Modules can be enabled or disabled, as needed.
• Statistics are updated every fifteen minutes via cron.
• Mother site, e.g. the site where your forum is hosted, will never be listed as a referrer.
• Only one extra query per forum page.
• Statistics can be sorted either numerically or alphabetically.
• Automatic Integration with Referrer Statistics (if installed)[/i]
• Configurable number of last items for top threads / forums.
• Allows to specify whether to show data from from forums with post counts disabled.
• Configurable number of last items in referring site statistics.
• Configurable gender field and responses (optional).
• Configurable list of usergroups that can access statistics (optional).
• Referring site statistics censure (by word) and individual site blockage.
• Detects most popular browsers and operating systems.
• Requires cookies for screen color depth and resolution tracking.

VERSION HISTORY

1.6.2 - stable
[+] Added recognition for Windows 7-based user agents.

1.6.1 - stable
[+] The navbar link is now shown only to those who can access the statistics.

1.6.0 - stable
[+] Added two new modules:

• Referrer Statistics
• Age & Gender Demographics

[+] Added Google Chrome recognition (this includes a matching icon).
[+] Added Setting to control whether top threads & forums data is gathered from forums where posts are not counted towards user post counts.
[+] Updated ip to country databases to the most recent ones.
[F] Corrected a few phrasing bugs and phrased things that were not phrased in 1.5.2.

1.5.2 - stable
[+] Added a few more translations (thanks to those who submitted these).
[+] Fixed a few bugs that were mentioned for 1.5.1.

1.5.1 - stable
[+] Added option to globally enable / disable any stats modules.
[+] Fixed bug with some staff being counted twice.
[+] All numbers are now nicely formatted.

1.5.0 - stable
[+] No more template edits, just upload the files and the product and you're ready to go.
[+] vBulletin 3.5.x - 3.8.x compatible.

1.4.4 - stable
[+] Referrering Sites now have a JS and an XML feed capability.
[+] Added a better firefox icon into the icon database.
[F] Fixed a couple of minor bugs.

1.4.3 - stable
[+] The hack now respects vBulletin's "Show Active Users" setting.
[+] The hack is now valid XHTML (all templates must be reverted).

1.4.2 - stable
[+] Corrected a bug, which caused incorrect number of moderators to be displayed.
[+] gb.gif renamed to uk.gif.

1.4.1 - stable
[+] The front page has been revised, and it now loads four times as fast regardless of the size of your database.
[+] The ip database has been updated to the revision of October 18, 2006.
[+] The flag database has been updated to the revision of October 18, 2006.
[+] Referrer statistics hack, if installed, is now detected automatically. No configuration required.
[+] Added an option to specify a custom board creation date, if not reflected correctly by the joindate of a user with lowest userid.

1.4.0 - stable
[+] The system is now cross-compatible with both vBulletin 3.5 and 3.6.
[+] The ip database has been updated to the revision of August 30, 2006.
[+] A variety of code optimizations and template corrections have been done.
[+] To keep statistics legible, the engine now only tracks the major and the minor browser versions. X.Y.Z.E is now stripped to become X.Y.

1.3.2 - stable
[+] Since more browsers support cookies than ajax, screendepth/screenresolution tracking is now done via cookies.
[+] One less query per page - e.g. now it's only one query per page.
[+] Statistics are now updated every 15 minutes rather than every hour.
[F] Fixed faulty count in cron and in region stats screen.
[F] Fixed bug which prevented the cron from being created in some cases.
[F] Fixed minor bugs which caused errors of the E_NOTICE level.

1.3.1 - stable
[F] Fixed AJAX coding error, which used synchronous ajax instead of asynchronous, evidently causing higher load.
[F] Fixed WOL error that was caused by AJAX stats update and that made it look like everyone is browsing the stats.
[F] Fixed a query in update cron.

1.3.0 - stable
[+] Yay, no more file edits!
[+] No more multiple queries. The hack now takes up a maximum of two (2) queries per page in contemporary browsers, and one (1) query in non-javascript browsers.
[+] New vBulletin Option: Statistics can be configured either numerically or alphabetically.
[+] Since Opera is updated way too often, the engine, from now on, will only identify the primary version marker (7,8,9 as opposed to 7.01, 7.02, 7.21, 8.01 etc).
[+] Templates and phrases are now in their own separate group.
[F] Eliminated a bug which prevented bot identification from working properly.

1.2.8 - beta
[+] Removed 5 queries during the stats recording.
[+] Countries mapped to valid cia abbreviations and the countrybit template has been updated.
[+] Added missing Fiji flag (Thanks to firstrebel!)
[F] Eliminated a bug which prevented the thread and post limit from having any effect.
[F] Eliminated a bug which prevented the browser identification from working properly in some cases.

1.2.7
[F] Corrected online users count.
[F] Corrected staff count.
[F] Eliminated the style stats bug, which originated from users who had invalid styles saved in their profiles.
[F] Eliminated the posts per day bug.
[F] Renamed "gb.gif" to "uk.gif"
[F] Renamed "Oceania" region to "Australia and Oceania".
[F] Only threads and forums to which the user has permissions are displayed in top threads and forums.

1.2.6
[F] Minor template fix.

1.2.5
[+] Rewritten Browser and OS detection (previosly somebody else's module was used).
[F] For staff stats, only primary usergroups are now counted.

1.2.1
[+] New "Usergroups Allowed To View Stats" admincp option.
[+] Navigation split in two rows, for greater view
[+] Crawlers now have a browser picture associated with them.
[F] Eliminated Tracking of Empty Referrers.
[F] Updated country database with a number of missing countries, including UK.
[F] Minor fix to style statistics screen, applying to styles that are not used.

1.2.0
First Official Release.

1.1.0
Development release; Excess queries, phrases, and code has been removed.

1.0.0
Development release; All phrases that were hard-coded were moved into the phrase system.

The installation manual is contained within the attached file.

IF YOU LIKE MY HACK, PLEASE CLICK INSTALL

[akanevsky]

@MPDev Yorixz, Thank you

@MPDev:
The changes you introduced are not correct. Please look at the following two lines:

$deletions[] = "(`cdata` = '$ident[1]' AND `count` = '$data[0]' AND `ctype` = '$ident[0]' AND `cmisc` = '$data[2]')";
$insertions[] = "('$ident[1]', '" . ($data[0] + $data[1]) . "', '$ident[0]', '$data[2]')";

$data[0]+$data[1] is where the old count is added to the new count.

[akanevsky]

Originally Posted by gothicuser

I was referring to your response! I used to drive coaches professionally for almost 35 years and if I spent my time throwing my toys out the pram at some of the stupid/silly/annoying comments from my passengers I would have been driving an empty coach all that time.
Relax, take it all with a pinch of salt. Of course you are going to get frustrating little tic-turds pestering you all the time, but thats life! Learn to live with it without embarrassing yourself.

And I was referring to you pointing a finger at yourself
I'm sorry, son, but you are not in a position to preach anything to anyone around here.
You have been added to my ignore list.

[gothicuser]

Never mind, you carry on the way you do, the rest of the world will carry on ignoring you!

[MPDev]

Originally Posted by Psionic Vision

@MPDev:
The changes you introduced are not correct. Please look at the following two lines:

$deletions[] = "(`cdata` = '$ident[1]' AND `count` = '$data[0]' AND `ctype` = '$ident[0]' AND `cmisc` = '$data[2]')";
$insertions[] = "('$ident[1]', '" . ($data[0] + $data[1]) . "', '$ident[0]', '$data[2]')";

$data[0]+$data[1] is where the old count is added to the new count.

I believe you are incorrect - the reason is, as I pointed out for example, there are multiple US lines being printed; so for every US line, it overwrites the "old" value with a "new" value without taking into consideration that the results need to be cumulative during processing. In other words $data[1] needs to be a sum of all the results returned and not just the last result processed.

After alot of testing on my very busy site, I believe they are correct.

[akanevsky]

I believe you are incorrect - the reason is, as I pointed out for example, there are multiple US lines being printed; so for every US line, it overwrites the "old" value with a "new" value without taking into consideration that the results need to be cumulative during processing. In other words $data[1] needs to be a sum of all the results returned and not just the last result processed.

Hmm... Yes. I see what you mean now.
However, the initial query is supposed to group the results and there should be only one single row per value... And there should not be more than one row per value in cache anyway. Let me check it...

[akanevsky]

Yup, the query on my table gives the following results:

count   ctype           cdata
2	country	        19x.12x.5x.25x	
1	country	        213.6.250.24x	
1	country	        24.18x.34.5x	
1	country	        8x.201.64.9x	
2	country	        84.17x.214.16x	
2	country	        87.24x.1x.1	
1	referrer	yandex.ru	
2	useragent	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)	
3	useragent	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)	
1	useragent	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MRA 4.1 (build 00975))	
1	useragent	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MRA 4.1 (build 00975); .NET CLR 1.1.4322)	
2	useragent	Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; MyIE2; MRA 4.3 (build 01218))

As you can see, there are no two same rows, so there should not be a need to count it twice.

[MPDev]

Look closer at your data.

MSIE 6.0; Windows NT 5.1

                                $updatecache["browser_||_$browser[bstring]"] = array($existingcache["browser_||_$browser[bstring]"]['count'], $itemcnt, $existingcache["browser_||_$browser[bstring]"]['cmisc']);

Given your example with multiple MSIE 6.0 entries, how does your code keep a cumulation of those entries?

You do not resolve your IP addresses to countries until you process the cache. So, again, where are your results cumulative for, say, US based IP addresses as you process your cache? (Again, looking at the data I posted, you can see how you only account for a single US ip address per cache processing versus the sum of all US based ips.

[akanevsky]

So are you saying that there can be two same entries with the same sessionid?

Given your example with multiple MSIE 6.0 entries, how does your code keep a cumulation of those entries?

Oh. Now I am convinced Will implement your fix.

[MPDev]

As I just edited, same goes for IP addresses (as everything else); since you post process country codes you do not track the cumulative totals for country results; you only process the last entry (which is most likely only 1).

                                        $updatecache["country_||_$coucode"] = array($existingcache["country_||_$coucode"]['count'], $item['count'], $existingcache["country_||_$coucode"]['cmisc']);

country_||_US will be processed hundreds of times, but since you don't increment the results, only the last record processed is every recorded.

[MPDev]

Just as an FYI, because of the ip include scripts; larger sites like mine will need to process their cache with much more frequency than 1 hour or the memory requirements can be HUGE. (I had 15,000 records and not even 999MB of memory allocation would allow me to process it).

I'm a total data-junkie and love stuff like this - I hope you don't mind me picking it apart and finding this stuff because I want as accurate of results as possible; this kind of info is very important to advertisers and webmasters who really want to understand who is using their sites.

I'll probably add a referrer tracker to this because I really want to track actual links into my site from external sources.