vBusy: Last minute session stats analyzer for admins

[tamarian]

There's also a vB 3.5 version: https://vborg.vbsupport.ru/showthread.php?s=&threadid=82788

What this hack does: This hack will allow you to have a quick glance at what's going on in the forum, based on the session table. The session table contains short-term information, that can be useful in finding out if things are more busy than usual. This idea come from this thread: How to view the busiest threads? (https://vborg.vbsupport.ru/showthread.php?t=82596) You will have the option to view the bussiest threads, or all activities, in the last 1, 5 or 10 minutes. You can enter addiitonal minute values in the URL.

Features:
Lists is descending order busiets/most active:

Threads
Forums
PHP Scripts
Hosts IP's
Resolved IP's
Useragents and bots
Lists all session data, with sort options based on (time, user, host, useragent, location.

Per minute chart of sessions table activities.

Installation: : Just copy the file vbusy.php into your admincp directory.

Optional: If you want to have it integrated in the admin panel menu, just do this simple edit:

In ./admincp/index.php:

Find:
construct_nav_option($vbphrase['scheduled_task_log'], 'cronlog.php?do=choose', '<br />');

Add after it:
// vBusy hack
construct_nav_option('vBusy', 'vbusy.php', '<br />');
// vBusy end

Click install to receive auto updates about new features/bugs.

[Corriewf]

Any screen shots?

[Corriewf]

How could this be used in a practical sense?

[tamarian]

How could this be used in a practical sense?

It's mainly useful for busy forums, or to see if you've been slash-dotted, which thread is being slash-dotted etc. See this thread:

https://vborg.vbsupport.ru/showthread.php?t=82596

[Corriewf]

It's mainly useful for busy forums, or to see if you've been slash-dotted, which thread is being slash-dotted etc. See this thread:

https://vborg.vbsupport.ru/showthread.php?t=82596

Slash dotted?

[tamarian]

Slash dotted?

It's a saying, for quick spikes in traffic, like what might happen when your site is mentioned in a slashdot ( http://slashdot.org/ ) article, or a ddos attack, etc.

[Corriewf]

I guess what I meant by my previous question is what kind of action could be taken to correct the problem based off data from this hack.

[tamarian]

I guess what I meant by my previous question is what kind of action could be taken to correct the problem based off data from this hack.

That's a tough question :) Defending against a ddos is very complicated, and outside the scope of this thread. This hack may give you a hint as to what range of IP's should be blocked, if their activity level is suspicious. If it shows a thread being targeted, and bringing the server to it's knees, you may want to hide the thread for a while.

The hack is simply a monitoring/diagnostic tool, what you do with the information is up to you. But it's better to know what might be happening behind the scene, than not having any idea why things seem too busy.

[Spinball]

I requested such a hack because we occasionally get spikes in traffic. We chug along at about 800 or 900 visitors online, but occasionally it will spike to over 1,200 and at unusual times. This hack will help me see what they are looking at. As tamarian says, it's usually because one if your threads has found interest on another busy site and people are clicking through in droves.
Well done tamarian :up: It's nice and fast, too.
Unlike who's online !

[Reeve of shinra]

Interesting and no harm in having as it can be usefule *click*

[tamarian]

I think I'll add a few more goodies. One thing that might pinpoint spikes is a one-minute chart, link for each minute to isolate what was going on. Also stats for most active hosts, scripts, spiders... stay tuned :)

Wa'il

[Corriewf]

What would be nice is to see query specs like whats the most run right now ect ect to troubleshoot even the queries being used.

[Abe1]

You say to add:
// vBusy hack
construct_nav_option('vBussy', 'vbusy.php', '<br />');
// vBusy end
Shouldn't it be:
// vBusy hack
construct_nav_option('vBusy', 'vbusy.php', '<br />');
// vBusy end

You spelled vBusy with 2 's'.

[tamarian]

You say to add:
// vBusy hack
construct_nav_option('vBussy', 'vbusy.php', '<br />');
// vBusy end
Shouldn't it be:
// vBusy hack
construct_nav_option('vBusy', 'vbusy.php', '<br />');
// vBusy end

You spelled vBusy with 2 's'.

You are correct :) I'll correct the spelling, but it should still work, since the spelling error is in the title, not the script name.

[Spinball]

The most useful thing would be to see where the majority of people are coming from. Within the thread with the most views, show the referers by their frequency.

[tamarian]

The most useful thing would be to see where the majority of people are coming from. Within the thread with the most views, show the referers by their frequency.

The referrer information is not stored in the session table. It's not that hard to add, it just requires adding a new field to the session table, and trivial code changes to the /includes/session.php file. So, it is doable. But there are a few caveats, that may make it not so desirable. The session data is not cumilative, which means when the visitor clicks on a link (say to see the next page in the thread), the record will be replaced with this activity, and the referrer will be your site/forum, so it will not give you the data you really want, unless you run the script at the righ time. This means if you run the script, a couple minutes later, you may not find the actual referrer.

The best thing to do, IMHO, is that once you identify the time when the spike occured, is to look at the server logs for that time period (you will have the exact time down to the second), and just look at the "referrer" tag in the apache log.

If you still want the referrer though, I can send you to required steps by PM, to avoid having others here do the database updates if they'd rather not have this feature. Or if there's enough demand, I can list the steps here.

[Paul M]

Referrers would be useful.

[Spinball]

tamarian, the situation which leads to the scenario I am taking about here is a link to a specific thread on your site from another site. (or sometimes if you're lucky, more than one site) when something particularly interesting has caught their attention.
Their referer information*should* in the first instance hold the site containing the link to yours.
Perhaps the referer information could only be stored if it's off site.
Although having all the referer information would allow us to show the route people take though the site. Something which may be useful in analysing how well presented and usable your site is.
However, I don't think any of this fucntionality is worth slowing down the forums significantly.
I can't actually analyse our apache logs as they are too big.

[Edit:] My moderators have asked if the thread title rather than the thread number could be shown. I agree that this would be most useful.

[Freezerator]

Nice hacks, installed en works like it should.

[tamarian]

tamarian, the situation which leads to the scenario I am taking about here is a link to a specific thread on your site from another site. (or sometimes if you're lucky, more than one site) when something particularly interesting has caught their attention.
Their referer information*should* in the first instance hold the site containing the link to yours.
Perhaps the referer information could only be stored if it's off site.
Although having all the referer information would allow us to show the route people take though the site. Something which may be useful in analysing how well presented and usable your site is.


The session table won't be helpful in analysing the viewing patterns, or route through the site, because it replaces the activity, rather than add a new record.

There's an easier option, if you want to know when a specific site is sending referrals to you. You can add a short peice of code in showthread.php, that "if the referral is site xyz.com, then email this address that they've come". This would have the advanatage the information is not lost, and gives you an email alert. It also has the advanatage of no database access, and no added processing, unless the referral you're interested in has dropped by.

However, I don't think any of this fucntionality is worth slowing down the forums significantly.

Exactly. The session table is heavily used, and any additional fields and processing in the session file will be executed on every single vBulletin function.

I can't actually analyse our apache logs as they are too big.

You might want to consider adding Urchin (http://www.urchin.com/) to your server. It's very effecient in crunching the logs and making useful reports out of it. Most hosting companies can install it for a one time license fee of $20 or so.


[Edit:] My moderators have asked if the thread title rather than the thread number could be shown. I agree that this would be most useful.

That would be easy to add.

[bigcurt]

Lol dont let these guys get ya down, I really like this hack..it will work with 3.0.3 correct?

[bigcurt]

/me installs.

[tamarian]

Lol dont let these guys get ya down, I really like this hack..it will work with 3.0.3 correct?

Should be no problems with any 3.0.x :)

[Spinball]

Very,very good.

[tamarian]

For the brave souls who'll upgrade to vB 3.5 beta, this has now been ported:

https://vborg.vbsupport.ru/showthread.php?threadid=82788

[Spinball]

Got a bug here tamarian in the minute report. No bar images. When I do a view source:
<td width="100%" align="left" nowrap="nowrap"><img src="../http://www.avforums.com/forums/images/polls/bar3-l.gif" height="10" /><img src="../http://www.avforums.com/forums/images/polls/bar3.gif" width="33.783783783784%" height="10" /><img src="../http://www.avforums.com/forums/images/polls/bar3-r.gif" height="10" /></td>
vBadvanced requires http:// type paths to the various image directories.

[tamarian]

Got a bug here tamarian in the minute report. No bar images. When I do a view source:

vBadvanced requires http:// type paths to the various image directories.

Doesn't look like a bug from vBusy. Seems like vBadvanced (or something) is inserting http in the wrong place.

img src="../http://www.....

I haven't used vBadvanced, so I don't know why it's doing it, or how to fix it.

[Spinball]

Doesn't like a bug from vBusy. Seems like vBadvanced (or something) is inserting http in the wrong place.

img src="../http://www.....

I haven't used vBadvanced, so I don't know why it's doing it, or how to fix it.

No, it's because vBAdvanced requires you to put http://www.yoursite.com/forums/images/buttons
for the button folder rather than /images/buttons
so that the vBadvanced pages can be located anywhere on your server and the buttons still appear.
This doesn't screw up the buttons on the rest of the forums, but it does make the bar image broken in the vBusy hack because you are adding '../' to the front of the folder name.
Just FYI - not a complaint, really since I'll just edit out the bit which adds the '../'.
But since a lot of other people use vBadvanced, it's something to bear in mind. :)

[tamarian]

No, it's because vBAdvanced requires you to put
http://www.yoursite.com/forums/images/buttons
for the button folder rather than
so that the vBadvanced pages can be located anywhere on your server and the buttons still appear.

I see. But not sure how to fix this, without breaking it for those not using vba. vBusy is basically using the image path from your style variable.

A question about vba, when you go to the admincp stats page, and view the stats for registration or new posts, do the bars show, or is it the same problem? If they do show properly, was the stats.php file hacked for it to work?

[dsboyce8624]

I see. But not sure how to fix this, without breaking it for those not using vba. vBusy is basically using the image path from your style variable.

A question about vba, when you go to the admincp stats page, and view the stats for registration or new posts, do the bars show, or is it the same problem? If they do show properly, was the stats.php file hacked for it to work?

Yes they work. And removing the '../' . fixed it.

I did also just notice the time is off by 4 hours.

[Moparx]

Invalid SQL:
SELECT count(inthread) as viewers, inthread, thread.title FROM vb3_session
LEFT JOIN vb3_thread ON thread.threadid=session.inthread
WHERE inthread > 0
AND lastactivity >= 1119493313
GROUP BY inthread ORDER BY viewers DESC
mysql error: Unknown table 'thread' in field list

mysql error number: 1109

there is no thread.title (or forum.title) in the sessions table. (i'm running 3.0.7)

[tamarian]

Invalid SQL:
SELECT count(inthread) as viewers, inthread, thread.title FROM vb3_session
LEFT JOIN vb3_thread ON thread.threadid=session.inthread
WHERE inthread > 0
AND lastactivity >= 1119493313
GROUP BY inthread ORDER BY viewers DESC
mysql error: Unknown table 'thread' in field list

mysql error number: 1109

there is no thread.title (or forum.title) in the sessions table. (i'm running 3.0.7)

Those are to read from the thread and forum tables for the tile.

I just updated the file, please re-download, and let me know if it fixes the problem. :)

[Moparx]

Database error in vBulletin 3.0.7:

Invalid SQL:
SELECT count(inthread) as viewers, inthread, thread.title FROM vb3_session
LEFT JOIN vb3_thread as thread ON thread.threadid=session.inthread
WHERE inthread > 0
AND lastactivity >= 1119495417
GROUP BY inthread ORDER BY viewers DESC
mysql error: Unknown table 'session' in on clause

mysql error number: 1109

works with . TABLE_PREFIX . added to to session.inforum ;)

[tamarian]

works with . TABLE_PREFIX . added to to session.inforum ;)

Well, while we're at it, try the new version :) I just re-uploaded. It was missing one "as" alias. Let me know if it fixes it.

[Moparx]

working perfectly :D

[tamarian]

working perfectly :D

Great. I need to have a table prefix on my test server, so I don't forget those aliases :)

[Hornstar]

tamarian, im going to pm you....but im very interested to know anything you can share on finding the IP of a person who is ddossing my site.

Secondly if i find the ip of the person, they are most likely using a proxy, could there be a way to detect if that ip is a proxy, and if yes reveal the true ip.

and lastly, will blocking that ip address from my site, stop the ddosing, or are most ddosing programs going to hide the ip anyway.

Anything on this would be really good.

Another question because you seem smart ^^ someone was mentioning something about find out where they came from, im also interested to find out where people are coming from b4 they get to my site, e.g other sites or google etc.

Thanks for any info you can help me with.