Add-On Releases - vBISpy - AJAX real-time feed of new posts/threads

After seeing the http://www.digg.com/spy application; I wondered if this could be done for vBulletin as well. I came across this webpage which had a "how to" on how they built a similiar application to do the same thing. Taking a page from their book (i.e. the code), I modified it for use with vBulletin.

During development, I also came across this post here on vBulletin.Org where the very subject had already been discussed.

So I put this together; a simple ZIP file with a single PHP script, three javascript files and 4 images.

Installation

. UNZIP the file into your forums directory
. That's it!

Virtually all the code to customize the page is in the vaispy.php script; there are no plugins, no templates and no phrases. You do not need to do anything further; this should work right out of the box.

In your browser, open the URL:

http://www.yourforum.com/forum/vaispy.php

You can see examples of this at:

http://www.viperalley.com/forum/vaispy.php
http://www.extremefitness.com/forum/vaispy.php

Note: I claim no ownership of this code except for the file vaispy.php - the rest of the files were using 'freely distributable' sources. As such, you may use these files as you wish, but please do not remove the copyrights.

This modification is for vBulletin 3.6 only, if you are running vBulletin 3.5 you will need to see this thread.

JOIN THE vBIspy Network!
Once you have this mod installed, you can join the vBIspy network to have your threads appear on this site; the vBIspy Network is a great way for people to see what's going on in various vBulletin communities and for forum owners to promote their sites and generate new traffic.

Optional add-ons

Who's Online mod for this: https://vborg.vbsupport.ru/showthread.php?t=126209

vBAdvanced Module: https://vborg.vbsupport.ru/showthread.php?t=126421

UPDATES
1.0.4
Added buro9's code from here

1.0.5 - 9.9.06
Modified some JavaScript in va_spy.js to remove split strings

1.0.6 - 9.10.06
Added date cuffoff code from here and changed init() to spyinit() - changed vaispy.php and va_spy.js files.

1.0.7 - 9.12.06
va_spy.js: Added a forum url variable to allow for integration with portals or other products outside the forum directory
va_spy.js: Added xmldelay variable at top of script for easier configuration or repolling
Updated files: vaispy.php, vb_spy.js

1.0.8 - 9.13.06
vaispy.php - changed code for checking for forum permissions to include password protected forums, added javascript tag for script code

1.0.9 - 9.16.06
vaispy.php
va_spy.js
va_effects.js
- modified to allow for proper display in Opera browsers and moved more html into vaispy.php to allow for stylevars, trimmed va_effects.js to remove unused code (ForumDog's suggestions, except not using templates yet), changed way threads are parsed (removes html as well)

1.0.10 - 9.17.06
va_prototype.js - removed 20kb of unused code

1.1.11 - 9.25.2006
vaispy.php
va_spy.js
- added SirAdrian's mods for thread status icons and alternating row colors
- added code to prepopulate the first 10 rows with existing threads allowing for 5 new ones to be added in scrolling mode (versus scrolling starting from the first thread).
- added option to display subscribed threads only

1.1.12 - 9.26.2006
vaispy.php
- added code to add empty rows if initial pull has less than 20 rows.

[soletrader]

Does anybody know if this hack is server intensive?

[buro9]

Originally Posted by soletrader

Does anybody know if this hack is server intensive?

The original version was, it did a full table scan of the thread table. The version currently attached to the thread only scans the last 24 hours of the thread table. If you're on a busy forum you should have it scan less, I scan only the last hour.

Aside from that, take a look... turn on debug mode and add explain=1 to the do=xml querystring for the php file. I'm seeing the page generated in 5 queries, all taking less than 0.01 seconds, and the largest query isn't the spy query, but the ones that fetch the datastore and templates (I'm using the template layer for spy and not the echo'd HTML that spy originally shipped with).

I also reduced the number of refreshes fired by the JavaScript, from 7 seconds down to 20 seconds. As that is the highest rate of posting on my forum around peak time, so it seemed daft to be firing off so many useless AJAX calls. I'm considering dropping it further to 30 seconds, which would be about the maximum on a large forum before you degrade the user experience of the spy.

In summary: Doesn't need to be intensive, but the first version was.

Oh, and the JavaScript is horrible. All kinds of dodgy 1.0 stuff and bizarre splitting of strings. I'm very much inclined to re-write that stuff using proper DOM stuff. vBulletin is largely XHTML, there's not much excuse to break that on this page through the use of innerHTML rather than DOM manipulation. Besides... use one or the other, both are fast, but combined they are cludgy. So I'm going to change most of mine to fully use DOM stuff, and get rid of those arrays of data when objects would be just as good, or using the original responseXML.

Anyway... rambling... low server load if you use the latest version.

[ForumDog]

Originally Posted by MPDev

I'd probably say that any code to support other plugins should not be standard; if you use that mod you can add the code as you documented.

Fair enough. I've added the hack title and author name to the post so others can find it if they search.

buro9, would you release your improvements? I also moved the code except the inside of the table rows to the templating system and changed the non-IE XHTML to divs and spans with CSS for better markup and to fix the Opera rendering failure, plus some other cosmetic changes/additions. It's currently driving me nuts that the HTML is all inside the JS and I can't use conditionals or anything like that.

It would be really great if I could and the code was vB consistent if I could twist your arm.

[buro9]

When I've done the DOM JavaScript stuff I'll post it here. It may be a few weeks though, I'm still running the old javascript on my site, the new stuff I haven't had tested in many browsers (just Firefox on Ubuntu), so I've yet to even get it past IE on Windows And I'm going to be distracted and slow working for a fortnight as my girl is coming to stay for a couple of weeks... this takes a lower priority

As for the basics to make things better now:
In vaispy.php:
REPLACE:

$daysprune = 1;

WITH:

$daysprune = 0.04;

That will take one days worth of queries down to 1 hour worth of queries (57.1 minutes to be precise). So even fewer rows in the thread table locked.

In va_spy.js:
REPLACE:

var xmldelay = 7000;

WITH:

var xmldelay = 20 * 1000;

AND

REPLACE THE TWO INSTANCES OF:

setTimeout("getXML()", 5000);

WITH:

setTimeout("getXML()", xmldelay);

That changes the time between polling for updates from 7 seconds to 20. I prefer doing the math like that as it makes it obvious it's 20 x 1000 milliseconds = 20 seconds.

Changing that you will have taken each users average 10 queries per minute down to 3 queries per minute. Change it to 30 seconds and they'll be down to 2 queries per minute. Any time span higher than this will reduce the beauty and usability of the spy page too much for my liking. But quiet forums could easily get it to 1 minute per poll.

I also don't like * queries in SQL, so I changed in vaispy.php:

    $getthreads = $db->query_read("
        SELECT thread.*, post.pagetext AS preview, post.userid AS lastpuserid
        FROM ".TABLE_PREFIX."thread AS thread
        LEFT JOIN ".TABLE_PREFIX."deletionlog AS deletionlog ON(thread.threadid = deletionlog.primaryid AND type = 'thread')
        LEFT JOIN ".TABLE_PREFIX."post AS post ON(post.postid = thread.lastpostid)
        WHERE open <> 10 AND thread.lastpostid > $lastpostid
        AND forumid NOT IN (0$blockforums)
        AND thread.visible = '1'
		AND post.visible = 1
        AND deletionlog.primaryid IS NULL
        $datecut
        ORDER BY lastpost DESC LIMIT 10");

TO:

    $getthreads = $db->query_read("
      SELECT thread.forumid,
             thread.firstpostid,
             thread.lastpost,
             thread.lastposter,
             thread.lastpostid,
             thread.replycount,
             thread.threadid,
             thread.title,
             thread.views,
             post.pagetext AS preview,
             post.userid AS lastpuserid
        FROM " . TABLE_PREFIX . "thread AS thread
             LEFT JOIN " . TABLE_PREFIX . "deletionlog AS deletionlog ON (thread.threadid = deletionlog.primaryid AND type = 'thread')
             LEFT JOIN " . TABLE_PREFIX . "post AS post ON (post.postid = thread.lastpostid)
       WHERE open <> 10
         $datecut
         AND forumid NOT IN (0$blockforums)
         AND thread.visible = '1'
         AND thread.lastpostid > $lastpostid
         AND post.visible = 1
         AND deletionlog.primaryid IS NULL
       ORDER BY thread.lastpost DESC LIMIT 10
    ");

The difference on that is neglible on performance, but I prefer to know what my queries return.

[bashy]

Hi peeps

Can someone please take a look at my vaispy.php as the opacity is not working this time. It worked the last time i edited the page to my liking, but then i updated and lot it all, so i have now done it again but the opacity is not working on the last few posts at all....

You can view it here untill i remove for guests http://www.bashys-place.com/forums/vaispy.php

I have enclosed the php for you to check if you have time please?

Also done a quick scout and cannot find how i could stop guests from using this please?
At present guests can only see 10 or so posts, memebsr can see 45, i dont want guests seeing any!

[ForumDog]

Originally Posted by bashy

Hi peeps

Can someone please take a look at my vaispy.php as the opacity is not working this time. It worked the last time i edited the page to my liking, but then i updated and lot it all, so i have now done it again but the opacity is not working on the last few posts at all....

You can view it here untill i remove for guests http://www.bashys-place.com/forums/vaispy.php

I have enclosed the php for you to check if you have time please?

Also done a quick scout and cannot find how i could stop guests from using this please?
At present guests can only see 10 or so posts, memebsr can see 45, i dont want guests seeing any!

There's a post near the beginning that says how restrict permissions to member's only, have a search.

The opacity was not correct in the first place, it's one of the fixes I made I'll post it with the rest soon. Thanks buro9 great to see people working together on this mod.

[MPDev]

Originally Posted by buro9

All kinds of dodgy 1.0 stuff and bizarre splitting of strings.

In the latest version I got rid of that code as well; for IE I just use a string.

I'll change the query in the next update so everyone can benefit from the improvement.

And nice to have others putting in the effort! Thanks.

[bashy]

I did have the opacity working the on 1st install, it was fine

Originally Posted by ForumDog

There's a post near the beginning that says how restrict permissions to member's only, have a search.

The opacity was not correct in the first place, it's one of the fixes I made I'll post it with the rest soon. Thanks buro9 great to see people working together on this mod.

[MPDev]

It works for IE, but not on FF; I assume it's a tag, but haven't had the time to go looking just yet.

[MPDev]

Originally Posted by ForumDog

I also moved the code except the inside of the table rows to the templating system and changed the non-IE XHTML to divs and spans with CSS for better markup and to fix the Opera rendering failure, plus some other cosmetic changes/additions.

The Opera fix would be helpful; as I am sure the others would be. Thanks for sharing.