Deferred Threadviews v1.00 (to reduce server load)

Hack name: Deferred Threadviews
Version: 1.00
Purpose:
To reduce server load by not updating the view count on every page view but instead keeping the it in memory and writing to the db only occasionally.
Functions:
? Logs threadviews to a temporary table, and periodically commits data to the thread table.
? Frequency of writes controlled by server load and a random number.
? Includes optional email notification.
vB version tested on: 2.2.6
PHP version tested on: 4.2.0
Acknowledgements: Thanks to MattR for the "store threadviews as list of non-unique ids and use count(*)" suggestion that removed 1 query per pageview.



The updating of threadviews has been identified as a main cause of high server loads. This hack basically writes threadviews to a HEAP table (i.e. temporary table residing in memory) and will only update the thread count when the load is below a specified value and a random # value is hit.

(The impetus for this hack was an email from my host: They threatened to boot me off my server due to CPU usage "abuse", so as part of a larger effort, I decided to write this hack. *crossing fingers*)

SUMMARY of changes:
1) Set 1 Admin CP option
2) Create MySQL table
3) Insert code into SHOWTHREAD.PHP

The code will optionally send you an email every time the threadviews are committed to the thread table.

More details are in the text file.


This is what the email you'll get looks like:

2395 Total Views (original number of write queries)
170 Threads (new number of write queries)

93% of Writes Avoided [2225 avoided]
14:1 Original Writes vs. Consolidated Writes

Version History:

0.9 - Initial Release
0.91 - removed need for reading of temp table before writing to it (thanks to MattR). Note that if you've been using 0.9, you must re-create the my_threadviews table as it's structure has changed.
0.92 - added reporting of the ratio of Original Writes vs. Consolidated Writes. Use this to help in tweaking the frequency of updates. If the ratio is low, (e.g. 2:1, 5:1, etc) it probably means updates are occuring too often. (The ratio will also be low if you have many separate threads that are viewed a small number of times.)
1.00 - No code changes. Changed version to 1.00 since it's been stable for the past 2 months.

[Tim Wheatley]

I've decided to keep this hack installed. But have not decided to keep the original settings.

I have removed the line which tells it to work under a certain server usage. And have just got it running so it moves them to the database periodically.

I've attached a text file which is a downloaded post from my admin forum on my forums, this contains over 50 e-mails sent by this hack.

I have now commented out the e-mail lines because I'm happy with it's performance at these settings.

[FFMania]

Have some problem......when I find this code....

PHP Code:

    if ($noshutdownfunc) {
      $DB_site->query("UPDATE thread SET views=views+1 WHERE threadid='$threadid'");
    } else {
      $shutdownqueries[]="UPDATE LOW_PRIORITY thread SET views=views+1 WHERE threadid='$threadid'";
    } 


I can't fiind it since I have lesane Store hack installed..so the code becomes...

PHP Code:

if ($noshutdownfunc) {
  $DB_site->query("UPDATE thread SET views=views+1 WHERE threadid='$threadid'");
  
  // Store hack by Lesane

  $storeview=$DB_site->query_first("select views, postuserid from thread WHERE threadid='$threadid'");
  $views=$storeview[views];
  $postuserid=$storeview[postuserid];

  $storeadmin = $DB_site->query_first("SELECT * FROM storeadmin");
  $view1=$storeadmin[view1];
  $view2=$storeadmin[view2];
  $view3=$storeadmin[view3];
  $view4=$storeadmin[view4];
  $view5=$storeadmin[view5];
  $viewpoint1=$storeadmin[vpoint1];
  $viewpoint2=$storeadmin[vpoint2];
  $viewpoint3=$storeadmin[vpoint3];
  $viewpoint4=$storeadmin[vpoint4];
  $viewpoint5=$storeadmin[vpoint5];

  if ($views == $view1) 
  {
    $DB_site->query("UPDATE user SET storep=storep+$viewpoint1 where userid='$postuserid'");
  }
  elseif ($views == $view2) 
  {
    $DB_site->query("UPDATE user SET storep=storep+$viewpoint2 where userid='$postuserid'");
  }
  elseif ($views == $view3) 
  {
    $DB_site->query("UPDATE user SET storep=storep+$viewpoint3 where userid='$postuserid'");
  }
  elseif ($views == $view4) 
  {
    $DB_site->query("UPDATE user SET storep=storep+$viewpoint4 where userid='$postuserid'");
  }
  elseif ($views == $view5) 
  {
    $DB_site->query("UPDATE user SET storep=storep+$viewpoint5 where userid='$postuserid'");
  }
  
} else {
  $shutdownqueries[]="UPDATE LOW_PRIORITY thread SET views=views+1 WHERE threadid='$threadid'"; 


Any idea what I should do?

[Courage]

I'm the only one with a problem..

After I installed this hack I got 200 Email errors in ~ 5 min

Database error in vBulletin :

Link-ID == false, connect failed
mysql error: Too many connections

mysql error number: 1040

I'm using Vb 2.2.2 witht A LOT of hacks

[Tim Wheatley]

Update! This is going to be uninstalled from my boards. Last nights busy period was very busy and all went well. Then it got to about 1am when the visitors start to go down and BOOM - the board was killed for 5mins. I kept trying to enter and when I eventually did the server stats said that usage had been:

10.40 7.00

This caused massive problems and also (I believe) all the view counts were lost.

It seems that when it comes to updating the stats it's such a massive undertaking that it kills the server pc.

[Tim Wheatley]

I have reinstalled this.

Now have it functioning correctly.

My 'normal users online' is around 100-150 at once. Peaking maybe at 250 at times. A lot of guests visit and a lot of people read threads and don't post, let me know if anyone wants to see my settings for it...

[Capt PPRuNe]

Errr... can someone shed some light on the reason every time I try to download this hack i get v0.91 instead of v0.92?

having installed it I am getting the emails but without the ratio bit added to v0.92.

Looks like it will be a great hack as i have over 50,000 registered users and at least twice that many guests with an average of 250-350 users on at any one time peaking to about 550 every day.

Dedicated server with average load at around 1.5-2.5 but can peak to about 6 or 7 and ocassionally even more.

[Jujubee]

Originally posted by stilger
Hello.. Im thinking about installing this hack on a site that I support that averages about 400 users logged in at a time. Sometimes peaking to around 2000 (well vbulletin reports the most ever as being 2410 I don't know how much you can trust this though). We currently spread our load over 2 servers using Round Robin DNS and have a separate server for all database functions. Will this hack work on multiple server site? I am thinking because of how this hack works there should be no problem but I wanted to make sure. It sounds like this hack would help with the occasional To Many Connections messages we get..... Thanks for the info.

Doh! I stopped getting notification emails... sorry for the long delay.

I don't see why this wouldn't work with multiple web servers, but I can't be 100% without intimate knowledge of your setup. (that's me covering my ass )

The hack makes cumulative updates to the db, so while the two webservers would update the db at different times, it should still work. You'll probably have to tweak the update frequency to get it to your liking, but otherwise, I don't forsee a problem.

[Jujubee]

Originally posted by FFMania
Have some problem......when I find this code....

PHP Code:

    if ($noshutdownfunc) {
      $DB_site->query("UPDATE thread SET views=views+1 WHERE threadid='$threadid'");
    } else {
      $shutdownqueries[]="UPDATE LOW_PRIORITY thread SET views=views+1 WHERE threadid='$threadid'";
    } 


I can't fiind it since I have lesane Store hack installed..so the code becomes...
<snip>
Any idea what I should do?

I'm not familiar with the Lesane hack, but it looks way more complicated than this hack. Maybe you could ask them...

[stilger]

I just installed this. Tweaking now.

I would like to see your settings....

Originally posted by Tim Wheatley
I have reinstalled this.

Now have it functioning correctly.

My 'normal users online' is around 100-150 at once. Peaking maybe at 250 at times. A lot of guests visit and a lot of people read threads and don't post, let me know if anyone wants to see my settings for it...

[Darth Cow]

Hmmm... thank you. Just installed and should do wonders for our forums .