vb.org Archive - Using Memcached to optimize vB hacks

As we all know, there are lots of great hacks here at vB.org, but some of them are certainly not written with large forums in mind. On my site, www.pentaxforums.com, which averages 1,200-2,000 simultaneous members, I would love to install every useful hack that I come across, but I can't really afford to have silly statistics and gimmicks add global queries to the database.

Many hacks have one or more of these issues:
-Add global database queries
-Use slow/redundant database queries
-Repetitively perform strenuous computations

I've therefore turned to Memcached to cache frequently-updated yet non-critical data in order to save queries and increase page generation time. I've applied this to the following hacks, just to name a few:
-Top poster on forum home (5 minute caching)
-Forumhome social group stats (5 minute caching)
-Moderated posts / subscribed threads in notifications (5 minute caching)
-Cyb advanced new posts (5 minute caching)

In addition, I use this for:
-Fully caching the current forum activity in index.php
-Fully caching the output of showgroups.php

All in all, I've saved a total of 5 queries on forumhome and 3 global queries by applying the cache, and also significantly reduced page generation time, which is the real biggie. For example, without caching, my forumhome would take about 0.30 seconds to generate. With caching, however, the generation time falls to about 0.07 seconds - that's over 4x faster!

I'd like to share with you the code I wrote to accomplish the caching. It assumes you have Memcached installed on your server. This particular code is for the forumhome top poster hack. As you can see, it's quite simple in structure, and can therefore easily be adapted to other hacks as well.

The general pseudocode is:

Code:

connect to cache

get data from cache

if data expired:

 get data from database

 update cache

PHP Code:


		
			
// Cache data for a certain time period to reduce queries!

    $cache = array();

    $memcache = new Memcache;

    $memcache->connect('MEMCACHED SERVER IP GOES HERE', 'MEMCACHED PORT GOES HERE');



    // Set for how long to cache, in seconds

    $cache['limit'] = 600;



    // The keys used for caching

    $cache['datafile'] = 'some_string';



    // Check cache state

    $data = $memcache->get($cache['datafile']);



    if ($data === false)

    {

        // Begin main add-on code

        $postcount = $db->query_first("

            SELECT COUNT(*) AS count

            FROM " . TABLE_PREFIX . "moderation AS moderation

            INNER JOIN " . TABLE_PREFIX . "post AS post ON (post.postid = moderation.primaryid)

            WHERE moderation.type = 'reply'

        ");

        

        $threadcount = $db->query_first("

            SELECT COUNT(*) AS count

            FROM " . TABLE_PREFIX . "moderation AS moderation

            INNER JOIN " . TABLE_PREFIX . "thread AS thread ON (thread.threadid = moderation.primaryid)

            WHERE moderation.type = 'thread'

        ");

        // End main add-on code



        // Save the data we just fetched to the cache

        $memcache->set($cache['datafile'],array($postcount['count'],$threadcount['count']),MEMCACHE_COMPRESSED,$cache['limit']);

        

    }

    else

    {

        // Use data from the cache

        list($postcount['count'],$threadcount['count']) = $data; 

    }

    

    // Code that uses the data

    $vbulletin->userinfo['poststomoderate'] = $postcount['count'];

            $notifications['poststomoderate'] = array(

                'phrase' => $vbphrase['posts_awaiting_moderation'],

                'link'   => 'http://www.pentaxforums.com/forums/moderation.php?do=viewposts&type=moderated' . $vbulletin->session->vars['sessionurl_q'],

                'order'  => 10

            );

    $vbulletin->userinfo['threadstomoderate'] = $threadcount['count'];

            $notifications['threadstomoderate'] = array(

                'phrase' => $vbphrase['threads_awaiting_moderation'],

                'link'   => 'http://www.pentaxforums.com/forums/moderation.php?do=viewthreads&type=moderated' . $vbulletin->session->vars['sessionurl_q'],

                'order'  => 10

            ); 



            // Close the connection

            $memcache->close();

Here, $postcount['count'] and $threadcount['count'] is the data from the queries which ends up being cached. The nice thing is that even if the data is fetched from the cache and not the database, it can be accessed through the same variable. This is because you can store anything in memcache- even templates that have already been evaluated.

If you currently don't have memcached installed on your server, it's quite easy to install using PECL. To check if you have it installed, upload a script to your server that contains the following code:

PHP Code:


		
			
die(class_exists('Memcache'));

A few notes:

it's good practice to add vbulletin options for the cache timeouts so you can keep things centralized, and $vbulletin->config for the memcached servers
you can substitute time() calls with the constant TIMENOW if you're inside vbulletin
only use my method for code with global queries or intensive computation! A trivial mysql query is usually faster than connecting to memcache and reading from it. Try running some basic benchmarks using microtime() to find out if caching is worth it.
Try to keep the individual data fragments you cache as small as possible. The smaller the data, the faster it can be fetched.

So, in conclusion, if used properly, this code can speed up your forum tremendously. If you have a big board, try giving it a spin! Also, if you're interested in reducing your server's memory load, look into installing APC for PHP. Note that on a production environment, it would be better to have a global memcache connection instead of initializing it every time you try fetching data.

I hope you guys found this useful!

X vBulletin 3.8.12 by vBS Debug Information
Page Generation 0.01228 seconds Memory Usage 1,771KB Queries Executed 10 (?)
More Information
Template Usage: (1)ad_footer_end (1)ad_footer_start (1)ad_header_end (1)ad_header_logo (1)ad_navbar_below (1)bbcode_code_printable (2)bbcode_php_printable (1)footer (1)gobutton (1)header (1)headinclude (6)option (1)pagenav (1)pagenav_curpage (2)pagenav_pagelink (1)post_thanks_navbar_search (1)printthread (10)printthreadbit (1)spacer_close (1)spacer_open Phrase Groups Available: global postbit showthread	Included Files: ./printthread.php ./global.php ./includes/init.php ./includes/class_core.php ./includes/config.php ./includes/functions.php ./includes/class_hook.php ./includes/modsystem_functions.php ./includes/class_bbcode_alt.php ./includes/class_bbcode.php ./includes/functions_bigthree.php Hooks Called: init_startup init_startup_session_setup_start init_startup_session_setup_complete cache_permissions fetch_threadinfo_query fetch_threadinfo fetch_foruminfo style_fetch cache_templates global_start parse_templates global_setup_complete printthread_start pagenav_page pagenav_complete bbcode_fetch_tags bbcode_create bbcode_parse_start bbcode_parse_complete_precache bbcode_parse_complete printthread_post printthread_complete
Messages: