Go Back   vb.org Archive > vBulletin Article Depository > Read An Article > General Articles
Reload this Page Using Memcached to optimize vB hacks
FAQ Members List Calendar Search Today's Posts Mark Forums Read

Reply
Page 1 of 3 1 23
 
Thread Tools
Using Memcached to optimize vB hacks
MoMan
Join Date: Oct 2005
Posts: 301

 

USA
Show Printable Version Email this Page Subscription
MoMan MoMan is offline 08-12-2010, 10:00 PM
As we all know, there are lots of great hacks here at vB.org, but some of them are certainly not written with large forums in mind. On my site, www.pentaxforums.com, which averages 1,200-2,000 simultaneous members, I would love to install every useful hack that I come across, but I can't really afford to have silly statistics and gimmicks add global queries to the database.

Many hacks have one or more of these issues:
-Add global database queries
-Use slow/redundant database queries
-Repetitively perform strenuous computations

I've therefore turned to Memcached to cache frequently-updated yet non-critical data in order to save queries and increase page generation time. I've applied this to the following hacks, just to name a few:
-Top poster on forum home (5 minute caching)
-Forumhome social group stats (5 minute caching)
-Moderated posts / subscribed threads in notifications (5 minute caching)
-Cyb advanced new posts (5 minute caching)

In addition, I use this for:
-Fully caching the current forum activity in index.php
-Fully caching the output of showgroups.php

All in all, I've saved a total of 5 queries on forumhome and 3 global queries by applying the cache, and also significantly reduced page generation time, which is the real biggie. For example, without caching, my forumhome would take about 0.30 seconds to generate. With caching, however, the generation time falls to about 0.07 seconds - that's over 4x faster!

I'd like to share with you the code I wrote to accomplish the caching. It assumes you have Memcached installed on your server. This particular code is for the forumhome top poster hack. As you can see, it's quite simple in structure, and can therefore easily be adapted to other hacks as well.

The general pseudocode is:

Code:
connect to cache
get data from cache
if data expired:
 get data from database
 update cache
PHP Code:
// Cache data for a certain time period to reduce queries!
    $cache = array();
    $memcache = new Memcache;
    $memcache->connect('MEMCACHED SERVER IP GOES HERE', 'MEMCACHED PORT GOES HERE');

    // Set for how long to cache, in seconds
    $cache['limit'] = 600;

    // The keys used for caching
    $cache['datafile'] = 'some_string';

    // Check cache state
    $data = $memcache->get($cache['datafile']);

    if ($data === false)
    {
        // Begin main add-on code
        $postcount = $db->query_first("
            SELECT COUNT(*) AS count
            FROM " . TABLE_PREFIX . "moderation AS moderation
            INNER JOIN " . TABLE_PREFIX . "post AS post ON (post.postid = moderation.primaryid)
            WHERE moderation.type = 'reply'
        ");
        
        $threadcount = $db->query_first("
            SELECT COUNT(*) AS count
            FROM " . TABLE_PREFIX . "moderation AS moderation
            INNER JOIN " . TABLE_PREFIX . "thread AS thread ON (thread.threadid = moderation.primaryid)
            WHERE moderation.type = 'thread'
        ");
        // End main add-on code

        // Save the data we just fetched to the cache
        $memcache->set($cache['datafile'],array($postcount['count'],$threadcount['count']),MEMCACHE_COMPRESSED,$cache['limit']);
        
    }
    else
    {
        // Use data from the cache
        list($postcount['count'],$threadcount['count']) = $data; 
    }
    
    // Code that uses the data
    $vbulletin->userinfo['poststomoderate'] = $postcount['count'];
            $notifications['poststomoderate'] = array(
                'phrase' => $vbphrase['posts_awaiting_moderation'],
                'link'   => 'http://www.pentaxforums.com/forums/moderation.php?do=viewposts&type=moderated' . $vbulletin->session->vars['sessionurl_q'],
                'order'  => 10
            );
    $vbulletin->userinfo['threadstomoderate'] = $threadcount['count'];
            $notifications['threadstomoderate'] = array(
                'phrase' => $vbphrase['threads_awaiting_moderation'],
                'link'   => 'http://www.pentaxforums.com/forums/moderation.php?do=viewthreads&type=moderated' . $vbulletin->session->vars['sessionurl_q'],
                'order'  => 10
            ); 

            // Close the connection
            $memcache->close(); 
Here, $postcount['count'] and $threadcount['count'] is the data from the queries which ends up being cached. The nice thing is that even if the data is fetched from the cache and not the database, it can be accessed through the same variable. This is because you can store anything in memcache- even templates that have already been evaluated.

If you currently don't have memcached installed on your server, it's quite easy to install using PECL. To check if you have it installed, upload a script to your server that contains the following code:
PHP Code:
die(class_exists('Memcache')); 
A few notes:
  • it's good practice to add vbulletin options for the cache timeouts so you can keep things centralized, and $vbulletin->config for the memcached servers
  • you can substitute time() calls with the constant TIMENOW if you're inside vbulletin
  • only use my method for code with global queries or intensive computation! A trivial mysql query is usually faster than connecting to memcache and reading from it. Try running some basic benchmarks using microtime() to find out if caching is worth it.
  • Try to keep the individual data fragments you cache as small as possible. The smaller the data, the faster it can be fetched.

So, in conclusion, if used properly, this code can speed up your forum tremendously. If you have a big board, try giving it a spin! Also, if you're interested in reducing your server's memory load, look into installing APC for PHP. Note that on a production environment, it would be better to have a global memcache connection instead of initializing it every time you try fetching data.

I hope you guys found this useful!
Reply With Quote
  #2  
Old 08-28-2010, 07:21 PM
ssslippy ssslippy is offline
 
Join Date: Jan 2006
Posts: 877
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
How hard is this to adapt to xcache. Running only 2 servers and no reason to run memcache yet.
Reply With Quote
  #3  
Old 08-28-2010, 09:18 PM
MoMan MoMan is offline
 
Join Date: Oct 2005
Location: USA
Posts: 301
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I found the xcache API here: http://xcache.lighttpd.net/wiki/XcacheApi

So you'd basically want to change the memcache set and get calls to xcache_set and xcache_get
Reply With Quote
  #4  
Old 09-01-2010, 09:56 PM
monkeyboy1916's Avatar
monkeyboy1916 monkeyboy1916 is offline
 
Join Date: Nov 2006
Posts: 59
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I'd love to see a method working with xCache. I'm not able to at the moment but will try when I can, so if someone figures out this method, please post it for others to make use of~

Great idea btw MoMan.
Reply With Quote
  #5  
Old 09-03-2010, 06:31 PM
abdobasha2004's Avatar
abdobasha2004 abdobasha2004 is offline
 
Join Date: Aug 2008
Posts: 541
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
great
thanks
Reply With Quote
  #6  
Old 09-06-2010, 01:06 AM
ssslippy ssslippy is offline
 
Join Date: Jan 2006
Posts: 877
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Would it be possible to see the code you used for the showgroups page?
Reply With Quote
  #7  
Old 09-11-2010, 02:40 PM
MoMan MoMan is offline
 
Join Date: Oct 2005
Location: USA
Posts: 301
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I'm not going to post the whole thing as it's site specific and would likely break things on others' sites, but the general steps I followed are (note that there are other approaches as well, such as caching the entire query, but I figured it would be best performance-wise to just cache everything):

1. Strip out the navbar/header/footer from the SHOWGROUPS template
2. In showgroups.php, cache the evaled output of SHOWGROUPS in a variable called $HTML using the code model at the top of this thread
3. At the end of the file, print the output using GENERIC_SHELL. This will prevent caching of the navbar/header/footer, so that they are always current.

Note that if you use multiple styles that don't share the same graphics, you either have to set $cache['datafile'] = 'something' . $styleid, or use a global replace for image folder paths.
Reply With Quote
  #8  
Old 09-15-2010, 05:22 AM
Shabcool Shabcool is offline
 
Join Date: Jul 2008
Posts: 9
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
thanks
Reply With Quote
  #9  
Old 09-15-2010, 02:41 PM
compwhizii compwhizii is offline
 
Join Date: Aug 2009
Posts: 44
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
You should make of the Datastore system: http://members.vbulletin.com/api/vBu...Datastore.html
Reply With Quote
  #10  
Old 09-18-2010, 04:15 AM
MoMan MoMan is offline
 
Join Date: Oct 2005
Location: USA
Posts: 301
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
I've found that going through the datastore is much slower than this on-the-fly approach (which is still fine from a design standpoint as long as you use $vbulletin->config for the server info and an option setting for the caching time), since memcached isn't good at dealing with large amounts of data.
Reply With Quote
Reply
Page 1 of 3 1 23

« Previous Thread | Next Thread »
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 08:49 AM.

Contact Us - Archive - Top

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.04572 seconds
  • Memory Usage 2,313KB
  • Queries Executed 23 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)bbcode_code
  • (2)bbcode_php
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)modsystem_article
  • (1)navbar
  • (4)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (2)pagenav_pagelink
  • (10)post_thanks_box
  • (3)post_thanks_box_bit
  • (10)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (1)post_thanks_postbit
  • (10)post_thanks_postbit_info
  • (9)postbit
  • (10)postbit_onlinestatus
  • (10)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 
Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • fetch_musername
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • post_thanks_function_fetch_thanks_bit_start
  • post_thanks_function_show_thanks_date_start
  • post_thanks_function_show_thanks_date_end
  • post_thanks_function_fetch_thanks_bit_end
  • post_thanks_function_fetch_post_thanks_template_start
  • post_thanks_function_fetch_post_thanks_template_end
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete