Dynamically Linked Source - Speed Up Your Multiple Site Network & Upgrades

[Jafo232]

For the purposes of this documentation, "Network" refers to a group of forums on a single server or cluster of servers.

When 3.7.1 PL2 was released, I posted over at vbulletin.com (http://www.vbulletin.com/forum/showthread.php?t=275894) lamenting the fact that I had 15 upgrades to perform and how it would be nice to have a multiple site license to lighten the burden of downloading each install separately. In the end, I spent much more time coming up with this system.

Dynamically Linked Source or DLS allows operators of two or more forums on the same server or cluster, for all intents and purposes, to use one set of source code instead of multiple sets. By using one set of source code, Op code caching is much more efficient as only one copy of the source is cached. This greatly reduces the amount of memory used caching the source code, and also makes it more efficient. It will lead to less fragmentation for systems using APC (http://www.php.net/apc), XCache (http://xcache.lighttpd.net/), eAccelerator (http://eaccelerator.net/), or other such methods of caching Op code.

The method uses symbolic links to a central repository of the source code. Therefore, upgrades to all sites on the network can be done quickly by just replacing the source. For patch level releases, this means the entire network can be patched with one upload. Upgrades to a new version can be done this way, or a new repository can be created and each site can be upgraded one at a time.

The issue of licensing has also been addressed. Once the product file is installed, and assuming the installation instructions were followed, each installation will report the proper license key when salting cookies, or reporting back to Jelsoft as Vbulletin does from time to time (mostly in the AdminCP).

I contacted the development team at Jelsoft and discussed this method before releasing it, and got the OK from Mike. I was initially concerned that it may break licensing or perhaps give away too much of the license reporting methodology, but now that I look back, it really does not do either.

In tests on a rather large network that contains seven forums with about 7 million posts, hundreds of thousands of registered users, and uses three load balanced servers the load has been reduced by almost 40%. You can see in the graph attached to this thread for the particulars.

I have tried to make the readme as clear as possible, but honestly that was never one of my strong points. I cannot emphasize more that before you implement this method, TEST it first in your sandbox. Also, make sure you backup your software and database before you use this.

I also have included a PDF that explains what is happening in greater detail.

Updates:

8/20/08 - This will work with the latest version of VBSEO (3.2.0 at this time). Add all the files to the DLS except the config file, and run the php-clone.php script.

This product was developed for Andy R (https://vborg.vbsupport.ru/member.php?u=7590) from Social Knowledge (http://www.socialknowledge.net/) and released here for others to enjoy.


Please don't forget to click install (https://vborg.vbsupport.ru/vborg_miscactions.php?do=installhack&threadid=188108) as I have worked quite a while on this. :)

[Jafo232]

Reserved...

[Zacarias]

This is very interesting. I'll be testing this one.

[GrendelKhan{TSU]

awesome. exactly what I need. will test. :D thank you!!


ps. er... how do I sandbox test this for multiple boards? do I ahve to set up a multi-board test as well? :(

[nirvana43]

Installed and rated! ;) thanks a million man.

[Jafo232]

awesome. exactly what I need. will test. :D thank you!!


ps. er... how do I sandbox test this for multiple boards? do I ahve to set up a multi-board test as well? :(

Well, you don't HAVE to setup multiple boards in the sandbox, but if you want to test how it works with multiple boards (which is the whole point) then yes. But if you want to see how it works, one will be enough. Once you get it working on one site, you should not have problems getting it to work on multiple sites.

Just remember, BACK UP EVERYTHING! The php-clone.php script DELETES your forum source code files and then replaces them with symbolic links. Should you configure it wrong, you could find yourself with no forum software. :) A quick backup of the source files will correct that quickly.

I also recommend backing up the database, not that this product/method does anything to the database, it is just for prudence sake...

[AzzidReign]

Wow, this is a great idea. I'm going to check this out once I get some free time.

[Reeve of shinra]

This looks great and its going to be a blessing for those running multiple forums. I assume each sites keeps its own DB? How are mods and plugins handled? I assume they are excluded?

[Jafo232]

This looks great and its going to be a blessing for those running multiple forums. I assume each sites keeps its own DB? How are mods and plugins handled? I assume they are excluded?

Yes, each site still keeps its own DB, and mods/plugins are handled the same way as they always were or you can include them in the DLS and make them available to the entire network.

You can link any file just by adding it to the DLS source and running the php-clone.php script.

[djr]

*VERY* interesting and I love the out-of-the-box thinking. Truly impressed. Do you know whether this will impact vbSEO? Would it be possible to also include vbSEO in the DLS or isn't that necessary?

[Jafo232]

The sites we tested on were using VBSEO and VBACMPS with no affect. You just have to make sure that you have the latest version of VBSEO..

I will be working on including VBSEO into the DLS. Initial work on it has yet to be of any success..

[djr]

The sites we tested on were using VBSEO and VBACMPS with no affect. You just have to make sure that you have the latest version of VBSEO..

I will be working on including VBSEO into the DLS. Initial work on it has yet to be of any success..Thanks Jafo! Again, much much respect for this idea. Now off to discuss with my fellow webmaster :cool:

[TheInsaneManiac]

Wait does this mean I only need one VBulletin license for all my domains now or no?

[briansol]

not at all.. you still need your proper licenses.

[Jafo232]

Wait does this mean I only need one VBulletin license for all my domains now or no?

No, you still need all the license #'s. If you read the readme, you will see where you insert the license number(s) (NOT THE CUSTOMER #).

[Jafo232]

Hmm, I got a notification saying someone asked "any chance for memcached support?" but it isn't here. Well, the answer is, it does support memcached.

[gigaenvy]

This is a premium add-on. Great effort and taking things into your own hands as many of us do.

[Reeve of shinra]

Hmm, I got a notification saying someone asked "any chance for memcached support?" but it isn't here. Well, the answer is, it does support memcached.

Off topic but yes, I hit 'new posts' saw a reply on 'page 2'. I clicked on page 1 in new window, read it, click on page 2 and it took me back to page 1. The post was probably deleted but I can't imagine why if thats all it said.

[Jafo232]

I think they did some sort of database restore, because I had a PM I know I read already, yet it showed as new again..

[Hornstar]

wow you have really gone and done a great job here. i currently have 2 forums running on my server, however as I expand I will really have to go and look into this more.

[Jafo232]

wow you have really gone and done a great job here. i currently have 2 forums running on my server, however as I expand I will really have to go and look into this more.

Even with just 2 forums, by installing this mod, you will cut your memory usage in half (providing your using an opcode cache like APC, Xcache, or eAcc)..

[Jafo232]

I do want to take an opportunity to clarify something I think may be unclear.

When you install this, you define a license key for each site using the DLS system by entering in the license #. This license # that you enter will be used throughout the script for the site it is defined for. This in no way bypasses any licensing methodology in the script. If you enter an invalid license # or a license for a different site than what you defined at vbulletin.com, it will still throw up the red flags for Jelsoft.

The only thing the plugin does is make sure the proper license (as you define it) is returned when called. We contacted the VB team about this and they detailed to us where these areas were so we could build this system without stepping on any licensing issues when it came to their piracy protection schemes.

If you try to use this to have multiple copies of the same license running, it will still be identified exactly the same way it would if they were separate installs.

[y2ksw]

I wonder if this would work on Windows servers ... I suppose not ... (sigh) ;)

[Rebecca217]

This is brilliant. Thank you! :)

[Jafo232]

I wonder if this would work on Windows servers ... I suppose not ... (sigh) ;)

Probably not, at least out of the box. I suppose you could use shortcuts instead of symbolic links and then it would work.

[Jafo232]

*VERY* interesting and I love the out-of-the-box thinking. Truly impressed. Do you know whether this will impact vbSEO? Would it be possible to also include vbSEO in the DLS or isn't that necessary?

We finally got this working with VBSEO... If you upload all the files for vbSEO to the DLS except for the config file, run the php-clone.php script, you should be all set. Really speeds things up!

[KevinL]

This really sounds great!

[Jafo232]

This really sounds great!

Great! Don't forget to mark installed if you find it useful. :)

[Beermonster]

Where do you put the path to the forum in? there is one line in php-clone.php that says Path to the forum not forums, which path is this?

Edit: and also how are you able to upgrade the databases or is it just files this updates?

[y2ksw]

Probably not, at least out of the box. I suppose you could use shortcuts instead of symbolic links and then it would work.
I've tried it. Shortcuts won't work. So that's a Linux-only solution ;)

[Jafo232]

Where do you put the path to the forum in? there is one line in php-clone.php that says Path to the forum not forums, which path is this?

Edit: and also how are you able to upgrade the databases or is it just files this updates?

$oldls = './public_html/forums';

you can use relative or absolute when it comes to the path i.e."

$oldls = '/some/pathto/public_html/forums';

This does nothing to the databases.

[AWS]

I've tried it. Shortcuts won't work. So that's a Linux-only solution ;)
What kind of problems are you having with shortcuts?. I use shortcuts to serve up files and resources all the time. Might be a little config tweaking needed.

[GoTTi]

great mod. so if i understand whats going on here, this mod will let me upgrade multiple forums on the same server, under different domain names (or accounts on the server), as long as i input the license information properly, it will upgrade all the forums running the software at one time?

seems like a really great mod here. i am old fashioned and enjoy upgrading the forums one at a time, and download each script for my domains individually. not all forums can be upgraded to the same version, due to hacks/addons from here that might be out dated, and forum owners dont wanna loose them...

but i do need a better understanding of this mod. does the server need any pre-req software or modifications to it or is this a out of the box kind of mod because i am not server smart, i understand real basic stuff, and i see things in here like OP code and that throws me for a loop like am i suppose to look for something to make this work right...the link in your PDF for opcode is not working so i cant get information on it.

anyways, if you could answer my question id appreciate it. this mod just lets me upgrade multiple forums that are installed on the server in 1 shot?

thanks and good job, been a while since i seen a mod like this that really benefits the server instead of its users.

[Jafo232]

You can use it to upgrade multiple forums at once if you want to, but I would recommend creating a new source repository and upgrading your forums one at a time just for the reasons you laid out.

Opcode caching is basically this. When a PHP script is executed on your server, the server reads the script, compiles it, and then executes the compiled code (op code). It does this EVERY time the script is called. Of course, there is a lot of overhead compiling the source over and over so caching will only compile it once and then the server will use the compiled code saving you A LOT of memory and CPU.

When you run multiple forums on a server, and you are using opcode caching, each forums file will be cached even though for all intents and purposes, they are the same files. By allowing all the forums to share one set of source files, they are only cached once.

So index.php on forum1 and index.php on forum2 are only cached one time when you use the DLS as opposed to twice. This cuts the memory usage for cached files in half.

If you are not comfortable with your server prowess, you may want to test this on some test installs before you try to do it live. ;)

[y2ksw]

What kind of problems are you having with shortcuts?. I use shortcuts to serve up files and resources all the time. Might be a little config tweaking needed.
I'm using Apache on Windows 2003 Servers. When using a shortcut (actually terminates with a .php.lnk buth shows .php) Apache returns a 404 page not found error. I tried this with the sendmessage.php file in the same domain, but different folder, just to see if it does anything besides a page not found error :)

It may work with IIS, but I won't use IIS for other reasons ;)

[Jafo232]

Hmm, I know very little about Windows servers, but could you not associate the ink file to run as PHP?

[y2ksw]

Hmm, I know very little about Windows servers, but could you not associate the ink file to run as PHP?
:) I tried that and it failed because PHP attempts to read directly the link file.

If I run the file first (then it's in the cache), the file link works. However after restarting Apache the cache is empty and it would fail with a 404 error.

Ok nevermind, it's a great solution for Linux, and there it'll surely apply :)

[sockwater]

You can use hard links on windows and it works (just tested). A hard link basically means that if you change either file, the other one will also be changed. You can create a hardlink on windows like this:
fsutil hardlink create newfile.php oldfile.phpAlso you might want to try using junctions (http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), which are basically directory symlinks. I don't know if this would work under apache or not.

EDIT: If you try using hardlinks, I'd like to know if the opcode caching sees them as the same file or separate files.

[Jafo232]

That would be ideal. Otherwise it is only good for patch releases.

[y2ksw]

You can use hard links on windows and it works (just tested). A hard link basically means that if you change either file, the other one will also be changed. You can create a hardlink on windows like this:
fsutil hardlink create newfile.php oldfile.phpAlso you might want to try using junctions (http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx), which are basically directory symlinks. I don't know if this would work under apache or not.

EDIT: If you try using hardlinks, I'd like to know if the opcode caching sees them as the same file or separate files.
I'll try that tonight ... and if it works I'll make a program which creates those links recursively for Windows Servers :)

[sockwater]

I just tested with junctions as well. Apache reads them just fine. Use this command, after downloading junction.exe from Microsoft (http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx).

junction -s newvbdir oldvbdirnewvbdir will now be a symlink to oldvbdir, including all files and directories recursively. It seems to work the same way as a hard link, if you change the contents of either file it changes the other.

The bottom line is both junctions and hard links should be tested to see if the opcode caching only caches one copy of the file or multiple copies.

[Jafo232]

Don't forget what you have to do with the config files.

[y2ksw]

I just tested with junctions as well. Apache reads them just fine. Use this command, after downloading junction.exe from Microsoft (http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx).

junction -s newvbdir oldvbdirnewvbdir will now be a symlink to oldvbdir, including all files and directories recursively. It seems to work the same way as a hard link, if you change the contents of either file it changes the other.

The bottom line is both junctions and hard links should be tested to see if the opcode caching only caches one copy of the file or multiple copies.
I tried this first. It seems as if the cache sees just one copy, thus theorically it would work. Practically however you can't use it for vBulletin because of the configuration file which can't be separated with junctions. junctions would make sense though if you had your board files in a different disk or folder - which may become handy when moving a site.

Now I'll try the fsutil solution.

[Jafo232]

I tried this first. It seems as if the cache sees just one copy, thus theorically it would work. Practically however you can't use it for vBulletin because of the configuration file which can't be separated with junctions. junctions would make sense though if you had your board files in a different disk or folder - which may become handy when moving a site.

Now I'll try the fsutil solution.

Well, you probably can if you follow the directions in this mod. You have to create a configs folder in the includes directory. Place your site configs in there and rename them uniquely. Then use the config file in this mod.

[y2ksw]

I've tried now both systems (junction, fsutil) on a clean test environment. Both systems are not cached once, but as if they were separate files. A clean 3.7.0 installation and copy produce each about 2.55 MB of uncompressed eAccelerator cached files on forum home (index.php).

The junction feature requires to rename the config.php files and is a one-liner.
The fsutil feature replaces the single files with a hardlink to the common files folder, and allows thus to keep some of the files in their original state. Thus, renaming of config.php and other files are not necessary. fsutil requires an utility to execute recursively. Please allow me to test it a little before releasing ;)
Both features require to repeat the junction/fsutil calls on each copy. It is not possible to copy and paste those links, and doing so, will just retrieve the original files.

[Jafo232]

Well, if they are not caching in your windows environment, I don't see much of a point other than speedier upgrades. Getting Linux of course would solve all of this for you. ;) (Couldn't resist)...

[UaECasher]

This mod is great :)

[y2ksw]

This utility automates the use if fsutil on Windows Servers. It has been tested on XP Pro and Windows Server 2003, and comes with the full source for Visual Basic 6.0.

Please try it first on a test environment and make a backup before linking the files.

I will not assume any responsibility for your test and/or production environment! Use on your own risk.

[sockwater]

I've tried now both systems (junction, fsutil) on a clean test environment. Both systems are not cached once, but as if they were separate files. A clean 3.7.0 installation and copy produce each about 2.55 MB of uncompressed eAccelerator cached files on forum home (index.php).
The junction feature requires to rename the config.php files and is a one-liner.
The fsutil feature replaces the single files with a hardlink to the common files folder, and allows thus to keep some of the files in their original state. Thus, renaming of config.php and other files are not necessary. fsutil requires an utility to execute recursively. Please allow me to test it a little before releasing ;)
Both features require to repeat the junction/fsutil calls on each copy. It is not possible to copy and paste those links, and doing so, will just retrieve the original files.
I wonder if APC handles junctions/hardlinks on Windows the same way eAccelerator seems to-- seeing the files as the same or different files. On the other hand, maybe this could be a feature request for eAccelerator/APC, maybe something can be tweaked in their code to make them recognize that hardlinks and junctions actually refer to the same file.

[thompson]

very interesting. will try it.

[Amenadiel]

this seems amazing, I'm just starting to understand its full extents.

By the way, I'm studying the integration of two boards. They are on separate webservers and I plan to have independents database servers for each of them. It seems this mod can help me combine both webservers so both boards can use one logical webserver which is the load balancing between the current ones, but won't help me integrate users and passwords since DBs will still be on separate machines, am I right?

[Jafo232]

Yes, that is correct. This will really do nothing as far as integration goes.

[993ti]

You can link any file just by adding it to the DLS source and running the php-clone.php script.

So if 2 forums have the same products that required uploads, it's best to include those files?

[Jafo232]

Yes, you can add them.

[sdsvtdriver]

When referring to Windows, are we referring to Windows running IIS or Apache (or is it not relevant)? If Apache, would using the alias command work?

Alias "/site1/forums" "C:/htdocs/forums"
Alias "/site2/forums" "C:/htdocs/forums"
Alias "/site3/forums" "C:/htdocs/forums"

Has anyone tried NtFS link? http://sourceforge.net/project/platformdownload.php?group_id=161885

linkd? http://support.microsoft.com/kb/205524

[sockwater]

NTFS Link creates hardlinks and junctions and linkd creates junctions. Both of these have been confirmed to not work. Well, they work but they are cached as if they were separate files.

[Jafo232]

Yeah, I doubt there is a Windows solution that will offer any cache relief. Of course, you could always go Linux, but who wants to start that discussion!

[sockwater]

Yeah, I doubt there is a Windows solution that will offer any cache relief. Of course, you could always go Linux, but who wants to start that discussion!
This might actually be a bug in the caching code that is not recoginzing Windows hardlinks/junctions in the same way it recognizes Linux symlinks. Since I use linux, I won't be pursuing this, but for those who have Windows servers, it might be worthwhile to submit this as a bug/feature request to APC/Xcache or whatever you're using.

[Jafo232]

If it happens with the big three opcode cache methods, I would then think it was something native with Windows. I dunno, not a .dll head here.. :)

[ManagerJosh]

Just wondering, are users/members of each forum independent? Or are they linked?

[Jafo232]

Just wondering, are users/members of each forum independent? Or are they linked?

Independent. This does not share any information between forums. It only uses the same SOURCE code, not data..

[azn_romeo_4u]

So you have to use windows to get this to work? Does it work apache? Or litespeedtech?

[Jafo232]

So you have to use windows to get this to work? Does it work apache? Or litespeedtech?

It does NOT work with Windows.

[sockwater]

If it happens with the big three opcode cache methods, I would then think it was something native with Windows. I dunno, not a .dll head here.. :)
Or all 3 of those opcode caching methods were designed for *nix operating systems.

[lkn]

can you it's for single forum on my VPS?
is it can use for 3.6.x ?
will it speed up 4room?

sorry my bad english

[sockwater]

can you it's for single forum on my VPS?
is it can use for 3.6.x ?
will it speed up 4room?

sorry my bad english
If you're only running one forum, there's no reason to use this.

[Jafo232]

If you're only running one forum, there's no reason to use this.

Precisely!

[BamaStangGuy]

So if all your sites don't use photoplog for example, you would just leave photoplog files in the original forums directory?

Edit: Just went through with it. Doesn't seem to work with 3.8. It doesn't like including the config.php or something. I always get this error: Configuration: includes/config.php exists, but is not in the 3.6+ format. Please convert your config file via the new config.php.new.

If I rename config_me.php (which is in configs folder) and put it in the includes folder as config.php it works. It's a 3.8 config file anyways. Not sure why $config = 0 and it's spitting that error.

I'd be willing to pay you for your time to get this to work with 3.8. I was looking forward to streamlining all my vbulletin installs into one set of files, along with some of my mods. I'm not sure if Andy is moving to 3.8 anytime soon or not, so I don't know what your game plan is with that.

[Jafo232]

I am not having any issue with it on 3.8.. Are you sure you have modified config.php correctly (as per the readme)? That is the only time I have ever seen that error is when config.php is incorrect.

[BamaStangGuy]

Here is what I did:

Uploaded vBulletin into /home/vbcore/ (fresh files from vBulletin.com. 3.8.1)
Created a folder called configs inside the includes which is inside /home/vbcore/
Copied my config.php from /home/evo/public_html/forum/includes/ into /home/vbcore/includes/configs/ and renamed it to config_me.php
Uploaded config.php from your zip file to /home/vbcore/config.php and edited it as follows:

## Change me
require_once(CORE_PATH . 'configs/config_me.php');

Along with changing my license info and putting my domain name in.

I then ran your php script and it created all the links.

I then opened my site up and got that error.

I know the config_me.php is fine, because its the same one I am using on my site right now. Plus, I renamed config_me.php to config.php and then placed it in /home/vbcore/includes/ overwriting your config.php. I then was able to get the site to load fine, using the DLS.

It's just when I use your config.php to include the file do I get that error.

[BamaStangGuy]

I just tried this on another one of my vBulletin installs and got the exact same results. :(

[1up_dave]

Just in case anyone was wondering, I just successfully set this up under 3.8.4, and it runs perfectly fine. I'm still tweaking some things, but yeah, I'm good to go.

[azn_romeo_4u]

I wonder if this will work with my single sign on system. Got 4 forums right now and it's a pain to update all 4 forums even though all of them share the same DB.

They are all on different domains. I'm confused on how this works...but I will be testing it probably tomorrow.

Awesome idea and mod BTW

[Jafo232]

If they are on the same physical server (or cluster) then yes it will work.