Large Installation

[shalsaid]

Hello

we have a vbulletin installation that gets about 1.5 million pageviews per day

we use one apache server and two db servers one master (8 cpus 16G RAM)and the other is slave (8 CPUs 16 G RAM)

we use mysql 5

we use this confogration
// If you have multiple database backends, this is the information for your slave
// server. If you are not 100% sure you need to fill in this information,
// do not change any of the values here.
$config['SlaveServer']['servername'] = 'x.y.z.a';
$config['SlaveServer']['port'] = 3306;
$config['SlaveServer']['username'] = 'xxxxxxx';
$config['SlaveServer']['password'] = 'yyyyyyy';
$config['SlaveServer']['usepconnect'] = 0;

we have a problem that there is lag between the master and slave and sometimes locks on peak hours, which cause the server to be down

how can we scale the DB? if we use another slave i belive the locks will be worse since master need to synch two slaves.

I read that some people use memcache we installed a memcache server and configured the forum to connect to it
// ****** DATASTORE CACHE CONFIGURATION *****
// Here you can configure different methods for caching datastore items.
// vB_Datastore_Filecache - for using a cache file
// $config['Datastore']['class'] = 'vB_Datastore_Filecache';
// vB_Datastore_Memcached - for using a Memcache server
// It is also necessary to specify the hostname or IP address and the port the server is listening on
$config['Datastore']['class'] = 'vB_Datastore_Memcached';
$config['Misc']['memcacheserver'] = '10.0.0.1';
$config['Misc']['memcacheport'] = '11211';

we noticed that it just conume 60M of memcache only. how can we store more data in the datastore? can we store other stuff in the memcache

since i read that somepeople have two server of memcache. so there is a way to cahce more data

your help is appreciated

thanks
shadi

[snakes1100]

You are aware that vbulletin only dumps off the search to the slave correct?

Have you tried using sphinx search to cut down on the search resource requirement?

You didnt provide to much data about how many posts etc are on the site, online users, how many searches are going on etc.

[shalsaid]

Thanks for quick reply
we have about 20000 online users at peak

18 million post

we used sphinx, But alot of users complain about wrong search results, since out forum is non english forum

we Intel(R) Xeon(R) CPU 2.13GHz 4 cpus and 8 G RAM on master and same specs for slave.

now we removed the sphnix

[snakes1100]

Unfortunately, without knowing all the specs of how every thing is already setup, i can only suggest that you check customizing vbulletin, there are a couple firms doing custom work in redesigning vbulletin and a few of its functions.

I forget the link, but there is a thread or two at vbulletin.com talking about it.

Im sure you have things like xcache running, an optimized my.cnf etc...

[shalsaid]

Yes we use xcache on apache, as i told you the apache is ok
please find the info attached

[shalsaid]

what about memcache? how can we increase the data stored in it to reduce the hits on DB?

[snakes1100]

There are no other default options in vbulletin to store other data, that would be custom as well.

[y2ksw]

The database settings are very well tuned and would say they are not a direct cause of locking up. Some vBulletin settings may make the difference, such as similar threads search, viewing users in forums and discussions, immediate view / visit update, some of the scheduled tasks, and of course some vBulletin products/plugins.

The largest board I've been looking after was 12 million posts, and reducing the extra features down to the bone helped to increase the speed noticeably, which also has a positive side effect on database operations. However that server had only a single master database, so I can't tell...

[shalsaid]

Thanks Giovanni

so what do you suggest? what should we do?

have two masters and two slaves would solve the problem?

what settings should we change?

i read some one has two servers for memcache, how can i store more data in memcache?

thanks

[y2ksw]

If possible I'd use as less database servers as possible and rather would try to get a true beast ;) with best performance. For example, a large RAM or Solid State disk for temporary tables and eAccelerator data storage would push the time critical procedures wide over the top, and because they are clean after booting, they never get too filled. You also could use a Solid State disk for the operating system and just a SATA storage disk for the data. The large Solid State disks are still very expensive, so I don't believe they would be a good investment right now.

For power servers the number of processors is not so much a question, but L1, L2 cache and RAM sizes are, and also the FSB (Front-side Bus) speed. Also the processors should not be overclocked and have a good speed margin. This particular avoids overheating and shutting down unadvertedly.

Also the networking should be as fast as possible, 1000 MBits LAN should be the one. Since a LAN is only as fast as the slowest link, make sure your servers are connected by high speed and eventually have a second network card to connect to the provider, if they run at 100 MBit or less.

If possible load your operating system fully into memory and disallow disk swapping for the Kernel. Disk swapping can make all operation very slow, and may happen at any moment, often at the very wrong time. It happens, when the system is already low on memory and if it swaps the database or other essential services on disk, the system seems to hang, although it isn't, it is just executing very, very slow. However if your temporary space is on RAM / Solid State disks, this problem is much less evident, because it still runs in fast memory.

Finally, the disk size and daily disk maintenance is essential. If your disks are constantly or daily defragmented, and the used space is about 10% of the total disk space, you boost your overall performance, because new entries can be stored without searching a free slot. Defragmenting on the other side boosts the read time, because data can be read in one block.

Last not least, check the best way how to format the data disks. If the interleave is wrong, it may take more time to read and write data. Also make sure not to connect optical (DVD/CD) drives on the same data bus as hard disks (critical to EIDE if used). Best would be not to have any optical or floppy drive mounted.

I think I haven't forgot anything, have a nice evening ;)

[snakes1100]

I'll touch on the drives, but you never did tell us all the server specs, this info is required as we dont know what all your running.

If you want top performance from the DB server, they should be in a Raid 10 Array, i would use SAS drives, that are 15k rpm, SCSI drives have the performance as well, but SAS drives will out perform them.

[shalsaid]

Yes we use SAS 15K

Seagate \ 300GB:SAS:15K RPM Cheetah \ ST3300655SS

The CPU is
Intel \ 2.13 GHz 1066FSB - Kentsfield \ Xeon 3210 (Quad Core)

RAM is 8G

we dont use RAID

On both servers

[snakes1100]

Is it Raid 10?

[shalsaid]

we dont use RAID

[snakes1100]

Well, thats a issue there, mysql loves Raid 10.

[y2ksw]

I'd say the disk is a little too small in order to have rapid access for 18 million posts + attachments. I estimate the database size about 130 GB, which is a very optimistic size (likely to be more), and about 50 GB of attachments. This would make a 300GB disk at 2/3 full and thus becomes already unefficient.

The suggested RAID configuration would give 2x5 disks of which 1x5 disks are serial and the other 1x5 array the backup (parallel). At 1.5 TB (at 300 GB each) the used space is about 2/15 or 14% which is just fine.

It may also work with a single drive, but please make sure to have daily or progressive backups. A disk failure may become a major issue if backups miss or lack quality, and data recovery from broken disks is very, very expensive ($10k+).

[shalsaid]

what if I use enterprise mysql would it help?

http://www.mysql.com/products/enterprise/server.html

Federated Storage Engine to create a single logical database from many physical servers

would that solve our problem?

thanks for your help

[y2ksw]

Inform yourself first (at mysql.com) ... it may solve the problems partly, but generally seen it makes no difference how your database is divided. It would be really best not to split the database at all and get a really good server, if performance is the main goal.