How many servers is your big board on?

[Lee]

My two largest forums are on 2 servers each. What is your set up like and with how many servers?

[Paul M]

Currently 1 server - 3GHz HT Xeon with 2GB RAM.

[AdminNation]

Not a Big-Board since we're still building, but call it a precautionary measure. We have a dedicated server which hosts one other small vB forum.

Specs:
P4 3.0 GHz
2MB Cache
64BIT HT 2GB of DDR 400 ram
Two 80GB Western Digital hard drives
Raid 1 (3ware Raid Card)
A 250GB Western Digital back up hard drive.

[BigSoccer Tech.]

4x Web servers: Intel Dual Xeon 2.8 GHz, 2 GB Ram, 2 X 80 GB with Raid 1
3x DB servers: Intel Dual Xeon 2.8 GHz, 8 GB Ram, 3 X 73 GB SCSI U320 10K
1x db server: dual P4 for backups
RPM with Raid 5
2x Load balancers: Dual P4 3.0 ghz, 1gb Ram

Apache/Linux
PHP 4.4.1
MySQL 4.0.24

[Erwin]

More than 5. :)

[The Prohacker]

4x Web servers: Intel Dual Xeon 2.8 GHz, 2 GB Ram, 2 X 80 GB with Raid 1
3x DB servers: Intel Dual Xeon 2.8 GHz, 8 GB Ram, 3 X 73 GB SCSI U320 10K
1x db server: dual P4 for backups
RPM with Raid 5
2x Load balancers: Dual P4 3.0 ghz, 1gb Ram

Apache/Linux
PHP 4.4.1
MySQL 4.0.24


That?s kind of where we are at. We own several large forums and host them across a rather similar setup. I can tell you, the one DB server we have tried RAID10 on is doing great! Its a Dual Xeon EM64T 3.2Ghz with 4Gb RAM and 4 x 146Gb U320 15k RPM drives. We custom compiled our kernel on that server to enable the deadline IO scheduler and it has done wonders. IOwait dropped from 60% to around 9%.

-Mat

[woodysfj40]

switched to duals earlier this year....

Web server:
Pentium 4, 2.4 GHz (single processor)
2 gig RAM
36g 10k SCSI primary drive
36g 10k SCSI backup drive

MySQL server:
Pentium III Xeon, 900MHz (quad processor)
2 gig RAM
four 18g 10k SCSIs (RAID-5)

Apache 2.0.40
PHP 4.4.2
MySQL 4.1.18-std

[Skyline_GT]

only 1 because I don't know how to have set up multi servers for a website..

[Carnage]

last time i looked the server specs were on the line of dual 2.8 Xeons 2 gigs of ram. (Gua)

and my new boards are sharing a server with the game they are forums for which has a spec of dual 3gig Xeons with 3 gig of ram, if anything thou, it'll be the game that forces an upgrade to the server not the forums

[RedWingFan]

One dedicated server for now--7200 members, 1.5million+ posts. We've hit as many as 970 users online and haven't had any performance issues, although a majority of the guest users were Yahoo Slurp at the time. Normal activity is now about 650 members/non-members online during peak hours, still no major issues.

I did get our server configured with 3.25G of memory, which has helped. Well, that and George (aka "eva2000") having helped configure my.cnf to keep it running efficiently. :) I have four tables as InnoDB, which really cleared up the table locking problems we've had. We're running a Celeron 2.8GHz right now, which isn't ideal, but it IS working well for us, and within our budget (fueled 100% by donations).

[drew010]

One dedicated server for now--7200 members, 1.5million+ posts. We've hit as many as 970 users online and haven't had any performance issues, although a majority of the guest users were Yahoo Slurp at the time. Normal activity is now about 650 members/non-members online during peak hours, still no major issues.

I did get our server configured with 3.25G of memory, which has helped. Well, that and George (aka "eva2000") having helped configure my.cnf to keep it running efficiently. :) I have four tables as InnoDB, which really cleared up the table locking problems we've had. We're running a Celeron 2.8GHz right now, which isn't ideal, but it IS working well for us, and within our budget (fueled 100% by donations).

wow , mind sharing some of the my.cnf file? ive been getting deadlocks every few days it seems and i usually only have 50-60 members online at once. and the forum is the only major site on the server with heavy database usage.

either way, im running a P4 2.8 ghz w/ HT, 1024mb ram, 120 sata single drive and it runs smoothly, just get that deadlock once in a while for some reason.

[Andreas]

Web
Dual Xeon 2.4 GhZ
4 GB RAM
2x 80 GB 7.2K IDE HD Non-Raid

DB
Dual Xeon 2.8Ghz
4 GB RAM
4x 36 GB SCSI HD RAID 10

Table Locks are fun ;)
Except that it seems to work fine, though I am a bit concerned about CPU load on the webserver which seems to be too high - 400+ Plugins might be too much.
When this starts to cause problems i might have to go back hacking the files.

[sross]

Is there a performance gain on multiple servers vs. a large multi-processor box? For example are you really better off with two dual proc boxes vs. a quad proc box? I am starting to see quad boxes going for around 450US a month. In a few more years we'll probably have quad proc dual core boxes for under 400US a month so wouldn't that be more than enough for a huge board?

[Lee]

68 votes with 50% of them being only 1 server. Very interesting.

[kmike]

Everyone's concept of the "big board" is different.

[Carnage]

I personally think the one server one of my boards is on isn't sufficient... Its getting rather sluggish

[woodysfj40]

I have four tables as InnoDB, which really cleared up the table locking problems we've had.

curious to which 4 tables you have as InnoDB....

[Paul M]

Those that have multiple web servers, how are you spreading the load ? - simple round robin dns or some other method ?

[The Prohacker]

Those that have multiple web servers, how are you spreading the load ? - simple round robin dns or some other method ?


We use a Cisco CSS to spread the load across our web servers. Initially we used round robin DNS but if we need to drop a single/group of sites from certain web servers we had to manually edit the zone files. We normally serve all of our sites from the full number of web servers except when we have a denial of service attack. We try to make the attack not affect other sites while we work on mitigating it.

[zlos]

LVS solution with 2 load balancers, 3 web machines and 1 strong mysql machine behind.

[kerplunknet]

It doesn't really matter how many servers you have. I could have 20 servers, but they all could be using Pentium 2 processors...

[alexi]

It doesn't really matter how many servers you have. I could have 20 servers, but they all could be using Pentium 2 processors...

This is true... it's more meaningful if you include how much traffic you handle. I'm pushing 2200-2300 users at peak on 2 servers.

[Freezerator]

Web P4 2,8Ghz 2GB ram and 2x 120gb SATA raid 1

DB P4 2,8Ghz 2GB ram and 4x 120gb SATA raid 1
Also a second nic's in both servers and a switch for the db network wich is seperate of the normal network.

[futuredood]

how do you split your database up over servers?

[bang]

4 dual core opteron 280's with 4gigs of RAM each for scripts, email and backup, 2 dual 3.6GHz xeon's with 6 gigs of RAM each for mysql and 1 64 athlon 4000+ with 1 gig of RAM for image serving. plus a hardware load balancer and a firewall

[COBRAws]

how do you split your database up over servers?
I wonder the same. I know how to split forum search into another db like the slave DB, but just as far as that :S
Would be great if some people around here explained or write down a guide or something. Im really interested, besides one of my boards is getting really big (well, not as big as some of the boards here)

[Freezerator]

4 dual core opteron 280's with 4gigs of RAM each for scripts, email and backup, 2 dual 3.6GHz xeon's with 6 gigs of RAM each for mysql and 1 64 athlon 4000+ with 1 gig of RAM for image serving. plus a hardware load balancer and a firewall

Hi,

wich forum is this?

[bang]

it's a private russian board about urban exploration and speleology. laws about trespassing are quite strict in Russia/ex-USSR countries and getting a precautionary shot in the head is not the least of our worries. so information like maps, entrance points, photos and details about our gatherings is best kept private.

community was actually started back in 1988 as a BBS running PCB MetaWorlds. much later we had to go through quite a lot of shit to convert the old db into vBulletin format.

[almukmin]

I'm on Dual Xeon 2.8GHZ with 2GB Ram.

Board has 10k+ users with around 100+ members online at anytime.

3 months old.

[letsjoy]

currrently on 1

[Farjad]

I'm in the process of moving to 2 servers.. but really unsure how to do it.. Anyways currently on 1 server
Dual Xeon 3.2Ghz
2GB Ram
2x73GB SCSI HDD

If anyone can tell me how to seperate the apache & mysql to 2 different servers, i will be very thankful :)

[Dark Zero]

now i have a "simple" server for my forum
p4 2.8 ghz, 1gb ram, 120gb hd with 1.1 millon post, 8k users and average of 60 users online :P

[Troy Roberts]

Over 5.

I have a cluster setup, an image server and a db server.

All of my sites are hosted on them. Serve up over 20GB per second in bandwidth. They seem to be holding up well for the most part. At least it's easy to plug in more power.

[precharge]

4x Web servers: Intel Dual Xeon 2.8 GHz, 2 GB Ram, 2 X 80 GB with Raid 1
3x DB servers: Intel Dual Xeon 2.8 GHz, 8 GB Ram, 3 X 73 GB SCSI U320 10K
1x db server: dual P4 for backups
RPM with Raid 5
2x Load balancers: Dual P4 3.0 ghz, 1gb Ram

Apache/Linux
PHP 4.4.1
MySQL 4.0.24

What do you use for load bancing the db server?

[jason|xoxide]

At this point, we've got 1 server but I'm looking into moving to multiple or to one larger box because we're having real problems with table locking during searches.

The current hardware is Dual 2.8GHz Xeon CPUs, 4GB of RAM and (3) 73GB SCSI drives in RAID5. The biggest board on this box has 3M posts, 370K threads, and 53K members. There are generally 250-300 concurrent users during the day and ~400 in the evenings (default cookie timeout).

I'm considering moving to a Dual, dual-core Opteron 265 box w/ 6GB of RAM and (4) 73GB Drives in RAID5 but I don't know how much that's actually going to help the issues.

[scotty]

5x webserver (athlon XP3200+, 1 GB Ram, 2x80 MB Raid0)
[one of these is working as stagingserver and as attachment server]
1x dbserver (dual opterin 265, 4 GB RAM, 2x160 MB Raid0)
1x seperate (premium) webspace for all images except attachments & avatars
Load balancing is simply made by DNS round robin.

[dnerty]

Our forum has: Threads: 204,011, Posts: 9,818,856, Members: 86,900, Currently Active Users: 2364 (310 members and 2054 guests)
Most users ever online was 3,750, 28.08.2007. at 12:15.
Currently Active Users: 2364 (310 members and 2054 guests) (aprox. 2000-3000)
post.MYD is 4294967244 big ..

All that has been running on one server since 11/2006:
SuperMicro H8DCR-3 AMD Opteron 270 with 12GB RAM, 2x73 GB SCSI 10k
Now we will setup second, MySQL server..

[TECK]

Everyone's concept of the "big board" is different.
Very good point. I see 89 votes for 1 server. Those are not big boards, IMO.
In my books, a big board has at least 4 servers. :)

[dnerty]

? In my books, a big board has a lot of members, forums, posts.. You can have forum with 10 members on 10 servers. And you can't tell that is big board. :rolleyes:

[TECK]

You are missing completely my point.
We are not talking about storage facilities, but live boards with millions of posts and thousands of users online.

[dnerty]

I think my forum (https://vborg.vbsupport.ru/showpost.php?p=1410013&postcount=37)is quite big, and it wass running on one server till today. Now I have two servers online :).

[TECK]

Ya, those are nice numbers.. how can you run almost 4,000 users only in one box?
You are a hero. :)

I'm used to spread the server load through workers. Just curious, how high is the server load when you have 3,500 users online? Thanks for the info. I noticed your semi-automatic guns (10k drives and 12GB of RAM). Try if you can the new SAS 15K drives.

[dnerty]

Load is 10-15 with 3000+ users online.. Today I setup a new woodcrest with 2x73GB 15K HDD's :), as MySQL server. Now I need some time for optimization and I beleive those servers could handle 5000 users online.

[EricGT]

The thing that surprises me is the storage solutions I see some of you describing. I've always assumed that interaction with the hard drives would be the bottleneck. I spend a lot of money on really fast RAID systems. My current box has an Adaptec 3805 SATAII RAID card, with 8 Seagate 7,200RPM 750GB SATAII drives, in a RAID 10 configuration. The setup is fast as hell, with decent fault tolerance. This saved my butt a couple of days after I deployed this brand new box. A contractor at my data center arced something in the power box supplying my cabinet and spiked the box. One drive was taken out and the RAID card was damaged, but the system kept running after a restart (albeit with a fair amount of complaining) until I could get down and replace the damaged components. I have drives that big because they weren't much more than the smaller ones and because I am about to start offering image hosting.

What do you guys think is the bear minimum hard drive setup required to run a busy vB system, with both acceptable performance and fault tolerance? Also, what type of external storage is fast enough to use with a vB DB server? Eric

[dnerty]

I would like to have HDD's and RAID like you, but have to take care about every dollar. My clients are crying for money ;).. and, if you are ready to spend some time with your server you really can save some money.

[TECK]

Load is 10-15 with 3000+ users online.. Today I setup a new woodcrest with 2x73GB 15K HDD's :), as MySQL server. Now I need some time for optimization and I beleive those servers could handle 5000 users online.
With 5,000 users, you need 6 servers. 2 for web, 2 for db and 2 for load balancing (1 is failsafe). Then your server load will be around 0.5-1.5. You cannot run an efficient board while having the load at 15. However, you will obtain much better results if you use a proxy array, instead of spending extra money on 2 cheap balancers.. Serving the data will be ultra fast.

--------------- Added 1199154184 at 1199154184 ---------------

The thing that surprises me is the storage solutions I see some of you describing. I've always assumed that interaction with the hard drives would be the bottleneck. I spend a lot of money on really fast RAID systems. My current box has an Adaptec 3805 SATAII RAID card, with 8 Seagate 7,200RPM 750GB SATAII drives, in a RAID 10 configuration. The setup is fast as hell, with decent fault tolerance. This saved my butt a couple of days after I deployed this brand new box. A contractor at my data center arced something in the power box supplying my cabinet and spiked the box. One drive was taken out and the RAID card was damaged, but the system kept running after a restart (albeit with a fair amount of complaining) until I could get down and replace the damaged components. I have drives that big because they weren't much more than the smaller ones and because I am about to start offering image hosting.

What do you guys think is the bear minimum hard drive setup required to run a busy vB system, with both acceptable performance and fault tolerance? Also, what type of external storage is fast enough to use with a vB DB server? Eric
You should bump the drives to 15K SAS RAID10, especially for your db servers.
You are killing your servers with SATA, as we speak.

And instead of load balancer hardware, you should try nginx with few workers set in proxy.
It will blow your mind, speed wise.

[EricGT]

With 5,000 users, you need 6 servers. 2 for web, 2 for db and 2 for load balancing (1 is failsafe). Then your server load will be around 0.5-1.5. You cannot run an efficient board while having the load at 15. However, you will obtain much better results if you use a proxy array, instead of spending extra money on 2 cheap balancers.. Serving the data will be ultra fast.

--------------- Added 1199154184 at 1199154184 ---------------


You should bump the drives to 15K SAS RAID10, especially for your db servers.
You are killing your servers with SATA, as we speak.

And instead of load balancer hardware, you should try nginx with few workers set in proxy.
It will blow your mind, speed wise.

This is a powerful server, but it was still built on a tight budget and the extra expense could not be justified. At any rate, the current setup is lightning fast and handling the load just fine. I'll check out nginx, but when I need load balancing again, I will probably use the system built into the Astaro firewall I use. Thanks for the tip though. Eric

[Hornstar]

Yeah, I am in the process of learning as much as I can about splitting the load on 2 servers, as I am ready to go with 2 servers myself. Any useful links on the web about this stuff would be handy. For now i'll just keep reading each and every thing I see about it ^^

[user_not_found]

Currently 1 server - 2,4GHz HT with 2GB RAM.

[dnerty]

With 5,000 users, you need 6 servers. 2 for web, 2 for db and 2 for load balancing (1 is failsafe). Then your server load will be around 0.5-1.5. You cannot run an efficient board while having the load at 15. However, you will obtain much better results if you use a proxy array, instead of spending extra money on 2 cheap balancers.. Serving the data will be ultra fast.

--------------- Added 1199154184 at 1199154184 ---------------

Just finished with moving my databases to a new server.. Now I have 3500 users online with 2-3 load on web server, 0.3 - 0-6 on mysql server :).. Forum.hr (http://www.forum.hr)And didn't finished with optimisation yet, I will have lower values in a few days :D

[lauxanh]

What webserver are you using?
Litespeed is great and Ngix too.

Btw, That FORUM would make tons of money from adsense :D

[TECK]

I prefer Nginx. With the money I save from Litespeed Enterprise license, I put another grand on top and buy a very good server. :)

[dnerty]

I use good old Apache 1.3 :)

[chuanjer]

We are having 5 web servers and 1 db for our board.
Our board generates about 20+M pageviews/mth with approx 1300 - 2500 users online (as reported by vB on a 900 secs timeout)

However, the access is pretty slow and recently our db load is very erratic. It can jump to 100!!!
Refer to http://www.vbulletin.com/forum/showthread.php?p=1485653

Anyone can share with us on how to optimize our setup? Our servers are installed with off-the-box RHEL4. We do not have knowledge on how to custom build kernels or modules.

Apparently that way seems to be the most efficient to keep load down and serve out fast.
Anyone can share where to find such info.

For those who indicated 2 dbs, how is your setup like? How to have 2 dbs in 1 vB installation?

[dnerty]

It's hard to say without deep into servers..

[GotVtec]

V6Performance Networks (http://www.v6performance.net)
Users: 23k+
Threads: 82k+
Posts: 1.2 million
http://www.big-boards.com/board/1015/

All on 1 server.

[Digma]

Users: 14k+
Threads: 40k+
Posts: 1.5m+

Server(s): 1

Specs:
- Xeon Quad Core 3220 @ 2.4ghz
- 2GB RAM
- 147GB SA-SCSI 15k RPM
- 250GB SATA II (backup)
- 2TB bandwidth.

[PSS]

http://photography-on-the.net/forum

Big-boards rank ca. 250 and going up.
Alexa rank ca. 12,500
2000-3000 users online 24/7
4,7 million posts
135,000 members

One server:
RHEL 5 64-bit
Sphinx search
Litespeed Enterprise Server 3
2 x Intel Xeon-Woodcrest 5148
8GB RAM
2x SAS-SCSI (15k rpm) + SATA + NAS

Server can handle much more than above needs, it's on light load now.

--------------- Added 1200784410 at 1200784410 ---------------

With 5,000 users, you need 6 servers.

Sorry but that is simply not true.

Then your server load will be around 0.5-1.5. You cannot run an efficient board while having the load at 15.

Load is essentially "processes in queue". 1*cores load means all works in about realtime. To interpret any loads higher than that is not that straighforward. What those processes are, are their queuing really slowing things down, how long do the queued processes need to execute etc?

[lauxanh]

Wow,

that is impressive.

Would you mind share us your config for Litespeed? i have a board with that much of member online but less post and thread. Running on Xeon 5335 with 4GB of ram and 73GB SCSI for both webserver and database.

But at peak it low down.

Thanks

[alexi]

http://photography-on-the.net/forum

Big-boards rank ca. 250 and going up.
Alexa rank ca. 12,500
2000-3000 users online 24/7
4,7 million posts
135,000 members

One server:
RHEL 5 64-bit
Sphinx search
Litespeed Enterprise Server 3
2 x Intel Xeon-Woodcrest 5148
8GB RAM
2x SAS-SCSI (15k rpm) + SATA + NAS

Server can handle much more than above needs, it's on light load now.

--------------- Added 1200784410 at 1200784410 ---------------



Sorry but that is simply not true.



Load is essentially "processes in queue". 1*cores load means all works in about realtime. To interpret any loads higher than that is not that straighforward. What those processes are, are their queuing really slowing things down, how long do the queued processes need to execute etc?

I can see this. I can't believe the load difference since we switched to a SAN. I would think NAS would have the same functionality. It really seems to be all about the drives. I'm not about to try it but I bet I could get away with not having seperate front end servers.