Load Averages

Hi,

My vbulletin setup consists in two webservers with load balance (with Pound) and a MySQL server.

MySQL server load is low (around 0.25).

I am concerned with the load on the webservers:

server 1: load average: 3.46, 4.23, 3.58
server 2: load average: 1.87, 2.19, 2.12

As far as I know, any load above 1 is considered high, as the system starts queuing processes. I may be wrong. Some people say to not worry about server load and consider only after it surpasses 4.

Any thoughs on that? And any tips on how could I reduce the load on my webservers (without adding another webserver to the balancing, of course)?

A full description of my setup including all PHP/MySQL configuration files I am using can be found here:

http://www.vbulletin.com/forum/showthread.php?t=231085

Thanks for your thoughts!

Cheers,
Gabriel.

[vbullfan]

Gabrielt, we have experienced a similar situation on our site, where db load is significantly lower than web load. Our mysql server usually hovers under 0.20, yet the web server can go quite high (as high as 10-20.0 with about 7K users online). I think this is normal.

I checked out your server specs, and your load seems a little high given the setup. We are only running two servers, one web and one db. Our web server is a double-dual 250 AMD Opterons with 4GB RAM, and under normal load conditions (2K users) load remains just below 1.0. You said in your SOR that your max users is 2.5K, so to be above 1.0 load on both of your web servers seems pretty high.

Have you taken a look at your switch traffic? If you are running a 100mb switch you could be maxxing it out and causing processes to cue.

[gabrielt]

Hi,

Thanks for your input. Some other place I read that Opterons give a better performance for vBulletin, unfortunately our hosting company does not provide Opteron servers. Upgrading to 4 GB may be a good option.

Also, our MySQL server is now at 0.20 load only after I upgraded from 2 GB to 4 GB. The load with 2 GB was also very high. At least here I hit bull's eye. Plus memcached made some difference.

I also switched from Pound to Haproxy. Haproxy provides a lower load and also provides a more efficient balancing, in my experience.

I also tried to move from Apache to lighttpd. During the night, when load is lighter, lighttpd was providing a lower load than Apache, but during the day load went to to the roof (5.0-10.0), so I switched back to Apache. Maybe it was just a matter of fine tuning, but I wasted a lot of time with no practical results.

Regarding the switch, I am monitoring the network traffic thru MRTG and it is peaking 12 Mbps on server 1 and 10 Mbps on server 2. The NICs are set at 100 Mbps and the switch is also 100 Mbps. I made some transfer tests and data is really being transferred at 100 Mbps between servers.

What is your forums URL by the way? I'd like to take a look to see its performance. Mine is http://forum.clubedohardware.com.br

What do you use for load balancing?

Cheers,
Gabriel.

[ekool]

Gabriel,

You should install mtop on your database server and see if you can tie any particular "database" activity to high loads on your web servers.

You have over 2.2 million posts. My guess is that when someone is searching your forums, you run into the very common table lock issue which causes your webservers to queue all connections... when they do this, they open many apache/php processes and just sit there.. memory use goes up and cpu activity goes up until eventually either the search query finally finishes and pages are served or your apache servers hit their queue/connection limit and start rejecting new connections.

My guess is that you are going to need to get the sphinx search hack working and once you do that, your webserver loads will go down as well.

First step tho, look at mtop. We ran into this issue recently and even tho the mysql server would shunt multiple hundreds of mysql queries into the WAITING state, the load never went high. However, it did have a direct impact on the load of the web servers.

Hope this helps.

[gabrielt]

Yes, it actually helps a lot, thanks. I got someone saying very good things about sphinx, so I am learning more about it right now.

Cheers,
Gabriel.

[R1lover]

Originally Posted by gabrielt

Yes, it actually helps a lot, thanks. I got someone saying very good things about sphinx, so I am learning more about it right now.

Cheers,
Gabriel.

I see >25 membros e 660 visitantes) on your site.... with 25 members online you are having an issue?

Also what is your session timeout set to?

I know it may be an off time for your site but you should be able to run with 700 people online with out much load at all. Most of those visitors are probably spiders as well.