Quad Opterons + FreeBSD + Lighttpd + FastCGI/PHP + APC = Fast vB :)

[Erwin]

Yup, I migrated my forums to new servers (dual xeon to quad opterons) using completely new software - FreeBSD, Lighttpd instead of Apache. Lighttpd is really amazing, and I recommend that everyone switch from Apache to Lighttpd. APC works fine on the setup too. Just sharing.

[Brad]

Good to hear everything is going well. ;)

[Erwin]

FreeBSD is quite a robust OS with a very good upgrade system. I highly recommend it as a good basis for a server.

[kmike]

Well, all benchmarks I saw suggest otherwise - FreeBSD scales worse than Linux in multiprocessor setup. YMMV, of course, but I also can't see how the FreeBSD ports system is superiour to the yum/apt repositories. And if you're into recompiling everything to tune up for your system, Gentoo's ebuilds are just for that.

edit: I see that you didn't make it to SoftLayer...

[FlyBoy73]

Sounds good, Erwin. Thanks for sharing your experience with this setup.
I'm wondering if I could get away with something like this. My new Adaptec SAS RAID controller card doesn't have the widest range of drivers available yet, thus I'm using CentOS 4.1 rather than the latest version.
I'm always down for going faster ;)

[mathias]

how is converting apache over to lighhtpd

[adalren]

Erwin, did you notice less load on the servers vs apache? Any optimizations? I converted to lighttpd but didn't see a noticable drop in load but the pages are generated slightly faster.

[DJ XtAzY]

hmm i guess im gonna switch from apache =]

but i like gentoo as my OS =]

[Erwin]

Well, all benchmarks I saw suggest otherwise - FreeBSD scales worse than Linux in multiprocessor setup. YMMV, of course, but I also can't see how the FreeBSD ports system is superiour to the yum/apt repositories. And if you're into recompiling everything to tune up for your system, Gentoo's ebuilds are just for that.

edit: I see that you didn't make it to SoftLayer...
Only if you compile the wrong kernel. :) FreeBSD 6.0 using AMD64 SMP kernel on a Quad Opteron just plain flies.

Ports are better than yum/rpm because ports automatically fixes all broken dependencies and installs everything automatically using 1 command: portinstall apache or portinstall mysql5

Can't get any easier than that. ;)

[Erwin]

Sounds good, Erwin. Thanks for sharing your experience with this setup.
I'm wondering if I could get away with something like this. My new Adaptec SAS RAID controller card doesn't have the widest range of drivers available yet, thus I'm using CentOS 4.1 rather than the latest version.
I'm always down for going faster ;)
The only downfall of FreeBSD is its lack of driver support - I had to change my RAID card and go with a RAID 10 rather than a 0+1 setup. But it's all good.

[Erwin]

how is converting apache over to lighhtpd
On FreeBSD, it's this: portinstall lighttpd.

;)

[Erwin]

Erwin, did you notice less load on the servers vs apache? Any optimizations? I converted to lighttpd but didn't see a noticable drop in load but the pages are generated slightly faster.
Maybe because your webserver wasn't badly overloaded? I can tell you that I was able to consolidate my multiple webservers into a single more powerful one which makes maintaining files and the datastore cache much easier.

[Zachery]

Gentoo's portage system does all of that too, and i personally would have gone with Gentoo over FreeBSD.

[Erwin]

Yeah but Gentoo is still Linux and it basically copied the BSD ports system. BSD is the oldest *nix around AFAIK. In any case, there is no one OS that suits everyone. :) They are all good in their own way. I just wanted to try something different. :)

[Zachery]

I'm glad its working for you, what is your current setup now, and would you mind sharing who you went with?

[Erwin]

For the database server, it's a dual processor/dual core Opteron 270 with 8Mb RAM, 8 x 15k SCSI PCI-X Raid 10 array, FreeBSD 6, MySQL 5 - runs like the wind. Loads never hit 0.5. :) The main webserver is the Quad Opteron with similar specs. We also have image server, irc, attachment, dns, and a slave search server.

We went with webnx in the end - best prices in town - look for Dan Paultz and tell him Erwin sent you - he'll look after you well. :)

[kmike]

Only if you compile the wrong kernel. :) FreeBSD 6.0 using AMD64 SMP kernel on a Quad Opteron just plain flies.
Well, but you haven't compared it to Linux, have you?
I already posted some links to FreeBSD/Linux comparisons in MySQL performance, here're some more:
FreeBSD 5.x: http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2005-06/0331.html
FreeBSD 6.x:
http://www.archivesat.com/FreeBSD_Performance_tuning/thread412787.htm
http://www.archivesat.com/FreeBSD_Performance_tuning/thread420280.htm


Ports are better than yum/rpm because ports automatically fixes all broken dependencies and installs everything automatically using 1 command: portinstall apache or portinstall mysql5

Can't get any easier than that. ;)
I see some confusion here: yum does resolve the missing dependencies! I'm not sure what do you mean by "broken" dependencies, probably 3rd party rpm packages? But the same goes for the ports - if you install some 3rd party program manually, outside the ports system (doing usual "./configure; make; make install"), you have to resolve its dependencies manually.

There's one fundamental difference between the ports and rpm-based package managers - rpm's are all binary, and you build your ports from the source when installing them. It could be considered as an advantage as you're supposedly tuning the package for your system when building it (with the right parameters), but some time ago I saw the tests of Gentoo (which is touting the same approach as an advantage) vs some binary-based distribution didn't reveal any substantial performance advantages.

And finally, you can easily rebuild RPM package from the source rpm with options tailored for your architecture, too (that's what I did for our install, rebuilt php rpms with Opteron optimizations). It's as easy as doing "rpmbuild --target=athlon --rebuild yourpackage.src.rpm"

[Erwin]

Thanks for the links, it's very specific stuff there. Interesting.

I know about yum - I ran RHEL servers for 4 years. :)

I'm not here to start an OS-debate. :) I'm just posting my experiences which mirrors the experience of many others. :)

[T.S.]

For the database server, it's a dual processor/dual core Opteron 270 with 8Mb RAM, 8 x 15k SCSI PCI-X Raid 10 array, FreeBSD 6, MySQL 5 - runs like the wind. Loads never hit 0.5. :) The main webserver is the Quad Opteron with similar specs. We also have image server, irc, attachment, dns, and a slave search server.

We went with webnx in the end - best prices in town - look for Dan Paultz and tell him Erwin sent you - he'll look after you well. :)
I would have went with a little more ram. :p

[Erwin]

My swap is still zero. :) Loads are still less than 1. I think it's okay for now considering all it has is the one database as a dedicated database server, MINUS any search or IP searches - we have another dedicated slave database Dual Xeon with 4 MB or RAM to do just search or IP searches. :)

[T.S.]

My swap is still zero. :) Loads are still less than 1. I think it's okay for now considering all it has is the one database as a dedicated database server, MINUS any search or IP searches - we have another dedicated slave database Dual Xeon with 4 MB or RAM to do just search or IP searches. :)
Hopefully you mean GB rather than MB. ;)

[Erwin]

LOL!

Yeah, Gb. The last time I had 4 Mb RAM was on a 386... My 286 had 1 Mb RAM and my XT had 64kb.

[eva2000]

For the database server, it's a dual processor/dual core Opteron 270 with 8Mb RAM, 8 x 15k SCSI PCI-X Raid 10 array, FreeBSD 6, MySQL 5 - runs like the wind. Loads never hit 0.5. :) The main webserver is the Quad Opteron with similar specs. We also have image server, irc, attachment, dns, and a slave search server.

We went with webnx in the end - best prices in town - look for Dan Paultz and tell him Erwin sent you - he'll look after you well. :)
Those cpu loads are very nice :D

[Xorlev]

I recommend giving Litespeed Webserver a try as well...I much prefer it over Lighttpd.

[Zachery]

Theres that whole cost thing.
Enterprise Edition (4 CPU) $1199 ;)

[Xorlev]

The standard version is free and has no limits as long as it keeps to one CPU (or a dual core) and 300 concurrent users. But the benchmarks speak for themselves when you're under high load.

I do love both however. :)

[Erwin]

Lighttpd works very well if the config is tweaked. I'm using the Quake.net config example - works great.

[kerplunknet]

Lighttpd works very well if the config is tweaked. I'm using the Quake.net config example - works great.
Where would a person find this "Quake.net config" ?
:)

[Erwin]

Someone on vB.com posted a link to it in the large forums thread. :) Should still be there.

[bang]

gotta back up Zachery here. Gentoo is plain awesome. upgrading just doesn't get easier than Gentoo has it - execute "emerge -u world" in screen and forget about it. it's very stable - I have a single-CPU Gentoo box that has 260 days uptime and handles ~15k uniques a day. and needless to say, Gentoo logo is the best. Pacman forever! :D

[kerplunknet]

Ah, thanks ;)

http://nopaste.php-q.net/206712

[Erwin]

Mind you, my lighttpd is causing server errors at the moment, and I'm trying to troubleshoot it. It's possible APC doesn't want to work it. I can't get eA to work with it.

[Zachery]

You are trying to run both at the same time erwin? eA and APC accomplish the same thing.

I've got lighttpd running on a new server just for serving images, and large downloads (E3 is a killer on the site in question). Has made a huge differnece offloading just the images to another server. Loads on the three servers are below 1 and its running great.

[Erwin]

No, not at the same time - one or the other.

[Erwin]

I had to remove the quake.net tweaks in the end. :) I was getting server errors with them for some reason...

[Erwin]

By the way, my servers are with www.webnx.com and they are by far the cheapest providers of top end hardware I've found. We're talking Quad Opterons or Dual/Dual Opterons with 15k SCSI hardware RAID arrays for a few hundred bucks - and their service has been great so far - unmanaged servers but they help out wherever they can. I highly recommend them. :) You can contact Dan Paultz, who's a great guy, if you're interested in some high-end boxes to lease. The provisioning of the boxes take a while but once they're up they're good.

[Zachery]

Erwin, which method did you decide to go for when doing virtial hosts? If you have any?

[Erwin]

Virtual hosts are flexible in lighttpd - I don't do them at the moment but if I did I would be IP based.

[Zachery]

I was refering to the simple version or the conditional based version.

[adalren]

Mind you, my lighttpd is causing server errors at the moment, and I'm trying to troubleshoot it. It's possible APC doesn't want to work it. I can't get eA to work with it.

What kind of problems are you having with lighttpd?

We're using lighttpd + phpfcgi + 64bit CentOS and lately experiencing segfaults from the php processes. I see a few bug listings regarding segfaults on APC (http://pecl.php.net/bugs/search.php?cmd=display&status=Open&package_name[]=APC) and have a hunch it might be that.

BTW, eA does not work on a 64bit OS.

[Zachery]

What kind of problems are you having with lighttpd?

We're using lighttpd + phpfcgi + 64bit CentOS and lately experiencing segfaults from the php processes. I see a few bug listings regarding segfaults on APC (http://pecl.php.net/bugs/search.php?cmd=display&status=Open&package_name[]=APC) and have a hunch it might be that.

BTW, eA does not work on a 64bit OS.
I think it was digitalpoint on vB.com who commented about segfaults and APC under high load.

[Dan]

I've had a recent experience with FreeBSD servers and Erwin is right, it's really a good distro to use.

But with Lighttpd reference I haven't really had a chance to play with it much in a production enviroment, I'm currently playing around with it on my localhost. Hopefully one day my host can switch it or I could get my own server and use it.

[Erwin]

Ahhh... eA doesn't work with 64bit... didn't know that... that explains it.

[adalren]

I think it was digitalpoint on vB.com who commented about segfaults and APC under high load.
That can't be the case since the load is < 1. It seems like a child php process segfaults and serves error pages until the parent process kills it after the # of PHP_FCGI_MAX_REQUESTS. At least it recovers.

[Erwin]

Anyone want to share their lighttpd.conf? I can't use the quake.net one as my server crashes with it (server 500 error).

I'll share the tweaks I've made to mine:

Add these lines to the top:

server.max-keep-alive-requests = 0
server.max-fds = 2048
server.stat-cache-engine = "fam"


Basically, switch off keep-alive (well, it helps to speed things up on my webserver), double the maxed opened files from 1024 to 2048 (if you have a busy server) and use the stat cache system.

It seems to work well for me so far - let's see how it goes at peak times (it's off-peak now, only 1,000 users online).

More info on optimizing lighttpd here:

http://www.lighttpd.net/documentation/performance.html

[kmike]

Hmm, keep alive should help multiplexing web servers like lighttpd. Number of open files should not be an issue if you're not on the shared hosting, and there're no other downsides to enabling keep-alive.

[adalren]

How do you get fam/gamin to work? I installed gamin (yum install gamin) and there wasn't any daemons or init scripts to run.

Here is my conf file. It runs fine with apc 3.0.8 but segfaults with 3.0.10.

server.event-handler = "linux-sysepoll"
server.max-fds = 4096
server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4
server.max-read-idle = 20
server.max-write-idle = 180

[Erwin]

Hmm, keep alive should help multiplexing web servers like lighttpd. Number of open files should not be an issue if you're not on the shared hosting, and there're no other downsides to enabling keep-alive.
I thought keep alive is only useful for static files? I may be wrong obviously, I'm no expert. No. of files may be significant if there are lot of attachments, again from what I've read, so it's probably more relevant for an attachment server. Enabling keep-alive just means connections are kept longer - I have to say that when I was using apache, I never had keep alive on, even with my image server.

[kmike]

Well, I'm going to get to the basics on that.
The beauty of multiplexed web servers (lighttpd, thttpd, Zeus, nginx, etc) is in that number of connections isn't a choke point, their performance is usually disk (for static file serving) or CPU (for dynamic content) bound.
So there's _no_ reason not to enable keep-alive as it eliminates the need to re-open connection for the next request(s), and managing idle connections does not take up any resources for a multiplexed web server utilizing one of the advanced solutions to c10K problem (http://www.kegel.com/c10k.html) like epoll, rtsig or kqueue.

Back to Apache, everyone's disabling keep-alive in Apache because every connection, even an idle one, requires one active Apache process or thread to manage it, taking up memory.

I hope it answers your questions.

[Erwin]

For some reason, the default

server.max-write-idle = 360

is too short... I've increased that to 720 and it's still too slow according to the error log.

I'll give:

server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4

a try then. :) I'll report back.

In relation to kqueue, I'm using FreeBSD so I have this line:

server.event-handler = "freebsd-kqueue"

That covers that yeah?

[Xorlev]

Probably should bump up the keep-alive idle time, as not everyone is able to load, read, and make a new request in that time, unless you count the several requests for images and the like. I keep it at about 15 seconds myself.

[adalren]

Isn't 6minutes kind of long for server.max-write-idle? I see them in the logs but most of them are errors on transfering very small 10kbyte files. The server tries to send the file over and if it idles for that long, it's a good assumption that the user has canceled.

For some reason, the default

server.max-write-idle = 360

is too short... I've increased that to 720 and it's still too slow according to the error log.

I'll give:

server.max-keep-alive-requests = 4
server.max-keep-alive-idle = 4

a try then. :) I'll report back.

In relation to kqueue, I'm using FreeBSD so I have this line:

server.event-handler = "freebsd-kqueue"

That covers that yeah?

[Erwin]

Has anyone figured out how to switch logging off in lighttpd? :)

[adalren]

Removing mod_accesslog in your server.modules should disable logging.

[Erwin]

Removing mod_accesslog in your server.modules should disable logging.
Good one.

So far lighttpd is going well with the short keep-alive time. My logs are filled with the write-idle timeout thing but otherwise it's fine.

Mmm... I disabled mod_accesslog but logging still occurred. Any other ideas?

[adalren]

Are you trying to disable error logging because I don't think you can unless you specify /dev/null as the file output. Removing mod_accesslog only disables accesslog and that works fine for me.

[Erwin]

Yeah, disabling error logging is what I'm trying to achieve. But I'm rotating the logs at the moment so it's not a big deal. Funny how you can't do it in the config.

[turkforum]

Erwin may i ask who configured the server , is this steven... cause i also work with dan for almost 6 months. Bu t i have to switch from 2 server system to 1 server system as my hits dropped almost 1000% after we have been banned by the freaking nonsenseTurkish Internet Laws and had to change domain name...
Reason being i am asking whether it was steven i can give your site as a reference to himm..
thnx

[Erwin]

Dan Paultz helps a lot but he also gets Steven Ciaburri from rack911 to assist him - Steven helped me set up my irc server on one of my boxes. Sure, you can give Steven a reference to my site or to me, they'll know.

[DirkW]

Are you using Lighttpd on a multiple CPU server? If so, are you running just one instance? Are you using a PHP optimizer? What is your memory usage like?

[nso]

Erwin, you could experiment with (or implement) cpu_polling in your kernel to decrease the time it takes for the CPU to communicate with your network card(s) -which can be a bottleneck on busy sites. Fortunately FreeBSD has this option.

Add the following to your kernel and rebuild it, I've been using it with great success on my Dual Xeon system;
#Device polling
options DEVICE_POLLING
options HZ=1200