2 x Load balanced severs + 1 x database server - Rsync..?? Any tips?

[Make Money?]

Not having much luck with this on VB forums so I thought I would ask here.

We want 2 hardware load balanced servers that are round robin and Rsync'ed and the database on a 3rd server with NAS back up

Redhat 64bit
MYSQL 5

The idea is to use
a) round robin and
b) if one server fails a heath check, to have it fails over to the back up Rsynced copy server.

We have photo and Mp3 uploads, 25 gigs of attachments and growing

Is there any trick to getting attachments rsynced nicely?

Will there be issues with someone uploading an attachment, then returning to check it and it being not being Synced yet on the 2nd server because of lack of instant rsync?

Looking for advice / pitfalls / things to watch out for, I am sure some of you have done this before.

Many thanks in advance,:up:

[SoftDux]

Make sure all three servers are on the same network switch, if possible, and that the rsync is done via a seperate network card, this way it won't interfere with internet traffic.

Bear in mind, this way it the whole DC goes down, so does the site. So it might be worthwhile to try and put each server @ a different DC, but on at least a 100Mbps NIC + switch. I know some hosting providers (Softlayer / Rackspace? I can't remember which other does) offers what you want as a standard service, so it might be worthwhile looking into

[Make Money?]

Thanks for that SoftDux

I got these additional suggestions too

Is anyone out there using any of these methods with good results?

1) "sync is probably not viable - for large numbers of files, there's a big CPU and IO hit when you do the sync. Too much to run frequently enough to avoid round-robin problems. There are other tools that are better at synchronizing multiple servers, but I'm not aware of a good real-time option.

If you do find or build a fast sync tool, you could session affinity at the load balancer to keep a user on their web head for a particular period of time (5 minutes is standard for HTTPS connections - same approach is fine for HTTP too). This would avoid the short-term "where'd my file go" problem, but there's still a risk of someone viewing a page before the files hit their webserver.

2) Another approach is to stick all the files on your database box and have it export them to the web heads via NFS. NFS has its issues, but this is pretty simple to set up, especially over your servers private network. Still a single point of failure, but hopefully your database box has RAID to be a bit more on the safe side.

3) You could go completely external and upload your files to Amazon S3. It would be pretty simple to set up a local caching proxy that keeps frequently-requested files on local disk. I'd be surprised if someone isn't working on this for Vbulletin. Big benefit is the reliability of S3.

There's also low-level disk replication, like DRBD. You'd need a spare disk in each web head that would be synced in real-time between boxes. A bit harder to set up, but would definitely handle this and probably work pretty well over a gigabit private link.

The last general class of fixes are distributed filesystems of various ilks. From GFS and Lustre all the way down to MogileFS (not a real filesystem, but a fantastic file synchronization handler). If someone wrote MogileFS hooks into Vbulletin, that would be a great option. The others I think are just too much of a headache to set up on a small set of servers."

[drauk]

We do this sort of stuff pretty routinely. Works fine.

I wouldn't rsync the files between the machines. DRBD won't help you either; you can only read from one of the two mirrored pair with DRBD (i.e. you could send the file from disk A to disk B, but you can't read from Disk B whilst it's connected to Disk A).

The NFS solution will work perfectly well. Start with it on your MySQL server and if you need more performance later you can scale it up by using a dedicated NFS server. You can use RSync to back it up though. Or if you want to spend the money and get even more resilience, use DRBD to do real-time backups of your database and NFS server.

Why do you want/need to use Round Robin if you've got hardware load balancers?

More generally, don't even think about using Apache. Lighttpd and XCache will give you a lot more capacity.

[Make Money?]

Why do you want/need to use Round Robin if you've got hardware load balancers?


I dunno, its probably not needed, I am just looking for tips for my sys admin. I get lost in the jargon myself, I am swimming out of my depth now for sure

Sincere thanks for the tips..