[mute]

Here's the route we took with our multi-webserver setup:

1) Initially we tried rsync, but found that it sort of sucked for users when they'd upload an avatar or attachment and the link would be broken for some people until the rsync happened (1 minute is still too long to lag IMO).

2) We setup FASD or whatever it is called to monitor directories for changes and rsync based on activity. This worked OK, but it was sort of prone to problems, and still lagged a wee bit.

3) We decided to add a bunch of webservers, so we moved to a setup where we have an image server (running thttpd) that serves all the images and most of the static content (js, etc) on the site. This server also runs NFSd, and serves our webroot out to all of our webservers. The webservers mount the webroot via NFS, have rw access to avatars and attachments but only ro access to everything else.

#3 has been working well for us for a couple of years now. Sure, it's a single point of failure, but it has the least amount of headache compared to some of the other solutions I've seen (hacking code so image uploads go to a specific server, etc).

In the near future I'd like to start thinking about provisioning a better disaster recovery system where the webservers rsync a copy off the NFS to their local disks, and a new virtualhost so in the event of an imageserver failure, I can point the images url at our loadbalancer as opposed to the images server and move the local webroots into place and the webservers will pick up the slack until our images server is back online.