Big Boarders & Backups..

My last few projects have been with boards that fit the big board description. Hundreds of thousands of members, millions of posts, and tons of activity.

Usually I am hired on to help with fixing issues with plugins, writing custom plugins, optimizing, etc.. My last three clients, in addition to other things, all wanted consultation on backing up data. For the most part, they all had in common one feeling and that was that they had little to no faith in their dedicated hosting provider when it came to the backups. Either they did not understand it, or were somehow discouraged when they tried to use it last.

My last/current client actually bought me a drive and begged me to backup his data for him (for a fee of course).. Well, I have always had my own system for backing up my servers which allows me to have incremental backups so you could have hourly/weekly/monthly/yearly backups. Using rsync and hard links, I have multiple backups without using a ton of storage. Basically if something happened to my data, I could get back the last hours version of it. So I set him up with the same thing and he was ecstatic.. Gave him FTP access to the server where I stored his backups (using the drive he bought ) and he could see, and download all of his files at any interval..

Got me thinking about the demand for such a thing. Is this common with big boarders here? What would happen to you tomorrow if your servers hard drive(s) suffered some ill fate (power surge, water damage, dead drive, etc..)? Are you confident in your backup strategy?

The reason why I ask is, if there is a demand out there for it, I might consider running a small service to do it, but before I even bother, I thought of asking here first..

[Jafo232]

Originally Posted by amcd

this thread is in the big boards forum.

And??

[TECK]

Originally Posted by Jafo232

By using incremental backups with hard linking, you can keep snapshots on the backup server to allow for rollbacks to certain times (hourly, daily, weekly, monthly)..

You can keep snapshots with rcync, there is no need to complicate your life with incremental backups.
Only a few resources (out of many available):
Mike Rubel's rsync snapshots - the original rsync script
rsnapshot - based off Mike Rubel's article, my preferate
Dirvish - the most feature-filled of these programs

Output on rsnapshot:

[root@localhost /]# rsnapshot -v daily
echo 1842 > /var/run/rsnapshot.pid
/bin/rm -rf /.snapshots/daily.6/
mv /.snapshots/daily.5/ /.snapshots/daily.6/
mv /.snapshots/daily.4/ /.snapshots/daily.5/
mv /.snapshots/daily.3/ /.snapshots/daily.4/
mv /.snapshots/daily.2/ /.snapshots/daily.3/
mv /.snapshots/daily.1/ /.snapshots/daily.2/
mv /.snapshots/daily.0/ /.snapshots/daily.1/
mv /.snapshots/hourly.5/ /.snapshots/daily.0/
rm -f /var/run/rsnapshot.pid
[root@localhost /]#

Disk usage report on a subdirectory over time:

[root@localhost /]# rsnapshot du example.com/home/
5.2G    /.snapshots/hourly.0/example.com/home/
5.4M    /.snapshots/hourly.1/example.com/home/
12M     /.snapshots/hourly.2/example.com/home/
5.3M    /.snapshots/hourly.3/example.com/home/
5.4M    /.snapshots/hourly.4/example.com/home/
4.9M    /.snapshots/hourly.5/example.com/home/
5.4M    /.snapshots/daily.0/example.com/home/
728M    /.snapshots/daily.1/example.com/home/
4.8M    /.snapshots/daily.2/example.com/home/
4.8M    /.snapshots/daily.3/example.com/home/
6.2M    /.snapshots/daily.4/example.com/home/
4.8M    /.snapshots/daily.5/example.com/home/
4.8M    /.snapshots/daily.6/example.com/home/
6.0G    total

You see? It is possible.

Have your pick, or if you are not happy with this solution, simply use rdiff-backup. It is based on rsync and stores incremental rdiff deltas with the backup, with which it is possible to recreate any backup point. You problem is solved in 2 seconds.
Compared to rdiff-backup, rsync is faster, so it is often the better choice when pure mirroring is required. Also, rdiff-backup does not have a separate server like rsyncd (instead it relies on ssh-based networking and authentication). However, rdiff-backup uses much less memory than rsync, on large directories.

Originally Posted by Jafo232

And??

What amcd meant is: if you are a big board owner, you either know your terminal really well or... you have money to hire an admin as company employee. He will do the job for you. Either ways, the person will be very capable to complete the rsync job himself.

[Jafo232]

Actually, I have done custom work for quite a few big boarders who barely had knowledge of FTP, much less the terminal, negating both points I think.

[TECK]

That corresponds with my above statement: "you have money to hire an admin..."
In other words, you were hired to do a backup by the big board owner. If I have money, I would never bother to learn all those "rsync thingies", I will hire you to do it for me. I will spend my time to get more clients who will advertise on my board, because I'm a business guy, not a Linux lover.

[EricGT]

I wrote a script that shuts down Apache and MySQL each night, does an rsync of the DB directory to a backup directory on the server, restarts Apache & MySQL and then rsync's the backup directory to dedicated backup server. I maintain an encrypted tunnel between a server at home and the firewall of my remote network and rsync the backup directory of the backup server to a system at my home nightly. I also run a mysqldump of the DB once a week and rsync it to the backup server and home. There is no site without the data and I've been doing this long enough to fear and respect Murphy's Law. Eric

[TECK]

There is not need to shut down your server. You are basically deny access to users during that backup time.
Use rsnapshot, it is transparent... and it does incremental backups hourly.

[Jafo232]

I can understand shutting down httpd/mysqld I guess, when you are backing up the database binaries, myself though, I just do an automated mysql dump and let rsync back up the dump file.

[EricGT]

Originally Posted by Jafo232

I can understand shutting down httpd/mysqld I guess, when you are backing up the database binaries, myself though, I just do an automated mysql dump and let rsync back up the dump file.

I can back up and restore the binaries in a fraction of the time it takes to do a dump or restore it. I shut the MySQL server down so I know I get a clean copy. I also do a weekly mysqldump, to backstop the binary rsync.

TecK, I haven't heard of rsnapshot. I'll give a look. Thanks for the tip. Eric

[TECK]

My pleasure, Eric.

link: http://www.rsnapshot.org/
screenshots: http://www.rsnapshot.org/screenshots.html