Big Boarders & Backups..

[Jafo232]

My last few projects have been with boards that fit the big board description. Hundreds of thousands of members, millions of posts, and tons of activity.

Usually I am hired on to help with fixing issues with plugins, writing custom plugins, optimizing, etc.. My last three clients, in addition to other things, all wanted consultation on backing up data. For the most part, they all had in common one feeling and that was that they had little to no faith in their dedicated hosting provider when it came to the backups. Either they did not understand it, or were somehow discouraged when they tried to use it last.

My last/current client actually bought me a drive and begged me to backup his data for him (for a fee of course).. Well, I have always had my own system for backing up my servers which allows me to have incremental backups so you could have hourly/weekly/monthly/yearly backups. Using rsync and hard links, I have multiple backups without using a ton of storage. Basically if something happened to my data, I could get back the last hours version of it. So I set him up with the same thing and he was ecstatic.. Gave him FTP access to the server where I stored his backups (using the drive he bought :) ) and he could see, and download all of his files at any interval..

Got me thinking about the demand for such a thing. Is this common with big boarders here? What would happen to you tomorrow if your servers hard drive(s) suffered some ill fate (power surge, water damage, dead drive, etc..)? Are you confident in your backup strategy?

The reason why I ask is, if there is a demand out there for it, I might consider running a small service to do it, but before I even bother, I thought of asking here first..

[Hornstar]

I'd be interested in such a service, but depending on how much it would cost would be a factor, because my dedicated server is costing me enough already.

[Marco van Herwaarden]

I doubt many want a backup service that is not located at the same physical place or at least have a direct highspeed network connection to such a location.

I would not have to push my backups over the regular internet conneections.

[Jafo232]

Funny, I always thought the idea of having a remote backup a selling point. If say that same physical place caught on fire and burned down what good would having a backup system be there?

My backups transmit over the regular Internet yes, via SSH..

I haven't even considered a price..

[weeno]

Obviously, the big issue with big-boards is regular backups to the mysql db. For me, mysqlhotcopy took too long to use frequently. I ended up having to go with binary logs and weekly mysqlhotcopies.

arn

[Marco van Herwaarden]

A physical remote location is good in case of fire etc., but you should ensure that there is a fast direct connection between the 2 locations.

[ssslippy]

Ive noticed its become common for a hosting company to have a backup on remote these days.

[Ted S]

Ive noticed its become common for a hosting company to have a backup on remote these days.
Exactly. Virtually every hosting company now offers off-site backup solutions for those who want them. At the very least, a host will provide an off-machine backup or NAS access to do a more secure and cluster/machine independent backup. There are however companies that offer these services and do well at it... but to enter the marketplace, you'd be competing against established players and need to do a lot to insure your system's reliability and security. The last thing a site needs is to be outsourcing their backup solution to someone running a "small service" that ends up crashing.

[Jafo232]

I can understand that.. Many of the dedicated solutions I have seen include some sort of backup service.. Mine would be compliment that backup and offer some level of redundancy..

Thinking of prices, I was contemplating about $100/mo to backup 25gb of data... Opinions?

[amcd]

I do a full mysqldump of the db once a day at the least busy hour and the .tgz is rsynced to another server 3000 kilometers away.

[Jafo232]

I do a full mysqldump of the db once a day at the least busy hour and the .tgz is rsynced to another server 3000 kilometers away.

I do something similar, except I do a backup of a slave so as to keep the dump from blocking..

[SEOvB]

I can understand that.. Many of the dedicated solutions I have seen include some sort of backup service.. Mine would be compliment that backup and offer some level of redundancy..

Thinking of prices, I was contemplating about $100/mo to backup 25gb of data... Opinions?

You may want to shop around to see what other people are charging. I used Bqinternet (http://www.bqinternet.com/backup/) for my backup services when i required them, and it was A+ for a low price.

[Hornstar]

You may want to shop around to see what other people are charging. I used Bqinternet (http://www.bqinternet.com/backup/) for my backup services when i required them, and it was A+ for a low price. Thanks for that link, that looks like quite a good service.

[Jafo232]

It doesn't look bad, although I only see daily backups. Doesn't appear to cover hourly, weekly or monthly.. Could just be me.. In fact, it only appears to have one backup.. The daily one..

The samba share feature is a good idea (provided it is read only).. However, I could see people abusing the "No data transfer limit" feature in combination with the share. You could basically use the share to serve large files (i.e. videos).

[TECK]

Why pay money when you can do it yourself, like the pro's?
Just use rsync, it is the only method that will not break your busy servers while doing a backup.
What do you think they use when they do your backup? Heh...

Edit: Check what bqinternet.com (http://www.bqinternet.com/backup/) uses as procedure.
Features:
Daily backup using rsync technology
SSH encryption
On-the-fly compression
Get back online instantly with a Samba mount
FTP access
No data transfer limit
No setup fee

[Jafo232]

If you peruse the mods forum, you can see how technically apt your average Vbulletin owner is. Asking them to handle rsync would be an interesting follow..

[Lynne]

If you peruse the mods forum, you can see how technically apt your average Vbulletin owner is. Asking them to handle rsync would be an interesting follow..

http://www.oasisinet.com/forum/images/smiles/icon_lol.gif

[TECK]

If the average vBulletin owner is not capable to perform a rsync process, then the above listed service will come handy to them.
However, you have to look at rsync process like you would copy files from a folder to another... really easy. Setting up your own local rsync mirror is just a matter of configuring the rsyncd daemon to make your portage directory available for syncing. That's all. :)

[Jafo232]

Backing up however needs to be more than just mirroring the site. Lets say, for whatever reason, config.php is wiped out, or corrupted. The mirror server will just copy it, or delete it I suppose. Or a file gets hacked, it will get copied to the mirror.

By using incremental backups with hard linking, you can keep snapshots on the backup server to allow for rollbacks to certain times (hourly, daily, weekly, monthly)..

I guess what I am saying, is rsync alone is not all that helpful.

[amcd]

If you peruse the mods forum, you can see how technically apt your average Vbulletin owner is. Asking them to handle rsync would be an interesting follow..
this thread is in the big boards forum.

[Jafo232]

this thread is in the big boards forum.

And??

[TECK]

By using incremental backups with hard linking, you can keep snapshots on the backup server to allow for rollbacks to certain times (hourly, daily, weekly, monthly)..
You can keep snapshots with rcync, there is no need to complicate your life with incremental backups.
Only a few resources (out of many available):
Mike Rubel's rsync snapshots (http://www.mikerubel.org/computers/rsync_snapshots/) - the original rsync script
rsnapshot (http://www.rsnapshot.org/) - based off Mike Rubel's article, my preferate
Dirvish (http://www.dirvish.org/) - the most feature-filled of these programs

Output on rsnapshot:
[root@localhost /]# rsnapshot -v daily
echo 1842 > /var/run/rsnapshot.pid
/bin/rm -rf /.snapshots/daily.6/
mv /.snapshots/daily.5/ /.snapshots/daily.6/
mv /.snapshots/daily.4/ /.snapshots/daily.5/
mv /.snapshots/daily.3/ /.snapshots/daily.4/
mv /.snapshots/daily.2/ /.snapshots/daily.3/
mv /.snapshots/daily.1/ /.snapshots/daily.2/
mv /.snapshots/daily.0/ /.snapshots/daily.1/
mv /.snapshots/hourly.5/ /.snapshots/daily.0/
rm -f /var/run/rsnapshot.pid
[root@localhost /]#

Disk usage report on a subdirectory over time:
[root@localhost /]# rsnapshot du example.com/home/
5.2G /.snapshots/hourly.0/example.com/home/
5.4M /.snapshots/hourly.1/example.com/home/
12M /.snapshots/hourly.2/example.com/home/
5.3M /.snapshots/hourly.3/example.com/home/
5.4M /.snapshots/hourly.4/example.com/home/
4.9M /.snapshots/hourly.5/example.com/home/
5.4M /.snapshots/daily.0/example.com/home/
728M /.snapshots/daily.1/example.com/home/
4.8M /.snapshots/daily.2/example.com/home/
4.8M /.snapshots/daily.3/example.com/home/
6.2M /.snapshots/daily.4/example.com/home/
4.8M /.snapshots/daily.5/example.com/home/
4.8M /.snapshots/daily.6/example.com/home/
6.0G total
You see? It is possible. ;)

Have your pick, or if you are not happy with this solution, simply use rdiff-backup. It is based on rsync and stores incremental rdiff deltas with the backup, with which it is possible to recreate any backup point. You problem is solved in 2 seconds. :)
Compared to rdiff-backup, rsync is faster, so it is often the better choice when pure mirroring is required. Also, rdiff-backup does not have a separate server like rsyncd (instead it relies on ssh-based networking and authentication). However, rdiff-backup uses much less memory than rsync, on large directories.

And??
What amcd meant is: if you are a big board owner, you either know your terminal really well or... you have money to hire an admin as company employee. He will do the job for you. Either ways, the person will be very capable to complete the rsync job himself.

[Jafo232]

Actually, I have done custom work for quite a few big boarders who barely had knowledge of FTP, much less the terminal, negating both points I think.

[TECK]

That corresponds with my above statement: "you have money to hire an admin..."
In other words, you were hired to do a backup by the big board owner. If I have money, I would never bother to learn all those "rsync thingies", I will hire you to do it for me. I will spend my time to get more clients who will advertise on my board, because I'm a business guy, not a Linux lover. :)

[EricGT]

I wrote a script that shuts down Apache and MySQL each night, does an rsync of the DB directory to a backup directory on the server, restarts Apache & MySQL and then rsync's the backup directory to dedicated backup server. I maintain an encrypted tunnel between a server at home and the firewall of my remote network and rsync the backup directory of the backup server to a system at my home nightly. I also run a mysqldump of the DB once a week and rsync it to the backup server and home. There is no site without the data and I've been doing this long enough to fear and respect Murphy's Law. Eric

[TECK]

There is not need to shut down your server. You are basically deny access to users during that backup time. :)
Use rsnapshot, it is transparent... and it does incremental backups hourly.

[Jafo232]

I can understand shutting down httpd/mysqld I guess, when you are backing up the database binaries, myself though, I just do an automated mysql dump and let rsync back up the dump file.

[EricGT]

I can understand shutting down httpd/mysqld I guess, when you are backing up the database binaries, myself though, I just do an automated mysql dump and let rsync back up the dump file.

I can back up and restore the binaries in a fraction of the time it takes to do a dump or restore it. I shut the MySQL server down so I know I get a clean copy. I also do a weekly mysqldump, to backstop the binary rsync.

TecK, I haven't heard of rsnapshot. I'll give a look. Thanks for the tip. Eric

[TECK]

My pleasure, Eric.

link: http://www.rsnapshot.org/
screenshots: http://www.rsnapshot.org/screenshots.html