Go Back   vb.org Archive > vBulletin Article Depository > Read An Article > General Articles
Reload this Page Ultimate Vbulletin Backup Guide || With or Without ROOT ACCESS with over 5GB Backups
FAQ Members List Calendar Search Today's Posts Mark Forums Read

Reply
Page 1 of 2 1 2
 
Thread Tools
Ultimate Vbulletin Backup Guide || With or Without ROOT ACCESS with over 5GB Backups
intensecool
Join Date: Feb 2007
Posts: 69

 

Show Printable Version Email this Page Subscription
intensecool intensecool is offline 01-04-2011, 10:00 PM
Hey Guys,

I own a VBULLETIN , its kind of big one at this time.
Here are the stats as i write the guide :
Threads :: 979,845
Posts :: 1,159,042
Members :: 181,012

All of us know, BACKING up a big board or even a medium one is literally a big pain.
All of us need a system to backup forums to a place , of course AUTOMATICALLY.

I searched a lot of thing, tried everything possible to backup things.
But yes, i faced many issues like these :
  1. Incomplete or Faulty backups due to database being continuously used by traffic.
  2. The website is already loading the servers & with the MYSQL Being dumped makes the servers heavy leaving the sites inaccessible for longer time before it becomes normal.
  3. If you do this manually means when the website is getting hits, you end up with a backup that many times wont restore properly.
Yes, it has happened with me so many times that i was fed up with it and finally decided to make a short working script for myself. Now, its confirmed working i am sharing it here with you guys.

Considering the comments , the script is now working :

1. With SSH LOGIN to Backup Servers.
2. Without SSH Login to Backup Servers
3. Can be RUN VIA CRON Setup, scroll down for instructions.

Mission :

We will backup the database locally and then finally back it up to the REMOTE BACKUP server.

With SSH LOGIN :

Requirements :

1. Dedicated Servers.
2. Root access of course.
3. A BackUp Server with root access.


Process :

As everything is to be done AUTOMATED, we need to set up the SOURCE SERVER & DEST SERVER to have PASSWORD LESS logins.

How to do that :

Quote:
Lets say you want to copy between two hosts host_src and host_dest. host_src is the host where you would run the scp, ssh or rsyn command, irrespective of the direction of the file copy!

1.On host_src, run this command as the user that runs scp/ssh/rsync

$ ssh-keygen -t rsa

This will prompt for a passphrase. Just press the enter key. It'll then generate an identification (private key) and a public key. Do not ever share the private key with anyone! ssh-keygen shows where it saved the public key. This is by default ~/.ssh/id_rsa.pub:

Your public key has been saved in <your_home_dir>/.ssh/id_rsa.pub

2. Transfer the id_rsa.pub file to host_dest by either ftp, scp, rsync or any other method.

3. On host_dest, login as the remote user which you plan to use when you run scp, ssh or rsync on host_src.

4. Copy the contents of id_rsa.pub to ~/.ssh/authorized_keys

$ cat id_rsa.pub >>~/.ssh/authorized_keys
$ chmod 700 ~/.ssh/authorized_keys

If this file does not exists, then the above command will create it. Make sure you remove permission for others to read this file. If its a public key, why prevent others from reading this file? Probably, the owner of the key has distributed it to a few trusted users and has not placed any additional security measures to check if its really a trusted user.

5. Note that ssh by default does not allow root to log in. This has to be explicitly enabled on host_dest. This can be done by editing /etc/ssh/sshd_config and changing the option of PermitRootLogin from no to yes. Don't forget to restart sshd so that it reads the modified config file. Do this only if you want to use the root login.

Well, thats it. Now you can run scp, ssh and rsync on host_src connecting to host_dest and it won't prompt for the password. Note that this will still prompt for the password if you are running the commands on host_dest connecting to host_src. You can reverse the steps above (generate the public key on host_dest and copy it to host_src) and you have a two way setup ready!
I prefer to get a TWO WAY SETUP Done cause sometimes You may need it.

Now, we move to Backups :

All of us know about the MYSQL DUMP Command, so here is the command that you can use directly :

Quote:
mysqldump --user Your_DB_User_Here --password=Your_DB_PASS_here Your_DB_NAME_Here > /path/to/your/backup/directory/backup_YourSiteName__`date '+%m-%d-%Y'`.sql
backup_YourSiteName__`date '+%m-%d-%Y'`.sql
Backup SQL file with website name & date of backups.

So, this command will back up your database to the BACKUP directory & include your Website name & DATE of course you can change the file name as you want to.

Now, we need to take the backups in a way that :
  1. It Does not OVERLOAD the servers.
  2. It does not allow the DB access at all else the backups are ruined.
To solve this, we can close the forums , put the SITE UNDER MAINTENANCE Pages, BUT EVERYTHING failed for me.. I did all this, it gives me backup but of course loads the servers as the SERPS traffic keep on visiting the forums and you can not stop them at all.

So, i finally decided to let the FORUMS BECOME UNREACHABLE to take the backups & its a matter of minutes to back up things.
Now, to automate this thing, i just wrote a small script :

Quote:
cd /path/to/your/backup/directory/
cp -r /home/backup/* /home/backup_backup/
service httpd stop
rm ./*
mysqldump --user Your_DB_User_Here --password=Your_DB_PASS_here Your_DB_NAME_Here > /path/to/your/backup/directory/backup_YourSiteName__`date '+%m-%d-%Y'`.sql
service httpd start
scp *.sql root@xxx.xxx.xxx.xxx:/backup/
Now,what are these steps doing :

1. cd /path/to/your/backup/directory/
Goes to your backup directory.

2. cp -r /home/backup/* /home/backup_backup/
Copies the last done backup to ANOTHER BACKUP DIRECTORY, IN Case of my forums, i prefer this, you can leave it out.

3. service httpd stop
STOPS THE HTTPD Services leaving the sites inaccessible & REDUCING THE SERVER LOADS TO BARE MINIMUM.

4. rm ./*
mysqldump --user Your_DB_User_Here --password=Your_DB_PASS_here Your_DB_NAME_Here > /path/to/your/backup/directory/backup_YourSiteName__`date '+%m-%d-%Y'`.sql

Removes the OLD BACK in that directory, takes a new backup, stores the new backup as a file with DATE.
I DID NOT TAKE A GZIP backup as i have sometimes got CORRUPT BACKUPs with it, SQL files always worked for me, its your choice.

5. service httpd start
Starts the server httpd again, making the sites accessible.

6. scp *.sql root@xxx.xxx.xxx.xxx:/backup/

Now, we did the PASSWORD LOGIN SETUP ABOVE, this command copies your backup securely to the backup servers via SCP , so no worries.

Advantages :

1. The backups are always good & worked fine for me. I never had a faulty backup or corrupt sql file with this method.

2. The server loads NEVER SHOOT SKY HIGH as only MYSQLDUMP works at a time before the servers are open again.

3. Yes, the site becomes inaccessible for a couple of minutes in my case 1 Minute 20 seconds at present. But, its better then to show TIMED OUT ERRORS for 10 minutes in which SERVER LOAD is so high that the site lags badly.

4. Backups are safe in local storage as well as the REMOTE STORAGE.

WITHOUT SSH Login:

How Do I Setup a Cron Job To Backup Data Automatically?

Just add cron job as per your requirements:
Quote:
13 0 * * * /home/admin/bin/ftpbackup.sh >/dev/null 2>&1

Easy to remember format:
* * * * * command to be executed
- - - - -
| | | | |
| | | | ----- Day of week (0 - 7) (Sunday=0 or 7)
| | | ------- Month (1 - 12)
| | --------- Day of month (1 - 31)
| ----------- Hour (0 - 23)
------------- Minute (0 - 59)



By default the output of a command or a script (if any produced), will be email to your local email account. To stop receiving email output from crontab you need to append >/dev/null 2>&1.
For example:
0 3 * * * /root/backup.sh >/dev/null 2>&1

To mail output to particular email account let us say support@buyrdp.com you need to define MAILTO variable to your cron job:
MAILTO="support@buyrdp.com"
0 3 * * * /root/backup.sh >/dev/null 2>&1

How can i use the FTP CLIENT on this ..??

If you guys want to use FTP Clients for the back up & do not have ROOT access to set up SCP access, you can use this one :

1. Install the NCFTP Client.

Use yum or apt-get package manager to install ftp client called ncftp.

2. Replace the command for SCP with NCFTP :

Use ncftp command instead of SCP :

Code:
ncftp -u"your_ftp_user" -p"your_ftp_password" your_FTP_server
cd /path/to/your/backup/directory
mput *
quit
This will login via NCFTP Client , then move to your backup directory & use the MPUT command to dump the backup to the FTP & on completion will quit the session.


And about GRACEFULLY CLOSING THE Vbulletin :

If your VB gets around medium traffic , it can be tried but I will always prefer you to SHUT DOWN the HTTPD for just few minutes.

I know it will make the website inaccessible but will ENSURE THAT Backups are complete and can be SAFELY restored without any errors , as you know when the dump stats the DATABASE is still in use for one or other queries & this also lags the backups & may lead to corrupt backups.

The script worked for me , so i just shared with you.
Of course, you can take my help in anything above i will gladly help you with things. But I WONT BE RESPONSIBLE FOR ANY DATA LOSS occurring by application of the above things.

I hope you will use the script and if you do, I WOULD REQUEST A WORD "THANKS" in this post.

I also provide professional VBULLETIN HOSTING & Solutions, now backups too.
You can reach me at : support@buyrdp.com

Enjoy with VB.
Regards
Reply With Quote
  #2  
Old 01-06-2011, 10:13 PM
MrEyes MrEyes is offline
 
Join Date: Nov 2004
Posts: 380
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Great article, many thanks for posting.

3 comments though...

1) Many admins can't afford a secondary backup server, however many dedicated hosts provide a free FTP backup server - it might be worth adding some details of performing an FTP copy of the backup.

2) Rather than shutting down the webserver it is possible to gracefully close vBulletin, have a look at this:

https://vborg.vbsupport.ru/showthread.php?t=254529

3) It is also probably worth detailing how to setup the backup script as a cron job
Reply With Quote
  #3  
Old 01-07-2011, 04:58 AM
intensecool intensecool is offline
 
Join Date: Feb 2007
Posts: 69
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Quote:
Originally Posted by MrEyes View Post
Great article, many thanks for posting.

3 comments though...

1) Many admins can't afford a secondary backup server, however many dedicated hosts provide a free FTP backup server - it might be worth adding some details of performing an FTP copy of the backup.

2) Rather than shutting down the webserver it is possible to gracefully close vBulletin, have a look at this:

https://vborg.vbsupport.ru/showthread.php?t=254529

3) It is also probably worth detailing how to setup the backup script as a cron job

Thanks for appreciating this article.
I have done editing as per needed.

1. Added the FTP scripts on the post.

2. Shutting the WEB SERVER is a VERY GOOD option for those who get medium to high traffic. Loosing 2 minutes of audience is always better then getting a corrupt backup.
And it depends on the guy taking backups, you can always remove it.

3. Cron have been added and explained.
Reply With Quote
  #4  
Old 01-07-2011, 11:36 AM
TheLastSuperman's Avatar
TheLastSuperman TheLastSuperman is offline
Senior Member
  
Join Date: Sep 2008
Location: North Carolina
Posts: 5,844
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Nice Article, 5 Stars .

And my 2 cents? Always close the board on a backup, always no matter what - we have too many fearless "coders" out here who think they know what's best - NO! Common sense says don't have anyone interacting with the database while your making a backup for various reasons.
Reply With Quote
  #5  
Old 01-07-2011, 12:53 PM
intensecool intensecool is offline
 
Join Date: Feb 2007
Posts: 69
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Quote:
Originally Posted by TheLastSuperman View Post
Nice Article, 5 Stars .

And my 2 cents? Always close the board on a backup, always no matter what - we have too many fearless "coders" out here who think they know what's best - NO! Common sense says don't have anyone interacting with the database while your making a backup for various reasons.

Agreed to you , Thanks a lot for this appreciating 5 Stars

And yes, Stopping total Interaction with the database is always USEFUL :

1. It gets you a complete working backup.
2. It never lets you OVERLOAD the server which means your website is back online within few seconds of SWITCHING HTTPD service back on as the server load never goes high up while backups are on.

I can not find any mod which can do this & i dont think it can be really done except what i did above for my method.
Thanks a lot for all appreciation, hope the guys using this post a word of Thanks here.
Reply With Quote
  #6  
Old 01-21-2011, 10:17 AM
sross sross is offline
 
Join Date: Mar 2004
Posts: 355
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Sounds cool. I currently run a script called automysqlbackup.sh (http://sourceforge.net/projects/automysqlbackup/), my db is 570meg gzip. I then rsync my site to a local ubuntu box which in turn brings down the dumped db. I extract the db's from time to time and they extract without error. I also have used the dbs to build dev servers. I have not run into any problems so far. Do you think this is ok? I am also wondering about stopping http, even if you do this are there still not sessions attached to the mysql db? Does stopping http nuke all connections to the db? I also run lightspeed webserver so does my command to stop that before my backup differ? Thanks!
Reply With Quote
  #7  
Old 01-21-2011, 11:12 AM
Angel-Wings's Avatar
Angel-Wings Angel-Wings is offline
 
Join Date: Sep 2007
Posts: 206
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Or - more easy - install MySQL on your local comp and setup replica. Then power it on, wait until it's in sync, lock the tables (e.g. stop the slave), do the backup and start the slave again.

Result is you won't need a root access for SSH (rather dangerous to allow this), your SSH keys still use a passphrase (required to have) and your backups will be clean as well.

And nope - your local MySQL don't need to run all the time
Reply With Quote
  #8  
Old 01-23-2011, 06:08 AM
intensecool intensecool is offline
 
Join Date: Feb 2007
Posts: 69
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Quote:
Originally Posted by sross View Post
Sounds cool. I currently run a script called automysqlbackup.sh (http://sourceforge.net/projects/automysqlbackup/), my db is 570meg gzip. I then rsync my site to a local ubuntu box which in turn brings down the dumped db. I extract the db's from time to time and they extract without error. I also have used the dbs to build dev servers. I have not run into any problems so far. Do you think this is ok? I am also wondering about stopping http, even if you do this are there still not sessions attached to the mysql db? Does stopping http nuke all connections to the db? I also run lightspeed webserver so does my command to stop that before my backup differ? Thanks!
To me ,
The method is cool, i have used that one lots of times for backups.
But there is a difference with your DB of 570MB & 5GB.

Mine are over 6.5 GB now & when you try to dump that big server loads shoots up high , i have seen loads of over 50 at times.
This is why we need solution to backup without shooting that load which of course messes up DB , lags the server and hangs your website for a longer time period , while my method safely takes the backup, then does an scp that backup to remote server and turns your server back on.
There are several methods to dump small DB's but for big ones you need something like that one.

Quote:
Originally Posted by Angel-Wings View Post
Or - more easy - install MySQL on your local comp and setup replica. Then power it on, wait until it's in sync, lock the tables (e.g. stop the slave), do the backup and start the slave again.

Result is you won't need a root access for SSH (rather dangerous to allow this), your SSH keys still use a passphrase (required to have) and your backups will be clean as well.

And nope - your local MySQL don't need to run all the time
Think you can not do it on automation , you need to manually do it everytime.
For my method, its all automated.
Reply With Quote
  #9  
Old 01-23-2011, 05:22 PM
Angel-Wings's Avatar
Angel-Wings Angel-Wings is offline
 
Join Date: Sep 2007
Posts: 206
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Hmm - dunno what's so hard typing "net start mysql" or "/etc/rc.d/mysql start" on a command line. Automated SSH clients like Tunnelier or some SSH Agent can even automate the SSH login process while giving some more kind of security than using keys without a passphrase.

And - advantage with Replica is you don't need to lock tables on the server so there's no need to set the board offline. With some very easy client side script it all can be done without much user interaction
Reply With Quote
  #10  
Old 02-09-2011, 07:21 PM
abumohamed abumohamed is offline
 
Join Date: Mar 2005
Location: Saudi Arabia
Posts: 16
Благодарил(а): 0 раз(а)
Поблагодарили: 0 раз(а) в 0 сообщениях
Default
Great article
Reply With Quote
Reply
Page 1 of 2 1 2

« Previous Thread | Next Thread »
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT. The time now is 03:01 PM.

Contact Us - Archive - Top

Powered by vBulletin® Version 3.8.12 by vBS
Copyright ©2000 - 2024, vBulletin Solutions Inc.
X vBulletin 3.8.12 by vBS Debug Information
  • Page Generation 0.12569 seconds
  • Memory Usage 2,314KB
  • Queries Executed 23 (?)
More Information
Template Usage:
  • (1)SHOWTHREAD
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_header_end
  • (1)ad_header_logo
  • (1)ad_navbar_below
  • (1)ad_showthread_beforeqr
  • (1)bbcode_code
  • (8)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)modsystem_article
  • (1)navbar
  • (4)navbar_link
  • (120)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (1)pagenav_pagelink
  • (10)post_thanks_box
  • (10)post_thanks_button
  • (1)post_thanks_javascript
  • (1)post_thanks_navbar_search
  • (10)post_thanks_postbit_info
  • (9)postbit
  • (10)postbit_onlinestatus
  • (10)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 
Phrase Groups Available:
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files:
  • ./showthread.php
  • ./global.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/class_hook.php
  • ./includes/modsystem_functions.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_post_thanks.php 
Hooks Called:
  • init_startup
  • init_startup_session_setup_start
  • init_startup_session_setup_complete
  • cache_permissions
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • style_fetch
  • cache_templates
  • global_start
  • parse_templates
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • forumjump
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • post_thanks_function_post_thanks_off_start
  • post_thanks_function_post_thanks_off_end
  • post_thanks_function_fetch_thanks_start
  • post_thanks_function_fetch_thanks_end
  • post_thanks_function_thanked_already_start
  • post_thanks_function_thanked_already_end
  • fetch_musername
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • post_thanks_function_can_thank_this_post_start
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • showthread_complete