posts tagged as backup

ssh and rsync

Monday, 29 March 2010 7:42 am by noel
posted in tech | tags: , , ,

i use rsync to backup files within the local network and also through the internet. rsync, by default, does not encrypt the data it transmits so to backup files via the internet one has to encrypt the data or the port that the data travels in using another software. for this purpose i use ssh (secure shell) to create an encrypted “tunnel” between the transmitting and receiving computers.

the conventions i’ll be using: remote means the server where i will be copying files from and local means the server where i will be copying files to. local is also where the rsync backup script is located and initiated.

my assumptions are that both remote and local servers are running linux and both have rsync and openssh installed.

when initiating a backup run from the local server, the remote server would normally ask for a password. this is obviously not good especially when i need to schedule unattended backup runs at odd hours of the day (or night). so for the script not to ask a password i need to generate a public/private pair of keys on the local server to be used with ssh.

to generate a public/private key pair, log in to the console in the local server:

$ ssh-keygen -t dsa -b 2048 -f /home/localbackupuser/local-rsync-key
Generating public/private dsa key pair.
Enter passphrase (empty for no passphrase): [press enter here]
Enter same passphrase again: [press enter here]
Your identification has been saved in /home/localbackupuser/local-rsync-key.
Your public key has been saved in /home/localbackupuser/local-rsync-key.pub.
The key fingerprint is:
94:87:e5:c3:d0:06:e4:09:3a:76:a2:d2:d7:9b:2e:cc localbackupuser@local

so now we have keys to use to authenticate between the local and remote servers. we now have to copy the contents of the local-rsync-key.pub into the authorized_keys file of the remote server (/home/remoteuser/.ssh/authorized_keys)

i would normally mount the remote server’s drive using sshfs (secure shell filesystem) and edit the authorized_keys file as if it was in my workstation. another way to do this is via a remote console.

for added security, you can limit the computer(s) connecting to the remote server by specifying the ip address of the local server along with the contents of the public key generated above (details here). this would be very useful if the ip address of the local server doesn’t change. unfortunately mine does.

next step is to test the backup script on the local server via ssh. if the backup script starts syncing with the remote server then all that is left to do is add and entry in crontab to automatically start the backup at the time you specify.

links:
using rsync and ssh
sshfs

  • Share/Bookmark
-->

no comments

scripting rsync

Tuesday, 9 February 2010 6:45 am by noel
posted in tech | tags: , ,

here’s the script i use on secure backup runs using rsync—with a little modification to the names and ip addresses for security reasons—but everything else is essentially the same. i have to run this through cron because the server is headless—no monitor. i just access it via the ssh console or browser from another computer.

#!/bin/sh

DATETMP=`date +%Y.%m.%d`
RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
# the private ssh key of the local computer
KEY=/rsyncuser/.ssh/id_local
RHOST=remote.ip.addr.ess
RPATH=/remotedata/
LPATH=/localdata/
LOGFILE=/rsyncuser/rlog.$DATETMP.log
# contains the file extensions of files to be excluded from the backup
EXCLUDES=/rsyncuser/localexcludes
OPTS="--exclude-from=$EXCLUDES"

# check if rsync is already running
RUN=`ps x | grep rsync | grep -v grep | wc -l`
if [ "$RUN" -gt 0 ]; then
echo rsync already running
exit 1
fi

$RSYNC -avz -e "$SSH -i $KEY" $OPTS $RHOST:$RPATH $LPATH >> $LOGFILE

i leave an entry in crontab to run the script once each day. sometimes a backup run goes longer than 24 hours so i needed to check if rsync is already running in the server. if the script doesn’t check, it will run another instance of the script and would slow down the server or, worse, brings it down completely.

i have to encrypt all data that gets transferred between the two computers via ssh just in case a naughty third party is “listening in”. i use 2048-bit encryption. processing the data—encrypt at remote; decrypt at local—is a little slower but i am more confident that the data will be safe from eavesdroppers.

i use the exclude-from option to exclude files that shouldn’t be backed up—music and movies—or else the backup will take too long especially on just a dsl line.

if you notice anything wrong with the script, please leave a comment.

note: this is repost from my old blog.

  • Share/Bookmark
-->

1 comment

restoring a wordpress database

Saturday, 26 July 2008 11:39 pm by noel
posted in tech | tags: , , , ,

every wordpress blogger, me included, assumes that everything in his/her blog will go right or at least the blogger hopes and prays that things will go right. most times, it does. but occasionally, the sun burps a weird flare at the exact time a cosmic solar flare concentrator is pointed at your server and your wp_options table gets corrupt and marked “in use” so nothing else can touch it with the exception of dropping it. “dropping a table” in sql terms is equivalent to deleting a table of the database — a database is composed of at least one table.

anyway, one of the tables got blitzed — as my brother would call it — and it had to be replaced with a backup. unfortunately, the last backup was last july 16 — the table got blitzed somewhere around july 24.

one documentation i read was that i had to drop or delete all the wordpress tables in the database and then recreate and repopulate them using the backup. this made me a little queasy simply because i’m deleting the guts of a database and there’s no undo button to be seen within a 10 kilometer radius. so yes, that fact alone got my heart pumping a bit. but after the ‘x’ button was pushed there was this calmness that usually preceded a storm — i have to perform a restore. now.

the dang server does a 500 whenever i tried uploading the .sql backup. error 500 is a server error. uh oh. this is not good. uncompressed, the backup file is about 3.2 megabytes and that’s just text. my guess is that the server is timing out. the suggested solution was to copy-paste the .sql backup file in smaller parts — its just a text file so you can open it in a text editor — into the sql command window in phpmyadmin. its seriously tedious but it worked.

i eventually got the whole backup uploaded in about an hour. that was good.

what was bad was that a week’s worth of posts and comments were lost. some were recovered through google cache but there are still some that were missing. we do have the wp-db-backup plugin installed and enabled but unfortunately, scheduled e-mailing of the backup to a predefined e-mail address does not work. i’m not sure if the problem stems from my using a subdomain instead of a subdirectory for my blog. i don’t even get notices that there’s a comment awaiting moderation even though my wordpress is set to do so. i know its not the fault of the plugin because i have the same thing installed in another wordpress blog with the same host and it is installed in a subdirectory and not a subdomain and that setup works perfectly. so far its a toss up between wordpress being the culprit or my hosting provider — yahoo. i still have to do some more investigating.

in the meantime i can still use the plugin to perform backups for download to my computer. very manual and inelegant, i know, but at least that part works.

  • Share/Bookmark
-->

no comments