Guy Rutenberg » Bash

Improved FTP Backup for WordPress

Guy — Sun, 28 Feb 2010 06:38:44 +0000

This script backups both the database and files of a WordPress blog into a remote FTP server (while keeping a local copy). It’s an update of my WordPress Backup to FTP script. The main changes are auto-detecting database settings and better support for caching plugins (specifically WP-Cache). The new version makes it easier to backup multiple WordPress blogs to the same FTP server.

Usage is pretty simple after a short initial configuration. First, save the the script and make it executable:

chmod +x wp-backup

(assuming you saved it under the name wp-backup). After saving it edit the file with your favorite editor and set the 5 configuration variable to whatever is appropriate for you. BACKUP_DIR is the folder to save the local backups to. FTP_HOST, FTP_USER, FTP_PASS control the FTP host, username and password, respectively, for the remote backup server. FTP_BACKUP_DIR sets the folder on the FTP server to save the remote backup to.

Now that the initial configuration is done, all you need to do is execute the script and give the path to the blog as an argument. For example:

./wp-backup /home/someuser/myblog

And that it, the script will backup your files (excluding cache) and database to both a local and remote locations. This allows using the same script to backup multiple WordPress blogs, unlike the previous script which had to be modified for each blog.

And now the script itself:

#!/bin/bash
 
# Copyright 2008, 2010 Guy Rutenberg 
# WordPress FTP backup 2.0
#
# Easily backup wordpress instances via ftp.
#
# Change Log:
# ===========
# 2.0:
#  - Auto-detect database settings.
#  - Exclude cache data from backups.
 
BACKUP_DIR=
FTP_HOST=
FTP_USER=
FTP_PASS=
FTP_BACKUP_DIR=
 
# end of configuration - you probably don't need to touch anything bellow
 
PROG=`basename "$0"`
print_usage () {
    echo "USAGE: ${PROG} [options] BLOG_ROOT"
    echo "Backup a WordPress blog"
}
 
print_help ()  {
    print_usage
    cat << EOF
 
Options:
    -h, --help          show this help message and exit
 
EOF
}
 
TEMP=`getopt -o h --long help -n "$PROG" -- "$@"`
if (($?)); then
    print_usage
    exit 1
fi
 
eval set -- "$TEMP"
 
while true ; do
    case "$1" in
        -h|--help) print_help; exit ;;
        --) shift; break;;
    esac
done
 
if [ -z "$1" ]
then
 print_usage > /dev/stderr
 exit 1
fi
 
BLOG_DIR=$1
DB_NAME=`echo "\"${BLOG_DIR}/wp-config.php\"); echo DB_NAME;" | php`
DB_USER=`echo "\"${BLOG_DIR}/wp-config.php\"); echo DB_USER;" | php`
DB_PASS=`echo "\"${BLOG_DIR}/wp-config.php\"); echo DB_PASSWORD;" | php`
DB_HOST=`echo "\"${BLOG_DIR}/wp-config.php\"); echo DB_HOST;" | php`
 
BLOG_DIR=`dirname "$BLOG_DIR"`/`basename "$BLOG_DIR"`
BACKUP_DIR=`dirname "$BACKUP_DIR"`/`basename "$BACKUP_DIR"`
 
echo -n "dumping database... "
DUMP_NAME=${DB_NAME}-$(date +%Y%m%d).sql.bz2
mysqldump --user=${DB_USER} --password=${DB_PASS} --host=${DB_HOST} \
 --databases ${DB_NAME} \
 | bzip2 -c > ${BACKUP_DIR}/${DUMP_NAME}
if [ "$?" -ne "0" ]; then
	echo "failed!"
	exit 1
fi
echo "done"
 
echo -n "Creating tarball... "
TAR_NAME=${BLOG_DIR##*/}-$(date +%Y%m%d).tar.bz2
tar -cjf ${BACKUP_DIR}/${BLOG_DIR##*/}-$(date +%Y%m%d).tar.bz2 --exclude cache ${BLOG_DIR}
if [ "$?" -ne "0" ]; then
	echo "failed!"
	exit 2
fi
echo "done"
 
echo -n "Uploading SQL dump and tarball to FTP... "
lftp -u ${FTP_USER},${FTP_PASS} ${FTP_HOST} <
if [ "$?" -ne "0" ]; then
	echo "failed!"
	exit 3
fi
echo "done"

Batch Renaming Using sed

Guy — Mon, 12 Jan 2009 13:10:22 +0000

I was reorganizing my music library and decided to change the naming convention I’ve used. This task is just asking to be automated. Since the filename change could be described using regular expression, I looked for a way to use sed for the renaming process.

The files I had, had the following pattern as filename ARTIST – SONG – TRACK – ALBUM

James Brown - I Got You (I Feel Good).ogg  - 01 - Classic James Brown

I wanted to rename it to ARTIST – ALBUM – TRACK – NAME

James Brown - Classic James Brown - 01 - I Got You (I Feel Good).ogg

Describing the change as a sed program is easy:

s/\(.*\) - \(.*\) - \(.*\) - \(.*\).ogg/\1 - \4 - \3 - \2.ogg/

Now all that has to be done is to pass each filename to mv and pass it again after it went through the sed script. This could be done like this:

for i in *; do
  mv "$i" "`echo $i | sed "s/\(.*\) - \(.*\) - \(.*\) - \(.*\).ogg/\1 - \4 - \3 - \2.ogg/"`";
done

The important part is the

`echo $i | sed "s/\(.*\) - \(.*\) - \(.*\) - \(.*\).ogg/\1 - \4 - \3 - \2.ogg/"`

which pipes the filename to sed and returns it as an argument for mv.

To see what renaming will be done one can alter a bit the above command, and get

for i in *; do
  echo "$i" "->" "`echo $i | sed "s/\(.*\) - \(.*\) - \(.*\) - \(.*\).ogg/\1 - \4 - \3 - \2.ogg/"`";
done

While will effectively print a list of lines of the form oldname -> newname.

Of course this technique isn’t limited to the renaming I’ve done. By changing the pattern given to sed, one can do any kind of renaming that can be described as a regular expression replacement. Also one can change the globbing (the *) in the for loop to operate only on specific files, that match a given pattern, in the directory instead of all of them.

WordPress Backup to FTP

Guy — Tue, 06 Jan 2009 13:46:33 +0000

Update: A newer version of the script is available.

This script allows you to easily backup your WordPress blog to an FTP server. It’s actually a modification of my Wordpress Backup to Amazon S3 Script, but instead of saving the backup to Amazon S3 it uploads it to an FTP server. Another update is that now the SQL dump includes the database creation instructions so you don’t need to create it manually before restoring from the backup.

Although I’ve written it with WordPress in mind (to creates backups of my blog), it isn’t WordPress specific. It can be used to backup any website that consists of a MySQL database and files. I’ve successfully used it to backup MediaWiki installation.

#!/bin/bash
 
# (C) 2008 Guy Rutenberg
# This is a script that creates backups of my blog.
 
DB_NAME=guy_blog
DB_USER=******
DB_PASS=******
DB_HOST=localhost
 
BLOG_DIR=/home/guyru/guyrutenberg.com/
BACKUP_DIR=/home/guyru/backups/
 
FTP_HOST=88.198.42.4
FTP_USER=******
FTP_PASS=******
FTP_BACKUP_DIR=guyru/
 
# end of configuration - you probably don't need to touch anything bellow
BLOG_DIR=`dirname "$BLOG_DIR"`/`basename "$BLOG_DIR"`
BACKUP_DIR=`dirname "$BACKUP_DIR"`/`basename "$BACKUP_DIR"`
 
echo -n "dumping database... "
DUMP_NAME=${DB_NAME}-$(date +%Y%m%d).sql.bz2
mysqldump --user=${DB_USER} --password=${DB_PASS} --host=${DB_HOST} \
 --databases ${DB_NAME} \
 | bzip2 -c > ${BACKUP_DIR}/${DUMP_NAME}
if [ "$?" -ne "0" ]; then
	echo "failed!"
	exit 1
fi
echo "done"
 
echo -n "Creating tarball... "
TAR_NAME=${BLOG_DIR##*/}-$(date +%Y%m%d).tar.bz2
tar -cjf ${BACKUP_DIR}/${BLOG_DIR##*/}-$(date +%Y%m%d).tar.bz2 ${BLOG_DIR}
if [ "$?" -ne "0" ]; then
	echo "failed!"
	exit 2
fi
echo "done"
 
echo -n "Uploading SQL dump and tarball to FTP... "
lftp -u ${FTP_USER},${FTP_PASS} ${FTP_HOST} <
if [ "$?" -ne "0" ]; then
	echo "failed!"
	exit 3
fi
echo "done"

The configuration is done using the variables in the beginning of the script (until the “end of configuration” line). They are pretty self explanatory, and you should adjust them for your site before using the script. If there are variables you don’t understand what they do, see the post about the previous script, and don’t hesitate to comment.

Backup a SourceForge hosted SVN repository – sf-svn-backup

Guy — Fri, 12 Dec 2008 17:49:30 +0000

SourceForge urges their users to backup the code repositories of their projects. As I have several projects hosted with SourceForge, I should do it too. Making the backups isn’t complicated at all, but because it isn’t automated properly, I’ve been lazy with it.

sf-svn-backup was written in order to simply automate the process. The script is pretty simple to use, just pass as the first argument the project name and the script will write down to stdout the dump file.

For example:

sf-svn-backup openyahtzee > openyahtzee.dump

The project name should be it’s UNIX name (e.g. openyahtzee and not Open Yahtzee). Because the script writes the dump file directly to stdout it’s easy to pipe the output first through a compression program such as gzip to create compressed SVN dump files.

tarsum – Calculate Checksum for Files inside Tar Archive

Guy — Fri, 24 Oct 2008 20:02:05 +0000

Update: I’ve released tarsum-0.2, a new version of tarsum.

Some time ago, I got back a hard disk back from data recovery. One of the annoying issues I encountered with the recovered data was corrupted files. Some files looked like they were recovered successfully but their content was corrupted. The ones that were configuration files, where usually easy to detect, as it raised errors in programs that tried to use them. But when such error occurs in some general text file, (or inside the data of an SQL dump), the file may seem correctly fine unless closely inspected.

I have an habit of storing old backups on CDs (they are initially made to online storage), I do it in order to reduce backup costs. But the recovered/corrupted data issue raised some concerns about my ability to recover using this disks. Assuming that I have a disk failure, and I couldn’t recover from my online backups for reason, how can I check the integrity of my CD backups?

Only storing and comparing hash signature for the whole archive, is almost useless. It allows you to validate whether all the files are probably fine, but it can’t tell apart one corrupted file in the archive from a completed corrupted archive. My idea was to calculate checksum (hash) for each file in the data and store the signature in a way that would allow me to see which individual files are corrupted.

This is where tarsum comes to the rescue. As it’s name applies it calculate checksum for each file in the archive. You can download tarsum from here.

Using tarsum is pretty straight forward.

tarsum backup.tar > backup.tar.md5

Calculates the MD5 checksums of the files. You can specify other hashes as well, by passing a tool that calculates it (it must work like md5sum).

tarsum --checksum=sha256sum backup.tar > backup.tar.sha256

To verify the integrity of the files inside the archive we use the diff command:

tarsum backup.tar | diff backup.tar.md5 -

where backup.tar.md5 is the original signature file we created. This is possible because the signatures are sorted alphabetically by the file name inside the archive, so it the order of the files is always the same.

Note that if you use an updated version of GNU tar, tarsum can also operate directly on compressed archives (e.g. tar.bz2, tar.gz).

s3backup – Easy backups of Folders to Amazon S3

Guy — Tue, 21 Oct 2008 19:24:49 +0000

This is an updated version of my previous backups script – Backup Directories to Amazon S3 Script. The new script works much better and safer. Unlike the old script, the new one creates the tarballs in a temporary file under /tmp, and allows more control over the backup process.

#! /bin/bash
 
# Copyright (C) 2008 by Guy Rutenberg
 
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the
# Free Software Foundation, Inc.,
# 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
 
PROG=s3backup
DESCRIPTION="Backup directories to Amazon S3"
SYNOPSIS="${PROG} [options] DIR BUCKET"
VERSION=0.2
 
print_usage () {
    echo "Usage: ${SYNOPSIS}"
    echo "Save a backup of DIR to Amazon S3 bucket BUCKET"
}
print_help ()  {
    print_usage
    cat << EOF
 
Options:
    -h, --help          show this help message and exit
    --version           show program's version number and exit
    -f, --full-path     use the path to dir in archive instead of only dir
    -v, --verbose       be verbose about progress
 
Example:
    s3backup ~/Documents guyru-backup/
EOF
}
 
print_version () {
    echo "$PROG $VERSION"
    echo "Copyright (C) 2008 Guy Rutenberg "
}
 
# print only if verbose
vecho () {
    if (( "$VERBOSE" >= "$1" )) ; then
        echo "$2"
    fi
}
VERBOSE=0
TEMP=`getopt -o hvf --long help,version,verbose,full-path -n '${PROG}' -- "$@"`
if [ $? != 0 ] ; then
    print_usage
    exit 1
fi
 
eval set -- "$TEMP"
 
while true ; do
    case "$1" in
        -h|--help) print_help; exit ;;
        --version) print_version; exit ;;
        -f|--full-path) FULL_PATH=1; shift ;; 
        -v|--verbose) let VERBOSE++ ; shift ;;
        --) shift; break;;
    esac
done
 
# no directory passed
if [ -z "$1" ]; then
    echo "$PROG: missing DIR"
    echo ""
    print_usage
    exit 1
fi
 
# no bucket passed
if [ -z "$2" ]; then
    echo "$PROG: missing BUCKET"
    echo ""
    print_usage
    exit 2
fi
BUCKET=$2
 
 
DIR=$(basename "$1")
DIR_PATH=$(dirname "$1")
 
DATE=$(date +%Y%m%d)
 
 
if [ -z "$FULL_PATH" ] ; then
    cd "$DIR_PATH"
    TAR_PATH="$DIR"
else
    TAR_PATH="$1"
fi
 
vecho 1 "Creating tarball"
TARNAME=`mktemp`
TAR_OPTIONS="-c"
if (( $VERBOSE > 0 )); then
    TAR_OPTIONS+="v"
fi
tar $TAR_OPTIONS "$TAR_PATH" | bzip2 -9 -c > "${TARNAME}"
 
vecho 1 "Sending to Amazon S3"
S3_OPTIONS=""
if (( $VERBOSE > 0 )); then
    S3_OPTIONS+="v"
fi
s3cmd put $S3_OPTIONS "$TARNAME" "s3://${BUCKET}$DIR-$DATE.tar.bz2"
 
vecho 1 "Deleting intermediate files"
rm "$TARNAME"
 
# vim:sw=4:softtabstop=4:expandtab

The script can also be downloaded from s3backup.gz.

Script usage is pretty straight forward, just pass the folder to backup as the first argument to the script, and an S3 bucket name, to which the backup will be saved as the second. You can also consult s3backup --help.

Note that you can add a prefix to your backups by adding it to the S3 bucket. For example

s3backup ~/Documents bucketname/prefix/

Wordpress Backup to Amazon S3 Script

Guy — Sat, 11 Oct 2008 09:19:22 +0000

This is an updated version of my Wordpress Backup Script. The new version basically does the same thing: backup up a wordpress blog (actually any site that consists of files and a MySQL database). The new thing about the script is that instead of only saving the backup locally, it also uploads it to Amazon S3.

#!/bin/bash
 
# (C) 2008 Guy Rutenberg
# This is a script that creates backups of my blog.
 
DB_NAME=database_name
DB_USER=database_user
DB_PASS=****
DB_HOST=localhost
 
#no trailing slash
BLOG_DIR=/path/to/blog
BACKUP_DIR=/path/to/backups
S3BUCKET=s3://bucket-name/
 
# end of configuration - you probably don't need to touch anything bellow
DUMP_NAME=${DB_NAME}-$(date +%Y%m%d).sql.bz2
 
echo -n "dumping database... "
mysqldump --user=${DB_USER} --password=${DB_PASS} --host=${DB_HOST} ${DB_NAME} \
 | bzip2 -c > ${BACKUP_DIR}/${DUMP_NAME}
if [ "$?" -ne "0" ]; then
	echo "failed!"
	exit 1
fi
echo "done"
 
TAR_NAME=${BLOG_DIR##*/}-$(date +%Y%m%d).tar.bz2
echo -n "Creating tarball... "
tar -cjf ${BACKUP_DIR}/${BLOG_DIR##*/}-$(date +%Y%m%d).tar.bz2 ${BLOG_DIR}
if [ "$?" -ne "0" ]; then
	echo "failed!"
	exit 1
fi
echo "done"
 
echo -n "Uploading SQL dump to Amazon S3... "
s3cmd put ${BACKUP_DIR}/${DUMP_NAME} ${S3BUCKET}${DUMP_NAME}
if [ "$?" -ne "0" ]; then
	echo "failed!"
	exit 1
fi
echo "done"
 
echo -n "Uploading tarball to Amazon S3... "
s3cmd put ${BACKUP_DIR}/${TAR_NAME} ${S3BUCKET}${TAR_NAME}
if [ "$?" -ne "0" ]; then
	echo "failed!"
	exit 1
fi
echo "done"

As with the older Amazon S3 backup script, this one also required you to install the s3cmd package.

The first six variables:

DB_NAME
DB_USER
DB_PASS
DB_HOST
BLOG_DIR
BACKUP_DIR

Are the pretty self explanatory and have the same meaning as in the old script.

The script has one new variable: S3BUCKET. This should be set to the S3 bucket to which the backups should be saved. If you want to add some prefix to the backups files on S3 this is the place.

The script keeps the backups both in the local backup directory and the Amazon S3.

WordPress Backup Script

Guy — Wed, 07 May 2008 08:15:52 +0000

This is a small script I’ve written to automate my server-side backups of my blogs. It creates a backup of both the database and the actual WordPress files.

#!/bin/bash
 
# (C) 2008 Guy Rutenberg - http://www.guyrutenberg.com
# This is a script that creates backups of blogs.
 
DB_NAME=guy_blog
DB_USER=guy_root
DB_PASS=****
DB_HOST=localhost
 
#no trailing slash
BLOG_DIR=/home/guyru/guyrutenberg.com
BACKUP_DIR=/home/guyru/backups
 
 
echo -n "dumping database... "
mysqldump --user=${DB_USER} --password=${DB_PASS} --host=${DB_HOST} ${DB_NAME} \
 | bzip2 -c > ${BACKUP_DIR}/${DB_NAME}-$(date +%Y%m%d).sql.bz2
if [ "$?" -ne "0" ]; then
	echo -e "\nmysqldump failed!"
	exit 1
fi
echo "done"
 
 
echo -n "Creating tarball... "
tar -cjf ${BACKUP_DIR}/${BLOG_DIR##*/}-$(date +%Y%m%d).tar.bz2 ${BLOG_DIR}
if [ "$?" -ne "0" ]; then
	echo -e "\ntarball creation failed!"
	exit 1
fi
echo "done"

Copy the above code and save it to a file called backup-blog. Before you can use it there are some configurations to be done. At the beginning of the script (right after the copyright notice), you will find some variable you need to set.

The first set of variables is the database configuration. Set DB_NAME to the name of your database, DB_USER to your database username, DB_PASS to the user’s password and DB_HOST to the hostname of the database server. For most users the DB_HOST should be set for localhost (but some users need to specify a different value here).

The next variable is BLOG_DIR. Set it to the full path of the directory where you installed WordPress. BACKUP_DIR should be set to the full path of the directory you to keep the backups in, make sure the user which will run the script has write access to it. Notice that both paths shouldn’t end with a trailing ‘/’ (slash).

Now that the configuration is over, make the script executable:
chmod +x backup-blog
Now you can test that everything works ok by running the script: ./backup-blog. If the script failed for some reason make sure your configuration is correct.

The next step is to automate the backup process using cron. Run crontab -e to add an entry to the backup script. Append the following line to the crontab:
30 1 * * 4 /path/to/backup-blog
This will run the script automatically for you once a week, on 01:30 AM every Friday. Be sure to check up the backup folder from time to time to make sure everything is running smooth.

Note that while the script was originally intended to create backups of WordPress blogs it can also be used to backup any kind of website that needs both files and database backups, like MediaWiki, Drupal, or almost any kind of CMS and blogging platform.

Convert PNMs to DjVu

Guy — Tue, 11 Mar 2008 20:47:38 +0000

I’ve decided to scan some notebooks. After researching a bit, I’ve decided to use DjVu (instead of PDF which I normally use). I’ve chose to use DjVu because it offered great quality with very good compression rate (~26KB per page) in lineart (black and white).

While XSane can natively save a multipage project into PDF it can’t do so for DjVu. So, the solution is to use the PNMs generated by XSane and convert them using the command line tools offered by DjVuLibre to bundle them together to a DjVu file. As you can guess doing this manually is pretty hard work. To make this task easier I’ve written a small bash script to automate the process.

#!/bin/bash
# pnm2djvu 
#(C) 2008 Guy Rutenberg
#pnm2djvu - A small script that takes a directory with PNMs and converts them to 
#monochrome (bitonal) djvu file with the PNMs ordered alphabetically.
 
if [ -z "$1" ]; then
	echo "Directory of PNMs reguired"
	exit
fi
if [ -z "$2" ]; then
	echo "filename for output required"
	exit
fi
 
pnms=`ls "$1/" | grep ".pnm$"`
pagenum=1
for i in ${pnms}; do
	echo "Processing page number ${pagenum}"
	cjb2 "$1/$i"  "page$i.djvu"
	if [ -f "$2" ]; then
		djvm -i "$2" "page$i.djvu"
	else
		djvm -c "$2" "page$i.djvu"
	fi
	rm "page$i.djvu"
	pagenum=`expr $pagenum + 1`
done;

Using the script is pretty simple. Pass as the first argument a path to a directory holding the PNMs (it can also be “.”), and the name of the output DjVu file. If the file already exist the script will append all the PNMs to it, otherwise it will just create a new one.

I’ve released this script following the “release early” practice in the open-source community. This script is far from perfect and I plan t improve when I’ve some more time. Currently this script has some major drawbacks such as fixed DPI (300) and it only support bitonal (black and white) compression. As I wrote above, I plan to address this issues and make the script more flexible in the future.

Despite its drawbacks I hope you will still find this script useful. If you have any questions or suggestions please comment.

Backup Directories To Amazon S3 Script

Guy — Sat, 01 Mar 2008 20:12:27 +0000

This is a small script I wrote today, to automate my backups, which I do on Amazon S3. This is fairly short, yet useful bash script that utilize the s3cmd to do the actual sending of the files.

#! /bin/bash
 
# 2008 (C) Guy Rutenberg
#This is a script to backup a given folder and send them to Amazon S3
 
#string trailing slash (if exists)
DIR="${1/%\//}"
TARNAME="${DIR}-$(date +%Y%m%d).tar.bz2"
if [ -e ${TARNAME} ]
then
	echo "Refusing to overwrite existing file - ${TARNAME}"
	exit
fi
echo "Creating tarball"
tar -vc "$1" | bzip2 -9 -c > "${TARNAME}"
 
echo "Sending to Amazon S3"
s3cmd put -v "${TARNAME}" s3://guyru-backup/
 
echo "Deleting intermediate files"
rm "${TARNAME}"

To use the script just save it somewhere and make it executable. Remember to substitute guyru-backup with your own bucket name. Notice that you must have s3cmd installed (for Gentoo it’s available under net-misc/s3cmd). s3cmd is one of the best interfaces to the S3 service I’ve seen, so you’ll probably want to install it anyway.

The script usage instructions are pretty simple. Just call the script and pass the path to the folder you want to backup as the first argument.

I intend to use this script to fully automate my backups using cron (maybe this way I’ll remember to the backups regularly).

I hope you will find this script useful. If you have any suggestions or comment please send them.