Linux Time Machine With Rsync

The Idea

Time Machine is Apple's backup solution for Mac. It makes incremental backups and provides a GUI for browsing and restoring data from arbitrary points in time. I use it for my Macs and it's nice, but my important computers run Linux and I want something similar for them.

No need for the GUI, but incremental backups that are easy to access. The magic bullet: rsync with the --link-dest option. This makes rsync create hard links for every file that is unchanged between backup versions. A hard link takes virtually no space, so the backup only uses disk for files that actually changed between snapshots.

How --link-dest Works

When syncing source to destination, rsync checks each file against a reference directory (the --link-dest argument). If the file is identical, it creates a hard link instead of a copy. If the file has changed, it transfers the new version. The result: each snapshot looks like a complete backup, but only modified files consume extra disk.

The Original Script

Here's the script I started with. It backs up the root filesystem to an NFS-mounted QNAP NAS and creates date-stamped snapshot directories with a -current symlink pointing to the latest.

backup script (original inline version)

#!/bin/sh

BACKUP_MOUNTPOINT="/mnt/qnap-backup"
BACKUP_NAME="`hostname`-backup"
BACKUP_EXCLUDE="~/local/etc/backup_exclude.conf"

if [ `id -u` != "0" ]; then
    echo "You need to be root (sudo) to run system backups"
    exit 1
fi

VERBOSE_ARGS=""
if [ "x$1" = "x-v" ]; then
    VERBOSE_ARGS="--progress -v"
fi

TODAY=`date +"%Y-%m-%d"`

CURRENT_BACKUP=$BACKUP_MOUNTPOINT/${BACKUP_NAME}-current
NEW_BACKUP=$BACKUP_MOUNTPOINT/${BACKUP_NAME}-$TODAY

mkdir -p $NEW_BACKUP

if [ ! -d $NEW_BACKUP ]; then
    echo "No such directory: $NEW_BACKUP"
    exit 1
fi

rsync $VERBOSE_ARGS -a --delete --relative --one-file-system \
    --numeric-ids --exclude-from=$BACKUP_EXCLUDE \
    --link-dest=$CURRENT_BACKUP/ / $NEW_BACKUP/

# new symlink to current
[ -h $CURRENT_BACKUP ] && rm -f $CURRENT_BACKUP
ln -s $NEW_BACKUP $CURRENT_BACKUP

My NAS is mounted over NFS:

$ grep qnap-backup /etc/fstab
qnap:/backup  /mnt/qnap-backup nfs auto,user 0 0

The exclude file contains patterns for things you don't want backed up — one per line, see man rsync for details:

/home/johan/Videos/*
/tmp/*

Schedule it to run as root. I run mine every night at 3 am. After a few nights, the backup directory looked like this (piano is my hostname):

piano:/mnt/qnap-backup$ ls -1
piano-backup-2013-02-20
piano-backup-2013-02-21
piano-backup-2013-02-22
piano-backup-2013-02-23
piano-backup-2013-02-24
piano-backup-2013-02-25
piano-backup-2013-02-26
piano-backup-current

The last entry is a symlink to the most recent backup. Every new run compares against -current, so only new or changed files are actually stored. Everything else is hard-linked.

The GitHub Version

The inline script eventually evolved into a proper project on GitHub. The main improvements:

Separate config file (backup.conf) instead of hardcoded variables
Separate exclude file (backup_exclude.conf)
Support for backing up multiple filesystems as arguments: do_incremental_rsync.sh /home /
Since rsync uses --one-file-system, each mount needs its own argument

Test with the verbose flag before scheduling:

sudo ./do_incremental_rsync.sh -v /home

Verifying Hard Links

To confirm that rsync is actually hard-linking unchanged files, check the inode numbers across snapshots:

$ stat piano-backup-2013-02-25/home/johan/.bashrc | grep Links
Links: 7

$ stat piano-backup-2013-02-26/home/johan/.bashrc | grep Links
Links: 7

Same inode, same physical file on disk. Zero extra space.

Going Further

This is a starting point. The script should probably be extended to:

Erase old backups over a certain age
Check that the backup storage is not full
Log success or failure somewhere
Alert you if there is a problem

Requirements

rsync — probably already installed on your machine
Somewhere to put the backups: NAS, external disk, another partition. The target filesystem must support hard links (ext4, XFS, etc. — not FAT-32 or NTFS)
The backup target must be locally mounted (NFS, USB, etc.)