Backing up of KVMs with BorgBackup and BlockFuse Pt. 2

Backing up KVMs with BorgBackup

This is part 2 of a 2 part series, if you have not read part one yet you can do so here: Backing up of KVMs with BorgBackup and BlockFuse Pt. 1

We use a lot of Virtual Machines (VM) to run ours infrastructure, these are on multiple host machines running Logical Volume Management (LVM) with each VM on its on its own Logical Volume (LV). After faffing around with multiple backup strategies and programs I settled on using BorgBackup (short: Borg) due to its special handling of block devices along with its deduplication and compression capabilities.

Borgs’ documentation is vast and can be found here: https://borgbackup.readthedocs.io/en/stable/

BlockFuse is quite an ancient program and was written by Eric Wheeler initially to do rdiff-backups, but works perfectly in this case as well to mount blockdevice. More information can be found here: https://www.linuxglobal.com/blockfuse-to-the-rescue-rdiff-backup-of-lvm-snapshots-and-block-devices/

BlockFuse:

BlockFuse is a relatively old piece of code which hasn’t been maintained for some time, it was originally created for use with rdiff-backup but works great with Borg as well.

BlockFuse takes two arguments:

# ./block-fuse
usage: ./block-fuse /dev/directory /mnt/point

BlockFuse is quite simple: it enumerates the content of the mount-source directory, and exports all block devices with non-zero size as a file with 0400 permissions, owned by your fuse user (probably root for this).

Notes:

  • BlockFuse uses the mount-time as the modification time (st_mtime) for the mounted filesystem. This will force Borg to scan the block devices for changes. Therefore you must unmount and re-mount your BlockFuse filesystem after updating your snapshots. If you do not, Borg will skip the “files” because their modification time-stamp had not changed since the last backup.

Backup Process:

The first task in the backup process was to check for and mount your Logical Volumes:

if [[ ! -d /mnt/block-devices ]]; then
    sudo mkdir -p /mnt/block-devices
    cd /etc/block-fuse
    ./block-fuse /dev/mapper /mnt/block-devices
else
    cd /etc/block-fuse
    ./block-fuse /dev/mapper /mnt/block-devices
fi

In our backup process we are backing up to an external drive so we need to check to see if that is mounted as well:

if [[ ! -d /mnt/borgbackups ]]; then
    sudo mkdir -p /mnt/borgbackups
    sudo mount -o rw,$KEY -t cifs [your_external_storage_address] /mnt/borgbackups
else
    sudo mkdir -p /mnt/borgbackups
    sudo mount -o rw,$KEY -t cifs [your_external_storage_address] /mnt/borgbackups
fi

We are backing up multiple vmhosts so we have to verify that there is a separate backup folder for this machine:

if [[ ! -d /mnt/borgbackups/$HOSTNAME ]]; then
sudo mkdir -p /mnt/borgbackups/$HOSTNAME
fi

The only machines we want to back up are those in service, so we will make an array to pass through of all the running instances on the host:

name(){
	virsh list | awk '( $1 ~ /^[0-9]+$/ ) { print $2 }'
}
arr=($( name ))

Now by passing-over the array we can make the magic happen!:

for v in ${arr[*]}; do
	if sudo lvdisplay | grep -q $v; then
		#Take snapshots of VMs 
		lvcreate --size 5G -s -n $v-snap /dev/main-vg/$v
		borgrepo=/mnt/borgbackups/$HOSTNAME/$v-bbyo
		#Check to see if vm has been backed up before and then create backup
		if [[ ! -d $borgrepo ]]; then
	                borg init --encryption=none $borgrepo
	                borg create -C zlib,6 "$borgrepo::$v_{now:%Y-%m-%d}" /mnt/block-devices/main--vg-$v--snap
		else
	                borg create -C zlib,6 "$borgrepo::$v_{now:%Y-%m-%d}" /mnt/block-devices/main--vg-$v--snap

		fi
		#Remove snapshot
		lvremove -f /dev/main-vg/$v-snap
	fi
done

And finally we will clean up our workspace in preparation for the next scheduled back up:

umount /mnt/block-devices
umount /mnt/borgbackups

#Finish
echo "VM backup complete"

In the next part we will look at how we can recover our VMs.

The complete script for this, along with the BlockFuse files can be found here: https://github.com/donbale/blockbusterborg

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.