Pull-based Backups using OpenBSD base*

*The approach and scripts discussed here use mysqldump to back up a database at one point. Yes I know this isn't in OpenBSD base, but this was added just for a specific system. It's easy not to add it, and everything else is done through tools from the base system.

With any system, it's important backups and restores work properly. With #Chargen.One, I wanted to protect user data, and be able to easily restore. The important things for the backup were:

Most services like borg backup work best with a backup server visible from the server being backed up. I use a huge NAS to store my backups, and a separate server to store backups of backups. The NAS is behind a Firewall and the other server can see the NAS but not the Internet. As such, I need a backup system that lets me use the NAS to pull from Chargen.One onto the NAS, and then allow my isolated backup server to pull from the NAS. If that sounds a little paranoid, at least you understand why I use OpenBSD.

Backups are taken by root on a nightly basis, and put into a folder belonging to a dedicated backup user. Early in the morning, the backup is pulled from a system which deletes the backup from chargen.one. The remote backup system will store 30 days' worth of backups.

Configuring a backup account

The /home partition is the largest on the VPS, so I created an account to store backups in a temporary folder, create an archive, delete the temporary folder and change permissions so the remote backup system can pull and remove the backup archive.

To set up the backup user account, use the following commands (as root):

# useradd -m backup
# chmod 700 /home/backup
# su - backup
$ cd .ssh

On the backup system, generate an ssh keypair using ssh-keygen -t ed25519. Copy the contends of id_ed25519.pub from the backup system into /home/backup/.ssh/authorized_keys on the server being backed up.

SSH into the backup account on the server from the NAS to make sure everything works.

Each backup archive is created by a script that creates and stores content in /home/backup/backup/. Once backed up, the script will create a timestamped archive file and delete the /home/backup/backup/ directory. The script starts off very simply:

#!/bin/sh

mkdir /home/backup/backup
# Add stuff below here


# Don't add stuff below here
rm -rf /home/backup/backup

Backing up MySQL data

If you want to implement my backup scheme and don't run MariaDB or MySQL, then skip this section and backup using commands from base only.

Because MySQL is configured to use passwords, a /root/.my.cnf file containing credentials for the mysqldump command is needed.

[mysqldump]
user=root
password=your_password_here

The mysqldump command fully backs up all mysql databases, routines, events and triggers.

Add the following to the backup script (all one line):

mysqldump -A -R -E --triggers --single-transaction  > /home/backup/backup/mysql.gz

The --single-transaction option causes the backup to take place without locking tables.

Backing up a package list

OpenBSD uses it's own package management system called pkg. To create a backup of installed packages add the following to the backup script:

pkg_info -mz > /home/backup/backup/packages.txt

This can then be restored from a backup using pkg_add -l packages.txt.

Backing up files

The following files and directories should be backed up:

Use the tar command to create backups. A discussion of the tar command is best left to man tar, but as the backup isn't very large, I'm not using incremental backups, which keeps things simple...up to a point.

OpenBSD's tar implementation doesn't add the --exclude option as it's a GNU extension. Other BSDs such as FreeBSD do add the option, but the OpenBSD team prefer not to have it. I could've added the GNU tar package, but one of the stated goals of the script is to not require additional software to keep things portable. Paths such as /home/backup are excluded using shell expansion instead.

To test this, try the following command:

# tar cvf bk.tar /home/!(backup)

The exclamation mark means exclude anything in the parentheses. For multiple directories, separate the names with a pipe symbol, e.g. !(backup|user) to exclude both backup and user directories.

There are complications and error messages will be shown on each backup if absolute paths are added to the tar command. This means that an email would be generated every night, even if the backup succeeds. Ain't nobody got time for that.

As a workaround, changing to the root directory at the start makes all paths relative, and allows the shell expansion to work. The -C switch can be used instead, but this breaks shell expansion.

The final commands to go in the script look like this:

cd /
tar cf /home/backup/backup/files.tar etc/!(spwd.db) root \
	var/www var/log home/!(backup) /var/cron \
	usr/local/bin/writefreely usr/local/bin/backup.sh \
	usr/local/share/writefreely 

I've used backslashes to break up the lines for readability, but all the paths could be put on a single line if preferred.

I've excluded /etc/spwd.db from the backup because OpenBSD's built-in tar uses a feature called pledge that restricts access to certain files. The file isn't particularly important to this specific backup, but contains the shadow password database, which I'm happy to recreate as part of the restore process.

At this point you might wonder why gzip compression isn't being used in the tar archive. This is because the final archive will be compressed, and there's no point in compressing twice.

Creating the final archive

To distinguish between backups by date, I used a timestamp, generated by the date command. By default there are spaces and colons, neither of which are good for interoperability across Operating Systems and filesystems. Use date +%F_%H%M%S to generate a more reasonable format. Using tar's -C switch changes the tar working directory to /home/backup and stops a leading / error message appearing in the backup.

The final tar command in the backup script should look like this:

tar zcf /home/backup/c1_$(date +%F_%H%M%S).tgz -C /home/backup backup

It's also important to change the file ownership to the backup user so the remote system can delete the backup after it's been created.

chown backup:backup /home/backup/c1_*

The full backup script on chargen.one looks like this:

#!/bin/sh

mkdir /home/backup/backup
# Add stuff below here

# MySQL Backup
mysqldump -u root -A -R -E --triggers --single-transaction | gzip -9 > /home/backup/backup/mysql.gz

# Packages backup
pkg_info -mz > /home/backup/backup/packages.txt

# Files backup
cd /
tar cf /home/backup/backup/files.tar etc/!(spwd.db) root \
	var/www var/log home/!(backup) /var/cron \
	usr/local/bin/writefreely usr/local/bin/backup.sh \
	usr/local/share/writefreely 

# Final archive
tar zcf /home/backup/c1_$(date +%F_%H%M%S).tgz \
	-C /home/backup backup

# Fix permissions
chown backup:backup /home/backup/c1_*

# Don't add stuff below here
rm -rf /home/backup/backup

Automating the backup

As root (via su -, not doas), use crontab -e and add the following entry:

0 1 * * * /usr/local/bin/backup.sh

A new backup is created at 1am, every morning. On the remote server, a cron job calls a script at 3am to pull the backup down via scp using the following:

#!/bin/sh

find /Backups/c1/ -mtime +30 -exec rm {} \;
scp -q backup@chargen.one:./c1_*.tgz /Backups/c1/
ssh backup@chargen.one rm c1_*.tgz

And that's it! The secondary backup server pulls down the contents of /Backups from the NAS, so there's nothing left to do.

I'll write a separate post about restoring, as this post is already getting long, but hopefully it's useful to people who want pull, rather than push backups.