Tape Backup for TrueNAS Core

Introduction

I use LTO-5 Tape for cold backup and the important sets of data I have are less than 1.5TB, thus it fits to a single LTO-5 tape. I also would like to backup TrueNAS Core time to time, but, not surprisingly, it is much more than 1.5TB. Thus, a multi-volume backup is needed. The issue is TrueNAS Core runs on FreeBSD and at least the default installation does not have stenc (to enable tape encryption) and GNU tar that supports multiple volumes. Here is how to make this work.

Setup

I think the easiest way to compile the required programs without changing TrueNAS Core is to create a jail. A jail is basically an isolated FreeBSD instance running inside TrueNAS Core (OS level virtualization). I have created a default jail (clone jail), 13.2-release with DHCP autoconfigure. Then, I installed the following packages:

$ pkg install git autoconf libtool automake pkgconf hs-pandoc

I am using the same jail to compile both stenc and GNU tar. After the builds, the jail can be deleted or kept in case you need it later.

Compiling stenc

Simply cloning the stenc repo first and checking out the tag 2.0.0:

$ git clone https://github.com/scsitape/stenc.git
$ cd stenc
$ git checkout 2.0.0

then building it with:

$ autoreconf --install
$ ./autogen.sh && ./configure
$ make check
$ make

results an stenc executable in src folder.

stenc binary should be copied from the jail to TrueNAS by running something like this in TrueNAS:

$ cp <zfs_pool_path>/iocage/jails/<jailname>/root/root/stenc/src/stenc .

If the tape drive is attached (I have an external drive), it can be run for a simple check:

$ ./stenc

Status for /dev/nsa0 (QUANTUM ULTRIUM-HH5 H971)
--------------------------------------------------
Reading:                         Not decrypting
Writing:                         Not encrypting
Key instance counter:            0
Supported algorithms:
1    AES-256-GCM-128
     Key descriptors allowed, maximum 32 bytes
     Raw decryption mode allowed, raw read enabled by default
2    AES-256-GCM-128
     Key descriptors allowed, maximum 32 bytes
     Raw decryption mode allowed, raw read enabled by default

Compiling GNU tar

First fetching the latest GNU tar package (1.35) and extracting it:

$ fetch https://ftp.gnu.org/gnu/tar/tar-latest.tar.gz
$ tar xvf tar-latest.tar.gz
$ cd tar-1.35

then building it with:

$ setenv FORCE_UNSAFE_CONFIGURE 1
./configure LDFLAGS=/usr/local/lib LIBS=-lintl
make
make check

results a tar executable in src folder. Setting FORCE_UNSAFE_CONFIGURE is required because it is run as root. Also, LDFLAGS and LIBS has to be passed like this, otherwise make terminates with linker errors.

Like stenc, tar binary should be copied from jail to TrueNAS:

$ cp <zfs_pool_path>/iocage/jails/<jailname>/root/root/tar-1.35/src/tar gnutar

I copied it as gnutar because the system already has tar.

It works fine:

$ ./gnutar --version

tar (GNU tar) 1.35
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

Making a multi-volume backup

I have two main and two helper scripts. I also have TAPE environment variable set to /dev/nsa0.

One of the helpers (configure.sh) is to configure the tape before backup:

#!/bin/bash
set -e

STENC=./stenc

mt load
mt comp 1
mt blocksize 131072

$STENC -e on -d on --no-allow-raw-read -k tape.key -a 1

mt status > mt.log
$STENC > stenc.log

mt rewind

This loads the tape, enables compression, sets the block size to 128K (there is a reason for this, I will tell soon), then enable the encryption, save the tape and the encryption status to log files which I also add to the backup and finally rewind the tape.

The main backup script (backup.sh) takes an argument pointing to a file or a folder (reformatted for display purposes):

#!/bin/bash
set -e

if [ $# -ne 1 ];
then
        echo "use backup.sh <path>"
        exit 1
else
        SRC=$1
fi

GNUTAR=./gnutar

./configure.sh

echo `date +"%Y%m%d%H%M%S"` > timestamp

$GNUTAR --create --checkpoint=4000 
        --checkpoint-action=echo="#%u: %{w}T" 
        --blocking-factor=256 --format=pax 
        --totals=SIGUSR1 --multi-volume 
        --info-script=./switch.sh 
        --file=$TAPE 
        timestamp mt.log stenc.log $SRC

mt offline

This calls the configure first, then creates a timestamp which I also add to the backup, and runs GNU tar with multi volume options. TrueNAS Core gives a warning if I use more than 256 as blocking factor, which is equal to 128KB, that is why I also set the tape blocksize to 128KB.

I use the checkpoint and checkpoint action options to print write throughput approx. every half GB.

I use another helper script switch.sh that is called when a volume is full:

mt offline
echo -n "Switch the volume and press enter: "
read dummy
mt load
mt rewind

This just ejects the tape and waits for enter. Meanwhile, I replace the tape, and pressing enter loads and rewinds it. It does not matter if you load the tape (by pushing it yourself) or it is loaded by the tape drive (with mt load command). Both works.

Naturally it takes a lot of time to backup. One tape is 1.5TB (assuming it cannot be compressed at all), and the write throughput is around 130MB/s. This takes around 3h for one tape. In order to test this script, I added --tape-length option and made a small backup spanning 4 tapes, and it works (and restores) fine.

The other main script (restore.sh) is naturally for restoring (also reformatted below for display purposes) and it is run with an argument pointing to a folder where the backup will be extracted:

#!/bin/bash
set -e

if [ $# -ne 1 ];
then
        echo "use restore.sh <path>"
        exit 1
else
        DST=$1
fi

GNUTAR=./gnutar

./configure.sh

$GNUTAR --extract --checkpoint=4000 
        --checkpoint-action=echo="#%u: %{r}T" 
        --blocking-factor=256  --format=pax 
        --totals=SIGUSR1 --multi-volume 
        --info-script=./switch.sh 
        --file=$TAPE 
        --directory=$DST

mt offline

This also calls configure first since decryption etc. has to be configured. before. Then, GNU tar is run similarly for extract.