Restore First: How to Build Backups You Have Actually Tested

Karani GeoffreyJune 30, 20267 min read

Ask a small team whether they have backups and they will say yes. Ask them how long a full restore takes and you get silence, then a guess, and the guess is always wrong by a factor of five. I have watched a team discover, during an actual outage, that their nightly dump had been writing zero-byte files for eleven weeks because the disk filled and nobody checked the exit code.

My thesis: a backup is not a file, it is a rehearsed procedure with a measured duration. Which leads to the method - write the restore script first. If you cannot describe how the data comes back, the fact that it is going somewhere is irrelevant.

Answer three questions before you install anything

RPO - how much data can we lose? If the answer is "a day", nightly dumps are fine. If it is "an hour", you need binlogs or WAL shipping and you should know that now, not later.
RTO - how long may we be down? This is where bandwidth bites in East Africa. 40GB from Backblaze over a 20 Mbps link is roughly four and a half hours of download before you even start importing. If your RTO is two hours, your offsite copy alone cannot meet it and you need a local copy too.
Who has the keys? If your restic password is only in /root/.restic-pass on the server that just died, you do not have backups. You have encrypted noise.

Step 1: write restore.sh

Before the backup job exists. It will be wrong, and that is the point - fixing it teaches you what the backup must contain.

#!/usr/bin/env bash
# /usr/local/sbin/restore-drill.sh
# Restores the latest snapshot into a scratch database and verifies it.
set -euo pipefail

SNAP="${1:-latest}"
WORK=$(mktemp -d /var/tmp/restore.XXXXXX)
trap 'rm -rf "$WORK"' EXIT

export RESTIC_REPOSITORY="s3:s3.eu-central-003.backblazeb2.com/karani-backups"
export RESTIC_PASSWORD_FILE=/etc/restic/password
export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY
source /etc/restic/s3.env

echo "== 1. Pulling snapshot $SNAP"
time restic restore "$SNAP" --target "$WORK" --include /var/backups/db

DUMP=$(find "$WORK" -name 'billing-*.sql.zst' | sort | tail -1)
test -s "$DUMP" || { echo "FATAL: dump is empty or missing"; exit 1; }
echo "== dump: $DUMP ($(du -h "$DUMP" | cut -f1))"

echo "== 2. Loading into scratch schema"
sudo mariadb -e "DROP DATABASE IF EXISTS restore_test; CREATE DATABASE restore_test;"
time zstd -dc "$DUMP" | sudo mariadb restore_test

echo "== 3. Verifying"
sudo mariadb restore_test -N -e "
  SELECT CONCAT('tables=', COUNT(*)) FROM information_schema.tables
   WHERE table_schema='restore_test';"
sudo mariadb restore_test -N -e "
  SELECT CONCAT('newest_invoice=', IFNULL(MAX(creation),'NONE')) FROM \`tabSales Invoice\`;"

echo "== RESTORE DRILL OK"

Notice step 3. Counting tables proves the file parsed. Checking the newest row's timestamp proves the data is current - I have seen a perfectly valid restore of a dump that was six months stale because a path changed and nobody noticed the mtime.

Step 2: dumps that are actually consistent

A dump taken while the app is writing can be internally inconsistent, and you will only find out when foreign keys fail on import.

# MariaDB / MySQL - InnoDB only
mariadb-dump --single-transaction --quick --routines --triggers --events \
  --default-character-set=utf8mb4 --databases billing \
  | zstd -3 -o /var/backups/db/billing-$(date +%F-%H%M).sql.zst

# PostgreSQL - custom format, parallel restore later
pg_dump -Fc -Z6 -f /var/backups/db/billing-$(date +%F-%H%M).dump billing

--single-transaction gives you a consistent snapshot without locking writers, but only for transactional engines. If any table is still MyISAM, that table is dumped outside the transaction and can be inconsistent with the rest, silently. Check first:

SELECT table_name, engine
FROM information_schema.tables
WHERE table_schema = 'billing' AND engine <> 'InnoDB';

And never put the password on the command line where it lands in ps and your shell history. Use a credentials file:

# /root/.my.cnf  (chmod 600)
[client]
user=backup
password=change-this-now

Frappe and ERPNext specifically

cd /home/frappe/frappe-bench
bench --site erp.example.co.ke backup --with-files --compress

This writes to sites/<site>/private/backups/ and produces a database dump plus two tarballs for public and private files. The killer detail: the database is encrypted at rest for some fields, and the key lives in sites/<site>/site_config.json as encryption_key. Restore the database without that file and your saved passwords, email account credentials and API secrets are permanently unreadable. Back up site_config.json and common_site_config.json as first-class items, not as an afterthought.

Step 3: the backup job

#!/usr/bin/env bash
# /usr/local/sbin/backup.sh
set -euo pipefail

source /etc/restic/s3.env
export RESTIC_REPOSITORY RESTIC_PASSWORD_FILE=/etc/restic/password

STAMP=$(date +%F-%H%M)
OUT=/var/backups/db
install -d -m 0700 "$OUT"

# 1. Fail loudly if the disk cannot hold the dump
AVAIL=$(df --output=avail -k "$OUT" | tail -1)
[ "$AVAIL" -gt 5242880 ] || { echo "FATAL: less than 5G free on $OUT"; exit 1; }

# 2. Dump, then assert non-trivial size
mariadb-dump --single-transaction --quick --routines --triggers --events \
  --databases billing | zstd -3 -o "$OUT/billing-$STAMP.sql.zst"
SIZE=$(stat -c%s "$OUT/billing-$STAMP.sql.zst")
[ "$SIZE" -gt 1048576 ] || { echo "FATAL: dump only $SIZE bytes"; exit 1; }

# 3. Push to object storage
restic backup --tag nightly --host web01 \
  "$OUT" /etc/nginx /etc/systemd/system /srv/billing/uploads

# 4. Retention
restic forget --tag nightly --host web01 \
  --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 2 --prune

# 5. Keep only 2 local copies; object storage holds history
find "$OUT" -name 'billing-*.sql.zst' -printf '%T@ %p\n' \
  | sort -rn | tail -n +3 | cut -d' ' -f2- | xargs -r rm -f

# 6. Dead man switch
curl -fsS -m 10 --retry 3 "https://hc-ping.com/YOUR-UUID" > /dev/null

Step 1 and step 2 are the two checks that would have caught the eleven-week zero-byte disaster. set -euo pipefail plus an explicit size assertion beats hope.

Step 4: schedule it so failures are visible

# /etc/systemd/system/backup.service
[Unit]
Description=Nightly database and config backup
OnFailure=notify-failure@%n.service

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup.sh
Nice=10
IOSchedulingClass=idle
TimeoutStartSec=3h

# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup nightly

[Timer]
OnCalendar=*-*-* 01:30:00 Africa/Nairobi
RandomizedDelaySec=900
Persistent=true

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer
systemctl list-timers backup.timer
systemd-analyze calendar "*-*-* 01:30:00 Africa/Nairobi"

Persistent=true means if the VPS was off at 01:30 the job runs at next boot instead of being silently skipped. IOSchedulingClass=idle stops the dump from starving your app of disk I/O. And the dead man switch matters more than the alert: a job that fails pages you, but a job that never runs at all is invisible unless something expects a heartbeat.

Step 5: verify the repository, not just the job

restic snapshots --host web01 --tag nightly
restic check                                  # structure and metadata, fast
restic check --read-data-subset=5%            # downloads and hashes 5% of packs
restic check --read-data-subset=1/12          # one twelfth, rotate monthly
restic stats latest --mode restore-size

Running --read-data-subset=1/12 monthly means you have verified every byte of the repository once a year without ever paying for a full download in one go. Put it on its own weekly timer and treat a failure as a page.

Step 6: the drill, on a calendar, timed

Once a month, on a scratch VPS or a container, run the restore from scratch as if the main server no longer exists. Two rules: you may not SSH into production during the drill, and you must use only what is written in the runbook.

Stage	Target	Actual (Jun)	Actual (Jul)
Provision fresh VPS	10 min	12 min	9 min
Download snapshot	45 min	1h 20m	48 min
Import database	25 min	31 min	27 min
App up and serving	20 min	55 min	22 min
Total RTO	1h 40m	2h 58m	1h 46m

That table is the deliverable. It turns "we have backups" into a number you can put in front of a client, and it shows improvement over time. June was slow because nobody had written down the Redis config; July was faster because the drill produced a fix.

The failure modes that actually kill people

The key died with the server. Store the restic password and the Frappe encryption_key in a password manager and on paper in a safe. Test that a second person can retrieve them.
Backups run as root and the credentials file is world-readable. chmod 600 everything under /etc/restic/ and /root/.my.cnf. Consider append-only credentials on the storage side so ransomware on the host cannot delete history.
You backed up the database but not the uploads. Invoices with missing attachments are a legal problem, not a technical one.
The restore imports but the app will not start. Because you never captured /etc/nginx, the systemd units, cron entries or the .env. Config is data.
Everything is in one region of one provider. Keep the offsite copy at a different vendor to your VPS. Account suspensions are a real failure mode, not a hypothetical.
Nobody tested restoring an old snapshot. Ransomware and silent corruption are discovered late. Drill a 30-day-old snapshot at least once.

The only meaningful definition of a backup: data you have successfully restored, on a different machine, within a time you have measured, using instructions someone else can follow.

Start tonight with the smallest possible version. Write restore-drill.sh, run it once, time it, and write the number down. Everything else is refinement.

#backups#restic#linux#disaster-recovery#devops