Panama/server/home/RESTORE.md

Restoring ~/Server on server.gib from the ZFS-pool backup

Written 2026-09-17. Read this before you need it.

What this protects against

The NVMe/LUKS btrfs volume (/, /home, and /var/lib/docker) dying. The backup lives on the ZFS pool in the same machine, so it does not protect against fire, theft, or the whole box failing. It is a disk-failure backup, not an offsite one.

Layout

/home/gib/Public/Server_Backup/Containers/server.gib/current/

One directory, overwritten daily at 02:00 by server-backup.timer. History lives in ZFS: Media/Server_Backup has zfs-auto-snapshot keeping 14 dailies, taken by cron.daily between roughly 03:22 and 03:52. The 02:00 schedule is deliberate — it must finish before that window, or a snapshot captures a half-written backup as that day's history. It also staggers against the VPS backup, which pushes into this same pool at 01:00.

Older copies are under the hidden snapshot dir (snapdir=hidden, so ls will not show it, but you can cd in):

ls /home/gib/Public/Server_Backup/.zfs/snapshot/
ls /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/

The 03-16-2026 and 03-31-2026 directories are leftovers from the old manual script.

Why this runs as root

This box runs Docker, so most container data under ~/Server is root-owned. The old script used sudo rsync, which worked but meant it could never be automated without a sudoers rule. The replacement runs as root via server-backup.service, a systemd system unit — already root, so ownership is preserved natively and no sudoers entry is needed at runtime. If you see /etc/sudoers.d/99-claude-setup on this machine, it was temporary setup scaffolding and should be deleted.

(This differs from the VPS, which is rootless podman and needs rsync --fake-super to carry subuid ownership through. Nothing like that is needed here.)

The script lives at /usr/local/sbin/server-backup, deliberately not in ~/Server — a root-executed script inside a user-writable directory is a footgun. ~/Server/backup is a thin wrapper that triggers the systemd unit so manual runs behave identically to nightly ones.

What is in a backup

Path	What it is
Media/, Frigate/, Home/, OpenWebUI/, ...	Raw bind-mount data per service
_backup_state/sqlite/<original path>	Online .backup of every SQLite DB — authoritative
_backup_state/convex/<container>.zip	npx convex export snapshot of each SQLite-backed Convex deployment (gbrown) — authoritative

Use _backup_state/ for databases, not the raw copies. Containers keep running during the backup, so the raw files are crash-consistent at best. The dumps use SQLite's online backup API and are what will actually restore cleanly. That covers the ones that matter: Plex's com.plexapp.plugins.library.db (watch history, ratings), Jellyfin, Frigate, Home Assistant, OpenWebUI, Audiobookshelf, and every *arr config.

Convex is the exception. It writes to its SQLite file constantly, so a raw .backup never completes. Its _backup_state/convex/ export is the restore source, and the raw db.sqlite3 is not dumped at all.

What is deliberately NOT backed up

• Agents/Gitea/volumes/runner-{1,2,3}/dind — ~203 GB. Docker-in-Docker containerd image layers for the CI runners. Pure cache that churns constantly; a rebuilt runner just re-pulls it. The parts that matter — runner-N/data/.runner (the 648-byte registration token) and certs/ — are backed up.

  Measured as root. A plain du as gib reports these as ~13 MB because gib cannot descend into them. Do not trust non-root sizes here.

• Docker named volumes (/var/lib/docker/volumes, 47 in use, ~61 GB): Sentry (13) and Plausible (4) were consciously excluded — Sentry's event data is not worth restoring and Plausible had nothing in it. Romm looked like a gap but is not: its anonymous /romm volume is only a parent directory, with assets/, config/ and resources/ bind-mounted from ~/Server and library/ from the pool.
• Nextcloud AIO — has its own borg backup at ~/Public/Nextcloud/Backups, run by hand from the AIO admin UI. Last run 2026-09-17 (finished 09-18). Runs are slow and AIO stops the Nextcloud containers while one is going.
• node_modules, build output, *.log, cache/tmp dirs, *.mp4/*.mkv.

Restoring

Stop the stack first so nothing overwrites what you restore:

cd ~/Server/<service> && docker compose down     # per service, or stop docker entirely

Pull it back — as root, so ownership is restored:

sudo rsync -a --numeric-ids --delete \
  /home/gib/Public/Server_Backup/Containers/server.gib/current/ \
  /home/gib/Server/

For an older day, use a snapshot path instead:

/home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/

Then put the databases back from the dumps. For each one, stop the owning container, copy the dump over the live file, and match the original ownership. Plex, for example:

docker compose -f ~/Server/Media/compose.yml stop plex
D=~/Server/_backup_state/sqlite/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
L=~/Server/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
sudo cp "$D/com.plexapp.plugins.library.db" "$L/"
sudo chown --reference="$L/com.plexapp.dlna.db" "$L/com.plexapp.plugins.library.db"
docker compose -f ~/Server/Media/compose.yml start plex

chown --reference copies ownership from a neighbouring file, which saves looking up what uid each container expects.

For Convex, start the backend (it can be empty) and import the export through it. This replaces every table with the snapshot:

Z=~/Server/_backup_state/convex/gbrown-backend.zip
KEY=$(docker exec gbrown-backend ./generate_admin_key.sh | tail -1)
docker run --rm -it --network nginx-bridge -v "$Z:/backup.zip:ro" \
  -e CONVEX_SELF_HOSTED_URL=http://gbrown-backend:3210 -e CONVEX_SELF_HOSTED_ADMIN_KEY="$KEY" \
  docker.io/library/node:22-alpine sh -ec 'mkdir /w && cd /w && echo "{}" > package.json &&
    npm install --silent convex && npx convex import --replace-all /backup.zip'

Functions and schema are not in the export. Redeploy them from the app's repo as usual.

After a full restore the CI runners will have no dind storage. That is fine — they re-pull images on the next job. The registration in runner-N/data/.runner is intact, so they reconnect to Gitea without re-registering.

Checking it is working

systemctl list-timers server-backup.timer
systemctl status server-backup.service
tail -40 ~/Server/logs/backup.log

A healthy run ends with === backup complete ===. Watch for:

• ZFS history OK — confirms snapshots are still being taken. If this ever says WARNING, versioning is broken and current/ is just a mirror: a mistake would propagate with nothing to roll back to. Fix zfs-auto-snapshot before anything else.
• rsync exit 24 is a NOTE, not a failure — a file changed mid-transfer, normal on a live box.

Optional: put an uptime-kuma push URL in ~/Server/.backup-heartbeat-url and successful runs will ping it, turning a silent failure into a down monitor.