# Restoring ~/Server on server.gib from the ZFS-pool backup Written 2026-09-17. Read this *before* you need it. ## What this protects against The NVMe/LUKS btrfs volume (`/`, `/home`, and `/var/lib/docker`) dying. The backup lives on the ZFS pool in the same machine, so it does **not** protect against fire, theft, or the whole box failing. It is a disk-failure backup, not an offsite one. ## Layout /home/gib/Public/Server_Backup/Containers/server.gib/current/ **One directory, overwritten daily at 02:00** by `server-backup.timer`. History lives in ZFS: `Media/Server_Backup` has `zfs-auto-snapshot` keeping **14 dailies**, taken by `cron.daily` between roughly 03:22 and 03:52. The 02:00 schedule is deliberate — it must finish before that window, or a snapshot captures a half-written backup as that day's history. It also staggers against the VPS backup, which pushes into this same pool at 01:00. Older copies are under the hidden snapshot dir (`snapdir=hidden`, so `ls` will not show it, but you can `cd` in): ls /home/gib/Public/Server_Backup/.zfs/snapshot/ ls /home/gib/Public/Server_Backup/.zfs/snapshot//Containers/server.gib/current/ The `03-16-2026` and `03-31-2026` directories are leftovers from the old manual script. ## Why this runs as root This box runs Docker, so most container data under `~/Server` is root-owned. The old script used `sudo rsync`, which worked but meant it could never be automated without a sudoers rule. The replacement runs as root via `server-backup.service`, a systemd **system** unit — already root, so ownership is preserved natively and **no sudoers entry is needed at runtime**. If you see `/etc/sudoers.d/99-claude-setup` on this machine, it was temporary setup scaffolding and should be deleted. (This differs from the VPS, which is rootless podman and needs `rsync --fake-super` to carry subuid ownership through. Nothing like that is needed here.) The script lives at `/usr/local/sbin/server-backup`, deliberately **not** in `~/Server` — a root-executed script inside a user-writable directory is a footgun. `~/Server/backup` is a thin wrapper that triggers the systemd unit so manual runs behave identically to nightly ones. ## What is in a backup | Path | What it is | |---|---| | `Media/`, `Frigate/`, `Home/`, `OpenWebUI/`, ... | Raw bind-mount data per service | | `_backup_state/sqlite/` | Online `.backup` of every SQLite DB — **authoritative** | | `_backup_state/convex/.zip` | `npx convex export` snapshot of each SQLite-backed Convex deployment (gbrown) — **authoritative** | > **Use `_backup_state/` for databases, not the raw copies.** Containers keep running during > the backup, so the raw files are crash-consistent at best. The dumps use SQLite's online > backup API and are what will actually restore cleanly. That covers the ones that matter: > Plex's `com.plexapp.plugins.library.db` (watch history, ratings), Jellyfin, Frigate, Home > Assistant, OpenWebUI, Audiobookshelf, and every *arr config. > > **Convex is the exception.** It writes to its SQLite file constantly, so a raw `.backup` > never completes. Its `_backup_state/convex/` export is the restore source, and the raw > `db.sqlite3` is not dumped at all. ## What is deliberately NOT backed up - **`Agents/Gitea/volumes/runner-{1,2,3}/dind` — ~203 GB.** Docker-in-Docker containerd image layers for the CI runners. Pure cache that churns constantly; a rebuilt runner just re-pulls it. The parts that matter — `runner-N/data/.runner` (the 648-byte registration token) and `certs/` — **are** backed up. > Measured as root. A plain `du` as `gib` reports these as ~13 MB because gib cannot > descend into them. Do not trust non-root sizes here. - Docker **named volumes** (`/var/lib/docker/volumes`, 47 in use, ~61 GB): Sentry (13) and Plausible (4) were consciously excluded — Sentry's event data is not worth restoring and Plausible had nothing in it. Romm looked like a gap but is not: its anonymous `/romm` volume is only a parent directory, with `assets/`, `config/` and `resources/` bind-mounted from `~/Server` and `library/` from the pool. - **Nextcloud AIO** — has its own borg backup at `~/Public/Nextcloud/Backups`, run by hand from the AIO admin UI. Last run 2026-09-17 (finished 09-18). Runs are slow and AIO stops the Nextcloud containers while one is going. - `node_modules`, build output, `*.log`, `cache`/`tmp` dirs, `*.mp4`/`*.mkv`. ## Restoring Stop the stack first so nothing overwrites what you restore: cd ~/Server/ && docker compose down # per service, or stop docker entirely Pull it back — as root, so ownership is restored: sudo rsync -a --numeric-ids --delete \ /home/gib/Public/Server_Backup/Containers/server.gib/current/ \ /home/gib/Server/ For an older day, use a snapshot path instead: /home/gib/Public/Server_Backup/.zfs/snapshot//Containers/server.gib/current/ Then put the databases back from the dumps. For each one, stop the owning container, copy the dump over the live file, and match the original ownership. Plex, for example: docker compose -f ~/Server/Media/compose.yml stop plex D=~/Server/_backup_state/sqlite/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases L=~/Server/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases sudo cp "$D/com.plexapp.plugins.library.db" "$L/" sudo chown --reference="$L/com.plexapp.dlna.db" "$L/com.plexapp.plugins.library.db" docker compose -f ~/Server/Media/compose.yml start plex `chown --reference` copies ownership from a neighbouring file, which saves looking up what uid each container expects. For Convex, start the backend (it can be empty) and import the export through it. This replaces every table with the snapshot: Z=~/Server/_backup_state/convex/gbrown-backend.zip KEY=$(docker exec gbrown-backend ./generate_admin_key.sh | tail -1) docker run --rm -it --network nginx-bridge -v "$Z:/backup.zip:ro" \ -e CONVEX_SELF_HOSTED_URL=http://gbrown-backend:3210 -e CONVEX_SELF_HOSTED_ADMIN_KEY="$KEY" \ docker.io/library/node:22-alpine sh -ec 'mkdir /w && cd /w && echo "{}" > package.json && npm install --silent convex && npx convex import --replace-all /backup.zip' Functions and schema are not in the export. Redeploy them from the app's repo as usual. After a full restore the CI runners will have no `dind` storage. That is fine — they re-pull images on the next job. The registration in `runner-N/data/.runner` is intact, so they reconnect to Gitea without re-registering. ## Checking it is working systemctl list-timers server-backup.timer systemctl status server-backup.service tail -40 ~/Server/logs/backup.log A healthy run ends with `=== backup complete ===`. Watch for: - `ZFS history OK` — confirms snapshots are still being taken. If this ever says **WARNING**, versioning is broken and `current/` is just a mirror: a mistake would propagate with nothing to roll back to. Fix `zfs-auto-snapshot` before anything else. - `rsync exit 24` is a NOTE, not a failure — a file changed mid-transfer, normal on a live box. Optional: put an uptime-kuma push URL in `~/Server/.backup-heartbeat-url` and successful runs will ping it, turning a silent failure into a down monitor.