141 lines
7.3 KiB
Markdown
141 lines
7.3 KiB
Markdown
# Restoring ~/Server on server.gib from the ZFS-pool backup
|
|
|
|
Written 2026-09-17. Read this *before* you need it.
|
|
|
|
## What this protects against
|
|
|
|
The NVMe/LUKS btrfs volume (`/`, `/home`, and `/var/lib/docker`) dying. The backup lives on
|
|
the ZFS pool in the same machine, so it does **not** protect against fire, theft, or the
|
|
whole box failing. It is a disk-failure backup, not an offsite one.
|
|
|
|
## Layout
|
|
|
|
/home/gib/Public/Server_Backup/Containers/server.gib/current/
|
|
|
|
**One directory, overwritten daily at 02:00** by `server-backup.timer`. History lives in ZFS:
|
|
`Media/Server_Backup` has `zfs-auto-snapshot` keeping **14 dailies**, taken by `cron.daily`
|
|
between roughly 03:22 and 03:52. The 02:00 schedule is deliberate — it must finish before
|
|
that window, or a snapshot captures a half-written backup as that day's history. It also
|
|
staggers against the VPS backup, which pushes into this same pool at 01:00.
|
|
|
|
Older copies are under the hidden snapshot dir (`snapdir=hidden`, so `ls` will not show it,
|
|
but you can `cd` in):
|
|
|
|
ls /home/gib/Public/Server_Backup/.zfs/snapshot/
|
|
ls /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
|
|
|
|
The `03-16-2026` and `03-31-2026` directories are leftovers from the old manual script.
|
|
|
|
## Why this runs as root
|
|
|
|
This box runs Docker, so most container data under `~/Server` is root-owned. The old script
|
|
used `sudo rsync`, which worked but meant it could never be automated without a sudoers
|
|
rule. The replacement runs as root via `server-backup.service`, a systemd **system** unit —
|
|
already root, so ownership is preserved natively and **no sudoers entry is needed at
|
|
runtime**. If you see `/etc/sudoers.d/99-claude-setup` on this machine, it was temporary
|
|
setup scaffolding and should be deleted.
|
|
|
|
(This differs from the VPS, which is rootless podman and needs `rsync --fake-super` to carry
|
|
subuid ownership through. Nothing like that is needed here.)
|
|
|
|
The script lives at `/usr/local/sbin/server-backup`, deliberately **not** in `~/Server` — a
|
|
root-executed script inside a user-writable directory is a footgun. `~/Server/backup` is a
|
|
thin wrapper that triggers the systemd unit so manual runs behave identically to nightly ones.
|
|
|
|
## What is in a backup
|
|
|
|
| Path | What it is |
|
|
|---|---|
|
|
| `Media/`, `Frigate/`, `Home/`, `OpenWebUI/`, ... | Raw bind-mount data per service |
|
|
| `_backup_state/sqlite/<original path>` | Online `.backup` of every SQLite DB — **authoritative** |
|
|
| `_backup_state/convex/<container>.zip` | `npx convex export` snapshot of each SQLite-backed Convex deployment (gbrown) — **authoritative** |
|
|
|
|
> **Use `_backup_state/` for databases, not the raw copies.** Containers keep running during
|
|
> the backup, so the raw files are crash-consistent at best. The dumps use SQLite's online
|
|
> backup API and are what will actually restore cleanly. That covers the ones that matter:
|
|
> Plex's `com.plexapp.plugins.library.db` (watch history, ratings), Jellyfin, Frigate, Home
|
|
> Assistant, OpenWebUI, Audiobookshelf, and every *arr config.
|
|
>
|
|
> **Convex is the exception.** It writes to its SQLite file constantly, so a raw `.backup`
|
|
> never completes. Its `_backup_state/convex/` export is the restore source, and the raw
|
|
> `db.sqlite3` is not dumped at all.
|
|
|
|
## What is deliberately NOT backed up
|
|
|
|
- **`Agents/Gitea/volumes/runner-{1,2,3}/dind` — ~203 GB.** Docker-in-Docker containerd
|
|
image layers for the CI runners. Pure cache that churns constantly; a rebuilt runner just
|
|
re-pulls it. The parts that matter — `runner-N/data/.runner` (the 648-byte registration
|
|
token) and `certs/` — **are** backed up.
|
|
> Measured as root. A plain `du` as `gib` reports these as ~13 MB because gib cannot
|
|
> descend into them. Do not trust non-root sizes here.
|
|
- Docker **named volumes** (`/var/lib/docker/volumes`, 47 in use, ~61 GB): Sentry (13) and
|
|
Plausible (4) were consciously excluded — Sentry's event data is not worth restoring and
|
|
Plausible had nothing in it. Romm looked like a gap but is not: its anonymous `/romm`
|
|
volume is only a parent directory, with `assets/`, `config/` and `resources/` bind-mounted
|
|
from `~/Server` and `library/` from the pool.
|
|
- **Nextcloud AIO** — has its own borg backup at `~/Public/Nextcloud/Backups`, run by
|
|
hand from the AIO admin UI. Last run 2026-09-17 (finished 09-18). Runs are slow and
|
|
AIO stops the Nextcloud containers while one is going.
|
|
- `node_modules`, build output, `*.log`, `cache`/`tmp` dirs, `*.mp4`/`*.mkv`.
|
|
|
|
## Restoring
|
|
|
|
Stop the stack first so nothing overwrites what you restore:
|
|
|
|
cd ~/Server/<service> && docker compose down # per service, or stop docker entirely
|
|
|
|
Pull it back — as root, so ownership is restored:
|
|
|
|
sudo rsync -a --numeric-ids --delete \
|
|
/home/gib/Public/Server_Backup/Containers/server.gib/current/ \
|
|
/home/gib/Server/
|
|
|
|
For an older day, use a snapshot path instead:
|
|
|
|
/home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
|
|
|
|
Then put the databases back from the dumps. For each one, stop the owning container, copy the
|
|
dump over the live file, and match the original ownership. Plex, for example:
|
|
|
|
docker compose -f ~/Server/Media/compose.yml stop plex
|
|
D=~/Server/_backup_state/sqlite/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
|
|
L=~/Server/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
|
|
sudo cp "$D/com.plexapp.plugins.library.db" "$L/"
|
|
sudo chown --reference="$L/com.plexapp.dlna.db" "$L/com.plexapp.plugins.library.db"
|
|
docker compose -f ~/Server/Media/compose.yml start plex
|
|
|
|
`chown --reference` copies ownership from a neighbouring file, which saves looking up what
|
|
uid each container expects.
|
|
|
|
For Convex, start the backend (it can be empty) and import the export through it. This
|
|
replaces every table with the snapshot:
|
|
|
|
Z=~/Server/_backup_state/convex/gbrown-backend.zip
|
|
KEY=$(docker exec gbrown-backend ./generate_admin_key.sh | tail -1)
|
|
docker run --rm -it --network nginx-bridge -v "$Z:/backup.zip:ro" \
|
|
-e CONVEX_SELF_HOSTED_URL=http://gbrown-backend:3210 -e CONVEX_SELF_HOSTED_ADMIN_KEY="$KEY" \
|
|
docker.io/library/node:22-alpine sh -ec 'mkdir /w && cd /w && echo "{}" > package.json &&
|
|
npm install --silent convex && npx convex import --replace-all /backup.zip'
|
|
|
|
Functions and schema are not in the export. Redeploy them from the app's repo as usual.
|
|
|
|
After a full restore the CI runners will have no `dind` storage. That is fine — they re-pull
|
|
images on the next job. The registration in `runner-N/data/.runner` is intact, so they
|
|
reconnect to Gitea without re-registering.
|
|
|
|
## Checking it is working
|
|
|
|
systemctl list-timers server-backup.timer
|
|
systemctl status server-backup.service
|
|
tail -40 ~/Server/logs/backup.log
|
|
|
|
A healthy run ends with `=== backup complete ===`. Watch for:
|
|
|
|
- `ZFS history OK` — confirms snapshots are still being taken. If this ever says **WARNING**,
|
|
versioning is broken and `current/` is just a mirror: a mistake would propagate with
|
|
nothing to roll back to. Fix `zfs-auto-snapshot` before anything else.
|
|
- `rsync exit 24` is a NOTE, not a failure — a file changed mid-transfer, normal on a live box.
|
|
|
|
Optional: put an uptime-kuma push URL in `~/Server/.backup-heartbeat-url` and successful runs
|
|
will ping it, turning a silent failure into a down monitor.
|