Sync both server snapshots with what the live machines run, and keep their backup tooling

server/home/RESTORE.md

@@ -0,0 +1,140 @@
+# Restoring ~/Server on server.gib from the ZFS-pool backup
+
+Written 2026-09-17. Read this *before* you need it.
+
+## What this protects against
+
+The NVMe/LUKS btrfs volume (`/`, `/home`, and `/var/lib/docker`) dying. The backup lives on
+the ZFS pool in the same machine, so it does **not** protect against fire, theft, or the
+whole box failing. It is a disk-failure backup, not an offsite one.
+
+## Layout
+
+    /home/gib/Public/Server_Backup/Containers/server.gib/current/
+
+**One directory, overwritten daily at 02:00** by `server-backup.timer`. History lives in ZFS:
+`Media/Server_Backup` has `zfs-auto-snapshot` keeping **14 dailies**, taken by `cron.daily`
+between roughly 03:22 and 03:52. The 02:00 schedule is deliberate — it must finish before
+that window, or a snapshot captures a half-written backup as that day's history. It also
+staggers against the VPS backup, which pushes into this same pool at 01:00.
+
+Older copies are under the hidden snapshot dir (`snapdir=hidden`, so `ls` will not show it,
+but you can `cd` in):
+
+    ls /home/gib/Public/Server_Backup/.zfs/snapshot/
+    ls /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
+
+The `03-16-2026` and `03-31-2026` directories are leftovers from the old manual script.
+
+## Why this runs as root
+
+This box runs Docker, so most container data under `~/Server` is root-owned. The old script
+used `sudo rsync`, which worked but meant it could never be automated without a sudoers
+rule. The replacement runs as root via `server-backup.service`, a systemd **system** unit —
+already root, so ownership is preserved natively and **no sudoers entry is needed at
+runtime**. If you see `/etc/sudoers.d/99-claude-setup` on this machine, it was temporary
+setup scaffolding and should be deleted.
+
+(This differs from the VPS, which is rootless podman and needs `rsync --fake-super` to carry
+subuid ownership through. Nothing like that is needed here.)
+
+The script lives at `/usr/local/sbin/server-backup`, deliberately **not** in `~/Server` — a
+root-executed script inside a user-writable directory is a footgun. `~/Server/backup` is a
+thin wrapper that triggers the systemd unit so manual runs behave identically to nightly ones.
+
+## What is in a backup
+
+| Path | What it is |
+|---|---|
+| `Media/`, `Frigate/`, `Home/`, `OpenWebUI/`, ... | Raw bind-mount data per service |
+| `_backup_state/sqlite/<original path>` | Online `.backup` of every SQLite DB — **authoritative** |
+| `_backup_state/convex/<container>.zip` | `npx convex export` snapshot of each SQLite-backed Convex deployment (gbrown) — **authoritative** |
+
+> **Use `_backup_state/` for databases, not the raw copies.** Containers keep running during
+> the backup, so the raw files are crash-consistent at best. The dumps use SQLite's online
+> backup API and are what will actually restore cleanly. That covers the ones that matter:
+> Plex's `com.plexapp.plugins.library.db` (watch history, ratings), Jellyfin, Frigate, Home
+> Assistant, OpenWebUI, Audiobookshelf, and every *arr config.
+>
+> **Convex is the exception.** It writes to its SQLite file constantly, so a raw `.backup`
+> never completes. Its `_backup_state/convex/` export is the restore source, and the raw
+> `db.sqlite3` is not dumped at all.
+
+## What is deliberately NOT backed up
+
+- **`Agents/Gitea/volumes/runner-{1,2,3}/dind` — ~203 GB.** Docker-in-Docker containerd
+  image layers for the CI runners. Pure cache that churns constantly; a rebuilt runner just
+  re-pulls it. The parts that matter — `runner-N/data/.runner` (the 648-byte registration
+  token) and `certs/` — **are** backed up.
+  > Measured as root. A plain `du` as `gib` reports these as ~13 MB because gib cannot
+  > descend into them. Do not trust non-root sizes here.
+- Docker **named volumes** (`/var/lib/docker/volumes`, 47 in use, ~61 GB): Sentry (13) and
+  Plausible (4) were consciously excluded — Sentry's event data is not worth restoring and
+  Plausible had nothing in it. Romm looked like a gap but is not: its anonymous `/romm`
+  volume is only a parent directory, with `assets/`, `config/` and `resources/` bind-mounted
+  from `~/Server` and `library/` from the pool.
+- **Nextcloud AIO** — has its own borg backup at `~/Public/Nextcloud/Backups`, run by
+  hand from the AIO admin UI. Last run 2026-09-17 (finished 09-18). Runs are slow and
+  AIO stops the Nextcloud containers while one is going.
+- `node_modules`, build output, `*.log`, `cache`/`tmp` dirs, `*.mp4`/`*.mkv`.
+
+## Restoring
+
+Stop the stack first so nothing overwrites what you restore:
+
+    cd ~/Server/<service> && docker compose down     # per service, or stop docker entirely
+
+Pull it back — as root, so ownership is restored:
+
+    sudo rsync -a --numeric-ids --delete \
+      /home/gib/Public/Server_Backup/Containers/server.gib/current/ \
+      /home/gib/Server/
+
+For an older day, use a snapshot path instead:
+
+    /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
+
+Then put the databases back from the dumps. For each one, stop the owning container, copy the
+dump over the live file, and match the original ownership. Plex, for example:
+
+    docker compose -f ~/Server/Media/compose.yml stop plex
+    D=~/Server/_backup_state/sqlite/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
+    L=~/Server/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
+    sudo cp "$D/com.plexapp.plugins.library.db" "$L/"
+    sudo chown --reference="$L/com.plexapp.dlna.db" "$L/com.plexapp.plugins.library.db"
+    docker compose -f ~/Server/Media/compose.yml start plex
+
+`chown --reference` copies ownership from a neighbouring file, which saves looking up what
+uid each container expects.
+
+For Convex, start the backend (it can be empty) and import the export through it. This
+replaces every table with the snapshot:
+
+    Z=~/Server/_backup_state/convex/gbrown-backend.zip
+    KEY=$(docker exec gbrown-backend ./generate_admin_key.sh | tail -1)
+    docker run --rm -it --network nginx-bridge -v "$Z:/backup.zip:ro" \
+      -e CONVEX_SELF_HOSTED_URL=http://gbrown-backend:3210 -e CONVEX_SELF_HOSTED_ADMIN_KEY="$KEY" \
+      docker.io/library/node:22-alpine sh -ec 'mkdir /w && cd /w && echo "{}" > package.json &&
+        npm install --silent convex && npx convex import --replace-all /backup.zip'
+
+Functions and schema are not in the export. Redeploy them from the app's repo as usual.
+
+After a full restore the CI runners will have no `dind` storage. That is fine — they re-pull
+images on the next job. The registration in `runner-N/data/.runner` is intact, so they
+reconnect to Gitea without re-registering.
+
+## Checking it is working
+
+    systemctl list-timers server-backup.timer
+    systemctl status server-backup.service
+    tail -40 ~/Server/logs/backup.log
+
+A healthy run ends with `=== backup complete ===`. Watch for:
+
+- `ZFS history OK` — confirms snapshots are still being taken. If this ever says **WARNING**,
+  versioning is broken and `current/` is just a mirror: a mistake would propagate with
+  nothing to roll back to. Fix `zfs-auto-snapshot` before anything else.
+- `rsync exit 24` is a NOTE, not a failure — a file changed mid-transfer, normal on a live box.
+
+Optional: put an uptime-kuma push URL in `~/Server/.backup-heartbeat-url` and successful runs
+will ping it, turning a silent failure into a down monitor.