Sync both server snapshots with what the live machines run, and keep their backup tooling

2026-10-30 13:44:29 -04:00
1 parent 95fa480469
commit cb089b39ea
@@ -8,7 +8,9 @@ screen, plus rootless podman compose services managed as systemd user units.
 server/
  vps/            The catalog panama-server installs from, one directory
                  per service: compose.yml, the podman-<name>.service user
-                  unit, and an .env.example naming what the service needs told
+                  unit, and an .env.example naming what the service needs told.
+                  Loose files beside the directories are the VPS's own
+                  host tooling (see "Host tooling" below)
  home/           The home server's services in the same shape (see below)
  scripts/        update-containers, the nightly image updater
  systemd/        The units behind it, linked by setup-server
@@ -25,6 +27,8 @@ installable: panama-server reads `server/vps/` only, so a service moves into
 the catalog when it actually moves to podman.

 Paths keep their nesting (`server/home/Agents/Gitea` is `~/Server/Agents/Gitea`).
+`Media/compact-tv/` is the nightly TV re-encode job that runs beside the Media
+stack, with its own README.
 Three services are checkouts of upstream repositories. For those, only the
 local files are kept, and an `UPSTREAM` file names the repository and the exact
 commit the checkout sits at: `PostHog`, `Sentry` and `Plausible`.
@@ -38,7 +42,8 @@ compose and its copy here can differ on those lines alone.

 The repository carries the definitions; the machine carries the state.
 `panama server enable <Name>` creates `~/Server/<Name>/` as a **real
-directory** and symlinks only `compose.yml` into it; the unit symlinks into
+directory** and symlinks the service's visible top-level files into it:
+`compose.yml`, plus any helper script or README. The units symlink into
 `~/.config/systemd/user/`. The `.env` (seeded once from `.env.example`, then
 yours) and the bind-mounted `data/` live in `~/Server/<Name>/` and are never
 inside the checkout — so nothing git can do, `clean -fdx` included, can reach
@@ -99,6 +104,39 @@ existing service. The unit is the standard oneshot `podman compose up -d`
 shape (`WorkingDirectory=%h/Server/<Name>`); the shape contract pins the
 details. Secrets go in `.env.example` as `KEY=CHANGE_ME` lines.

+A service that needs a scheduled job of its own carries it beside the
+compose unit: a helper `<job>.service` and a `<job>.timer` with the same
+name, plus the script it runs. Enable links all three and enables the timer.
+Disable stops the timer and unlinks the units. `UseSend` is the example: its
+`usesend-cert-refresh` timer keeps the SMTP proxy's certificate in step with
+Nginx Proxy Manager's.
+
+## Host tooling
+
+Each server's own scripts and runbooks sit as loose files at the top of its
+tree. They are reference copies for rebuilding a machine. Nothing installs
+them.
+
+- `vps/backup`, `vps/server-backup.service`, `vps/server-backup.timer`: the
+  VPS's nightly backup, run as a user unit. It dumps every database into
+  `~/Server/_backup_state/`, then rsyncs `~/Server` to the home server's ZFS
+  pool with `--fake-super`, so subuid ownership survives the trip.
+- `vps/verify-rootless`: the post-reboot health check for the rootless
+  stack. It covers linger, the unprivileged port floor, unit state and boot
+  enablement, postgres, endpoints, and admin ports bound to the WireGuard
+  address.
+- `vps/AGENTS.md`: the VPS runbook, with both public IPs redacted.
+- `home/RESTORE.md`: how to restore the home server's `~/Server` from the
+  ZFS backup.
+- `home/backup` and `home/systemd/`: the home server's backup trigger and its
+  root system units (`server-backup` nightly, `media-backup` weekly). The
+  scripts those units run live in `/usr/local/sbin` on that machine. They are
+  readable by root alone, so this tree has no copy of them.
+
+The VPS units carry `RequiresMountsFor=/home/gib/Media` because that host
+keeps its podman graphroot on an attached volume mounted there. The setting
+is harmless on a machine without the mount.
+
 ## Nightly updates

 `server/scripts/update-containers` pulls images and restarts changed
@@ -19,7 +19,7 @@ x-runner-env: &runner-env
  TZ: America/New_York
  GITEA_INSTANCE_URL: https://git.gbrown.org
  GITEA_RUNNER_REGISTRATION_TOKEN: ${GITEA_RUNNER_REGISTRATION_TOKEN}
-  GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04"
+  GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04,build:docker://docker.gitea.com/runner-images:ubuntu-latest"
  DOCKER_TLS_VERIFY: "1"
  DOCKER_CERT_PATH: /certs/client
  CONFIG_FILE: /data/config.yml
@@ -18,7 +18,7 @@ services:
  #   restart: unless-stopped

  portainer-agent:
-    image: portainer/agent:2.33.2
+    image: portainer/agent:latest
    container_name: portainer-agent
    hostname: portainer-agent
    networks:
@@ -43,8 +43,8 @@ volumes:

 services:
  gbrown-backend:
-    # Pinned to the build the VPS Convex stacks already run. Bump on purpose.
-    image: ghcr.io/get-convex/convex-backend@sha256:d715e9ec088784407ca4ba2d3db592702cd328d02c76cdca3852c0018f2a76b4
+    # Pinned to the build the VPS Convex stacks run (bumped 2026-10-28). Bump on purpose.
+    image: ghcr.io/convex/convex-backend@sha256:207cf95def42f01614b6e3af0af5c20d0dcaa5ff2adf258ee7bc9555576644fa
    container_name: gbrown-backend
    hostname: gbrown-backend
    networks: ['nginx-bridge']
@@ -70,7 +70,7 @@ services:

  gbrown-dashboard:
    # Released alongside the backend; pinned with it.
-    image: ghcr.io/get-convex/convex-dashboard@sha256:f85cf0d0448b9c835ae3df5c7c1c6f0145dd4c8f0f92eb0245daf304efdd9f1a
+    image: ghcr.io/convex/convex-dashboard@sha256:a55bf8abca1332f787bc4d4a79850182175027be7b70aa230be4389a1d3268e2
    container_name: gbrown-dashboard
    hostname: gbrown-dashboard
    networks: ['nginx-bridge']
@@ -0,0 +1,109 @@
+# compact-tv
+
+Sonarr and Radarr each have exactly four quality profiles. Two of them (**Standard** and
+**Mini**) also have their files shrunk by `compact-tv.py`, which runs every night.
+
+| Profile | Used for | What gets kept |
+|---|---|---|
+| **Mini** | shows I demote hard | TV: 720p at 800 kbps (~0.42 GB per hour). Movies: 720p at 1500 kbps |
+| **Standard** | **every request from other users (Seerr default)**, and anything I demote | TV: 1080p at 1200 kbps (~0.6 GB per hour). Movies: 1080p at 2500 kbps (~1.2 GB per hour) |
+| **HD** | my own library, unchanged | whatever downloads, up to 1080p |
+| **UHD** | only me, through Seerr's 4K servers | 4K |
+
+SD sources keep their resolution, at 600 kbps for TV and 900 kbps for movies. Movies get
+higher rates because dark scenes, grain and action break down first at low bitrates. Nothing is ever
+upscaled. Audio becomes Opus: 96 kbps for stereo, 160 for 5.1, 192 for 7.1. Subtitles,
+chapters and attachments are copied as they are. I picked the rates from sample encodes of
+Shameless S05E06 (VMAF ~91 for Standard, ~86 for Mini).
+
+## How the profiles work
+
+- **Standard and Mini** are copies of HD that upgrade only toward small releases (see below),
+  so Sonarr and Radarr never replace a file with a bigger one.
+- **Small downloads first.** Both profiles prefer small releases when they grab:
+  - The "Small Encode" format (MeGusta, PSA, ELiTE, GalaxyTV/RG, YTS/YIFY) scores +3000.
+  - LQ scores 0. In HD and UHD it scores -10000, which keeps those groups out of HD.
+  - Standard merges HDTV/WEB/Bluray at each resolution, so the format score decides between
+    sources rather than the source type.
+  - Mini merges 720p and 1080p into one group and adds +1000 for 720p.
+  - Many files therefore arrive already small, and the job skips them.
+- **Minimum sizes are shared by every profile.** For 720p I lowered them to 5 MB per minute
+  and for 1080p to 6 MB per minute (8 for Bluray-1080p), so small encodes are not rejected.
+  HD still never takes the LQ groups.
+- **Seerr:**
+  - The "Sonarr - HD" and "Radarr" servers default to Standard. Only my admin account sees
+    the profile picker, so other users always get Standard. I pick HD or UHD per request.
+  - Other users no longer have "Request 4K".
+- **Configarr is opt-in** (`docker compose --profile configarr up configarr`). Its templates
+  would recreate the old TRaSH profiles and size limits.
+- Backups from before all this: `backups/pre-four-profiles-20261028/`. That covers both
+  apps' profiles, custom formats and size definitions, which title was on which profile,
+  and Seerr's settings. `setup-small-profiles.py` reapplies the Standard and Mini settings.
+
+## How the job works
+
+The nightly run (00:30 to 08:00) has two phases.
+
+**1. Search first.** It asks Sonarr and Radarr to search titles on Standard or Mini that it
+hasn't searched before, biggest first: 5 shows and 20 movies a night, so downloads and
+indexer use stay spread out. Standard and Mini upgrade only toward small releases:
+- Every allowed quality sits in one group, so no resolution or source upgrade can happen.
+- An upgrade needs a custom-format score jump of at least 2500, and upgrades stop at 3000.
+  Only gaining "Small Encode" clears that bar.
+
+So a search replaces a big file only with a small encode, never with a big HEVC release.
+After that, the apps' regular RSS checks keep picking up small releases on their own.
+
+**2. Encode what's left.** A title becomes eligible for encoding 7 days after its search,
+and only its files that are still big get encoded. Files that arrive after the title's
+search work differently:
+- This covers new episodes of ongoing shows, and fresh requests.
+- The first night they're seen, they get their own search (one episode or movie at a time).
+- They're encoded only if they're still big 3 days later.
+
+Downloads never wait for any of this, so new episodes and requests land in the library as
+soon as they download. Only encoding waits. For each file, the job encodes to
+`~/.cache/compact-tv/work.mkv` on the NVMe and then checks the result:
+- the duration matches the original
+- the audio and subtitle track counts match
+- the file is smaller
+
+It then renames the new file over the original, keeping the same name with `.mkv`, and asks
+Sonarr or Radarr to rescan. Plex picks the change up through its folder watcher, and the
+episode keeps its Plex ID, so watch history stays. If a small release shows up later, the
+app may still replace an encoded file with it, which costs bandwidth, not GPU.
+
+Other details:
+- Each output carries a `COMPACT_TIER` tag. A file is skipped if re-encoding would save less
+  than 30%. Ruled-out files are cached by size in `state.json`, so they aren't probed every
+  night. Search dates and failures are kept there too.
+- A downloaded HEVC file no bigger than twice the target size counts as done. That is the
+  small release a search brought in (a MeGusta 1080p episode runs about 2.3 Mbps against
+  Standard's 1.2), and it is neither re-encoded nor searched again. The job's own tagged
+  outputs are exempt, so Standard to Mini still works.
+- **Skipped:** HDR and Dolby Vision sources (they need tone mapping), subtitle tracks ffprobe
+  can't identify, and files that have failed twice.
+- Speed is about 0.19 hours of GPU per hour of 1080p video (NVENC is the bottleneck). The job
+  stops starting new encodes at 08:00.
+- `cost-report.py` estimates, per title, the GPU hours the job would need against the size of
+  a fresh small download.
+- A `PAUSE` file in this folder makes the nightly run do nothing.
+
+## Going back up
+
+Moving down works: Standard to Mini re-encodes the Standard copy. Moving up does not
+restore quality. To get quality back, I set the title to HD and search, which redownloads
+it at full size.
+
+## Running it
+
+    ./compact-tv.py --dry-run                      # what tonight would do
+    ./compact-tv.py --search-all --limit 0         # search every unsearched title now, encode nothing
+    touch PAUSE                                    # skip nightly runs until PAUSE is deleted
+    ./compact-tv.py --series "Shameless (US)" --limit 1
+    systemctl status compact-tv.timer compact-tv.service
+    tail -f compact-tv.log
+
+The installed units are copies of `compact-tv.service` and `compact-tv.timer` from this
+folder. `ExecStart` goes through `/usr/bin/python3` because SELinux won't let systemd
+execute a file labeled as home content.
@@ -0,0 +1,348 @@
+#!/usr/bin/env python3
+"""Shrink shows and movies that Sonarr/Radarr have on the "Standard" or "Mini" profile.
+
+Demoting something is just picking one of those profiles. Each night this job first asks
+Sonarr/Radarr to search a few titles it hasn't searched before. Both profiles upgrade only
+toward small releases, so the search swaps whatever big files have a small release for
+bandwidth instead of GPU. A week after a title's search, the job re-encodes whatever is
+still big to HEVC on the GPU (NVENC) at the tier's bitrate, verifies the result, swaps it in
+place of the original, and has Sonarr/Radarr rescan. Plex and Jellyfin pick up changes
+through their own folder watchers.
+
+Moving something back up means switching it to HD and searching, which redownloads it at
+full size. See README.md.
+
+Usage:
+  compact-tv.py [--until HH:MM] [--series TITLE] [--limit N] [--dry-run]
+
+--series matches a show or movie title. Runs as gib (files are gib-owned) from
+compact-tv.service, nightly.
+"""
+import argparse, datetime, fcntl, json, os, pathlib, re, shutil, subprocess, sys, time, urllib.request
+
+HERE = pathlib.Path(__file__).resolve().parent
+STATE = HERE / "state.json"
+LOG = HERE / "compact-tv.log"
+WORK = pathlib.Path.home() / ".cache" / "compact-tv"   # NVMe, so half-done encodes never land in a ZFS snapshot
+VOLUMES = pathlib.Path.home() / "Server/Media/volumes"
+
+# Per app: API port, container path -> host path, and how to list one title's files.
+APPS = {
+    "sonarr": {"port": 8989, "paths": ("/tv/", "/home/gib/Public/TV/"), "titles": "/series",
+               "files": lambda call, t: call(f"/episodefile?seriesId={t['id']}"),
+               "rescan": lambda t: {"name": "RescanSeries", "seriesId": t["id"]}},
+    "radarr": {"port": 7878, "paths": ("/movies/", "/home/gib/Public/Movies/"), "titles": "/movie",
+               "files": lambda call, t: [t["movieFile"]] if t.get("movieFile") else [],
+               "rescan": lambda t: {"name": "RescanMovie", "movieId": t["id"]}},
+}
+
+# Video bitrate (kbps) per output class, and the widest output each tier allows.
+# TV floors come from watching samples: 1200 kbps at 1080p, 800 at 720p, 600 at SD.
+# Movies get roughly double: dark scenes, grain and action break down first at low rates,
+# and a 2-hour movie at 2500 kbps is still ~2.3 GB (a quarter of a typical 1080p file).
+# That also leaves already-small releases (YTS-style, ~1.5-2 GB) under the 30% bar.
+# Nothing is ever upscaled: a 720p or SD source keeps its size and uses that class's rate.
+TIERS = {
+    "Standard": {"max_width": 1920, "kbps": {"sonarr": {"1080": 1200, "720": 1200, "sd": 600},
+                                             "radarr": {"1080": 2500, "720": 1800, "sd": 900}}},
+    "Mini":     {"max_width": 1280, "kbps": {"sonarr": {"1080": 800,  "720": 800,  "sd": 600},
+                                             "radarr": {"1080": 1500, "720": 1500, "sd": 900}}},
+}
+MIN_SAVING = 0.30            # leave a file alone unless re-encoding saves at least this much
+MAX_FAILURES = 2             # after this many failed attempts a file is skipped for good
+
+# Search first, encode later. Standard/Mini upgrade only toward small releases (see
+# setup-small-profiles.py), so asking the *arr to search a title swaps whatever big files
+# have a small release, for bandwidth instead of GPU. Each night searches a few titles not
+# searched before, biggest first, so downloads and indexer use stay spread out. A title's
+# files are only encoded once its search is SEARCH_GRACE_DAYS old.
+#
+# Files that arrive after their title's search (new episodes, fresh requests) were never
+# searched for a small copy, so the job searches that episode or movie on its own first,
+# and encodes it only if it is still big FILE_SEARCH_GRACE_DAYS later. Downloads are never
+# delayed by any of this; only encoding waits.
+SEARCHES_PER_NIGHT = {"sonarr": 5, "radarr": 20}
+SEARCH_GRACE_DAYS = 7
+FILE_SEARCH_GRACE_DAYS = 3        # small encodes usually appear within hours to a day; 3 leaves room
+HDR_TRANSFERS = {"smpte2084", "arib-std-b67"}
+
+
+def log(msg):
+    line = f"{datetime.datetime.now():%Y-%m-%d %H:%M:%S}  {msg}"
+    print(line, flush=True)
+    with open(LOG, "a") as f:
+        f.write(line + "\n")
+
+
+# ── Sonarr / Radarr ─────────────────────────────────────────────────────────
+def arr(app):
+    """Return a call(path, body=None) for the app's v3 API."""
+    key = re.search(r"<ApiKey>([^<]+)", (VOLUMES / app / "config.xml").read_text()).group(1)
+    ip = subprocess.run(["docker", "inspect", app, "--format",
+                         "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"],
+                        capture_output=True, text=True, check=True).stdout.split()[0]
+    base = f"http://{ip}:{APPS[app]['port']}/api/v3"
+
+    def call(path, body=None):
+        req = urllib.request.Request(base + path, headers={"X-Api-Key": key, "Content-Type": "application/json"},
+                                     data=json.dumps(body).encode() if body is not None else None)
+        with urllib.request.urlopen(req, timeout=60) as r:
+            return json.load(r)
+    return call
+
+
+# ── probing and planning ────────────────────────────────────────────────────
+def probe(path):
+    out = subprocess.run(["ffprobe", "-v", "error", "-print_format", "json", "-show_format", "-show_streams", str(path)],
+                         capture_output=True, text=True)
+    return json.loads(out.stdout) if out.returncode == 0 else None
+
+
+def subtitles(info):
+    """Subtitle tracks ffmpeg can carry over. Tracks ffprobe can't identify are dropped."""
+    return [s for s in info["streams"] if s["codec_type"] == "subtitle" and s.get("codec_name")]
+
+
+def audio_kbps(channels):
+    return 96 if channels <= 2 else 160 if channels <= 6 else 192
+
+
+def plan(info, tier, app):
+    """Return (ffmpeg args, estimated output bytes) for one file, or (None, reason) to skip it."""
+    t = TIERS[tier]
+    video = next((s for s in info["streams"] if s["codec_type"] == "video"
+                  and not s.get("disposition", {}).get("attached_pic")), None)
+    if not video:
+        return None, "no video stream"
+    if video.get("color_transfer") in HDR_TRANSFERS or any("Dolby Vision" in str(sd) for sd in video.get("side_data_list", [])):
+        return None, "HDR/Dolby Vision source (not handled yet)"
+    if (info["format"].get("tags") or {}).get("COMPACT_TIER") == tier:
+        return None, f"already {tier}"
+
+    src_w = int(video["width"])
+    out_w = min(src_w, t["max_width"])
+    cls = "1080" if out_w >= 1700 else "720" if out_w >= 1200 else "sd"
+    vkbps = t["kbps"][app][cls]
+    duration = float(info["format"]["duration"])
+    audios = [s for s in info["streams"] if s["codec_type"] == "audio"]
+    est = int((vkbps + sum(audio_kbps(int(a.get("channels") or 2)) for a in audios)) * 1000 / 8 * duration * 1.03)
+    if int(info["format"]["size"]) * (1 - MIN_SAVING) < est:
+        return None, f"already small ({int(info['format']['size']) / 1e9 * 3600 / duration:.2f} GB/h)"
+    # A downloaded HEVC file within 2x the target is a small release that already replaced a
+    # big one. Re-encoding it would cost GPU and quality for little space. Our own outputs
+    # (tagged) still go through, so moving a title from Standard to Mini works.
+    tags = info["format"].get("tags") or {}
+    if video.get("codec_name") == "hevc" and "COMPACT_TIER" not in tags and int(info["format"]["size"]) <= 2 * est:
+        return None, f"small HEVC release ({int(info['format']['size']) / 1e9 * 3600 / duration:.2f} GB/h)"
+
+    vf = []
+    if video.get("field_order") not in (None, "progressive", "unknown"):
+        vf.append("bwdif")
+    if out_w < src_w:
+        vf.append(f"scale={out_w}:-2:flags=lanczos")
+    args = ["-map", f"0:{video['index']}"]
+    if vf:
+        args += ["-filter:v", ",".join(vf)]
+    args += ["-c:v", "hevc_nvenc", "-preset", "p7", "-tune", "hq", "-profile:v", "main10", "-pix_fmt", "p010le",
+             "-rc", "vbr", "-multipass", "fullres", "-spatial-aq", "1", "-temporal-aq", "1", "-rc-lookahead", "32",
+             "-bf", "4", "-b_ref_mode", "middle",
+             "-b:v", f"{vkbps}k", "-maxrate", f"{vkbps * 2}k", "-bufsize", f"{vkbps * 4}k"]
+    for i, a in enumerate(audios):
+        ch = int(a.get("channels") or 2)
+        layout = {1: "mono", 2: "stereo", 6: "5.1", 8: "7.1"}.get(ch)
+        args += ["-map", f"0:{a['index']}", f"-c:a:{i}", "libopus", f"-b:a:{i}", f"{audio_kbps(ch)}k",
+                 f"-filter:a:{i}", f"aformat=channel_layouts={layout or 'stereo'}"]
+    args += ["-mapping_family", "1"]
+    for i, s in enumerate(subtitles(info)):
+        args += ["-map", f"0:{s['index']}", f"-c:s:{i}", "srt" if s["codec_name"] == "mov_text" else "copy"]
+    args += ["-map", "0:t?", "-c:t", "copy", "-metadata", f"COMPACT_TIER={tier}"]
+    return args, est
+
+
+# ── encode, verify, swap ────────────────────────────────────────────────────
+def encode(src, args, tier):
+    WORK.mkdir(parents=True, exist_ok=True)
+    tmp = WORK / "work.mkv"
+    tmp.unlink(missing_ok=True)
+    cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-hwaccel", "cuda", "-i", str(src),
+           *args, "-metadata", f"COMPACT_SOURCE={src.name} ({src.stat().st_size} bytes)", str(tmp)]
+    r = subprocess.run(cmd, capture_output=True, text=True)
+    if r.returncode != 0:
+        tmp.unlink(missing_ok=True)
+        raise RuntimeError("ffmpeg failed: " + r.stderr.strip()[-300:])
+    return tmp
+
+
+def verify(src_info, tmp):
+    new = probe(tmp)
+    if not new:
+        raise RuntimeError("output does not probe")
+    a, b = float(src_info["format"]["duration"]), float(new["format"]["duration"])
+    if abs(a - b) > max(2.0, a * 0.005):
+        raise RuntimeError(f"duration mismatch {a:.1f}s vs {b:.1f}s")
+    counts = lambda i: (sum(1 for s in i["streams"] if s["codec_type"] == "audio"), len(subtitles(i)))
+    if counts(src_info) != counts(new):
+        raise RuntimeError(f"audio/subtitle track counts {counts(src_info)} -> {counts(new)}")
+    if int(new["format"]["size"]) >= int(src_info["format"]["size"]):
+        raise RuntimeError("output is not smaller")
+    return int(new["format"]["size"])
+
+
+def swap(src, tmp):
+    """Copy the encode next to the original, then rename it over the original in one step."""
+    final = src.with_suffix(".mkv")
+    staged = src.with_name(f".{final.name}.compact-tmp")
+    shutil.copyfile(tmp, staged)
+    os.replace(staged, final)
+    if final != src:
+        src.unlink()
+    tmp.unlink(missing_ok=True)
+    return final
+
+
+# ── main ────────────────────────────────────────────────────────────────────
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--until", help="stop starting new encodes after this local time (HH:MM)")
+    ap.add_argument("--series", help="only this series title")
+    ap.add_argument("--limit", type=int, help="encode at most N files")
+    ap.add_argument("--dry-run", action="store_true", help="plan only, change nothing")
+    ap.add_argument("--search-all", action="store_true",
+                    help="search every not-yet-searched title now instead of a few per night")
+    opt = ap.parse_args()
+
+    if (HERE / "PAUSE").exists() and not opt.dry_run:
+        log("PAUSE file present, not encoding anything (delete it to resume)")
+        return
+
+    lock = open(HERE / ".lock", "w")
+    try:
+        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+    except BlockingIOError:
+        sys.exit("another compact-tv run is active")
+
+    deadline = None
+    if opt.until:
+        h, m = map(int, opt.until.split(":"))
+        now = datetime.datetime.now()
+        deadline = now.replace(hour=h, minute=m, second=0, microsecond=0)
+        if deadline <= now:
+            deadline += datetime.timedelta(days=1)
+
+    state = json.loads(STATE.read_text()) if STATE.exists() else {}
+    for k in ("failures", "searched", "skip"):
+        state.setdefault(k, {})
+    save = lambda: STATE.write_text(json.dumps(state, indent=1))
+    today = datetime.date.today()
+    work = []                                  # (app, call, title, tier); TV first, then movies
+    for app, a in APPS.items():
+        call = arr(app)
+        profiles = {p["id"]: p["name"] for p in call("/qualityprofile")}
+        work += [(app, call, t, profiles[t["qualityProfileId"]]) for t in call(a["titles"])
+                 if profiles.get(t["qualityProfileId"]) in TIERS and (not opt.series or t["title"] == opt.series)]
+    log(f"=== run start{' [dry run]' if opt.dry_run else ''}: {len(work)} title(s) on "
+        f"{'/'.join(TIERS)}{f', until {deadline:%H:%M}' if deadline else ''} ===")
+
+    # ── 1. search: ask the *arr for small replacements before spending GPU ──
+    size_of = lambda t: (t.get("statistics") or {}).get("sizeOnDisk") or t.get("sizeOnDisk") or 0
+    for app in APPS:
+        fresh = sorted((w for w in work if w[0] == app and f"{app}:{w[2]['id']}" not in state["searched"]
+                        and size_of(w[2]) > 0), key=lambda w: -size_of(w[2]))
+        fresh = fresh if opt.search_all else fresh[:SEARCHES_PER_NIGHT[app]]
+        for _, call, t, tier in fresh:
+            log(f"  {'would search' if opt.dry_run else 'search'} {t['title']} ({size_of(t) / 1e9:.0f} GB, {tier})")
+        if fresh and not opt.dry_run:
+            call = fresh[0][1]
+            if app == "sonarr":
+                for _, _, t, _ in fresh:
+                    call("/command", {"name": "SeriesSearch", "seriesId": t["id"]})
+            else:
+                call("/command", {"name": "MoviesSearch", "movieIds": [t["id"] for _, _, t, _ in fresh]})
+            for _, _, t, _ in fresh:
+                state["searched"][f"{app}:{t['id']}"] = today.isoformat()
+            save()
+
+    # ── 2. encode what is still big after its search ────────────────────────
+    age = lambda iso: (today - datetime.date.fromisoformat(iso)).days
+
+    def ready(app, ef, title_searched, to_search):
+        """True if this big file has had its chance to be replaced by a small download."""
+        if (ef.get("dateAdded") or "")[:10] <= title_searched:   # was on disk when the title was searched
+            return age(title_searched) >= SEARCH_GRACE_DAYS
+        key = f"{app}:file:{ef['id']}"                            # arrived later: search it on its own first
+        if key not in state["searched"]:
+            to_search.append(ef)
+            return False
+        return age(state["searched"][key]) >= FILE_SEARCH_GRACE_DAYS
+
+    def search_files(app, call, show, files):
+        if opt.dry_run:
+            log(f"  would search {len(files)} newer file(s) of {show['title']}"); return
+        if app == "sonarr":
+            ids = {f["id"] for f in files}
+            eps = [e["id"] for e in call(f"/episode?seriesId={show['id']}") if e.get("episodeFileId") in ids]
+            call("/command", {"name": "EpisodeSearch", "episodeIds": eps})
+        else:
+            call("/command", {"name": "MoviesSearch", "movieIds": [show["id"]]})
+        for f in files:
+            state["searched"][f"{app}:file:{f['id']}"] = today.isoformat()
+        log(f"  search {len(files)} newer file(s) of {show['title']} before encoding"); save()
+
+    done = saved = 0
+    for app, call, show, tier in work:
+        title_searched = state["searched"].get(f"{app}:{show['id']}")
+        if not title_searched:
+            continue                          # its title search comes first, on a later night
+        container, host = APPS[app]["paths"]
+        files = sorted(APPS[app]["files"](call, show), key=lambda f: f["path"])
+        changed, to_search = False, []
+        for ef in files:
+            if deadline and datetime.datetime.now() >= deadline:
+                log("deadline reached, stopping"); break
+            if opt.limit is not None and done >= opt.limit:
+                break
+            src = pathlib.Path(ef["path"].replace(container, host, 1))
+            if state["failures"].get(str(src), 0) >= MAX_FAILURES or not src.exists():
+                continue
+            # Files already ruled out at this tier and size need no re-probe (most of them, most nights).
+            seen = f"{tier}:{src.stat().st_size}"
+            if state["skip"].get(str(src)) == seen:
+                continue
+            info = probe(src)
+            if not info:
+                log(f"  SKIP {src.name}: does not probe"); continue
+            args, est = plan(info, tier, app)
+            if args is None:
+                state["skip"][str(src)] = seen  # reason in `est`; quiet, since most nights most files are done
+                continue
+            if not ready(app, ef, title_searched, to_search):
+                continue
+            old = int(info["format"]["size"])
+            if opt.dry_run:
+                log(f"  would encode {src.name}: {old / 1e6:.0f} MB -> ~{est / 1e6:.0f} MB ({tier})")
+                done += 1; continue
+            t0 = time.monotonic()
+            try:
+                tmp = encode(src, args, tier)
+                new = verify(info, tmp)
+                final = swap(src, tmp)
+            except Exception as e:
+                state["failures"][str(src)] = state["failures"].get(str(src), 0) + 1
+                save()
+                log(f"  FAIL {src.name}: {e}"); continue
+            done += 1; saved += old - new; changed = True
+            log(f"  {final.name}: {old / 1e6:.0f} MB -> {new / 1e6:.0f} MB "
+                f"(-{(1 - new / old) * 100:.0f}%, {time.monotonic() - t0:.0f}s, {tier})")
+        if to_search:
+            search_files(app, call, show, to_search)
+        if changed:
+            call("/command", APPS[app]["rescan"](show))
+        if (deadline and datetime.datetime.now() >= deadline) or (opt.limit is not None and done >= opt.limit):
+            break
+    if not opt.dry_run:
+        save()
+    log(f"=== run end: {done} file(s){' planned' if opt.dry_run else ''}, saved {saved / 1e9:.1f} GB ===")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,17 @@
+# Installed to /etc/systemd/system/. Runs as gib because the TV files are gib-owned,
+# so the job needs no root. A system unit (rather than a user unit) because gib has
+# no lingering session on this headless box. ExecStart goes through python3 because
+# SELinux will not let systemd execute a file labeled as home content (user_home_t).
+[Unit]
+Description=Re-encode TV shows on Sonarr's Compact/Supercompact profiles
+Documentation=file:///home/gib/Server/Media/compact-tv/README.md
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+User=gib
+Group=gib
+ExecStart=/usr/bin/python3 /home/gib/Server/Media/compact-tv/compact-tv.py --until 08:00
+Nice=15
+IOSchedulingClass=idle
@@ -0,0 +1,11 @@
+# Starts after the 00:00 hour and stops starting new encodes at 08:00 (see --until in
+# the service), so Plex and Frigate keep the GPU's encoder during the day.
+[Unit]
+Description=Nightly TV re-encode for Compact/Supercompact shows
+
+[Timer]
+OnCalendar=*-*-* 00:30:00
+Persistent=false
+
+[Install]
+WantedBy=timers.target
@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+"""Per-title cost of shrinking everything on Standard/Mini: re-encode vs redownload.
+
+For each show or movie on those profiles, probes every file with compact-tv's own planner
+and reports what the nightly job would do (encode, skip as already small, skip as HDR),
+the space it would save, and the GPU time it would take. Alongside that it estimates what a
+fresh small download would cost instead, using typical small-encode sizes.
+
+GPU time is measured, not guessed: the Shameless pilot took 669 s for 57.5 minutes of
+1080p (about 0.19 h of GPU per hour of video). 4K sources decode slower; the 1.5x for them
+is an estimate until the first 4K night is logged.
+
+Usage: cost-report.py > report.tsv     (also prints a summary to stderr)
+"""
+import importlib.util, json, pathlib, sys
+
+HERE = pathlib.Path(__file__).resolve().parent
+spec = importlib.util.spec_from_file_location("ct", HERE / "compact-tv.py")
+ct = importlib.util.module_from_spec(spec); spec.loader.exec_module(ct)
+
+GPU_H_PER_H = 669 / 3450                 # pilot: hours of GPU per hour of 1080p video
+UHD_FACTOR = 1.5                         # estimate for 2160p sources
+SMALL_GB_PER_H = {"Standard": 0.85, "Mini": 0.57}   # MeGusta-class 1080p / 720p HEVC, measured on S.W.A.T.
+
+rows = []
+for app, a in ct.APPS.items():
+    call = ct.arr(app)
+    profiles = {p["id"]: p["name"] for p in call("/qualityprofile")}
+    container, host = a["paths"]
+    for t in call(a["titles"]):
+        tier = profiles.get(t["qualityProfileId"])
+        if tier not in ct.TIERS:
+            continue
+        r = {"app": app, "title": t["title"], "tier": tier, "files": 0, "hours": 0.0, "size": 0,
+             "enc_files": 0, "enc_size": 0, "enc_est": 0, "gpu_h": 0.0, "hdr_size": 0, "small_size": 0}
+        for f in a["files"](call, t):
+            p = pathlib.Path(f["path"].replace(container, host, 1))
+            info = ct.probe(p) if p.exists() else None
+            if not info:
+                continue
+            size, dur = int(info["format"]["size"]), float(info["format"]["duration"]) / 3600
+            r["files"] += 1; r["hours"] += dur; r["size"] += size
+            args, why = ct.plan(info, tier, app)
+            if args:
+                w = max((s.get("width") or 0) for s in info["streams"] if s["codec_type"] == "video")
+                r["enc_files"] += 1; r["enc_size"] += size; r["enc_est"] += why
+                r["gpu_h"] += dur * GPU_H_PER_H * (UHD_FACTOR if w > 1920 else 1)
+            elif "HDR" in str(why):
+                r["hdr_size"] += size
+            else:
+                r["small_size"] += size
+        if r["files"]:
+            r["redownload_gb"] = round(r["hours"] * SMALL_GB_PER_H[tier], 1)
+            rows.append(r)
+
+rows.sort(key=lambda r: -(r["enc_size"] - r["enc_est"]))
+cols = ["app", "title", "tier", "files", "hours", "size", "enc_files", "enc_size", "enc_est", "gpu_h", "hdr_size",
+        "small_size", "redownload_gb"]
+print("\t".join(cols))
+for r in rows:
+    print("\t".join(str(round(r[c], 2) if isinstance(r[c], float) else r[c]) for c in cols))
+tot = lambda k: sum(r[k] for r in rows)
+print(f"{len(rows)} titles; encode {tot('enc_files')} files {tot('enc_size')/1e12:.2f} TB -> ~{tot('enc_est')/1e12:.2f} TB, "
+      f"{tot('gpu_h'):.0f} GPU hours; HDR skipped {tot('hdr_size')/1e12:.2f} TB; already small {tot('small_size')/1e12:.2f} TB",
+      file=sys.stderr)
@@ -0,0 +1,99 @@
+#!/usr/bin/env python3
+"""Make Standard and Mini prefer small releases, and upgrade only toward small ones.
+
+Run once per app; it is idempotent, so re-running updates everything in place.
+
+- Adds "Small Encode" (release groups known for small HEVC/x264 encodes) and "720p" /
+  "1080p" resolution formats. They score 0 in every profile except Standard and Mini.
+- HD and UHD are otherwise untouched: they keep LQ at -10000, so they never pick these.
+- Standard and Mini:
+  - All allowed qualities share ONE group, so no "better resolution" or "better source"
+    upgrade can ever download a big file. Only the custom-format score decides.
+  - Scores: Small Encode +3000, x265 +2000 (so first grabs still prefer HEVC), LQ 0,
+    release-tier formats 0 (one big release must never "beat" another big one).
+    Standard adds +500 for 1080p and +250 for 720p; Mini adds +1000 for 720p and +500 for
+    1080p. Those preferences stay below the upgrade threshold, so they never cause churn.
+  - Upgrades are on, but an upgrade needs a score jump of at least 2500, and upgrading
+    stops at 3000. In practice only gaining "Small Encode" clears that bar: a big H.264
+    file gets replaced by a small encode, but never by a big HEVC one.
+
+Usage: setup-small-profiles.py <sonarr|radarr>
+"""
+import json, re, subprocess, sys, urllib.request, pathlib
+
+APP = sys.argv[1]
+cfg = pathlib.Path.home() / f"Server/Media/volumes/{APP}/config.xml"
+key = re.search(r"<ApiKey>([^<]+)", cfg.read_text()).group(1)
+ip = subprocess.run(["docker", "inspect", APP, "--format", "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"],
+                    capture_output=True, text=True, check=True).stdout.split()[0]
+base = f"http://{ip}:{8989 if APP == 'sonarr' else 7878}/api/v3"
+
+def call(path, body=None, method=None):
+    req = urllib.request.Request(base + path, method=method, headers={"X-Api-Key": key, "Content-Type": "application/json"},
+                                 data=json.dumps(body).encode() if body is not None else None)
+    with urllib.request.urlopen(req, timeout=60) as r:
+        return json.load(r)
+
+GROUPS = r"^(MeGusta|PSA|ELiTE|GalaxyTV|GalaxyRG|GalaxyRG265|YIFY|YTS(\.(MX|LT|AG))?)$"
+resolution = lambda r: [{"name": f"{r}p", "implementation": "ResolutionSpecification",
+                         "negate": False, "required": True, "fields": [{"name": "value", "value": r}]}]
+WANTED = {
+    "Small Encode": [{"name": "Small encode groups", "implementation": "ReleaseGroupSpecification",
+                      "negate": False, "required": False, "fields": [{"name": "value", "value": GROUPS}]}],
+    "720p": resolution(720),
+    "1080p": resolution(1080),
+}
+TIERS = {
+    "Standard": {"Small Encode": 3000, "x265 (HD)": 2000, "LQ": 0, "LQ (Release Title)": 0, "1080p": 500, "720p": 250},
+    "Mini":     {"Small Encode": 3000, "x265 (HD)": 2000, "LQ": 0, "LQ (Release Title)": 0, "720p": 1000, "1080p": 500},
+}
+RELEASE_TIER = re.compile(r"Tier \d+|WEB Scene")
+MIN_UPGRADE_JUMP, UPGRADE_UNTIL = 2500, 3000
+
+cfs = {c["name"]: c for c in call("/customformat")}
+for name, specs in WANTED.items():
+    body = {"name": name, "includeCustomFormatWhenRenaming": False, "specifications": specs}
+    if name in cfs:
+        body["id"] = cfs[name]["id"]; call(f"/customformat/{body['id']}", body, "PUT")
+    else:
+        call("/customformat", body)
+cfs = {c["name"]: c["id"] for c in call("/customformat")}
+
+
+def flatten(items):
+    """Every quality in the profile with whether it is allowed, groups expanded, lowest rank first."""
+    for it in items:
+        for sub in (it["items"] or [it]):
+            yield sub["quality"], it["allowed"]
+
+
+def one_group(profile):
+    """Put every allowed quality into a single top-ranked group; leave disallowed ones as they are."""
+    flat = list(flatten(profile["items"]))
+    group = {"id": 1100, "name": "Any allowed", "allowed": True,
+             "items": [{"quality": q, "items": [], "allowed": True} for q, ok in flat if ok]}
+    profile["items"] = [{"quality": q, "items": [], "allowed": False} for q, ok in flat if not ok] + [group]
+    profile["cutoff"] = group["id"]
+    return profile
+
+
+for prof in call("/qualityprofile"):
+    have = {f["format"] for f in prof["formatItems"]}
+    for name in WANTED:
+        if cfs[name] not in have:
+            prof["formatItems"].append({"format": cfs[name], "name": name, "score": 0})
+    scores = TIERS.get(prof["name"])
+    if scores:
+        prof = one_group(prof)
+        prof.update(upgradeAllowed=True, minUpgradeFormatScore=MIN_UPGRADE_JUMP, cutoffFormatScore=UPGRADE_UNTIL)
+        for f in prof["formatItems"]:
+            if f["name"] in scores:
+                f["score"] = scores[f["name"]]
+            elif RELEASE_TIER.search(f["name"]):
+                f["score"] = 0
+    call(f"/qualityprofile/{prof['id']}", prof, "PUT")
+    if scores:
+        allowed = [q["quality"]["name"] for it in prof["items"] if it["allowed"] for q in it["items"]]
+        print(f"{APP} {prof['name']}: one group of {len(allowed)} qualities ({allowed[0]} .. {allowed[-1]}); "
+              f"upgrades on, jump >= {MIN_UPGRADE_JUMP}, until {UPGRADE_UNTIL}")
+        print("   nonzero scores: " + ", ".join(f"{f['name']}={f['score']}" for f in prof["formatItems"] if f["score"]))
@@ -427,3 +427,8 @@ services:
    # meant a completed sync was immediately restarted -- a hot loop of ~9
    # full syncs every 3 minutes. `no` lets it run once per `up` and stop.
    restart: "no"
+    # Opt-in only (`docker compose --profile configarr up configarr`). Its templates
+    # create the TRaSH profiles and size limits that were removed on 2026-10-28 in
+    # favour of four hand-managed profiles (Mini/Standard/HD/UHD), so a plain `up`
+    # must never run it. See Media/compact-tv/README.md.
+    profiles: [configarr]
@@ -41,8 +41,36 @@ services:
      # world is in backups/; upgrades rewrite the world and cannot be undone.
      - VERSION=26.1.2
      - DIFFICULTY=normal
-      - MAX_PLAYERS=10
-      - TYPE=VANILLA
+      - MAX_PLAYERS=30
+      # Paper since 2026-10-13. The last vanilla state is backups/gib-26.1.2-vanilla-*.
+      - TYPE=PAPER
+      # DonutSMP-style plugin set, pulled from Modrinth on each start. Every entry
+      # needs a 26.1.2 Paper build; check Modrinth before adding or bumping VERSION.
+      # Config lives in servers/gib/plugins/<Plugin>/. GrimAC only publishes alpha builds.
+      - MODRINTH_DOWNLOAD_DEPENDENCIES=required
+      - |
+        MODRINTH_PROJECTS=
+        luckperms
+        vaultunlocked
+        essentialsx
+        essentialsx-spawn
+        placeholderapi
+        playerpoints
+        zmenu
+        zauctionhouse
+        ultimateshop
+        smartspawner
+        wildrtp
+        ultimate-teams
+        asbounty
+        axafkzone
+        tab-was-taken
+        deluxemenus
+        coreprotect
+        worldedit
+        worldguard
+        grimac:alpha
+        chunky
      - MEMORY=12G
      - SIMULATION_DISTANCE=16
      - VIEW_DISTANCE=16
@@ -38,6 +38,7 @@ services:
      - ENABLE_OAUTH_SIGNUP=true
      - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
      #- ENABLE_OAUTH_ROLE_MANAGEMENT=true
+      #- OAUTH_ROLES_CLAIM=roles
      #- OAUTH_ADMIN_ROLES=admin,internal,superuser
      - OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID}
      - OAUTH_CLIENT_SECRET=${OAUTH_CLIENT_SECRET}
@@ -0,0 +1,140 @@
+# Restoring ~/Server on server.gib from the ZFS-pool backup
+
+Written 2026-09-17. Read this *before* you need it.
+
+## What this protects against
+
+The NVMe/LUKS btrfs volume (`/`, `/home`, and `/var/lib/docker`) dying. The backup lives on
+the ZFS pool in the same machine, so it does **not** protect against fire, theft, or the
+whole box failing. It is a disk-failure backup, not an offsite one.
+
+## Layout
+
+    /home/gib/Public/Server_Backup/Containers/server.gib/current/
+
+**One directory, overwritten daily at 02:00** by `server-backup.timer`. History lives in ZFS:
+`Media/Server_Backup` has `zfs-auto-snapshot` keeping **14 dailies**, taken by `cron.daily`
+between roughly 03:22 and 03:52. The 02:00 schedule is deliberate — it must finish before
+that window, or a snapshot captures a half-written backup as that day's history. It also
+staggers against the VPS backup, which pushes into this same pool at 01:00.
+
+Older copies are under the hidden snapshot dir (`snapdir=hidden`, so `ls` will not show it,
+but you can `cd` in):
+
+    ls /home/gib/Public/Server_Backup/.zfs/snapshot/
+    ls /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
+
+The `03-16-2026` and `03-31-2026` directories are leftovers from the old manual script.
+
+## Why this runs as root
+
+This box runs Docker, so most container data under `~/Server` is root-owned. The old script
+used `sudo rsync`, which worked but meant it could never be automated without a sudoers
+rule. The replacement runs as root via `server-backup.service`, a systemd **system** unit —
+already root, so ownership is preserved natively and **no sudoers entry is needed at
+runtime**. If you see `/etc/sudoers.d/99-claude-setup` on this machine, it was temporary
+setup scaffolding and should be deleted.
+
+(This differs from the VPS, which is rootless podman and needs `rsync --fake-super` to carry
+subuid ownership through. Nothing like that is needed here.)
+
+The script lives at `/usr/local/sbin/server-backup`, deliberately **not** in `~/Server` — a
+root-executed script inside a user-writable directory is a footgun. `~/Server/backup` is a
+thin wrapper that triggers the systemd unit so manual runs behave identically to nightly ones.
+
+## What is in a backup
+
+| Path | What it is |
+|---|---|
+| `Media/`, `Frigate/`, `Home/`, `OpenWebUI/`, ... | Raw bind-mount data per service |
+| `_backup_state/sqlite/<original path>` | Online `.backup` of every SQLite DB — **authoritative** |
+| `_backup_state/convex/<container>.zip` | `npx convex export` snapshot of each SQLite-backed Convex deployment (gbrown) — **authoritative** |
+
+> **Use `_backup_state/` for databases, not the raw copies.** Containers keep running during
+> the backup, so the raw files are crash-consistent at best. The dumps use SQLite's online
+> backup API and are what will actually restore cleanly. That covers the ones that matter:
+> Plex's `com.plexapp.plugins.library.db` (watch history, ratings), Jellyfin, Frigate, Home
+> Assistant, OpenWebUI, Audiobookshelf, and every *arr config.
+>
+> **Convex is the exception.** It writes to its SQLite file constantly, so a raw `.backup`
+> never completes. Its `_backup_state/convex/` export is the restore source, and the raw
+> `db.sqlite3` is not dumped at all.
+
+## What is deliberately NOT backed up
+
+- **`Agents/Gitea/volumes/runner-{1,2,3}/dind` — ~203 GB.** Docker-in-Docker containerd
+  image layers for the CI runners. Pure cache that churns constantly; a rebuilt runner just
+  re-pulls it. The parts that matter — `runner-N/data/.runner` (the 648-byte registration
+  token) and `certs/` — **are** backed up.
+  > Measured as root. A plain `du` as `gib` reports these as ~13 MB because gib cannot
+  > descend into them. Do not trust non-root sizes here.
+- Docker **named volumes** (`/var/lib/docker/volumes`, 47 in use, ~61 GB): Sentry (13) and
+  Plausible (4) were consciously excluded — Sentry's event data is not worth restoring and
+  Plausible had nothing in it. Romm looked like a gap but is not: its anonymous `/romm`
+  volume is only a parent directory, with `assets/`, `config/` and `resources/` bind-mounted
+  from `~/Server` and `library/` from the pool.
+- **Nextcloud AIO** — has its own borg backup at `~/Public/Nextcloud/Backups`, run by
+  hand from the AIO admin UI. Last run 2026-09-17 (finished 09-18). Runs are slow and
+  AIO stops the Nextcloud containers while one is going.
+- `node_modules`, build output, `*.log`, `cache`/`tmp` dirs, `*.mp4`/`*.mkv`.
+
+## Restoring
+
+Stop the stack first so nothing overwrites what you restore:
+
+    cd ~/Server/<service> && docker compose down     # per service, or stop docker entirely
+
+Pull it back — as root, so ownership is restored:
+
+    sudo rsync -a --numeric-ids --delete \
+      /home/gib/Public/Server_Backup/Containers/server.gib/current/ \
+      /home/gib/Server/
+
+For an older day, use a snapshot path instead:
+
+    /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
+
+Then put the databases back from the dumps. For each one, stop the owning container, copy the
+dump over the live file, and match the original ownership. Plex, for example:
+
+    docker compose -f ~/Server/Media/compose.yml stop plex
+    D=~/Server/_backup_state/sqlite/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
+    L=~/Server/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
+    sudo cp "$D/com.plexapp.plugins.library.db" "$L/"
+    sudo chown --reference="$L/com.plexapp.dlna.db" "$L/com.plexapp.plugins.library.db"
+    docker compose -f ~/Server/Media/compose.yml start plex
+
+`chown --reference` copies ownership from a neighbouring file, which saves looking up what
+uid each container expects.
+
+For Convex, start the backend (it can be empty) and import the export through it. This
+replaces every table with the snapshot:
+
+    Z=~/Server/_backup_state/convex/gbrown-backend.zip
+    KEY=$(docker exec gbrown-backend ./generate_admin_key.sh | tail -1)
+    docker run --rm -it --network nginx-bridge -v "$Z:/backup.zip:ro" \
+      -e CONVEX_SELF_HOSTED_URL=http://gbrown-backend:3210 -e CONVEX_SELF_HOSTED_ADMIN_KEY="$KEY" \
+      docker.io/library/node:22-alpine sh -ec 'mkdir /w && cd /w && echo "{}" > package.json &&
+        npm install --silent convex && npx convex import --replace-all /backup.zip'
+
+Functions and schema are not in the export. Redeploy them from the app's repo as usual.
+
+After a full restore the CI runners will have no `dind` storage. That is fine — they re-pull
+images on the next job. The registration in `runner-N/data/.runner` is intact, so they
+reconnect to Gitea without re-registering.
+
+## Checking it is working
+
+    systemctl list-timers server-backup.timer
+    systemctl status server-backup.service
+    tail -40 ~/Server/logs/backup.log
+
+A healthy run ends with `=== backup complete ===`. Watch for:
+
+- `ZFS history OK` — confirms snapshots are still being taken. If this ever says **WARNING**,
+  versioning is broken and `current/` is just a mirror: a mistake would propagate with
+  nothing to roll back to. Fix `zfs-auto-snapshot` before anything else.
+- `rsync exit 24` is a NOTE, not a failure — a file changed mid-transfer, normal on a live box.
+
+Optional: put an uptime-kuma push URL in `~/Server/.backup-heartbeat-url` and successful runs
+will ping it, turning a silent failure into a down monitor.
@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# Manual trigger for the daily backup.
+#
+# The real script lives at /usr/local/sbin/server-backup and runs as ROOT via
+# server-backup.service. It is kept out of ~/Server on purpose: ~/Server is
+# user-writable, and a root-executed script sitting in a user-writable directory
+# is an obvious footgun. Run it through systemd so a manual run behaves exactly
+# like the nightly one (same environment, same logging, same lock).
+#
+#   status:  systemctl status server-backup.service
+#   log:     tail -f ~/Server/logs/backup.log
+#   timer:   systemctl list-timers server-backup.timer
+#   restore: see ~/Server/RESTORE.md
+exec sudo systemctl start --wait server-backup.service
@@ -0,0 +1,19 @@
+[Unit]
+Description=Weekly copy of the Media pool to the Backup pool
+Documentation=file:///home/gib/Server/RESTORE.md
+After=zfs-mount.service
+Wants=zfs-mount.service
+RequiresMountsFor=/home/gib/Public /home/gib/.local/share/Backup
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/media-backup
+# 8.6 TB on the first run, deltas after. Generous, but bounded so a hung rsync
+# fails loudly instead of overlapping next week's fire.
+TimeoutStartSec=24h
+Nice=10
+IOSchedulingClass=best-effort
+IOSchedulingPriority=7
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,17 @@
+[Unit]
+Description=Weekly Media -> Backup pool copy
+
+[Timer]
+# Saturday 05:00. Clear of: the cron.daily zfs-auto-snapshot window (03:22-03:52),
+# timeshift (04:00), arr-queue-reconcile (04:31), and both ~/Server backups (01:00, 02:00).
+# Saturday specifically because Monday carries the Media scrub and Sunday has
+# raid-check (01:00), docker-prune (05:33) and dind-prune (06:03).
+# gib also runs Nextcloud's borg by hand roughly monthly and is, in his words,
+# not going to be doing that at 5 AM on a Saturday -- so the borg lock should be free.
+OnCalendar=Sat *-*-* 05:00:00
+RandomizedDelaySec=15m
+Persistent=true
+Unit=media-backup.service
+
+[Install]
+WantedBy=timers.target
@@ -0,0 +1,19 @@
+[Unit]
+Description=Back up /home/gib/Server to the ZFS pool
+Documentation=file:///home/gib/Server/RESTORE.md
+# The destination is on ZFS; do not run before the pool is up.
+After=zfs-mount.service docker.service
+Wants=zfs-mount.service
+RequiresMountsFor=/home/gib/Public
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/server-backup
+TimeoutStartSec=8h
+# This box is a daily driver running 145 containers. Stay out of the way.
+Nice=10
+IOSchedulingClass=best-effort
+IOSchedulingPriority=7
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,14 @@
+[Unit]
+Description=Daily backup of /home/gib/Server to the ZFS pool
+
+[Timer]
+# MUST finish before cron.daily fires zfs-auto-snapshot (~03:22-03:52); a snapshot that
+# catches a running rsync stores a partial backup as that day's history.
+# 02:00 also staggers against the VPS backup, which pushes into this same pool at 01:00.
+OnCalendar=*-*-* 02:00:00
+RandomizedDelaySec=10m
+Persistent=true
+Unit=server-backup.service
+
+[Install]
+WantedBy=timers.target
@@ -92,7 +92,7 @@ list_units() {
 # services therefore resolve no images and are skipped without a log line.
 #
 # Image tags frequently reference variables from the service's .env, e.g.
-#   ghcr.io/get-convex/convex-backend:${BACKEND_TAG:-latest}
+#   ghcr.io/convex/convex-backend:${BACKEND_TAG:-latest}
 #   git.gbrown.org/gib/${NEXT_CONTAINER_NAME}:latest
 # Those must be expanded or the pull is attempted against a literal "${...}" and
 # always fails. We expand in a SUBSHELL so the .env cannot leak into this script's
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: Authentik
 After=network-online.target podman.socket podman-postgresql.service
 Wants=network-online.target podman-postgresql.service
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: Beszel
 After=network-online.target podman.socket
 Wants=network-online.target
@@ -32,7 +32,7 @@ services:
    restart: unless-stopped

  bill-tracker-backend:
-    image: ${CONVEX_BACKEND_IMAGE:-ghcr.io/get-convex/convex-backend@sha256:1f2044e3eac463ac78973b136c0baf72d4ada602611d853d6f99f280e29e0a98}
+    image: ${CONVEX_BACKEND_IMAGE:-ghcr.io/convex/convex-backend@sha256:207cf95def42f01614b6e3af0af5c20d0dcaa5ff2adf258ee7bc9555576644fa}
    container_name: bill-tracker-backend
    hostname: bill-tracker-backend
    domainname: api.billtracker.gbrown.org
@@ -56,7 +56,7 @@ services:
    stop_signal: SIGINT

  bill-tracker-dashboard:
-    image: ${CONVEX_DASHBOARD_IMAGE:-ghcr.io/get-convex/convex-dashboard@sha256:284a2638e0c1a4ec0c2327d8219776f3a426ca5824b81686ae4d9454dc0ce8ed}
+    image: ${CONVEX_DASHBOARD_IMAGE:-ghcr.io/convex/convex-dashboard@sha256:a55bf8abca1332f787bc4d4a79850182175027be7b70aa230be4389a1d3268e2}
    container_name: bill-tracker-dashboard
    hostname: bill-tracker-dashboard
    domainname: dashboard.billtracker.gbrown.org
@@ -29,7 +29,7 @@ services:
    restart: unless-stopped

  convexmonorepo-backend:
-    image: ghcr.io/get-convex/convex-backend:${BACKEND_TAG:-latest}
+    image: ghcr.io/convex/convex-backend:${BACKEND_TAG:-latest}
    container_name: ${BACKEND_CONTAINER_NAME:-convex-backend}
    hostname: ${BACKEND_CONTAINER_NAME:-convex-backend}
    domainname: ${BACKEND_DOMAIN:-convex.gbrown.org}
@@ -58,7 +58,7 @@ services:
    stop_signal: SIGINT

  convexmonorepo-dashboard:
-    image: ghcr.io/get-convex/convex-dashboard:${DASHBOARD_TAG:-latest}
+    image: ghcr.io/convex/convex-dashboard:${DASHBOARD_TAG:-latest}
    container_name: ${DASHBOARD_CONTAINER_NAME:-convex-dashboard}
    hostname: ${DASHBOARD_CONTAINER_NAME:-convex-dashboard}
    domainname: ${DASHBOARD_DOMAIN:-dashboard.${BACKEND_DOMAIN:-convex.gbrown.org}}
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: iSponsorBlockTV
 After=network-online.target podman.socket
 Wants=network-online.target
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: Infisical
 After=network-online.target podman.socket podman-postgresql.service
 Wants=network-online.target podman-postgresql.service
@@ -19,7 +19,7 @@ services:
    - NODE_ENV=${NODE_ENV}
    - SENTRY_AUTH_TOKEN=${SENTRY_AUTH_TOKEN}
    - NEXT_PUBLIC_SITE_URL=${NEXT_PUBLIC_SITE_URL:-http://localhost:${NEXT_PORT:-3000}}
-    - NEXT_PUBLIC_CONVEX_URL=${NEXT_PUBLIC_CONVEX_URL:-http://${BACKEND_CONTAINER_NAME:-lashaddict-backend}:${BACKEND_PORT:-3210}}
+    - NEXT_PUBLIC_CONVEX_URL=${CONVEX_SERVER_URL:-http://${BACKEND_CONTAINER_NAME:-lashaddict-backend}:${BACKEND_PORT:-3210}}
    - NEXT_PUBLIC_PLAUSIBLE_URL=${NEXT_PUBLIC_PLAUSIBLE_URL:-https://plausible.gbrown.org}
    - NEXT_PUBLIC_SENTRY_DSN=${NEXT_PUBLIC_SENTRY_DSN}
    - NEXT_PUBLIC_SENTRY_URL=${NEXT_PUBLIC_SENTRY_URL}
@@ -33,7 +33,7 @@ services:
    stdin_open: true
    restart: unless-stopped
  lashaddict-backend:
-    image: ghcr.io/get-convex/convex-backend:${BACKEND_TAG:-latest}
+    image: ghcr.io/convex/convex-backend:${BACKEND_TAG:-latest}
    container_name: ${BACKEND_CONTAINER_NAME:-lashaddict-backend}
    # Same Cloudflare DNS pin as lashaddict-next (see its comment): the backend
    # fetches its own auth discovery URL, which Hetzner resolvers failed.
@@ -67,7 +67,7 @@ services:
    stop_grace_period: 10s
    stop_signal: SIGINT
  lashaddict-dashboard:
-    image: ghcr.io/get-convex/convex-dashboard:${DASHBOARD_TAG:-latest}
+    image: ghcr.io/convex/convex-dashboard:${DASHBOARD_TAG:-latest}
    container_name: ${DASHBOARD_CONTAINER_NAME:-lashaddict-dashboard}
    # Same Cloudflare DNS pin as lashaddict-next (see its comment): the backend
    # fetches its own auth discovery URL, which Hetzner resolvers failed.
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: N8n
 After=network-online.target podman-postgresql.service
 Wants=network-online.target
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: Nginx Proxy Manager
 After=network-online.target podman-postgresql.service
 Wants=network-online.target
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: Portainer
 After=network-online.target
 Wants=network-online.target
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: PostgreSQL
 After=network-online.target
 Wants=network-online.target
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: Rustdesk
 After=network-online.target podman.socket
 Wants=network-online.target
@@ -64,7 +64,7 @@ services:
    security_opt:
    - label:disable
  spoon-backend:
-    image: ghcr.io/get-convex/convex-backend:${BACKEND_TAG:-latest}
+    image: ghcr.io/convex/convex-backend:${BACKEND_TAG:-latest}
    container_name: ${BACKEND_CONTAINER_NAME:-spoon-backend}
    hostname: ${BACKEND_CONTAINER_NAME:-spoon-backend}
    domainname: ${BACKEND_DOMAIN:-convex.spoon.gbrown.org}
@@ -93,7 +93,7 @@ services:
    stop_grace_period: 10s
    stop_signal: SIGINT
  spoon-dashboard:
-    image: ghcr.io/get-convex/convex-dashboard:${DASHBOARD_TAG:-latest}
+    image: ghcr.io/convex/convex-dashboard:${DASHBOARD_TAG:-latest}
    container_name: ${DASHBOARD_CONTAINER_NAME:-spoon-dashboard}
    hostname: ${DASHBOARD_CONTAINER_NAME:-spoon-dashboard}
    domainname: ${DASHBOARD_DOMAIN:-dashboard.${BACKEND_DOMAIN:-spoon.gbrown.org}}
@@ -29,7 +29,7 @@ services:
    restart: unless-stopped

  stpeteit-backend:
-    image: ghcr.io/get-convex/convex-backend:${BACKEND_TAG:-latest}
+    image: ghcr.io/convex/convex-backend:${BACKEND_TAG:-latest}
    container_name: ${BACKEND_CONTAINER_NAME:-stpeteit-backend}
    hostname: ${BACKEND_CONTAINER_NAME:-stpeteit-backend}
    domainname: ${BACKEND_DOMAIN:-convex.stpeteit.com}
@@ -58,7 +58,7 @@ services:
    stop_signal: SIGINT

  stpeteit-dashboard:
-    image: ghcr.io/get-convex/convex-dashboard:${DASHBOARD_TAG:-latest}
+    image: ghcr.io/convex/convex-dashboard:${DASHBOARD_TAG:-latest}
    container_name: ${DASHBOARD_CONTAINER_NAME:-stpeteit-dashboard}
    hostname: ${DASHBOARD_CONTAINER_NAME:-stpeteit-dashboard}
    domainname: ${DASHBOARD_DOMAIN:-dashboard.${BACKEND_DOMAIN:-convex.stpete.com}}
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: Tools
 After=network-online.target podman.socket
 Wants=network-online.target
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: Uptime
 After=network-online.target
 Wants=network-online.target
@@ -1,4 +1,5 @@
 [Unit]
+RequiresMountsFor=/home/gib/Media
 Description=Podman Compose: Vaultwarden
 After=network-online.target podman.socket
 Wants=network-online.target
@@ -0,0 +1,194 @@
+#!/usr/bin/env bash
+#
+# Daily backup of ~/Server to the home server, as hardlinked dated snapshots.
+#
+# WHAT THIS FIXES vs. the old version (2026-09-17):
+#   1. OWNERSHIP. Rootless podman leaves most container data owned by subuids
+#      (gitea = 525288, postgres data = the userns root mapping). Plain `rsync -a`
+#      running as gib cannot chown on the far side, so every previous backup landed
+#      100% owned by uid 1000. Restoring that gives you the right bytes with the wrong
+#      owners: postgres refuses to start unless PGDATA is owned correctly, and gitea
+#      fails on its own data dir. Fixed with `-M--fake-super`, which makes the REMOTE
+#      rsync stash mode/uid/gid in a user.rsync.%stat xattr (verified working on the
+#      home server's ZFS). Restore with --fake-super on the read side to put them back.
+#   2. CONSISTENCY. 13 SQLite DBs and a live postgres were copied hot, mid-write. Now
+#      dumped properly first (sqlite3 .backup / pg_dumpall) into _backup_state/.
+#   3. SPACE / HISTORY. Every run was a full ~19G copy into a new dated directory.
+#      Now it rsyncs into ONE stable directory, `current/`, and versioning is left to
+#      the home server: Media/Server_Backup is its own ZFS dataset with zfs-auto-snapshot
+#      keeping 14 dailies. ZFS snapshots are block-level, so a database that changes a
+#      few pages costs a few pages -- not a whole new copy the way dated dirs (or even
+#      rsync --link-dest hardlinks, which are per-file) would.
+#
+# Containers keep running throughout. That is safe *because* of the dumps above --
+# the dumps, not the raw data dirs, are the authoritative restore path.
+#
+# See RESTORE.md for how to bring this back on a new machine.
+
+set -uo pipefail
+
+REMOTE_HOST="[email protected]"
+REMOTE_ROOT="/home/gib/Public/Server_Backup/Containers/lilman.gib"
+SOURCE="$HOME/Server/"
+STATE_DIR="$HOME/Server/_backup_state"
+DEST_NAME="current"                # One stable directory. ZFS snapshots provide history.
+LOG_DIR="$HOME/Server/logs"
+LOG="$LOG_DIR/backup.log"
+LOCK="${XDG_RUNTIME_DIR:-/tmp}/server-backup.lock"
+
+# Versioning is ZFS's job, not ours -- see the header. We only sanity-check that it is
+# actually still happening, so this can never silently degrade into a single mirror with
+# no history. Warn if the newest snapshot of the dataset is older than this.
+REMOTE_DATASET="Media/Server_Backup"
+SNAPSHOT_MAX_AGE_HOURS=48
+
+mkdir -p "$LOG_DIR"
+log() { printf '%s  %s\n' "$(date +'%Y-%m-%d %H:%M:%S')" "$*" | tee -a "$LOG"; }
+die() { log "FATAL: $*"; exit 1; }
+
+# Refuse to overlap with a still-running backup rather than corrupting a snapshot.
+exec 9>"$LOCK" || die "cannot open lock $LOCK"
+flock -n 9 || die "another backup is already running"
+
+EXCLUSIONS=(
+    # --- build noise -------------------------------------------------------
+    # NOTE: these are bare patterns, so they match at ANY depth. Today they only
+    # catch genuine scratch dirs (checked 2026-09-17). If you add a service whose
+    # real data lives in a dir called cache/build/dist/tmp, it will be silently
+    # skipped -- re-check this list when adding services.
+    "node_modules" ".git" "*.log" "*.tmp" ".cache" "cache" "tmp" "temp"
+    ".npm" ".yarn" "dist" "build" "target" ".venv" "venv" "__pycache__"
+    "*.pyc" ".DS_Store" "Thumbs.db" "vendor" "*.mp4" "*.mkv"
+
+    # --- unreadable to gib, and genuinely disposable ------------------------
+    # Redis/valkey cache dumps: mode 0600 under a subuid because the container
+    # chowns its own data. Pure cache (usesend + infisical), regenerated on boot.
+    "dump.rdb"
+    # Gitea runs WITHOUT userns_mode (s6 needs container-root -- see AGENTS.md §3),
+    # so these are subuid-owned and mode 0700. Excluding them is what lets this run
+    # without sudo. Nothing here is needed to restore gitea:
+    #   sessions/     ephemeral web logins; worst case everyone logs in again
+    #   issues.bleve  search index, rebuilt from the DB on demand
+    #   Gitea/ssh     a 25-byte `environment` file for the git shell
+    #
+    # ⚠️ The SSH HOST KEYS are NOT here -- they live in Gitea/volume/ssh, are
+    # gib-owned, and DO get backed up. Losing those = host-key mismatch for every client.
+    "sessions" "issues.bleve" "Gitea/ssh"
+
+    # Mode 0600 under subuid 525288, so unreadable to gib -- this single file is what
+    # made every previous run exit 23. It is NOT disposable (it holds the MCP bridge's
+    # registered OAuth clients), so it is captured via `podman exec` into
+    # _backup_state/files/ instead of being read off disk.
+    "BillTracker/data/mcp/bridge.json"
+
+    # Stirling-PDF writes this 0600 under subuid 525287 (since its 2026-09-25 update),
+    # which made every run since exit 23. Excluded rather than captured: losing it only
+    # means re-entering any credentials stored in Stirling-PDF.
+    "pdf-tools/extraConfigs/credential-encryption.key"
+)
+
+EXCLUDE_ARGS=()
+for e in "${EXCLUSIONS[@]}"; do EXCLUDE_ARGS+=( "--exclude=$e" ); done
+
+# ---------------------------------------------------------------------------
+# Phase 1: consistent dumps of everything transactional.
+# ---------------------------------------------------------------------------
+dump_state() {
+    log "Dumping databases to _backup_state/ ..."
+    rm -rf "$STATE_DIR"
+    mkdir -p "$STATE_DIR"/{postgres,sqlite,files}
+    printf 'Generated %s by ~/Server/backup. Authoritative restore source -- see RESTORE.md\n' \
+        "$(date -Is)" > "$STATE_DIR/README"
+
+    # Live postgres (serves npm, authentik, usesend, ... -- pg_dumpall gets all of them).
+    if podman ps --format '{{.Names}}' | grep -qx postgresql; then
+        if podman exec postgresql pg_dumpall -U npm 2>>"$LOG" | gzip -9 > "$STATE_DIR/postgres/all.sql.gz"; then
+            log "  postgres: $(du -h "$STATE_DIR/postgres/all.sql.gz" | cut -f1)"
+        else
+            log "  WARNING: pg_dumpall failed -- raw pg_data is still copied, but is a hot copy"
+        fi
+    else
+        log "  WARNING: postgresql container not running, skipping pg_dumpall"
+    fi
+
+    # SQLite: .backup uses the online backup API, so this is safe while containers write.
+    local n=0
+    while IFS= read -r db; do
+        # *.db also matches stirling-pdf's H2 database, which is not SQLite. Check magic.
+        head -c 16 "$db" 2>/dev/null | grep -q "SQLite format 3" || continue
+        local rel="${db#"$HOME"/Server/}" out
+        out="$STATE_DIR/sqlite/$rel"
+        mkdir -p "$(dirname "$out")"
+        if sqlite3 -cmd ".timeout 30000" "$db" ".backup '$out'" 2>>"$LOG"; then n=$((n+1)); else log "  WARNING: sqlite dump failed: $rel"; fi
+    done < <(find "$HOME/Server" -path "$STATE_DIR" -prune -o \
+                  -type f \( -name '*.db' -o -name '*.sqlite' -o -name '*.sqlite3' \) -print 2>/dev/null)
+    log "  sqlite: $n databases"
+
+    # Files gib cannot read off disk, pulled through their container instead.
+    if podman ps --format '{{.Names}}' | grep -qx bill-tracker-mcp; then
+        podman exec bill-tracker-mcp cat /data/bridge.json \
+            > "$STATE_DIR/files/BillTracker--data--mcp--bridge.json" 2>>"$LOG" \
+            && log "  bridge.json captured" \
+            || log "  WARNING: could not capture bridge.json"
+    fi
+}
+
+# ---------------------------------------------------------------------------
+# Phase 2: confirm ZFS is still providing history.
+#
+# We deliberately overwrite one directory instead of keeping dated copies, which
+# means ZFS snapshots are the ONLY thing standing between us and "a mirror of a
+# mistake". If auto-snapshot ever stops, that must be loud, not silent.
+# ---------------------------------------------------------------------------
+check_zfs_history() {
+    local newest now age_h
+    newest=$(ssh -o BatchMode=yes "$REMOTE_HOST" \
+        "zfs list -H -p -o creation -t snapshot -s creation -d 1 '$REMOTE_DATASET' 2>/dev/null | tail -1" || true)
+    if [ -z "$newest" ]; then
+        log "  ⚠️  WARNING: no ZFS snapshots on $REMOTE_DATASET -- this backup has NO history."
+        log "      It is a plain mirror right now. Check zfs-auto-snapshot on the home server."
+        return
+    fi
+    now=$(date +%s)
+    age_h=$(( (now - newest) / 3600 ))
+    if [ "$age_h" -gt "$SNAPSHOT_MAX_AGE_HOURS" ]; then
+        log "  ⚠️  WARNING: newest ZFS snapshot is ${age_h}h old (limit ${SNAPSHOT_MAX_AGE_HOURS}h) -- versioning may be broken."
+    else
+        log "  ZFS history OK (newest snapshot ${age_h}h old, 14 dailies retained)"
+    fi
+}
+
+# ---------------------------------------------------------------------------
+# Phase 3: mirror.
+# ---------------------------------------------------------------------------
+log "=== backup start -> $REMOTE_HOST:$REMOTE_ROOT/$DEST_NAME ==="
+ssh -o BatchMode=yes -o ConnectTimeout=15 "$REMOTE_HOST" "mkdir -p '$REMOTE_ROOT/$DEST_NAME'" \
+    || die "cannot reach $REMOTE_HOST"
+
+dump_state
+check_zfs_history
+
+# --delete/--delete-excluded keep `current` an exact mirror of the source. Anything
+# removed here still exists in yesterday's ZFS snapshot, so this is not destructive.
+rsync -a --numeric-ids -M--fake-super --delete --delete-excluded \
+      --human-readable --stats \
+      "${EXCLUDE_ARGS[@]}" \
+      --rsync-path="mkdir -p '$REMOTE_ROOT/$DEST_NAME' && rsync" \
+      "$SOURCE" "$REMOTE_HOST:$REMOTE_ROOT/$DEST_NAME/" 2>&1 | tee -a "$LOG"
+status=${PIPESTATUS[0]}
+
+# 24 = "source files vanished during transfer". Normal on a live system, not a failure.
+if [ "$status" -ne 0 ] && [ "$status" -ne 24 ]; then
+    die "rsync failed with exit $status -- 'current' may be half-updated; yesterday's ZFS snapshot is still good"
+fi
+[ "$status" -eq 24 ] && log "NOTE: rsync exit 24 (files vanished mid-run) -- expected on a live system"
+
+# Optional heartbeat so a silent failure cannot go unnoticed. Create the file with an
+# uptime-kuma push URL in it and this pings on success only; absent, nothing happens.
+if [ -r "$HOME/Server/.backup-heartbeat-url" ]; then
+    url=$(< "$HOME/Server/.backup-heartbeat-url")
+    curl -fsS -m 20 "$url" >/dev/null 2>&1 \
+        && log "Heartbeat sent" || log "NOTE: heartbeat ping failed (backup itself was fine)"
+fi
+
+log "=== backup complete ==="
@@ -0,0 +1,22 @@
+[Unit]
+Description=Back up ~/Server to the home server
+Documentation=file:///home/gib/Server/RESTORE.md
+After=network-online.target
+Wants=network-online.target
+# The backup reads podman-managed data and shells into containers for dumps.
+RequiresMountsFor=/home/gib/Media
+
+[Service]
+Type=oneshot
+ExecStart=%h/Server/backup
+# A stalled rsync should fail loudly, not hang until the next timer fires.
+TimeoutStartSec=6h
+# This box is IO-bound before it is CPU-bound (~18%% avg60 pressure), so stay
+# out of the way of the containers. best-effort/7 rather than idle, which can
+# starve the transfer outright when the volume is busy.
+Nice=10
+IOSchedulingClass=best-effort
+IOSchedulingPriority=7
+
+[Install]
+WantedBy=default.target
@@ -0,0 +1,18 @@
+[Unit]
+Description=Daily backup of ~/Server to the home server
+
+[Timer]
+# MUST finish before the home server's cron.daily fires zfs-auto-snapshot, which has
+# been landing between 03:22 and 03:52. If the snapshot catches a half-finished rsync,
+# that day's history entry is a partial backup. 01:00 leaves >2h of margin; the first
+# full run took well under that and daily deltas are far smaller.
+OnCalendar=*-*-* 01:00:00
+# Small jitter only -- must not push us into the snapshot window.
+RandomizedDelaySec=10m
+# If the VPS was down at 01:00, run once after it comes back rather than
+# silently skipping a day.
+Persistent=true
+Unit=server-backup.service
+
+[Install]
+WantedBy=timers.target
@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+# Post-reboot verification for the rootless podman stack.
+# Run as gib. Compares against ~/premigration_backup_2026-08-10/post_migration_baseline.txt
+set -uo pipefail
+
+fail=0
+ok()   { printf '  \033[32mOK\033[0m   %s\n' "$1"; }
+bad()  { printf '  \033[31mFAIL\033[0m %s\n' "$1"; fail=$((fail+1)); }
+
+echo "== linger =="
+[[ "$(loginctl show-user gib --property=Linger --value 2>/dev/null)" == "yes" ]] \
+  && ok "Linger=yes" || bad "linger is OFF - user units will not start at boot"
+
+echo "== unprivileged port start =="
+p=$(sysctl -n net.ipv4.ip_unprivileged_port_start 2>/dev/null)
+[[ "$p" == "80" ]] && ok "ip_unprivileged_port_start=80" \
+  || bad "ip_unprivileged_port_start=$p (NPM cannot bind :80/:443)"
+
+echo "== user units =="
+for s in postgresql bang completeuphoria sierraandtyler uptime portainer beszel authentik tools isponsorblocktv vaultwarden infisical rustdesk nginx-proxy-manager gbrown-agent; do
+  st=$(systemctl --user is-active "podman-$s.service" 2>/dev/null)
+  [[ "$st" == "active" ]] && ok "$s" || bad "$s is $st"
+done
+
+echo "== containers =="
+for c in postgresql nginx-proxy-manager uptime portainer bang completeuphoria sierraandtyler beszel beszel-agent authentik-server authentik-worker convertx it-tools pdf-tools bento-pdf iSponsorBlockTV vaultwarden infisical-redis infisical-backend rustdesk-server gbrown-agent; do
+  if podman ps --format '{{.Names}}' 2>/dev/null | grep -qx "$c"; then ok "$c"; else bad "$c not running"; fi
+done
+
+echo "== postgres health =="
+h=$(podman inspect postgresql --format '{{.State.Health.Status}}' 2>/dev/null)
+[[ "$h" == "healthy" ]] && ok "postgres healthy" || bad "postgres health=$h"
+
+echo "== endpoints =="
+chk() { local n=$1 exp=$2; shift 2; c=$(curl -sk -o /dev/null -w '%{http_code}' "$@" 2>/dev/null)
+        [[ "$c" == "$exp" ]] && ok "$n -> $c" || bad "$n -> $c (expected $exp)"; }
+# Admin ports moved to the WireGuard address 2026-08-12 (AGENTS.md §13.2).
+# They no longer listen on localhost, so probe 192.168.2.2.
+chk "NPM :81"   200 http://192.168.2.2:81
+chk "NPM :80"   200 http://localhost:80
+chk "NPM :443"  200 --resolve nginx.gibbyb.com:443:127.0.0.1 https://nginx.gibbyb.com
+chk "Uptime"    302 http://192.168.2.2:3001
+chk "Portainer" 200 https://192.168.2.2:9443
+
+# ─────────────────────────────────────────────────────────────────────────────
+# Boot-enablement check (added 2026-08-12)
+#
+# Five stacks were once found RUNNING but DISABLED — healthy in `podman ps`, yet they
+# would have disappeared on the next reboot. Starting a stack and enabling it are
+# separate actions; this catches the gap. See AGENTS.md §13.3.
+# ─────────────────────────────────────────────────────────────────────────────
+echo
+echo "== boot-enablement =="
+# podman's own units are meant to stay disabled; n8n is intentionally down.
+# podman's own units stay disabled by design; n8n is intentionally down;
+# podman-update.service is `static` (the timer activates it, so it is never enabled);
+# watchtower was RETIRED 2026-08-17 and must stay disabled (AGENTS.md §13.15).
+SKIP='podman-auto-update|podman-restart|podman-clean-transient|podman-kube@|podman-user-wait-network-online|podman-n8n|podman-update|podman-watchtower'
+bad=0
+for u in $(systemctl --user list-unit-files 'podman-*.service' --no-legend 2>/dev/null | awk '{print $1}'); do
+    echo "$u" | grep -qE "$SKIP" && continue
+    state=$(systemctl --user is-enabled "$u" 2>/dev/null)
+    if [ "$state" != "enabled" ]; then
+        echo "  WILL NOT SURVIVE REBOOT: $u ($state)  -> systemctl --user enable $u"
+        bad=$((bad+1))
+    fi
+done
+[ "$bad" -eq 0 ] && echo "  all service units enabled"
+
+echo
+echo "== nightly image updates (replaces watchtower — AGENTS.md §13.15) =="
+if [ "$(systemctl --user is-enabled podman-update.timer 2>/dev/null)" = "enabled" ]; then
+    echo "  timer enabled, next: $(systemctl --user list-timers podman-update.timer --no-pager 2>/dev/null | awk 'NR==2{print $1,$2,$3,$4}')"
+else
+    echo "  WARNING: podman-update.timer is NOT enabled — nothing will update"
+fi
+if [ "$(systemctl --user is-active podman-watchtower.service 2>/dev/null)" = "active" ]; then
+    echo "  WARNING: watchtower is RUNNING — it recreates containers outside their pod (§13.13)"
+fi
+# A container outside its compose pod is the signature of a watchtower-style recreate.
+podless=""
+for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do
+    [ -z "$(podman inspect "$c" --format '{{.Pod}}' 2>/dev/null)" ] && podless="$podless $c"
+done
+[ -n "$podless" ] && echo "  PODLESS (rebuild with systemctl --user restart):$podless" || echo "  all containers in their pod"
+
+echo
+echo "== admin ports bound to 0.0.0.0 (should be 192.168.2.2 — AGENTS.md §13.2) =="
+podman ps --format '{{.Names}} {{.Ports}}' | grep -E '0\.0\.0\.0:(81|80-81|3001|9443|5432)(->|-)' \
+    | sed 's/^/  EXPOSED: /' || echo "  none"
+
+echo
+if [[ $fail -eq 0 ]]; then
+  printf '\033[32mALL CHECKS PASSED\033[0m\n'
+else
+  printf '\033[31m%d CHECK(S) FAILED\033[0m\n' "$fail"
+  echo "Investigate: journalctl --user -u podman-<name>.service -n 50"
+fi
+exit $fail