diff --git a/server/README.md b/server/README.md index ff6a462..15072dc 100644 --- a/server/README.md +++ b/server/README.md @@ -8,7 +8,9 @@ screen, plus rootless podman compose services managed as systemd user units. server/ vps/ The catalog panama-server installs from, one directory per service: compose.yml, the podman-.service user - unit, and an .env.example naming what the service needs told + unit, and an .env.example naming what the service needs told. + Loose files beside the directories are the VPS's own + host tooling (see "Host tooling" below) home/ The home server's services in the same shape (see below) scripts/ update-containers, the nightly image updater systemd/ The units behind it, linked by setup-server @@ -25,6 +27,8 @@ installable: panama-server reads `server/vps/` only, so a service moves into the catalog when it actually moves to podman. Paths keep their nesting (`server/home/Agents/Gitea` is `~/Server/Agents/Gitea`). +`Media/compact-tv/` is the nightly TV re-encode job that runs beside the Media +stack, with its own README. Three services are checkouts of upstream repositories. For those, only the local files are kept, and an `UPSTREAM` file names the repository and the exact commit the checkout sits at: `PostHog`, `Sentry` and `Plausible`. @@ -38,7 +42,8 @@ compose and its copy here can differ on those lines alone. The repository carries the definitions; the machine carries the state. `panama server enable ` creates `~/Server//` as a **real -directory** and symlinks only `compose.yml` into it; the unit symlinks into +directory** and symlinks the service's visible top-level files into it: +`compose.yml`, plus any helper script or README. The units symlink into `~/.config/systemd/user/`. The `.env` (seeded once from `.env.example`, then yours) and the bind-mounted `data/` live in `~/Server//` and are never inside the checkout — so nothing git can do, `clean -fdx` included, can reach @@ -99,6 +104,39 @@ existing service. The unit is the standard oneshot `podman compose up -d` shape (`WorkingDirectory=%h/Server/`); the shape contract pins the details. Secrets go in `.env.example` as `KEY=CHANGE_ME` lines. +A service that needs a scheduled job of its own carries it beside the +compose unit: a helper `.service` and a `.timer` with the same +name, plus the script it runs. Enable links all three and enables the timer. +Disable stops the timer and unlinks the units. `UseSend` is the example: its +`usesend-cert-refresh` timer keeps the SMTP proxy's certificate in step with +Nginx Proxy Manager's. + +## Host tooling + +Each server's own scripts and runbooks sit as loose files at the top of its +tree. They are reference copies for rebuilding a machine. Nothing installs +them. + +- `vps/backup`, `vps/server-backup.service`, `vps/server-backup.timer`: the + VPS's nightly backup, run as a user unit. It dumps every database into + `~/Server/_backup_state/`, then rsyncs `~/Server` to the home server's ZFS + pool with `--fake-super`, so subuid ownership survives the trip. +- `vps/verify-rootless`: the post-reboot health check for the rootless + stack. It covers linger, the unprivileged port floor, unit state and boot + enablement, postgres, endpoints, and admin ports bound to the WireGuard + address. +- `vps/AGENTS.md`: the VPS runbook, with both public IPs redacted. +- `home/RESTORE.md`: how to restore the home server's `~/Server` from the + ZFS backup. +- `home/backup` and `home/systemd/`: the home server's backup trigger and its + root system units (`server-backup` nightly, `media-backup` weekly). The + scripts those units run live in `/usr/local/sbin` on that machine. They are + readable by root alone, so this tree has no copy of them. + +The VPS units carry `RequiresMountsFor=/home/gib/Media` because that host +keeps its podman graphroot on an attached volume mounted there. The setting +is harmless on a machine without the mount. + ## Nightly updates `server/scripts/update-containers` pulls images and restarts changed diff --git a/server/home/Agents/Gitea/compose.yml b/server/home/Agents/Gitea/compose.yml index 02da314..498788b 100644 --- a/server/home/Agents/Gitea/compose.yml +++ b/server/home/Agents/Gitea/compose.yml @@ -19,7 +19,7 @@ x-runner-env: &runner-env TZ: America/New_York GITEA_INSTANCE_URL: https://git.gbrown.org GITEA_RUNNER_REGISTRATION_TOKEN: ${GITEA_RUNNER_REGISTRATION_TOKEN} - GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04" + GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04,build:docker://docker.gitea.com/runner-images:ubuntu-latest" DOCKER_TLS_VERIFY: "1" DOCKER_CERT_PATH: /certs/client CONFIG_FILE: /data/config.yml diff --git a/server/home/Agents/Portainer/compose.yml b/server/home/Agents/Portainer/compose.yml index 3e93e0b..7ff1bed 100644 --- a/server/home/Agents/Portainer/compose.yml +++ b/server/home/Agents/Portainer/compose.yml @@ -18,7 +18,7 @@ services: # restart: unless-stopped portainer-agent: - image: portainer/agent:2.33.2 + image: portainer/agent:latest container_name: portainer-agent hostname: portainer-agent networks: diff --git a/server/home/Gbrown/compose.yml b/server/home/Gbrown/compose.yml index 01925d5..91a9fb4 100644 --- a/server/home/Gbrown/compose.yml +++ b/server/home/Gbrown/compose.yml @@ -43,8 +43,8 @@ volumes: services: gbrown-backend: - # Pinned to the build the VPS Convex stacks already run. Bump on purpose. - image: ghcr.io/get-convex/convex-backend@sha256:d715e9ec088784407ca4ba2d3db592702cd328d02c76cdca3852c0018f2a76b4 + # Pinned to the build the VPS Convex stacks run (bumped 2026-10-28). Bump on purpose. + image: ghcr.io/convex/convex-backend@sha256:207cf95def42f01614b6e3af0af5c20d0dcaa5ff2adf258ee7bc9555576644fa container_name: gbrown-backend hostname: gbrown-backend networks: ['nginx-bridge'] @@ -70,7 +70,7 @@ services: gbrown-dashboard: # Released alongside the backend; pinned with it. - image: ghcr.io/get-convex/convex-dashboard@sha256:f85cf0d0448b9c835ae3df5c7c1c6f0145dd4c8f0f92eb0245daf304efdd9f1a + image: ghcr.io/convex/convex-dashboard@sha256:a55bf8abca1332f787bc4d4a79850182175027be7b70aa230be4389a1d3268e2 container_name: gbrown-dashboard hostname: gbrown-dashboard networks: ['nginx-bridge'] diff --git a/server/home/Media/compact-tv/README.md b/server/home/Media/compact-tv/README.md new file mode 100644 index 0000000..ea0373c --- /dev/null +++ b/server/home/Media/compact-tv/README.md @@ -0,0 +1,109 @@ +# compact-tv + +Sonarr and Radarr each have exactly four quality profiles. Two of them (**Standard** and +**Mini**) also have their files shrunk by `compact-tv.py`, which runs every night. + +| Profile | Used for | What gets kept | +|---|---|---| +| **Mini** | shows I demote hard | TV: 720p at 800 kbps (~0.42 GB per hour). Movies: 720p at 1500 kbps | +| **Standard** | **every request from other users (Seerr default)**, and anything I demote | TV: 1080p at 1200 kbps (~0.6 GB per hour). Movies: 1080p at 2500 kbps (~1.2 GB per hour) | +| **HD** | my own library, unchanged | whatever downloads, up to 1080p | +| **UHD** | only me, through Seerr's 4K servers | 4K | + +SD sources keep their resolution, at 600 kbps for TV and 900 kbps for movies. Movies get +higher rates because dark scenes, grain and action break down first at low bitrates. Nothing is ever +upscaled. Audio becomes Opus: 96 kbps for stereo, 160 for 5.1, 192 for 7.1. Subtitles, +chapters and attachments are copied as they are. I picked the rates from sample encodes of +Shameless S05E06 (VMAF ~91 for Standard, ~86 for Mini). + +## How the profiles work + +- **Standard and Mini** are copies of HD that upgrade only toward small releases (see below), + so Sonarr and Radarr never replace a file with a bigger one. +- **Small downloads first.** Both profiles prefer small releases when they grab: + - The "Small Encode" format (MeGusta, PSA, ELiTE, GalaxyTV/RG, YTS/YIFY) scores +3000. + - LQ scores 0. In HD and UHD it scores -10000, which keeps those groups out of HD. + - Standard merges HDTV/WEB/Bluray at each resolution, so the format score decides between + sources rather than the source type. + - Mini merges 720p and 1080p into one group and adds +1000 for 720p. + - Many files therefore arrive already small, and the job skips them. +- **Minimum sizes are shared by every profile.** For 720p I lowered them to 5 MB per minute + and for 1080p to 6 MB per minute (8 for Bluray-1080p), so small encodes are not rejected. + HD still never takes the LQ groups. +- **Seerr:** + - The "Sonarr - HD" and "Radarr" servers default to Standard. Only my admin account sees + the profile picker, so other users always get Standard. I pick HD or UHD per request. + - Other users no longer have "Request 4K". +- **Configarr is opt-in** (`docker compose --profile configarr up configarr`). Its templates + would recreate the old TRaSH profiles and size limits. +- Backups from before all this: `backups/pre-four-profiles-20261028/`. That covers both + apps' profiles, custom formats and size definitions, which title was on which profile, + and Seerr's settings. `setup-small-profiles.py` reapplies the Standard and Mini settings. + +## How the job works + +The nightly run (00:30 to 08:00) has two phases. + +**1. Search first.** It asks Sonarr and Radarr to search titles on Standard or Mini that it +hasn't searched before, biggest first: 5 shows and 20 movies a night, so downloads and +indexer use stay spread out. Standard and Mini upgrade only toward small releases: +- Every allowed quality sits in one group, so no resolution or source upgrade can happen. +- An upgrade needs a custom-format score jump of at least 2500, and upgrades stop at 3000. + Only gaining "Small Encode" clears that bar. + +So a search replaces a big file only with a small encode, never with a big HEVC release. +After that, the apps' regular RSS checks keep picking up small releases on their own. + +**2. Encode what's left.** A title becomes eligible for encoding 7 days after its search, +and only its files that are still big get encoded. Files that arrive after the title's +search work differently: +- This covers new episodes of ongoing shows, and fresh requests. +- The first night they're seen, they get their own search (one episode or movie at a time). +- They're encoded only if they're still big 3 days later. + +Downloads never wait for any of this, so new episodes and requests land in the library as +soon as they download. Only encoding waits. For each file, the job encodes to +`~/.cache/compact-tv/work.mkv` on the NVMe and then checks the result: +- the duration matches the original +- the audio and subtitle track counts match +- the file is smaller + +It then renames the new file over the original, keeping the same name with `.mkv`, and asks +Sonarr or Radarr to rescan. Plex picks the change up through its folder watcher, and the +episode keeps its Plex ID, so watch history stays. If a small release shows up later, the +app may still replace an encoded file with it, which costs bandwidth, not GPU. + +Other details: +- Each output carries a `COMPACT_TIER` tag. A file is skipped if re-encoding would save less + than 30%. Ruled-out files are cached by size in `state.json`, so they aren't probed every + night. Search dates and failures are kept there too. +- A downloaded HEVC file no bigger than twice the target size counts as done. That is the + small release a search brought in (a MeGusta 1080p episode runs about 2.3 Mbps against + Standard's 1.2), and it is neither re-encoded nor searched again. The job's own tagged + outputs are exempt, so Standard to Mini still works. +- **Skipped:** HDR and Dolby Vision sources (they need tone mapping), subtitle tracks ffprobe + can't identify, and files that have failed twice. +- Speed is about 0.19 hours of GPU per hour of 1080p video (NVENC is the bottleneck). The job + stops starting new encodes at 08:00. +- `cost-report.py` estimates, per title, the GPU hours the job would need against the size of + a fresh small download. +- A `PAUSE` file in this folder makes the nightly run do nothing. + +## Going back up + +Moving down works: Standard to Mini re-encodes the Standard copy. Moving up does not +restore quality. To get quality back, I set the title to HD and search, which redownloads +it at full size. + +## Running it + + ./compact-tv.py --dry-run # what tonight would do + ./compact-tv.py --search-all --limit 0 # search every unsearched title now, encode nothing + touch PAUSE # skip nightly runs until PAUSE is deleted + ./compact-tv.py --series "Shameless (US)" --limit 1 + systemctl status compact-tv.timer compact-tv.service + tail -f compact-tv.log + +The installed units are copies of `compact-tv.service` and `compact-tv.timer` from this +folder. `ExecStart` goes through `/usr/bin/python3` because SELinux won't let systemd +execute a file labeled as home content. diff --git a/server/home/Media/compact-tv/compact-tv.py b/server/home/Media/compact-tv/compact-tv.py new file mode 100755 index 0000000..3236aa0 --- /dev/null +++ b/server/home/Media/compact-tv/compact-tv.py @@ -0,0 +1,348 @@ +#!/usr/bin/env python3 +"""Shrink shows and movies that Sonarr/Radarr have on the "Standard" or "Mini" profile. + +Demoting something is just picking one of those profiles. Each night this job first asks +Sonarr/Radarr to search a few titles it hasn't searched before. Both profiles upgrade only +toward small releases, so the search swaps whatever big files have a small release for +bandwidth instead of GPU. A week after a title's search, the job re-encodes whatever is +still big to HEVC on the GPU (NVENC) at the tier's bitrate, verifies the result, swaps it in +place of the original, and has Sonarr/Radarr rescan. Plex and Jellyfin pick up changes +through their own folder watchers. + +Moving something back up means switching it to HD and searching, which redownloads it at +full size. See README.md. + +Usage: + compact-tv.py [--until HH:MM] [--series TITLE] [--limit N] [--dry-run] + +--series matches a show or movie title. Runs as gib (files are gib-owned) from +compact-tv.service, nightly. +""" +import argparse, datetime, fcntl, json, os, pathlib, re, shutil, subprocess, sys, time, urllib.request + +HERE = pathlib.Path(__file__).resolve().parent +STATE = HERE / "state.json" +LOG = HERE / "compact-tv.log" +WORK = pathlib.Path.home() / ".cache" / "compact-tv" # NVMe, so half-done encodes never land in a ZFS snapshot +VOLUMES = pathlib.Path.home() / "Server/Media/volumes" + +# Per app: API port, container path -> host path, and how to list one title's files. +APPS = { + "sonarr": {"port": 8989, "paths": ("/tv/", "/home/gib/Public/TV/"), "titles": "/series", + "files": lambda call, t: call(f"/episodefile?seriesId={t['id']}"), + "rescan": lambda t: {"name": "RescanSeries", "seriesId": t["id"]}}, + "radarr": {"port": 7878, "paths": ("/movies/", "/home/gib/Public/Movies/"), "titles": "/movie", + "files": lambda call, t: [t["movieFile"]] if t.get("movieFile") else [], + "rescan": lambda t: {"name": "RescanMovie", "movieId": t["id"]}}, +} + +# Video bitrate (kbps) per output class, and the widest output each tier allows. +# TV floors come from watching samples: 1200 kbps at 1080p, 800 at 720p, 600 at SD. +# Movies get roughly double: dark scenes, grain and action break down first at low rates, +# and a 2-hour movie at 2500 kbps is still ~2.3 GB (a quarter of a typical 1080p file). +# That also leaves already-small releases (YTS-style, ~1.5-2 GB) under the 30% bar. +# Nothing is ever upscaled: a 720p or SD source keeps its size and uses that class's rate. +TIERS = { + "Standard": {"max_width": 1920, "kbps": {"sonarr": {"1080": 1200, "720": 1200, "sd": 600}, + "radarr": {"1080": 2500, "720": 1800, "sd": 900}}}, + "Mini": {"max_width": 1280, "kbps": {"sonarr": {"1080": 800, "720": 800, "sd": 600}, + "radarr": {"1080": 1500, "720": 1500, "sd": 900}}}, +} +MIN_SAVING = 0.30 # leave a file alone unless re-encoding saves at least this much +MAX_FAILURES = 2 # after this many failed attempts a file is skipped for good + +# Search first, encode later. Standard/Mini upgrade only toward small releases (see +# setup-small-profiles.py), so asking the *arr to search a title swaps whatever big files +# have a small release, for bandwidth instead of GPU. Each night searches a few titles not +# searched before, biggest first, so downloads and indexer use stay spread out. A title's +# files are only encoded once its search is SEARCH_GRACE_DAYS old. +# +# Files that arrive after their title's search (new episodes, fresh requests) were never +# searched for a small copy, so the job searches that episode or movie on its own first, +# and encodes it only if it is still big FILE_SEARCH_GRACE_DAYS later. Downloads are never +# delayed by any of this; only encoding waits. +SEARCHES_PER_NIGHT = {"sonarr": 5, "radarr": 20} +SEARCH_GRACE_DAYS = 7 +FILE_SEARCH_GRACE_DAYS = 3 # small encodes usually appear within hours to a day; 3 leaves room +HDR_TRANSFERS = {"smpte2084", "arib-std-b67"} + + +def log(msg): + line = f"{datetime.datetime.now():%Y-%m-%d %H:%M:%S} {msg}" + print(line, flush=True) + with open(LOG, "a") as f: + f.write(line + "\n") + + +# ── Sonarr / Radarr ───────────────────────────────────────────────────────── +def arr(app): + """Return a call(path, body=None) for the app's v3 API.""" + key = re.search(r"([^<]+)", (VOLUMES / app / "config.xml").read_text()).group(1) + ip = subprocess.run(["docker", "inspect", app, "--format", + "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"], + capture_output=True, text=True, check=True).stdout.split()[0] + base = f"http://{ip}:{APPS[app]['port']}/api/v3" + + def call(path, body=None): + req = urllib.request.Request(base + path, headers={"X-Api-Key": key, "Content-Type": "application/json"}, + data=json.dumps(body).encode() if body is not None else None) + with urllib.request.urlopen(req, timeout=60) as r: + return json.load(r) + return call + + +# ── probing and planning ──────────────────────────────────────────────────── +def probe(path): + out = subprocess.run(["ffprobe", "-v", "error", "-print_format", "json", "-show_format", "-show_streams", str(path)], + capture_output=True, text=True) + return json.loads(out.stdout) if out.returncode == 0 else None + + +def subtitles(info): + """Subtitle tracks ffmpeg can carry over. Tracks ffprobe can't identify are dropped.""" + return [s for s in info["streams"] if s["codec_type"] == "subtitle" and s.get("codec_name")] + + +def audio_kbps(channels): + return 96 if channels <= 2 else 160 if channels <= 6 else 192 + + +def plan(info, tier, app): + """Return (ffmpeg args, estimated output bytes) for one file, or (None, reason) to skip it.""" + t = TIERS[tier] + video = next((s for s in info["streams"] if s["codec_type"] == "video" + and not s.get("disposition", {}).get("attached_pic")), None) + if not video: + return None, "no video stream" + if video.get("color_transfer") in HDR_TRANSFERS or any("Dolby Vision" in str(sd) for sd in video.get("side_data_list", [])): + return None, "HDR/Dolby Vision source (not handled yet)" + if (info["format"].get("tags") or {}).get("COMPACT_TIER") == tier: + return None, f"already {tier}" + + src_w = int(video["width"]) + out_w = min(src_w, t["max_width"]) + cls = "1080" if out_w >= 1700 else "720" if out_w >= 1200 else "sd" + vkbps = t["kbps"][app][cls] + duration = float(info["format"]["duration"]) + audios = [s for s in info["streams"] if s["codec_type"] == "audio"] + est = int((vkbps + sum(audio_kbps(int(a.get("channels") or 2)) for a in audios)) * 1000 / 8 * duration * 1.03) + if int(info["format"]["size"]) * (1 - MIN_SAVING) < est: + return None, f"already small ({int(info['format']['size']) / 1e9 * 3600 / duration:.2f} GB/h)" + # A downloaded HEVC file within 2x the target is a small release that already replaced a + # big one. Re-encoding it would cost GPU and quality for little space. Our own outputs + # (tagged) still go through, so moving a title from Standard to Mini works. + tags = info["format"].get("tags") or {} + if video.get("codec_name") == "hevc" and "COMPACT_TIER" not in tags and int(info["format"]["size"]) <= 2 * est: + return None, f"small HEVC release ({int(info['format']['size']) / 1e9 * 3600 / duration:.2f} GB/h)" + + vf = [] + if video.get("field_order") not in (None, "progressive", "unknown"): + vf.append("bwdif") + if out_w < src_w: + vf.append(f"scale={out_w}:-2:flags=lanczos") + args = ["-map", f"0:{video['index']}"] + if vf: + args += ["-filter:v", ",".join(vf)] + args += ["-c:v", "hevc_nvenc", "-preset", "p7", "-tune", "hq", "-profile:v", "main10", "-pix_fmt", "p010le", + "-rc", "vbr", "-multipass", "fullres", "-spatial-aq", "1", "-temporal-aq", "1", "-rc-lookahead", "32", + "-bf", "4", "-b_ref_mode", "middle", + "-b:v", f"{vkbps}k", "-maxrate", f"{vkbps * 2}k", "-bufsize", f"{vkbps * 4}k"] + for i, a in enumerate(audios): + ch = int(a.get("channels") or 2) + layout = {1: "mono", 2: "stereo", 6: "5.1", 8: "7.1"}.get(ch) + args += ["-map", f"0:{a['index']}", f"-c:a:{i}", "libopus", f"-b:a:{i}", f"{audio_kbps(ch)}k", + f"-filter:a:{i}", f"aformat=channel_layouts={layout or 'stereo'}"] + args += ["-mapping_family", "1"] + for i, s in enumerate(subtitles(info)): + args += ["-map", f"0:{s['index']}", f"-c:s:{i}", "srt" if s["codec_name"] == "mov_text" else "copy"] + args += ["-map", "0:t?", "-c:t", "copy", "-metadata", f"COMPACT_TIER={tier}"] + return args, est + + +# ── encode, verify, swap ──────────────────────────────────────────────────── +def encode(src, args, tier): + WORK.mkdir(parents=True, exist_ok=True) + tmp = WORK / "work.mkv" + tmp.unlink(missing_ok=True) + cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-hwaccel", "cuda", "-i", str(src), + *args, "-metadata", f"COMPACT_SOURCE={src.name} ({src.stat().st_size} bytes)", str(tmp)] + r = subprocess.run(cmd, capture_output=True, text=True) + if r.returncode != 0: + tmp.unlink(missing_ok=True) + raise RuntimeError("ffmpeg failed: " + r.stderr.strip()[-300:]) + return tmp + + +def verify(src_info, tmp): + new = probe(tmp) + if not new: + raise RuntimeError("output does not probe") + a, b = float(src_info["format"]["duration"]), float(new["format"]["duration"]) + if abs(a - b) > max(2.0, a * 0.005): + raise RuntimeError(f"duration mismatch {a:.1f}s vs {b:.1f}s") + counts = lambda i: (sum(1 for s in i["streams"] if s["codec_type"] == "audio"), len(subtitles(i))) + if counts(src_info) != counts(new): + raise RuntimeError(f"audio/subtitle track counts {counts(src_info)} -> {counts(new)}") + if int(new["format"]["size"]) >= int(src_info["format"]["size"]): + raise RuntimeError("output is not smaller") + return int(new["format"]["size"]) + + +def swap(src, tmp): + """Copy the encode next to the original, then rename it over the original in one step.""" + final = src.with_suffix(".mkv") + staged = src.with_name(f".{final.name}.compact-tmp") + shutil.copyfile(tmp, staged) + os.replace(staged, final) + if final != src: + src.unlink() + tmp.unlink(missing_ok=True) + return final + + +# ── main ──────────────────────────────────────────────────────────────────── +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--until", help="stop starting new encodes after this local time (HH:MM)") + ap.add_argument("--series", help="only this series title") + ap.add_argument("--limit", type=int, help="encode at most N files") + ap.add_argument("--dry-run", action="store_true", help="plan only, change nothing") + ap.add_argument("--search-all", action="store_true", + help="search every not-yet-searched title now instead of a few per night") + opt = ap.parse_args() + + if (HERE / "PAUSE").exists() and not opt.dry_run: + log("PAUSE file present, not encoding anything (delete it to resume)") + return + + lock = open(HERE / ".lock", "w") + try: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + except BlockingIOError: + sys.exit("another compact-tv run is active") + + deadline = None + if opt.until: + h, m = map(int, opt.until.split(":")) + now = datetime.datetime.now() + deadline = now.replace(hour=h, minute=m, second=0, microsecond=0) + if deadline <= now: + deadline += datetime.timedelta(days=1) + + state = json.loads(STATE.read_text()) if STATE.exists() else {} + for k in ("failures", "searched", "skip"): + state.setdefault(k, {}) + save = lambda: STATE.write_text(json.dumps(state, indent=1)) + today = datetime.date.today() + work = [] # (app, call, title, tier); TV first, then movies + for app, a in APPS.items(): + call = arr(app) + profiles = {p["id"]: p["name"] for p in call("/qualityprofile")} + work += [(app, call, t, profiles[t["qualityProfileId"]]) for t in call(a["titles"]) + if profiles.get(t["qualityProfileId"]) in TIERS and (not opt.series or t["title"] == opt.series)] + log(f"=== run start{' [dry run]' if opt.dry_run else ''}: {len(work)} title(s) on " + f"{'/'.join(TIERS)}{f', until {deadline:%H:%M}' if deadline else ''} ===") + + # ── 1. search: ask the *arr for small replacements before spending GPU ── + size_of = lambda t: (t.get("statistics") or {}).get("sizeOnDisk") or t.get("sizeOnDisk") or 0 + for app in APPS: + fresh = sorted((w for w in work if w[0] == app and f"{app}:{w[2]['id']}" not in state["searched"] + and size_of(w[2]) > 0), key=lambda w: -size_of(w[2])) + fresh = fresh if opt.search_all else fresh[:SEARCHES_PER_NIGHT[app]] + for _, call, t, tier in fresh: + log(f" {'would search' if opt.dry_run else 'search'} {t['title']} ({size_of(t) / 1e9:.0f} GB, {tier})") + if fresh and not opt.dry_run: + call = fresh[0][1] + if app == "sonarr": + for _, _, t, _ in fresh: + call("/command", {"name": "SeriesSearch", "seriesId": t["id"]}) + else: + call("/command", {"name": "MoviesSearch", "movieIds": [t["id"] for _, _, t, _ in fresh]}) + for _, _, t, _ in fresh: + state["searched"][f"{app}:{t['id']}"] = today.isoformat() + save() + + # ── 2. encode what is still big after its search ──────────────────────── + age = lambda iso: (today - datetime.date.fromisoformat(iso)).days + + def ready(app, ef, title_searched, to_search): + """True if this big file has had its chance to be replaced by a small download.""" + if (ef.get("dateAdded") or "")[:10] <= title_searched: # was on disk when the title was searched + return age(title_searched) >= SEARCH_GRACE_DAYS + key = f"{app}:file:{ef['id']}" # arrived later: search it on its own first + if key not in state["searched"]: + to_search.append(ef) + return False + return age(state["searched"][key]) >= FILE_SEARCH_GRACE_DAYS + + def search_files(app, call, show, files): + if opt.dry_run: + log(f" would search {len(files)} newer file(s) of {show['title']}"); return + if app == "sonarr": + ids = {f["id"] for f in files} + eps = [e["id"] for e in call(f"/episode?seriesId={show['id']}") if e.get("episodeFileId") in ids] + call("/command", {"name": "EpisodeSearch", "episodeIds": eps}) + else: + call("/command", {"name": "MoviesSearch", "movieIds": [show["id"]]}) + for f in files: + state["searched"][f"{app}:file:{f['id']}"] = today.isoformat() + log(f" search {len(files)} newer file(s) of {show['title']} before encoding"); save() + + done = saved = 0 + for app, call, show, tier in work: + title_searched = state["searched"].get(f"{app}:{show['id']}") + if not title_searched: + continue # its title search comes first, on a later night + container, host = APPS[app]["paths"] + files = sorted(APPS[app]["files"](call, show), key=lambda f: f["path"]) + changed, to_search = False, [] + for ef in files: + if deadline and datetime.datetime.now() >= deadline: + log("deadline reached, stopping"); break + if opt.limit is not None and done >= opt.limit: + break + src = pathlib.Path(ef["path"].replace(container, host, 1)) + if state["failures"].get(str(src), 0) >= MAX_FAILURES or not src.exists(): + continue + # Files already ruled out at this tier and size need no re-probe (most of them, most nights). + seen = f"{tier}:{src.stat().st_size}" + if state["skip"].get(str(src)) == seen: + continue + info = probe(src) + if not info: + log(f" SKIP {src.name}: does not probe"); continue + args, est = plan(info, tier, app) + if args is None: + state["skip"][str(src)] = seen # reason in `est`; quiet, since most nights most files are done + continue + if not ready(app, ef, title_searched, to_search): + continue + old = int(info["format"]["size"]) + if opt.dry_run: + log(f" would encode {src.name}: {old / 1e6:.0f} MB -> ~{est / 1e6:.0f} MB ({tier})") + done += 1; continue + t0 = time.monotonic() + try: + tmp = encode(src, args, tier) + new = verify(info, tmp) + final = swap(src, tmp) + except Exception as e: + state["failures"][str(src)] = state["failures"].get(str(src), 0) + 1 + save() + log(f" FAIL {src.name}: {e}"); continue + done += 1; saved += old - new; changed = True + log(f" {final.name}: {old / 1e6:.0f} MB -> {new / 1e6:.0f} MB " + f"(-{(1 - new / old) * 100:.0f}%, {time.monotonic() - t0:.0f}s, {tier})") + if to_search: + search_files(app, call, show, to_search) + if changed: + call("/command", APPS[app]["rescan"](show)) + if (deadline and datetime.datetime.now() >= deadline) or (opt.limit is not None and done >= opt.limit): + break + if not opt.dry_run: + save() + log(f"=== run end: {done} file(s){' planned' if opt.dry_run else ''}, saved {saved / 1e9:.1f} GB ===") + + +if __name__ == "__main__": + main() diff --git a/server/home/Media/compact-tv/compact-tv.service b/server/home/Media/compact-tv/compact-tv.service new file mode 100644 index 0000000..9ac14c6 --- /dev/null +++ b/server/home/Media/compact-tv/compact-tv.service @@ -0,0 +1,17 @@ +# Installed to /etc/systemd/system/. Runs as gib because the TV files are gib-owned, +# so the job needs no root. A system unit (rather than a user unit) because gib has +# no lingering session on this headless box. ExecStart goes through python3 because +# SELinux will not let systemd execute a file labeled as home content (user_home_t). +[Unit] +Description=Re-encode TV shows on Sonarr's Compact/Supercompact profiles +Documentation=file:///home/gib/Server/Media/compact-tv/README.md +After=docker.service +Requires=docker.service + +[Service] +Type=oneshot +User=gib +Group=gib +ExecStart=/usr/bin/python3 /home/gib/Server/Media/compact-tv/compact-tv.py --until 08:00 +Nice=15 +IOSchedulingClass=idle diff --git a/server/home/Media/compact-tv/compact-tv.timer b/server/home/Media/compact-tv/compact-tv.timer new file mode 100644 index 0000000..9570751 --- /dev/null +++ b/server/home/Media/compact-tv/compact-tv.timer @@ -0,0 +1,11 @@ +# Starts after the 00:00 hour and stops starting new encodes at 08:00 (see --until in +# the service), so Plex and Frigate keep the GPU's encoder during the day. +[Unit] +Description=Nightly TV re-encode for Compact/Supercompact shows + +[Timer] +OnCalendar=*-*-* 00:30:00 +Persistent=false + +[Install] +WantedBy=timers.target diff --git a/server/home/Media/compact-tv/cost-report.py b/server/home/Media/compact-tv/cost-report.py new file mode 100644 index 0000000..ccda4b8 --- /dev/null +++ b/server/home/Media/compact-tv/cost-report.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +"""Per-title cost of shrinking everything on Standard/Mini: re-encode vs redownload. + +For each show or movie on those profiles, probes every file with compact-tv's own planner +and reports what the nightly job would do (encode, skip as already small, skip as HDR), +the space it would save, and the GPU time it would take. Alongside that it estimates what a +fresh small download would cost instead, using typical small-encode sizes. + +GPU time is measured, not guessed: the Shameless pilot took 669 s for 57.5 minutes of +1080p (about 0.19 h of GPU per hour of video). 4K sources decode slower; the 1.5x for them +is an estimate until the first 4K night is logged. + +Usage: cost-report.py > report.tsv (also prints a summary to stderr) +""" +import importlib.util, json, pathlib, sys + +HERE = pathlib.Path(__file__).resolve().parent +spec = importlib.util.spec_from_file_location("ct", HERE / "compact-tv.py") +ct = importlib.util.module_from_spec(spec); spec.loader.exec_module(ct) + +GPU_H_PER_H = 669 / 3450 # pilot: hours of GPU per hour of 1080p video +UHD_FACTOR = 1.5 # estimate for 2160p sources +SMALL_GB_PER_H = {"Standard": 0.85, "Mini": 0.57} # MeGusta-class 1080p / 720p HEVC, measured on S.W.A.T. + +rows = [] +for app, a in ct.APPS.items(): + call = ct.arr(app) + profiles = {p["id"]: p["name"] for p in call("/qualityprofile")} + container, host = a["paths"] + for t in call(a["titles"]): + tier = profiles.get(t["qualityProfileId"]) + if tier not in ct.TIERS: + continue + r = {"app": app, "title": t["title"], "tier": tier, "files": 0, "hours": 0.0, "size": 0, + "enc_files": 0, "enc_size": 0, "enc_est": 0, "gpu_h": 0.0, "hdr_size": 0, "small_size": 0} + for f in a["files"](call, t): + p = pathlib.Path(f["path"].replace(container, host, 1)) + info = ct.probe(p) if p.exists() else None + if not info: + continue + size, dur = int(info["format"]["size"]), float(info["format"]["duration"]) / 3600 + r["files"] += 1; r["hours"] += dur; r["size"] += size + args, why = ct.plan(info, tier, app) + if args: + w = max((s.get("width") or 0) for s in info["streams"] if s["codec_type"] == "video") + r["enc_files"] += 1; r["enc_size"] += size; r["enc_est"] += why + r["gpu_h"] += dur * GPU_H_PER_H * (UHD_FACTOR if w > 1920 else 1) + elif "HDR" in str(why): + r["hdr_size"] += size + else: + r["small_size"] += size + if r["files"]: + r["redownload_gb"] = round(r["hours"] * SMALL_GB_PER_H[tier], 1) + rows.append(r) + +rows.sort(key=lambda r: -(r["enc_size"] - r["enc_est"])) +cols = ["app", "title", "tier", "files", "hours", "size", "enc_files", "enc_size", "enc_est", "gpu_h", "hdr_size", + "small_size", "redownload_gb"] +print("\t".join(cols)) +for r in rows: + print("\t".join(str(round(r[c], 2) if isinstance(r[c], float) else r[c]) for c in cols)) +tot = lambda k: sum(r[k] for r in rows) +print(f"{len(rows)} titles; encode {tot('enc_files')} files {tot('enc_size')/1e12:.2f} TB -> ~{tot('enc_est')/1e12:.2f} TB, " + f"{tot('gpu_h'):.0f} GPU hours; HDR skipped {tot('hdr_size')/1e12:.2f} TB; already small {tot('small_size')/1e12:.2f} TB", + file=sys.stderr) diff --git a/server/home/Media/compact-tv/setup-small-profiles.py b/server/home/Media/compact-tv/setup-small-profiles.py new file mode 100644 index 0000000..8a47755 --- /dev/null +++ b/server/home/Media/compact-tv/setup-small-profiles.py @@ -0,0 +1,99 @@ +#!/usr/bin/env python3 +"""Make Standard and Mini prefer small releases, and upgrade only toward small ones. + +Run once per app; it is idempotent, so re-running updates everything in place. + +- Adds "Small Encode" (release groups known for small HEVC/x264 encodes) and "720p" / + "1080p" resolution formats. They score 0 in every profile except Standard and Mini. +- HD and UHD are otherwise untouched: they keep LQ at -10000, so they never pick these. +- Standard and Mini: + - All allowed qualities share ONE group, so no "better resolution" or "better source" + upgrade can ever download a big file. Only the custom-format score decides. + - Scores: Small Encode +3000, x265 +2000 (so first grabs still prefer HEVC), LQ 0, + release-tier formats 0 (one big release must never "beat" another big one). + Standard adds +500 for 1080p and +250 for 720p; Mini adds +1000 for 720p and +500 for + 1080p. Those preferences stay below the upgrade threshold, so they never cause churn. + - Upgrades are on, but an upgrade needs a score jump of at least 2500, and upgrading + stops at 3000. In practice only gaining "Small Encode" clears that bar: a big H.264 + file gets replaced by a small encode, but never by a big HEVC one. + +Usage: setup-small-profiles.py +""" +import json, re, subprocess, sys, urllib.request, pathlib + +APP = sys.argv[1] +cfg = pathlib.Path.home() / f"Server/Media/volumes/{APP}/config.xml" +key = re.search(r"([^<]+)", cfg.read_text()).group(1) +ip = subprocess.run(["docker", "inspect", APP, "--format", "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"], + capture_output=True, text=True, check=True).stdout.split()[0] +base = f"http://{ip}:{8989 if APP == 'sonarr' else 7878}/api/v3" + +def call(path, body=None, method=None): + req = urllib.request.Request(base + path, method=method, headers={"X-Api-Key": key, "Content-Type": "application/json"}, + data=json.dumps(body).encode() if body is not None else None) + with urllib.request.urlopen(req, timeout=60) as r: + return json.load(r) + +GROUPS = r"^(MeGusta|PSA|ELiTE|GalaxyTV|GalaxyRG|GalaxyRG265|YIFY|YTS(\.(MX|LT|AG))?)$" +resolution = lambda r: [{"name": f"{r}p", "implementation": "ResolutionSpecification", + "negate": False, "required": True, "fields": [{"name": "value", "value": r}]}] +WANTED = { + "Small Encode": [{"name": "Small encode groups", "implementation": "ReleaseGroupSpecification", + "negate": False, "required": False, "fields": [{"name": "value", "value": GROUPS}]}], + "720p": resolution(720), + "1080p": resolution(1080), +} +TIERS = { + "Standard": {"Small Encode": 3000, "x265 (HD)": 2000, "LQ": 0, "LQ (Release Title)": 0, "1080p": 500, "720p": 250}, + "Mini": {"Small Encode": 3000, "x265 (HD)": 2000, "LQ": 0, "LQ (Release Title)": 0, "720p": 1000, "1080p": 500}, +} +RELEASE_TIER = re.compile(r"Tier \d+|WEB Scene") +MIN_UPGRADE_JUMP, UPGRADE_UNTIL = 2500, 3000 + +cfs = {c["name"]: c for c in call("/customformat")} +for name, specs in WANTED.items(): + body = {"name": name, "includeCustomFormatWhenRenaming": False, "specifications": specs} + if name in cfs: + body["id"] = cfs[name]["id"]; call(f"/customformat/{body['id']}", body, "PUT") + else: + call("/customformat", body) +cfs = {c["name"]: c["id"] for c in call("/customformat")} + + +def flatten(items): + """Every quality in the profile with whether it is allowed, groups expanded, lowest rank first.""" + for it in items: + for sub in (it["items"] or [it]): + yield sub["quality"], it["allowed"] + + +def one_group(profile): + """Put every allowed quality into a single top-ranked group; leave disallowed ones as they are.""" + flat = list(flatten(profile["items"])) + group = {"id": 1100, "name": "Any allowed", "allowed": True, + "items": [{"quality": q, "items": [], "allowed": True} for q, ok in flat if ok]} + profile["items"] = [{"quality": q, "items": [], "allowed": False} for q, ok in flat if not ok] + [group] + profile["cutoff"] = group["id"] + return profile + + +for prof in call("/qualityprofile"): + have = {f["format"] for f in prof["formatItems"]} + for name in WANTED: + if cfs[name] not in have: + prof["formatItems"].append({"format": cfs[name], "name": name, "score": 0}) + scores = TIERS.get(prof["name"]) + if scores: + prof = one_group(prof) + prof.update(upgradeAllowed=True, minUpgradeFormatScore=MIN_UPGRADE_JUMP, cutoffFormatScore=UPGRADE_UNTIL) + for f in prof["formatItems"]: + if f["name"] in scores: + f["score"] = scores[f["name"]] + elif RELEASE_TIER.search(f["name"]): + f["score"] = 0 + call(f"/qualityprofile/{prof['id']}", prof, "PUT") + if scores: + allowed = [q["quality"]["name"] for it in prof["items"] if it["allowed"] for q in it["items"]] + print(f"{APP} {prof['name']}: one group of {len(allowed)} qualities ({allowed[0]} .. {allowed[-1]}); " + f"upgrades on, jump >= {MIN_UPGRADE_JUMP}, until {UPGRADE_UNTIL}") + print(" nonzero scores: " + ", ".join(f"{f['name']}={f['score']}" for f in prof["formatItems"] if f["score"])) diff --git a/server/home/Media/compose.yml b/server/home/Media/compose.yml index ddb7dc0..f6c0bbc 100644 --- a/server/home/Media/compose.yml +++ b/server/home/Media/compose.yml @@ -427,3 +427,8 @@ services: # meant a completed sync was immediately restarted -- a hot loop of ~9 # full syncs every 3 minutes. `no` lets it run once per `up` and stop. restart: "no" + # Opt-in only (`docker compose --profile configarr up configarr`). Its templates + # create the TRaSH profiles and size limits that were removed on 2026-10-28 in + # favour of four hand-managed profiles (Mini/Standard/HD/UHD), so a plain `up` + # must never run it. See Media/compact-tv/README.md. + profiles: [configarr] diff --git a/server/home/Minecraft/Java/compose.yml b/server/home/Minecraft/Java/compose.yml index cc95bd7..4a54dbc 100644 --- a/server/home/Minecraft/Java/compose.yml +++ b/server/home/Minecraft/Java/compose.yml @@ -41,8 +41,36 @@ services: # world is in backups/; upgrades rewrite the world and cannot be undone. - VERSION=26.1.2 - DIFFICULTY=normal - - MAX_PLAYERS=10 - - TYPE=VANILLA + - MAX_PLAYERS=30 + # Paper since 2026-10-13. The last vanilla state is backups/gib-26.1.2-vanilla-*. + - TYPE=PAPER + # DonutSMP-style plugin set, pulled from Modrinth on each start. Every entry + # needs a 26.1.2 Paper build; check Modrinth before adding or bumping VERSION. + # Config lives in servers/gib/plugins//. GrimAC only publishes alpha builds. + - MODRINTH_DOWNLOAD_DEPENDENCIES=required + - | + MODRINTH_PROJECTS= + luckperms + vaultunlocked + essentialsx + essentialsx-spawn + placeholderapi + playerpoints + zmenu + zauctionhouse + ultimateshop + smartspawner + wildrtp + ultimate-teams + asbounty + axafkzone + tab-was-taken + deluxemenus + coreprotect + worldedit + worldguard + grimac:alpha + chunky - MEMORY=12G - SIMULATION_DISTANCE=16 - VIEW_DISTANCE=16 diff --git a/server/home/OpenWebUI/compose.yml b/server/home/OpenWebUI/compose.yml index 7c0e451..f0d28d5 100644 --- a/server/home/OpenWebUI/compose.yml +++ b/server/home/OpenWebUI/compose.yml @@ -38,6 +38,7 @@ services: - ENABLE_OAUTH_SIGNUP=true - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true #- ENABLE_OAUTH_ROLE_MANAGEMENT=true + #- OAUTH_ROLES_CLAIM=roles #- OAUTH_ADMIN_ROLES=admin,internal,superuser - OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID} - OAUTH_CLIENT_SECRET=${OAUTH_CLIENT_SECRET} diff --git a/server/home/RESTORE.md b/server/home/RESTORE.md new file mode 100644 index 0000000..a25cb2e --- /dev/null +++ b/server/home/RESTORE.md @@ -0,0 +1,140 @@ +# Restoring ~/Server on server.gib from the ZFS-pool backup + +Written 2026-09-17. Read this *before* you need it. + +## What this protects against + +The NVMe/LUKS btrfs volume (`/`, `/home`, and `/var/lib/docker`) dying. The backup lives on +the ZFS pool in the same machine, so it does **not** protect against fire, theft, or the +whole box failing. It is a disk-failure backup, not an offsite one. + +## Layout + + /home/gib/Public/Server_Backup/Containers/server.gib/current/ + +**One directory, overwritten daily at 02:00** by `server-backup.timer`. History lives in ZFS: +`Media/Server_Backup` has `zfs-auto-snapshot` keeping **14 dailies**, taken by `cron.daily` +between roughly 03:22 and 03:52. The 02:00 schedule is deliberate — it must finish before +that window, or a snapshot captures a half-written backup as that day's history. It also +staggers against the VPS backup, which pushes into this same pool at 01:00. + +Older copies are under the hidden snapshot dir (`snapdir=hidden`, so `ls` will not show it, +but you can `cd` in): + + ls /home/gib/Public/Server_Backup/.zfs/snapshot/ + ls /home/gib/Public/Server_Backup/.zfs/snapshot//Containers/server.gib/current/ + +The `03-16-2026` and `03-31-2026` directories are leftovers from the old manual script. + +## Why this runs as root + +This box runs Docker, so most container data under `~/Server` is root-owned. The old script +used `sudo rsync`, which worked but meant it could never be automated without a sudoers +rule. The replacement runs as root via `server-backup.service`, a systemd **system** unit — +already root, so ownership is preserved natively and **no sudoers entry is needed at +runtime**. If you see `/etc/sudoers.d/99-claude-setup` on this machine, it was temporary +setup scaffolding and should be deleted. + +(This differs from the VPS, which is rootless podman and needs `rsync --fake-super` to carry +subuid ownership through. Nothing like that is needed here.) + +The script lives at `/usr/local/sbin/server-backup`, deliberately **not** in `~/Server` — a +root-executed script inside a user-writable directory is a footgun. `~/Server/backup` is a +thin wrapper that triggers the systemd unit so manual runs behave identically to nightly ones. + +## What is in a backup + +| Path | What it is | +|---|---| +| `Media/`, `Frigate/`, `Home/`, `OpenWebUI/`, ... | Raw bind-mount data per service | +| `_backup_state/sqlite/` | Online `.backup` of every SQLite DB — **authoritative** | +| `_backup_state/convex/.zip` | `npx convex export` snapshot of each SQLite-backed Convex deployment (gbrown) — **authoritative** | + +> **Use `_backup_state/` for databases, not the raw copies.** Containers keep running during +> the backup, so the raw files are crash-consistent at best. The dumps use SQLite's online +> backup API and are what will actually restore cleanly. That covers the ones that matter: +> Plex's `com.plexapp.plugins.library.db` (watch history, ratings), Jellyfin, Frigate, Home +> Assistant, OpenWebUI, Audiobookshelf, and every *arr config. +> +> **Convex is the exception.** It writes to its SQLite file constantly, so a raw `.backup` +> never completes. Its `_backup_state/convex/` export is the restore source, and the raw +> `db.sqlite3` is not dumped at all. + +## What is deliberately NOT backed up + +- **`Agents/Gitea/volumes/runner-{1,2,3}/dind` — ~203 GB.** Docker-in-Docker containerd + image layers for the CI runners. Pure cache that churns constantly; a rebuilt runner just + re-pulls it. The parts that matter — `runner-N/data/.runner` (the 648-byte registration + token) and `certs/` — **are** backed up. + > Measured as root. A plain `du` as `gib` reports these as ~13 MB because gib cannot + > descend into them. Do not trust non-root sizes here. +- Docker **named volumes** (`/var/lib/docker/volumes`, 47 in use, ~61 GB): Sentry (13) and + Plausible (4) were consciously excluded — Sentry's event data is not worth restoring and + Plausible had nothing in it. Romm looked like a gap but is not: its anonymous `/romm` + volume is only a parent directory, with `assets/`, `config/` and `resources/` bind-mounted + from `~/Server` and `library/` from the pool. +- **Nextcloud AIO** — has its own borg backup at `~/Public/Nextcloud/Backups`, run by + hand from the AIO admin UI. Last run 2026-09-17 (finished 09-18). Runs are slow and + AIO stops the Nextcloud containers while one is going. +- `node_modules`, build output, `*.log`, `cache`/`tmp` dirs, `*.mp4`/`*.mkv`. + +## Restoring + +Stop the stack first so nothing overwrites what you restore: + + cd ~/Server/ && docker compose down # per service, or stop docker entirely + +Pull it back — as root, so ownership is restored: + + sudo rsync -a --numeric-ids --delete \ + /home/gib/Public/Server_Backup/Containers/server.gib/current/ \ + /home/gib/Server/ + +For an older day, use a snapshot path instead: + + /home/gib/Public/Server_Backup/.zfs/snapshot//Containers/server.gib/current/ + +Then put the databases back from the dumps. For each one, stop the owning container, copy the +dump over the live file, and match the original ownership. Plex, for example: + + docker compose -f ~/Server/Media/compose.yml stop plex + D=~/Server/_backup_state/sqlite/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases + L=~/Server/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases + sudo cp "$D/com.plexapp.plugins.library.db" "$L/" + sudo chown --reference="$L/com.plexapp.dlna.db" "$L/com.plexapp.plugins.library.db" + docker compose -f ~/Server/Media/compose.yml start plex + +`chown --reference` copies ownership from a neighbouring file, which saves looking up what +uid each container expects. + +For Convex, start the backend (it can be empty) and import the export through it. This +replaces every table with the snapshot: + + Z=~/Server/_backup_state/convex/gbrown-backend.zip + KEY=$(docker exec gbrown-backend ./generate_admin_key.sh | tail -1) + docker run --rm -it --network nginx-bridge -v "$Z:/backup.zip:ro" \ + -e CONVEX_SELF_HOSTED_URL=http://gbrown-backend:3210 -e CONVEX_SELF_HOSTED_ADMIN_KEY="$KEY" \ + docker.io/library/node:22-alpine sh -ec 'mkdir /w && cd /w && echo "{}" > package.json && + npm install --silent convex && npx convex import --replace-all /backup.zip' + +Functions and schema are not in the export. Redeploy them from the app's repo as usual. + +After a full restore the CI runners will have no `dind` storage. That is fine — they re-pull +images on the next job. The registration in `runner-N/data/.runner` is intact, so they +reconnect to Gitea without re-registering. + +## Checking it is working + + systemctl list-timers server-backup.timer + systemctl status server-backup.service + tail -40 ~/Server/logs/backup.log + +A healthy run ends with `=== backup complete ===`. Watch for: + +- `ZFS history OK` — confirms snapshots are still being taken. If this ever says **WARNING**, + versioning is broken and `current/` is just a mirror: a mistake would propagate with + nothing to roll back to. Fix `zfs-auto-snapshot` before anything else. +- `rsync exit 24` is a NOTE, not a failure — a file changed mid-transfer, normal on a live box. + +Optional: put an uptime-kuma push URL in `~/Server/.backup-heartbeat-url` and successful runs +will ping it, turning a silent failure into a down monitor. diff --git a/server/home/backup b/server/home/backup new file mode 100755 index 0000000..8c118e7 --- /dev/null +++ b/server/home/backup @@ -0,0 +1,14 @@ +#!/usr/bin/env bash +# Manual trigger for the daily backup. +# +# The real script lives at /usr/local/sbin/server-backup and runs as ROOT via +# server-backup.service. It is kept out of ~/Server on purpose: ~/Server is +# user-writable, and a root-executed script sitting in a user-writable directory +# is an obvious footgun. Run it through systemd so a manual run behaves exactly +# like the nightly one (same environment, same logging, same lock). +# +# status: systemctl status server-backup.service +# log: tail -f ~/Server/logs/backup.log +# timer: systemctl list-timers server-backup.timer +# restore: see ~/Server/RESTORE.md +exec sudo systemctl start --wait server-backup.service diff --git a/server/home/systemd/media-backup.service b/server/home/systemd/media-backup.service new file mode 100644 index 0000000..c088800 --- /dev/null +++ b/server/home/systemd/media-backup.service @@ -0,0 +1,19 @@ +[Unit] +Description=Weekly copy of the Media pool to the Backup pool +Documentation=file:///home/gib/Server/RESTORE.md +After=zfs-mount.service +Wants=zfs-mount.service +RequiresMountsFor=/home/gib/Public /home/gib/.local/share/Backup + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/media-backup +# 8.6 TB on the first run, deltas after. Generous, but bounded so a hung rsync +# fails loudly instead of overlapping next week's fire. +TimeoutStartSec=24h +Nice=10 +IOSchedulingClass=best-effort +IOSchedulingPriority=7 + +[Install] +WantedBy=multi-user.target diff --git a/server/home/systemd/media-backup.timer b/server/home/systemd/media-backup.timer new file mode 100644 index 0000000..cc5d05a --- /dev/null +++ b/server/home/systemd/media-backup.timer @@ -0,0 +1,17 @@ +[Unit] +Description=Weekly Media -> Backup pool copy + +[Timer] +# Saturday 05:00. Clear of: the cron.daily zfs-auto-snapshot window (03:22-03:52), +# timeshift (04:00), arr-queue-reconcile (04:31), and both ~/Server backups (01:00, 02:00). +# Saturday specifically because Monday carries the Media scrub and Sunday has +# raid-check (01:00), docker-prune (05:33) and dind-prune (06:03). +# gib also runs Nextcloud's borg by hand roughly monthly and is, in his words, +# not going to be doing that at 5 AM on a Saturday -- so the borg lock should be free. +OnCalendar=Sat *-*-* 05:00:00 +RandomizedDelaySec=15m +Persistent=true +Unit=media-backup.service + +[Install] +WantedBy=timers.target diff --git a/server/home/systemd/server-backup.service b/server/home/systemd/server-backup.service new file mode 100644 index 0000000..59553f3 --- /dev/null +++ b/server/home/systemd/server-backup.service @@ -0,0 +1,19 @@ +[Unit] +Description=Back up /home/gib/Server to the ZFS pool +Documentation=file:///home/gib/Server/RESTORE.md +# The destination is on ZFS; do not run before the pool is up. +After=zfs-mount.service docker.service +Wants=zfs-mount.service +RequiresMountsFor=/home/gib/Public + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/server-backup +TimeoutStartSec=8h +# This box is a daily driver running 145 containers. Stay out of the way. +Nice=10 +IOSchedulingClass=best-effort +IOSchedulingPriority=7 + +[Install] +WantedBy=multi-user.target diff --git a/server/home/systemd/server-backup.timer b/server/home/systemd/server-backup.timer new file mode 100644 index 0000000..b46f7c8 --- /dev/null +++ b/server/home/systemd/server-backup.timer @@ -0,0 +1,14 @@ +[Unit] +Description=Daily backup of /home/gib/Server to the ZFS pool + +[Timer] +# MUST finish before cron.daily fires zfs-auto-snapshot (~03:22-03:52); a snapshot that +# catches a running rsync stores a partial backup as that day's history. +# 02:00 also staggers against the VPS backup, which pushes into this same pool at 01:00. +OnCalendar=*-*-* 02:00:00 +RandomizedDelaySec=10m +Persistent=true +Unit=server-backup.service + +[Install] +WantedBy=timers.target diff --git a/server/scripts/update-containers b/server/scripts/update-containers index 52c0aa9..985624a 100755 --- a/server/scripts/update-containers +++ b/server/scripts/update-containers @@ -92,7 +92,7 @@ list_units() { # services therefore resolve no images and are skipped without a log line. # # Image tags frequently reference variables from the service's .env, e.g. -# ghcr.io/get-convex/convex-backend:${BACKEND_TAG:-latest} +# ghcr.io/convex/convex-backend:${BACKEND_TAG:-latest} # git.gbrown.org/gib/${NEXT_CONTAINER_NAME}:latest # Those must be expanded or the pull is attempted against a literal "${...}" and # always fails. We expand in a SUBSHELL so the .env cannot leak into this script's diff --git a/server/vps/AGENTS.md b/server/vps/AGENTS.md new file mode 100644 index 0000000..b96df39 --- /dev/null +++ b/server/vps/AGENTS.md @@ -0,0 +1,1789 @@ +# AGENTS.md — `~/Server` container environment + +> Snapshot of `junior.gib:~/Server/AGENTS.md`. The two public addresses read +> `` and `` here. The live copy has the real ones. + +Operating guide for AI agents and humans working on this VPS. Read this **before** adding a +container, editing a compose file, or touching the firewall. Much of what follows is +non-obvious and was learned the hard way; deviating will break things. + +**Host:** Hetzner VPS, Fedora 44, kernel 7.1.7-200, `podman 5.8.4`, `podman-compose 1.6.0`, +SELinux **Enforcing** (targeted), firewalld active. +**User:** `gib` (uid 1000), subuid/subgid range `524288:65536`. + +> **This file is the document. `CLAUDE.md` is a symlink to it.** That way Claude Code loads +> it automatically, while the 20+ compose files and scripts that cite `AGENTS.md §N` keep +> resolving. Edit `AGENTS.md`; never replace either name with a second real copy, or the two +> will drift apart silently. +> +> The symlink previously ran the other way and a markdown formatter dissolved it on +> 2026-09-22 — saving through a symlink replaced it with a regular file and orphaned the +> original. If `CLAUDE.md` ever turns back into a real file, that is what happened. + +## Docs that deliberately live elsewhere + +Kept next to the thing they describe, because that is where you will be standing when you +need them: + +| Path | Covers | +| ------------------------------------- | ------------------------------------------------------- | +| `UseSend/certs/README.md` | Why the SMTP proxy needs a real cert, and refreshing it | +| `Authentik/blueprints/REPRODUCE.md` | Rebuilding the password-recovery flow from scratch | +| `Next_Apps/Completeuphoria/README.md` | That app — **its own git repo**, do not fold in | +| `Search/Bang/README.md` | That app — **its own git repo**, do not fold in | + +Completed history (migration plan, session reports, one-off agent prompts) moved to +`archive/` on 2026-09-22. Nothing there is current; consult it for _why_, never for _how_. + +--- + +## 1. The one rule that governs everything + +> **Everything is ROOTLESS podman. There is no root podman on this box and there must +> never be again.** + +`/var/lib/containers` is empty, all root systemd units are deleted, and root's store was +wiped with `podman system reset`. If you catch yourself typing `sudo podman`, stop — you are +about to create a second, invisible container store that duplicates disk and silently +diverges from the running stack. + +| Do | Never | +| ---------------------------------- | ----------------------------------- | +| `podman ...` as `gib` | `sudo podman ...` | +| `systemctl --user ...` | `sudo systemctl ...` for containers | +| units in `~/.config/systemd/user/` | units in `/etc/systemd/system/` | +| `WantedBy=default.target` | `WantedBy=multi-user.target` | + +--- + +## 2. Layout and conventions + +``` +~/Server/ +├── AGENTS.md <- this file +├── archive/ <- completed history; see archive/README.md +├── verify-rootless.sh <- run after ANY change; exits non-zero on failure +├── backup <- rsync to remote; runs unprivileged +├── podman-example.service <- user-unit template for new services +├── compose.example.yml <- compose template +└── / + ├── compose.yml + ├── .env <- secrets; never commit, never echo in full + ├── podman-.service <- copy of the installed user unit (reference) + └── volumes/ | data/ | / <- bind-mounted data, gib-owned +``` + +### Service inventory (as of 2026-08-12, end of migration) + +**40 containers across 22 active user units.** `podman ps` is the source of truth; this is +orientation. Verify enablement with `verify-rootless.sh` (§13.3), not by eye. + +| Unit | Directory | Notes | +| ------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------- | +| postgresql | PostgreSQL | **shared, 14 databases.** pgvector, `-trixie` base only (§13.4) | +| nginx-proxy-manager | Nginx_Proxy_Manager | the front door; admin on `nginx.gibbyb.com` | +| authentik | Authentik | SSO. Port 9000 on the WireGuard IP only. **Protects many hosts by forward auth (§13.9)** | +| gitea | Gitea | git + container registry. No `userns_mode` (s6). Packages on the volume | +| watchtower | Watchtower | auto-update; API on WireGuard only; excludes postgres + authentik | +| adminer | Adminer | DB admin UI; no auth of its own | +| vaultwarden | Vaultwarden | | +| infisical | Infisical | secrets; DB on the shared postgres | +| tools | Tools | convertx, it-tools, pdf-tools, bento-pdf | +| rustdesk | Rustdesk | **split DNS — see §7.** Relay needs raw TCP; key must never be regenerated | +| beszel | Beszel | hub + this host's agent | +| uptime | Uptime | Uptime Kuma; unconfined (socket) | +| portainer | Portainer | unconfined (socket). Public hostname is `port.gbrown.org` | +| bang | Search/Bang/docker | built locally; defaults to Google since searxng was removed | +| isponsorblocktv | ISponsorBlockTV | host network, `keep-id` | +| completeuphoria | Next_Apps/Completeuphoria/docker | built locally | +| sierraandtyler | Next_Apps/sierraandtyler/docker | built locally | +| convexmonorepo | ConvexMonorepo | Next + Convex backend + dashboard | +| spoon | Spoon | Next + Convex + agent-worker (unconfined, needs the socket) | +| stpeteit | StPeteIT | Next + Convex + dashboard | +| lashaddict | LashAddict | Next + Convex + dashboard. **lashaddictstudio.com** (own Cloudflare zone); see §13.17 for the DNS pin | +| usesend | UseSend | app + redis + smtp-server + RustFS storage (`storage.usesend.gbrown.org`, pinned tag; replaced MinIO 2026-10-27) | +| **n8n** | N8n | **intentionally stopped** — unit disabled, data kept. `systemctl --user enable --now podman-n8n.service` | + +**Convex stacks use two ports** — `api.*` → 3210 (client API), `convex.*` → 3211 (HTTP +actions). Getting these backwards produces a working-looking site that 500s. + +**Compose files present but NEVER STARTED** (no unit enabled): `Documenso`, `CalCom`, +`Penpot`. Each needs its role and database created first; Documenso additionally needs a +signing certificate. Instructions are in each compose file's header. + +Never migrated / stays home: nextcloud, immich, plex, jellyfin, \*arr, sentry, openwebui, +frigate, romm, obsidian-livesync, the KVMs. **Home runs Docker, not podman** (§13.8b). + +Service directory names are `TitleCase`; unit and container names are `lowercase`. +Some services nest under a category (`Search/Bang/docker`, `Next_Apps/Completeuphoria/docker`) — +the unit's `WorkingDirectory` is what matters, not depth. + +--- + +## 3. Adding a new container — the checklist + +### 3.1 compose.yml + +```yaml +networks: + nginx-bridge: + external: true # ALWAYS external; see §5 + +services: + myservice: + image: docker.io/vendor/image:tag + container_name: myservice + hostname: myservice + networks: ["nginx-bridge"] + environment: + - TZ=America/New_York # see §8 + volumes: + - ./data:/var/lib/whatever:Z # :Z is MANDATORY; see §6 + restart: unless-stopped +``` + +Do **not** publish ports unless the service must be reachable off-box. Everything on +`nginx-bridge` resolves by container name, so NPM proxies to `http://myservice:8080` +internally with no host port at all. Fewer published ports = less firewall surface. + +### 3.2 Does it need `userns_mode`? + +This is the subtlest part of the setup. Decide by how the image starts: + +```bash +podman inspect --format '{{.Config.User}}' # empty => starts as root +``` + +| Image behaviour | What to do | Host file ownership | +| ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | +| **Starts as root**, drops privileges internally (most images: nginx, postgres, uptime-kuma, influxdb, beszel) | Nothing. Default mapping puts container-root at host uid 1000. | `gib` for root-owned paths; the container's own chown pushes service-user files into subuid space (e.g. `525264`) | +| **Runs directly as a fixed non-root uid**, or you want host files to stay `gib`-owned | `userns_mode: "keep-id:uid=,gid="` | `gib` | + +**Never use `user: 1000:1000`.** Under a rootless userns that resolves to subuid `525287`, +not to `gib`, and the container loses access to its own gib-owned data. This exact mistake +is why n8n was broken. Use `keep-id` instead. + +Live examples: + +- `PostgreSQL` — `keep-id:uid=999,gid=999` (postgres runs as 999; keeps `pg_data` gib-owned) +- `N8n` — `keep-id:uid=1000,gid=1000` +- everything else — default mapping + +**uid mapping cheat sheet** (subuid base 524288): + +``` +container uid 0 -> host 1000 (gib) +container uid 999 -> host 525286 +container uid 1000 -> host 525287 +container uid N -> host 524288 + (N-1) +``` + +### 3.3 Host directory ownership + +Create data dirs **owned by gib** before first start: + +```bash +mkdir -p ~/Server/MyService/data # already gib-owned +``` + +If you ever import data written by another uid, `sudo chown -R gib:gib` it first. A uid +outside gib's subuid range appears as `nobody` inside the container and cannot be fixed by +the container itself. + +### 3.4 systemd user unit + +Copy `podman-example.service`, adjust `Description` and `WorkingDirectory`: + +```bash +cp ~/Server/podman-example.service ~/.config/systemd/user/podman-myservice.service +# edit WorkingDirectory=%h/Server/MyService +systemctl --user daemon-reload +systemctl --user enable --now podman-myservice.service +cp ~/.config/systemd/user/podman-myservice.service ~/Server/MyService/ # keep a copy +``` + +If it depends on Postgres, add to `[Unit]`: + +```ini +After=network-online.target podman-postgresql.service +Wants=podman-postgresql.service +``` + +> ⚠️ Ordering ≠ readiness. `podman compose up -d` returns when the container is _created_, +> not when Postgres accepts connections. Dependents must tolerate a brief DB-unavailable +> window (NPM and n8n both retry). If one doesn't, add an `ExecStartPre` polling +> `pg_isready`. + +### 3.5 Verify + +```bash +~/Server/verify-rootless.sh # add your service to the lists inside it +``` + +--- + +## 4. Boot behaviour (linger) + +User services start at boot **without anyone logging in** because lingering is enabled: + +```bash +loginctl show-user gib --property=Linger # must be Linger=yes +``` + +If linger is ever off, nothing starts until someone SSHes in — and logging in _masks_ the +failure. When testing boot behaviour, check from **outside** the box first, before SSHing in. + +Proven working: after reboot, `user@1000.service` starts ~9 s after boot and all containers +~11–13 s after, well before any login. + +--- + +## 5. Networking + +All services share the external network **`nginx-bridge` (172.18.0.0/24, gw 172.18.0.1)**. + +Created out-of-band so no stack races to create it at boot: + +```bash +podman network create --subnet 172.18.0.0/24 nginx-bridge +``` + +Every compose file declares it `external: true` — including NPM, which used to define it +inline. Containers resolve each other by `container_name` over this network. + +> **Rootless and root networks are entirely separate namespaces.** This is why the migration +> had to be a big-bang cutover. Irrelevant now that root podman is gone, but it explains why +> you cannot "test one service" against a differently-privileged stack. + +### Privileged ports + +NPM binds 80/443 without root because of `/etc/sysctl.d/99-rootless-ports.conf`: + +``` +net.ipv4.ip_unprivileged_port_start=80 +``` + +Host-wide: **any** unprivileged process can bind ≥80. Acceptable on a single-user VPS. If +this file is lost, NPM silently fails to bind and every site goes down. + +### ⚠️ Published ports lose the client IP + +Rootless publishes ports through the **`rootlessport` userspace proxy**, so the peer address +nginx sees is `172.18.0.x`, not the real client. **No header trick recovers this when there +is no proxy in front** — nothing upstream sets one. + +The only real fix is to put the hostname behind Cloudflare, so `CF-Connecting-IP` exists. +See §7. + +--- + +## 6. SELinux — Enforcing + +Every bind-mounted volume **must** carry `:Z` (or `:rw,Z` / `:ro,Z`): + +```yaml +volumes: + - ./data:/var/lib/whatever:Z +``` + +`:Z` makes podman relabel the host path to `container_file_t` with a **private MCS category** +unique to that container, re-derived on every start. Without it, Enforcing denies all access +and the container fails. + +### ⚠️ Shared mounts need `:z` (lowercase), not `:Z` + +`:Z` assigns a **private** MCS category unique to one container. If two containers mount the +**same host path**, the second to start relabels it and locks the first out. Use lowercase +`:z` for shared paths: + +```yaml +server: + volumes: + - ./volumes/media:/data/media:z # shared with worker -> lowercase + - ./volumes/server/custom-templates:/templates:Z # exclusive -> uppercase +worker: + volumes: + - ./volumes/media:/data/media:z # same path, same lowercase z + - ./volumes/worker/certs:/certs:Z # exclusive -> uppercase +``` + +Verify: a shared dir shows `container_file_t:s0` with **no** category; an exclusive dir's +category must match its container's `ProcessLabel`. + +```bash +ls -dZ volumes/media # want: container_file_t:s0 +podman inspect authentik-server --format '{{.ProcessLabel}}' +``` + +### ⚠️ Watchtower recreates DROP the :Z/:z bind flag (found 2026-08-13) + +When watchtower recreates a container after pulling a new image, the copied +bind loses its relabel option (`Binds` shows only `rprivate,rbind`). With `:Z` +the volume keeps the OLD container’s MCS category while the new container gets +a fresh pair — the service is locked out of its own data. agentchat hit this +as SQLITE_CANTOPEN in a crashloop; fixed by switching its bind to `:z`. + +**Any watchtower-updated container with a bind mount must use lowercase `:z`.** +After a recreate the files stay `container_file_t:s0` (no category), which a +confined container can still access — the only labeling that survives. +The Convex stacks (lashaddict / stpeteit / convexmonorepo / spoon) were +switched to `:z` and restarted 2026-08-13; all backends verified healthy. + +Two deliberate exceptions: + +1. **The podman socket mount gets NO `:Z`.** + ```yaml + - /run/user/1000/podman/podman.sock:/var/run/docker.sock # no :Z, ever + ``` + Relabeling the live socket would break podman itself. +2. **Containers mounting that socket need SELinux separation off**, because systemd + recreates the socket each boot as `user_tmp_t`: + ```yaml + security_opt: ["label:disable"] + ``` + Currently only **portainer** and **uptime**. They run unconfined (empty `ProcessLabel`); + the other nine are confined as `container_t:s0:c…`. Accept this only for + container-management tooling. + +Check confinement: + +```bash +podman inspect --format '{{.ProcessLabel}}' # empty = unconfined +``` + +> ### ⚠️ NEVER run `restorecon -F` against `~/Server` +> +> Plain `restorecon -r` is **safe** — it reports the `:Z` labels as "customized by admin" +> and leaves them alone. `-F` forces them back to `user_home_t` and every container loses +> access to its data simultaneously. + +Going Permissive→Enforcing needed no relabel here: `restorecon -nr` on `/etc /usr/sbin +/usr/bin /usr/lib/systemd /var/lib /root /boot` reported 0 mislabeled files. Re-check that +before flipping SELinux on any rebuild. + +--- + +## 7. Firewall and Cloudflare + +firewalld zones — **rules apply by arriving interface**: + +| Interface | Zone | Policy | +| ------------------------------- | --------- | ------------------------------------ | +| `eth0` (, public) | `public` | `ssh` + explicitly opened ports only | +| `Home` (192.168.2.2, WireGuard) | `trusted` | target ACCEPT — full LAN/VPN access | + +Binding `Home` to `trusted` is what keeps LAN devices working when `public` is locked down. +Before that it had **no zone** and silently inherited `public`. + +Deliberately closed on the public interface: **81** (NPM admin), 3001, 9443, **5432**. +Reach NPM admin via `https://nginx.gibbyb.com` or over WireGuard at `192.168.2.2:81`. +Never open 5432 to the internet. + +### ⚠️ Every VPS hostname must be Proxied (orange cloud) + +This is the single most important DNS rule here, and it has bitten repeatedly. Because +80/443 accept only Cloudflare ranges, **a DNS-only record pointing at is +unreachable** — it fails at connect, not with an error page. The only exception is +rustdesk, below, which uses ports the firewall opens explicitly. + +### Rustdesk — the one service that cannot be proxied + +RustDesk's ID/relay protocol is raw TCP/UDP on 21115-21119. Cloudflare's proxy carries only +HTTP/HTTPS, so it uses **two hostnames on purpose**: + +| Hostname | Cloud | Serves | +| --------------------- | ------------------------------------------- | ----------------------------- | +| `relay.gbrown.org` | **grey / DNS-only** → | ID + relay, ports 21115-21119 | +| `rustdesk.gbrown.org` | **proxied** → NPM → `rustdesk-server:21114` | web console + API, over TLS | + +Do not "simplify" these into one name — the API would then have to travel unencrypted, or +the relay would break. `relay.gbrown.org` publishes the VPS IP; that is accepted, because +80/443 remain Cloudflare-only so the IP grants no access to any web service. + +Firewall ports opened for it (public zone): `21114-21119/tcp` plus **`21116/udp`** — the UDP +one is easy to miss and heartbeats fail silently without it. + +The server keypair at `Rustdesk/volumes/server/id_ed25519` was carried over from home. Its +public half must keep matching `RUSTDESK_API_RUSTDESK_KEY` in the compose, or **every client +needs re-pairing**. Never regenerate it. + +### Split-horizon DNS — LAN clients need their own records + +The UniFi gateway answers `*.gbrown.org` → `192.168.1.2` (home) for every hostname. So after +migrating a service, LAN clients keep silently reaching the OLD copy — no error, just stale +data. Public DNS being correct does not help them. + +**For each migrated service, add a UniFi Host (A) record → `192.168.2.2`** (the VPS's +WireGuard address). UniFi honours exact matches over the wildcard, so the wildcard stays put +for everything still at home. Verified working: LAN clients reach `192.168.2.2:443`, the +certificate validates, and the firewall's `trusted` zone permits it. + +Do **not** point LAN records at `` — the firewall drops non-Cloudflare traffic. +Do **not** point them at Cloudflare edge IPs — those are not stable. + +Currently carrying UniFi overrides: `vault`, `tools`, `infisical`, `auth`. + +> **Never add `nginx.gbrown.org` → VPS.** That hostname reaches _home's_ proxy manager admin +> UI. Pointing it at the VPS locks you out of it. The VPS admin lives on `nginx.gibbyb.com`. + +### Advanced Certificate Manager + +**ACM is active on `gbrown.org` only** (~$10/mo, confirmed billed to that zone). Universal +SSL covers one subdomain level; ACM covers the multi-level wildcards. Active certificate +(expires 2026-11-09) carries seven SANs: + +``` +gbrown.org, *.gbrown.org, *.tools.gbrown.org, *.spoon.gbrown.org, +*.lashaddict.gbrown.org, *.convexmonorepo.gbrown.org, *.convex.gbrown.org +``` + +A **second** advanced certificate (ordered 2026-10-27) carries the two-label patterns added +later. Advanced certificates cannot be edited, so a new pattern means ordering another one, never +deleting the first (that would drop every name only it covers): + +``` +*.usesend.gbrown.org, usesend.gbrown.org, *.billtracker.gbrown.org, billtracker.gbrown.org +``` + +Without it, `convert.tools.gbrown.org` and the 12 other two-label hostnames cannot be +proxied at all. If a new two-label pattern appears, it needs adding to this certificate. + +**Ports 80/443 on `eth0` are restricted to Cloudflare only** (active since 2026-08-11), via +firewalld ipsets `cloudflare4` (15 ranges) and `cloudflare6` (7 ranges): + +``` +tcp dport 22 accept <- ssh, never remove +ip saddr @cloudflare4 tcp dport 80,443 accept +ip6 saddr @cloudflare6 tcp dport 80,443 accept + <- everything else: zone default reject +``` + +**Refresh the ranges periodically** — Cloudflare does change them, and a stale set silently +blocks legitimate traffic: + +```bash +curl -s https://www.cloudflare.com/ips-v4 -o /tmp/cf-v4.txt +curl -s https://www.cloudflare.com/ips-v6 -o /tmp/cf-v6.txt +sudo firewall-cmd --permanent --ipset=cloudflare4 --add-entries-from-file=/tmp/cf-v4.txt +sudo firewall-cmd --permanent --ipset=cloudflare6 --add-entries-from-file=/tmp/cf-v6.txt +sudo firewall-cmd --reload +``` + +> **You cannot test this restriction from the VPS itself.** Traffic from the host to its own +> public IP routes over loopback and never traverses the `eth0` INPUT path, so it always +> succeeds and proves nothing. Verify from an external machine: +> +> ```bash +> curl -k --max-time 10 --resolve any.gibbyb.com:443: https://any.gibbyb.com/ +> ``` +> +> **Verified 2026-08-11:** `curl: (7) Failed to connect ... after 46 ms: Could not connect`. +> A fast _rejection_ (tens of ms) is the correct signal — the zone's `default` target sends +> an ICMP reject. A 10 s hang would mean a DROP instead; a `200`/`302` would mean the rule +> is not taking effect. Meanwhile `https://any.gibbyb.com/` via Cloudflare returns normally. + +> ### ⚠️ Verify DNS is actually proxied before restricting 80/443 +> +> A Cloudflare-only rule black-holes any hostname whose DNS points straight at the origin. +> **This happened during setup**: `*.gibbyb.com` was DNS-only, and applying the rule took +> every one of those sites down until the ports were reopened. Check each hostname; do not +> trust an assumption that "everything is behind Cloudflare": +> +> ```bash +> resolvectl query uptime.gibbyb.com # = DIRECT (would be blocked) +> # 104.21.x / 172.67.x = Cloudflare (safe) +> ``` +> +> **Any new hostname pointed at this VPS must be Proxied (orange cloud) in Cloudflare +> before it will work at all.** DNS-only records are blocked by the firewall. +> +> Current DNS: `gibbyb.com` A → `` **Proxied**, `*.gibbyb.com` CNAME → +> `gibbyb.com` **Proxied**, so new subdomains inherit proxying automatically. +> Cloudflare SSL/TLS mode must be **Full (strict)** — NPM serves valid Let's Encrypt certs; +> "Flexible" causes redirect loops. +> +> Note `*.gbrown.org` resolves to `` — a **different machine** (home network). +> NPM here holds configs for those names but traffic does not arrive here. + +### Real client IPs + +`volumes/data/nginx/custom/server_proxy.conf` (included in every proxy host's **server** +block) contains: + +```nginx +real_ip_header CF-Connecting-IP; +``` + +It must be in _server_ context — `nginx.conf:81` already declares `real_ip_header` at http +level, and redeclaring there is a fatal duplicate-directive error. NPM's `nginx.conf` already +trusts the bridge via `set_real_ip_from 172.16.0.0/12`, so no `set_real_ip_from` is needed. + +This is now trustworthy **because** the firewall permits 80/443 only from Cloudflare ranges: +nothing else can reach nginx to forge the header. The two controls depend on each other — +if you ever reopen 80/443 to the internet, `CF-Connecting-IP` becomes forgeable again and +logged IPs can no longer be trusted for access lists or fail2ban. + +--- + +## 8. Timezone + +Host and all containers are **`America/New_York`**. New services should set +`TZ=America/New_York` in `environment:`, or their logs stamp UTC and correlating across +services becomes painful. + +--- + +## 9. Backups + +`~/Server/backup` rsyncs to a remote host and **runs unprivileged** — do not reintroduce +`sudo`. As of 2026-08-11 every file under `~/Server` is gib-owned and readable (verified: +4,307 entries traversed, 0 permission errors), so nothing is being silently skipped. + +It still carries `--exclude=dump.rdb` defensively. That was for valkey, which chowned its +own data to subuid 525286 at mode 600 inside the userns — unreadable to gib. Valkey went +with searxng, but any future service that writes as a non-root uid with restrictive modes +will hit the same thing. **If you add one, re-run the check above**; a `Permission denied` +in rsync means data is silently missing from your backups: + +```bash +rsync -avn --exclude=node_modules --exclude=.git ~/Server/ /tmp/probe/ 2>&1 | grep -i denied +``` + +Database backups are **not** covered by rsync in any useful form — take a logical dump: + +```bash +podman exec postgresql pg_dumpall -U npm > ~/pg_dumpall_$(date +%F).sql +``` + +Do this before any change touching `pg_data` ownership, permissions or SELinux labels. + +Pre-migration snapshot kept at `~/premigration_backup_2026-08-10/` (volume tarball + +4,043-entry ownership manifest + pg dump). + +--- + +## 10. Operational gotchas + +### ⚠️ Encrypted ClientHello breaks split-horizon DNS — keep `ech` OFF + +**This cost more time than everything else combined, and nothing about it is guessable.** + +Cloudflare publishes an HTTPS/SVCB record for proxied hostnames containing an `ech=` +parameter whose public name decodes to `cloudflare-ech.com`. Browsers read that record, +enable Encrypted ClientHello, and send **`cloudflare-ech.com` as the outer SNI**. With a +UniFi override sending that connection to our own nginx instead of Cloudflare, nginx has +never heard of `cloudflare-ech.com` and replies with **TLS alert 112, unrecognized_name**. +Browsers do not retry. + +Symptoms, all of which we chased down blind alleys: + +- Browser shows `ERR_SSL_UNRECOGNIZED_NAME_ALERT`; **`curl` and `openssl` return 200**. + They never query HTTPS RRs, so they send plaintext SNI and always succeed. +- Public access works fine — ECH terminates at Cloudflare, which understands it. +- Only LAN clients with a UniFi override are affected. +- A hostname with **no** HTTPS record (grey cloud, e.g. `sonarr`) works normally. That + contrast is the diagnostic. + +The zone setting is `ech`. It is **hard-disabled in the dashboard UI on the Free plan but +editable via the zone settings API**. It must stay `off` for as long as split-horizon DNS +points any proxied hostname at a local origin. + +Reproduce in one command: + +```bash +openssl s_client -connect 192.168.2.2:443 -servername cloudflare-ech.com +# -> tlsv1 unrecognized name ... SSL alert number 112 +dig vault.gbrown.org HTTPS # must show NO ech= parameter +``` + +> **Do not test browser problems with curl.** curl bypasses HTTPS RRs, ECH, HSTS, service +> workers and the HTTP cache. A green curl next to a failing browser means the difference +> between them _is_ the bug. Reproduce with the actual browser engine — headless works: +> `chrome --headless=new --dump-dom ` and read stderr for `net_error` codes. + +**nginx sends no SNI on `proxy_pass` by default.** Proxying to an HTTPS upstream that serves +multiple vhosts — e.g. home NPM → `https://192.168.2.2:443` — fails the TLS handshake with +no useful error, surfacing as a 502. Cost us hours. Always add: + +```nginx +proxy_ssl_server_name on; +proxy_ssl_name $host; +``` + +Symptom to recognise: `curl` to the bare IP gives `000`, but `curl --resolve host:443:IP` +gives a real status. That difference _is_ the diagnosis. + +**Scheme/port mismatches produce misleading errors.** We hit this three separate times: +`http` → port 9443, `https` → port 9000, `http` → port 443. All produced 400/502/000 rather +than anything naming the cause. When a proxy host misbehaves, check scheme against port +_first_. Authentik serves HTTP on 9000 and HTTPS on 9443; NPM serves HTTPS on 443. + +**NPM reloads nginx on every save, and requests during the reload get a 503 with an empty +body.** Wait ~10 s and retry before treating a 503 as a real failure. Multiple "outages" +this session were this. + +**A 401 is not a failure for CouchDB or for authentik's outpost.** CouchDB demands +credentials on every endpoint; browsers render the resulting auth prompt as an error page. +Check the nginx access log for the status nginx actually returned rather than trusting what +a browser displays. + +**Don't test forward-auth with bare curl.** The authentik outpost needs `X-Forwarded-*` +headers to reconstruct the original URL; without them it returns 500 and logs "failed to +detect a forward URL from nginx". That 500 is a bad test, not a broken outpost. + +**DNS negative caching outlives record changes.** After deleting and recreating a record, +`systemd-resolved` will keep serving NXDOMAIN. `sudo resolvectl flush-caches`. A hostname +that resolves with `dig @1.1.1.1` but fails with `curl` is this. + +**`nc -z` inside the NPM container is not a reliable upstream probe** — it reports false +negatives for containers that are demonstrably serving. Test the real path with `curl +--resolve host:443:127.0.0.1` instead. + +### Older gotchas + +**Check `systemctl --failed` and capture logs BEFORE vacuuming journals.** A +`journalctl --vacuum-size` during cleanup destroyed the logs explaining an n8n failure that +had been running for two months. Diagnose first, reclaim space second. + +**`podman image prune -a` is dangerous while the stack is stopped.** With no containers +running, `-a` considers every image unused and deletes images the stack depends on. Use plain +`podman image prune` (dangling only). + +**Env var names must match what the image expects.** `PostgreSQL/.env` defined +`POSTGRES_NAME` while compose passed `POSTGRES_DB`, so the healthcheck expanded to +`pg_isready -d -U npm` and failed 178,503 consecutive times while Postgres itself was fine. +Both keys are present now; keep them in sync. + +**Every service in a compose file needs its `networks:` key.** n8n's `task-runners` omitted +it, landed on the compose default network, and could never resolve +`N8N_RUNNERS_TASK_BROKER_URI=http://n8n:5679`. That single missing line kept n8n down for +two months. + +**Restart the full stack after a podman upgrade**, before rebooting, so failures are +attributable. Re-verify `keep-id` behaviour after a `podman-compose` version bump — it is +the mechanism postgres and n8n depend on: + +```bash +podman run --rm --userns keep-id:uid=999,gid=999 -v /tmp/t:/t:rw \ + docker.io/library/postgres:17 touch /t/probe # host file must be owned by uid 1000 +``` + +--- + +## 11. Hardware access — what rootless genuinely cannot do + +Some tools want raw hardware. Rootless cannot give it to them, and this is a real boundary, +not a configuration puzzle. Verified on this host 2026-08-11: + +```bash +podman run --rm --device /dev/sda:/dev/sda alpine ls -l /dev/sda +# ls: /dev/sda: Permission denied <- gib is not in the `disk` group +``` + +`--cap-add SYS_RAWIO` / `SYS_ADMIN` **appear** to work — the container starts and the bit +shows in `CapEff` — but the capability is granted only inside the user namespace. It confers +nothing over host devices. Do not read a successful start as proof of access; test the actual +operation. + +**Do not "fix" this** by adding `gib` to `disk`, running the container as root, or +reintroducing root podman. If a workload truly needs raw devices, run its collector on a host +that has them and keep only the UI here. + +### The split-deployment pattern (use this for monitoring tools) + +Most monitoring tools separate a **hub/web** component (no privileges) from an **agent/ +collector** (needs hardware or host access). Put the hub on the VPS; run agents wherever the +thing being monitored lives. + +| Service | On this VPS | Elsewhere | +| ------------ | ------------------------------------------------ | ------------------------------------- | +| **Beszel** | `beszel` hub + `beszel-agent` (monitors the VPS) | `beszel-agent` on each other host | +| **Scrutiny** | ❌ nothing — see below | full omnibus stays on the home server | + +Beszel's hub lives here (`beszel.gibbyb.com`) and agents report in. The home server's agent +reaches this hub over WireGuard; add it in the UI as host `192.168.1.2` port `45876`. The +VPS's own agent is host `host.docker.internal` port `45876`. Both agents carry the hub's +public key, derived with `ssh-keygen -y -f Beszel/volumes/hub/id_ed25519`. **If the hub is +ever rebuilt its keypair changes and every agent's `KEY` must be updated.** + +### ❌ Do not host Scrutiny here (decided 2026-08-11, twice over) + +It was built here, evaluated, and deliberately removed. Two independent reasons: + +1. **Scrutiny has no authentication at all.** Verified: `/web/dashboard` → 200 and + `/api/summary` → 200 with no credentials. Publishing it would expose disk models, serial + numbers and health to anyone who found the hostname. Beszel has real accounts; Scrutiny + does not. Do not put it behind Cloudflare without Cloudflare Access or NPM auth in front. +2. **There is no SMART data on this host.** `/dev/sda` (QEMU HARDDISK) and `/dev/sdb` + (Hetzner Volume) both report `SMART support is: Unavailable`, and rootless cannot reach + raw devices anyway (§11 above). + +The collector must run where the disks are, so the web half gains nothing by moving. It runs +as the full omnibus on the home server, LAN/VPN-only. Revisit only if a second disk-bearing +host appears and one pane of glass becomes worth adding auth for. + +### Authentik — migrated copy, NOT yet authoritative + +`~/Server/Authentik` is a 1:1 copy of the home server's `~/Server/auth`, built 2026-08-11. +**The home instance is still authoritative and still serving `auth.gbrown.org`.** This copy +has its own database restored from a point-in-time dump, so the two diverge from that moment +— it is a cutover target, not a hot standby. Changes made in one do not appear in the other. + +- DB: role + database `authentik` in the shared VPS `postgresql`, same `PG_PASS` as home, + so `.env` needed no edits. Restored 212 tables; verified 11 users / 37 apps / 15 flows / + 37 providers / 11 groups on both sides. +- `AUTHENTIK_SECRET_KEY` in `.env` (mode 600) **must** match the key the database was + encrypted with. Never regenerate it on a migrated database. +- No Redis — authentik 2026.x is Postgres-backed. Home has none either. +- Version pinned via `AUTHENTIK_TAG` in `.env` (2026.2.2). Keep both sides on the same tag + through a cutover; authentik runs DB migrations on start and they are not reversible. + +**Cutover, when you want it:** create an explicit `auth.gbrown.org` record pointing at the +VPS (an explicit record overrides the `*.gbrown.org` wildcard), add an NPM proxy host to +`http://authentik-server:9000`, and re-dump/restore the database immediately beforehand so +nothing is lost in the gap. Roll back by deleting the DNS record. + +### Adapting a Docker/root compose file to rootless + +Checklist when porting from the home server (which runs Docker as root): + +| Docker/root | Rootless podman here | +| -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `/var/run/docker.sock` | `/run/user/1000/podman/podman.sock` (rootless containers only), **no `:Z`**, plus `security_opt: ['label:disable']` | +| `privileged: true`, `devices:`, `cap_add: SYS_RAWIO/SYS_ADMIN` | Not available. Split the workload — see above. | +| `network_mode: host` | Works. Ports ≥80 bind fine; the agent is reachable from bridge containers via `extra_hosts: host.docker.internal:host-gateway` (resolves to `169.254.1.2`). | +| bind mounts | add `:Z` (§6) | +| `/etc/localtime:/etc/localtime:ro` | drop it; set `TZ=America/New_York` instead (§8) | +| host-specific env (e.g. `EXTRA_FILESYSTEMS=/boot`) | re-check against _this_ host — the VPS has no separate `/boot` partition | + +Verify a ported service is reachable the way NPM will reach it, before wiring a proxy host: + +```bash +podman run --rm --network nginx-bridge alpine \ + sh -c 'apk add -q curl && curl -s -o /dev/null -w "%{http_code}\n" http://:/' +``` + +--- + +## 12. Quick reference + +```bash +# state +podman ps +systemctl --user list-units 'podman-*' +~/Server/verify-rootless.sh + +# lifecycle (NEVER sudo) +systemctl --user restart podman-.service +systemctl --user stop podman-.service +podman logs + +# after editing a compose file +systemctl --user restart podman-.service + +# after editing a unit file +systemctl --user daemon-reload && systemctl --user restart podman-.service + +# SELinux +getenforce +podman inspect --format '{{.ProcessLabel}}' +sudo ausearch -m avc -ts recent # denials + +# firewall +sudo firewall-cmd --zone=public --list-all +sudo firewall-cmd --zone=trusted --list-all + +# nginx / NPM +podman exec nginx-proxy-manager nginx -t +podman exec nginx-proxy-manager nginx -s reload +``` + +**Temporary root for an agent** (revoke when done): + +```bash +echo 'gib ALL=(ALL) NOPASSWD: ALL' | sudo tee /etc/sudoers.d/99-claude-temp >/dev/null +sudo chmod 0440 /etc/sudoers.d/99-claude-temp +# revoke: +sudo rm /etc/sudoers.d/99-claude-temp +``` + +--- + +## §13. Findings from the 2026-08-12 post-migration audit + +### 13.1 Upstream Hetzner Cloud Firewall (not visible from this machine) + +There is a **cloud firewall in front of the VM** that firewalld cannot see and that +`firewall-cmd` will happily contradict. Verified 2026-08-12: + +| port | firewalld | nftables | service healthy | reachable from internet | +| ---------------------------- | --------- | -------- | --------------- | ----------------------- | +| 22 | allow | allow | yes | **yes** | +| 2222 (gitea SSH) | allow | allow | yes | **no** | +| 21115-21119 (rustdesk relay) | allow | allow | yes | **no** | + +Confirmed from an unrelated external IP and independently by portchecker.io. + +**Consequence:** `git clone ssh://git@git.gbrown.org:2222/...` works over WireGuard but +fails from any outside network, and RustDesk cannot relay for clients that are not on +the LAN or tunnel. + +⚠️ **When a port is open in firewalld but still unreachable, check the Hetzner console +before debugging anything on this box.** Two hours were spent proving the local stack +was correct. See `archive/BROWSER_AUDIT_PROMPT_3.md` §A1. + +**RESOLVED 2026-08-12** — the rules below were added and verified open from an external +IP and by portchecker.io. `ssh -p 2222 git@` now completes a handshake and +returns `Permission denied (publickey)`, which is the correct key-only response. + +⚠️ The console names this server **`Junior`** (id 110193938), while the OS hostname is +`LittleMan`. Same machine — do not go looking for a second server. + +**The rules** (Hetzner Cloud Console → Firewalls → `Default_Firewall`, id 10059209). +All **INBOUND**; source `0.0.0.0/0` + `::/0`: + +| protocol | ports | purpose | +| -------- | ----------- | ---------------------------------------------------------------- | +| TCP | 21115-21117 | RustDesk: NAT test (21115), ID/hole-punch (21116), relay (21117) | +| **UDP** | 21116 | RustDesk ID registration — without it clients never come online | +| TCP | 2222 | Gitea SSH (verified key-only: password auth is refused) | + +Verified listeners: `hbbs` on 21115/21116/21118 tcp + 21116 udp, `hbbr` on 21117/21119, +`apimain` on 21114. + +**Deliberately NOT opened:** + +- `21114/tcp` — the RustDesk web API. Already served via Cloudflare on 443 + (`rustdesk.gbrown.org`). Opening it publishes an admin API that currently sits behind + Cloudflare. **This is why the rule is 21115-21117 and not 21114-21119.** +- `21118-21119/tcp` — websocket variants, needed only by the browser client. + +⚠️ **Add no OUTBOUND rules.** Hetzner's firewall is stateful and permits all egress while +zero outbound rules exist; adding even one flips outbound to default-deny and every +destination must then be enumerated. + +⚠️ **Never add a broad inbound range.** Ports 81, 3001 and 9443 are bound to 0.0.0.0 on +this host (§13.2) and are safe only because this firewall permits nothing but 22/80/443 +plus the rows above. + +### 13.2 Publish admin ports to the WireGuard address, not 0.0.0.0 + +`ports: ['5432:5432']` bound all 14 databases to the public interface, with only +firewalld plus the Hetzner firewall in between. Fixed to `192.168.2.2:5432:5432`. + +**RESOLVED 2026-08-12.** All now bound to the WireGuard address: + +| container | port | binding | +| ------------------- | ----------- | ------------------ | +| postgresql | 5432 | `192.168.2.2:5432` | +| portainer | 9443 | `192.168.2.2:9443` | +| nginx-proxy-manager | **81 only** | `192.168.2.2:81` | +| uptime | 3001 | `192.168.2.2:3001` | + +⚠️ **NPM's 80 and 443 deliberately stay on `0.0.0.0`** — they are the public front door, +and firewalld already restricts them to Cloudflare ranges. Only the admin port moved. + +Verified after: public sites unaffected (git/vault/port 200), all admin UIs reachable over +WireGuard (npm:81 200, uptime:3001 302, portainer:9443 200). + +**Keep this in proportion.** None of these are reachable from the internet: the Hetzner +cloud firewall (§13.1) permits only 22/80/443, and firewalld does not open them either. +All three also have their own login. So this is a defense-in-depth observation, not an +exposure — the binding is wider than it needs to be, behind two firewalls and an +application login. + +It matters in exactly one scenario: **if the Hetzner firewall rules are ever broadened**, +these ports become internet-facing immediately and silently, with no further change on +this box. That is the reason to prefer `192.168.2.2:` bindings — it removes the dependency +on a rule set that lives in a web console someone else may edit. + +⚠️ **When adding Hetzner firewall rules, open only the specific ports intended** +(2222 for gitea SSH, 21115-21119 for the rustdesk relay). Do not open a broad range — +that would expose all three of the above. + +Postgres was the one genuine finding here and is fixed. The remaining three are a +one-line change each (`- "192.168.2.2:9443:9443"`), deliberately left alone: changing +NPM's 81 alters how the proxy itself is administered, and a mistake there locks you out +of the tool you would use to repair it. + +**Rule for new services: never publish an admin port as `NNNN:NNNN`.** Either bind it to +`192.168.2.2` or do not publish it at all and let NPM reach it over nginx-bridge. + +### 13.3 `systemctl --user enable` is a separate step from starting + +Five stacks (convexmonorepo, lashaddict, spoon, stpeteit, usesend) were **running but +`disabled`** — they would have vanished on the next reboot, silently, because everything +looked healthy in `podman ps`. `podman-compose up` starting the containers does not +enable the unit. + +**Always verify after adding a service:** + + systemctl --user is-enabled podman-.service # must print "enabled" + +`verify-rootless.sh` checks this now. Note `podman-auto-update`, `podman-restart`, +`podman-clean-transient` and `podman-kube@` are podman's own units and are correctly +disabled — do not enable them. + +### 13.4 Postgres image must stay on a -trixie base + +See the comment block in PostgreSQL/compose.yml. Switching to a bookworm-based image +changes glibc from 2.41 to 2.36, which changes text collation and can make existing +indexes return **silently wrong results**. Any base change requires +`REINDEX DATABASE` + `ALTER DATABASE ... REFRESH COLLATION VERSION` on all 14 databases. + +### 13.5 The backup exits non-zero for benign reasons + +Gitea runs without `userns_mode`, so `sessions/`, `issues.bleve` and `Gitea/ssh` are +owned by subuid 525288 and unreadable to gib. rsync therefore exited 23 on every run. +Now excluded — none are needed to restore. **The gitea SSH host keys are NOT among them** +(they are gib-owned in `Gitea/volume/ssh` and are backed up); losing those would give +every client a host-key mismatch. + +### 13.6 Cloudflare Load Balancers — DELETED 2026-08-13 (were: two unproxied LBs shadowing the tunnel) + +> **RESOLVED 2026-08-13: both load balancers were deleted.** The proxied `*.gbrown.org` +> and `gbrown.org` CNAMEs into the Cloudflare tunnel now carry all gbrown.org traffic, +> which was the "delete" option from §4.3 of `archive/SESSION_2026-08-12.md`. The pools (home-pool, +> vps-pool) still exist at the account level and are re-attachable if LBs are ever +> recreated. Full incident report: `archive/SESSION_2026-08-13.md`. +> +> What forced the decision: the LBs had been flipped to **proxied**, and home-pool's +> endpoint carried a **`Host: gbrown.org` header override**. A proxied LB applies that +> override to the SNI and Host of _every_ request — first causing 525s on the whole +> namespace (no `gbrown.org` server block existed on home NPM), then, once an apex host +> was added, routing every subdomain to the apex site. Unproxied LBs never applied it, +> which is why the override sat harmless for months. See §13.13 for the apex host that +> is now load-bearing. + +Historical analysis from 2026-08-12, kept because the shadowing/diagnostic lessons stand: + +> **Correction:** an earlier revision of this section blamed a "DNS-only wildcard record". +> That was wrong. There is no such record, and no `portainer.gbrown.org` record either — +> a browser audit enumerated all 53 gbrown.org records and found neither. The cause is +> **two Cloudflare Load Balancers**, which shadow DNS records of the same name. + +Measured 2026-08-12 against three public resolvers and Cloudflare's own authoritative +nameservers (`junade.ns.cloudflare.com`, `megan.ns.cloudflare.com`): + + portainer.gbrown.org. 30 IN A + definitelynotarealhost.gbrown.org. -> A + +A randomly invented name resolves to the same address. The mechanism: + +| LB (zone gbrown.org) | enabled | **proxied** | TTL | pools | fallback | +| -------------------- | ------- | ----------- | --- | ------------------- | -------- | +| `*.gbrown.org` | true | **false** | 30 | home-pool, vps-pool | vps-pool | +| `gbrown.org` | true | **false** | 30 | home-pool, vps-pool | vps-pool | + +| pool | origin | monitor | +| --------- | -------------- | ----------------- | +| home-pool | | **none attached** | +| vps-pool | | **none attached** | + +**Three consequences:** + +1. A load balancer takes precedence over a same-name DNS record, so the _proxied_ + `*.gbrown.org` and `gbrown.org` CNAMEs to `cfargotunnel.com` are **shadowed and never + take effect** — the tunnel carries no gbrown.org traffic today. +2. Because the LBs are `proxied: false`, they answer with the pool origin address + directly, publishing the home IP for the whole namespace including the apex and names + that do not exist. Cloudflare's own DNS page flags _"Your origin IP address is + partially exposed."_ +3. With **no monitor attached to either pool**, both are permanently "healthy", steering + always lands on the first default pool (home-pool), and **failover to the VPS can + never occur.** This is paid-for load balancing that structurally cannot fail over. + +Explicit hostnames (`port`, `git`, `auth`, `vault`, …) have their own proxied A records +at and win over the wildcard LB, so they are unaffected. + +⚠️ **Diagnostic rule:** before concluding a subdomain "has a stale record", query a name +that cannot exist (`dig +short thisdoesnotexist.gbrown.org`). If it resolves, you are +seeing catch-all behaviour — a wildcard _or an LB_ — not a per-host record. + +**Before changing the LBs, read §13.8** — the client-IP consequence is the part that +actually bites. + +### 13.7 Gitea container registry retention + +One cleanup rule (Site Administration -> Packages -> Cleanup Rules): container +packages keep the newest 10 versions, drop versions older than 7 days, and always +keep tags matching `latest|main|prod`. Gitea applies it with its built-in daily +package cleanup cron. On 2026-10-01 the registry held 136 versions across 10 +packages, 9.8 GB of blobs (down from 19 GB before the rule). + +Checking without a browser session: + + sqlite3 -readonly ~/Server/Gitea/volume/gitea/gitea.db \ + "SELECT count(*) FROM package_version; SELECT * FROM package_cleanup_rule;" + +### 13.8 Home's NPM real_ip — FIXED 2026-08-12 (was: no config at all) + +Verified 2026-08-12 on the home server: there are **no custom nginx snippets** under its +NPM data directory, i.e. no `set_real_ip_from` / `real_ip_header`. The VPS _does_ have +this (`Nginx_Proxy_Manager/volumes/data/nginx/custom/server_proxy.conf`, §7); home does +not. + +**Why this matters.** The home cloudflared tunnel is active and enabled, and its ingress +already covers the whole namespace: + + tunnel: e4f0d4d8-5e57-4bee-ab7b-56ef1ed0b7da (/etc/cloudflared/config.yml) + ingress: + - hostname: gbrown.org -> https://localhost:443 + - hostname: "*.gbrown.org" -> https://localhost:443 + - hostname: gibbyb.com / "*.gibbyb.com" -> https://localhost:443 + - hostname: madelinemcguigan.com / "*." -> https://localhost:443 + - hostname: completeuphoria.club -> https://localhost:443 + - service: http_status:404 (catch-all) + +So **deleting the LBs would NOT dark the home namespace** — the tunnel catch-all already +routes it. That risk, raised in the browser audit, is disproved. + +**The real consequence is the client IP.** Today the LBs hand out home's raw address, so +home's NPM sees the _actual client IP_ and IP-based access lists work. Change that and: + +| route | what home's NPM sees as the client | +| ------------------------------------- | ------------------------------------------------- | +| today: unproxied LB -> home IP direct | the real client IP | +| delete LBs -> tunnel | **127.0.0.1** (cloudflared dials `localhost:443`) | +| proxy the LBs | **a Cloudflare edge IP** | + +Both alternatives break every IP-based access list on the home NPM, because without +`real_ip` nginx has no way to recover the true client address. + +**This is also why `allow 127.0.0.1` once existed in the "private" access list, and why +removing it was correct.** With it present, _anyone on the internet_ arriving through the +tunnel would have matched a rule meant to mean "LAN only". Removing it closed that hole; +it also means tunnel-routed traffic now fails closed. + +⚠️ **Prerequisite for either LB change: configure real_ip on the home NPM first**, the +same way the VPS does. For tunnel traffic that means trusting the loopback: + + set_real_ip_from 127.0.0.1; + real_ip_header CF-Connecting-IP; + +For a proxied LB it means trusting Cloudflare's published ranges instead. Do this +_before_ touching the load balancers, not after — otherwise the KVMs and anything else +behind an access list go dark the moment DNS changes. + +**Safe to do independently, right now:** attach the existing `https /health` monitor to +both pools. It only adds health checking and moves no traffic while home-pool is up. + +#### 13.8b The fix, as applied (home server, 2026-08-12) + +⚠️ Home runs **Docker**, not podman. `docker exec nginx-proxy-manager ...`, not +`sudo podman`. NPM data volume: `/home/gib/Server/nginx/volumes/data` → `/data`. + +Two files, because the directive must be split across contexts: + +`/data/nginx/custom/http_top.conf` (http context — `set_real_ip_from` is additive): + + set_real_ip_from 127.0.0.1; + set_real_ip_from ::1; + +`/data/nginx/custom/server_proxy.conf` (server context — included by all 31 proxy hosts): + + real_ip_header CF-Connecting-IP; + +⚠️ **`real_ip_header` must NOT go in `http_top.conf`.** NPM's `nginx.conf:81` already +declares `real_ip_header X-Real-IP;` in the http context, and `http_top.conf` is included +into that same context — a second declaration is a duplicate directive and **nginx will +refuse to start**. The server block legally overrides it. The VPS is arranged the same +way for the same reason. + +**Verified after reload** (against `scrutiny` and `ncmaster`, both access-listed and +enabled): + +| request from loopback | result | +| -------------------------------- | -------------------------------------------- | +| no header | **403** — loopback is not in the allow list | +| `CF-Connecting-IP: 192.168.1.50` | **302** — LAN range allowed, real_ip working | +| `CF-Connecting-IP: 8.8.8.8` | **403** — public IP still denied | + +Access logs now record the forwarded client instead of `127.0.0.1`. All home sites +healthy afterwards (jellyfin/sonarr/radarr 302, immich 200, obsidian 401), NPM 0 restarts. + +Useful facts discovered while doing this: + +- NPM already trusts Cloudflare's ranges (`104.16.0.0/13`, `172.64.0.0/13`, + `103.21.244.0/22`) via `conf.d/include/ip_ranges.conf` — 265 entries. Only loopback was + missing, and the header was wrong. So **proxying the LBs would also now resolve + correctly**, not just the tunnel path. +- Home's "Private" access list is `192.168.1.0/24`, `192.168.2.0/24`, `192.168.112.1/24` + — confirmed **no `127.0.0.1`**. +- `nano`/`macnano` proxy hosts exist in home's DB but are **disabled** (no generated + conf); they are served by the VPS NPM (hosts 42/43) forwarding to `192.168.1.10/.11` + across WireGuard. + +### 13.9 Two ways hosts are protected — do NOT "fix" one into the other + +**Correction.** An earlier revision of this section claimed `prowlarr.gbrown.org` and +`comfyui.gbrown.org` were "not access-controlled despite the UI saying they are". **That +was wrong, and acting on it would have broken working SSO.** Both are protected — by +**authentik forward auth**, not by an IP access list. + +Evidence: + + $ curl -sk -D - https://prowlarr.gbrown.org + HTTP/2 302 + location: https://prowlarr.gbrown.org/outpost.goauthentik.io/start?rd=/ + + id=29 prowlarr -> 29 auth_request/authentik lines, 0 allow/deny + id=48 comfyui -> 21 auth_request/authentik lines, 0 allow/deny + id=116 scrutiny -> 0 authentik lines, 4 allow/deny + id=47 ncmaster -> 0 authentik lines, 4 allow/deny + +The generated config carries: + + auth_request /outpost.goauthentik.io/auth/nginx; + error_page 401 = @goauthentik_proxy_signin; + +**This environment deliberately uses two different protection mechanisms, per host:** + +| mechanism | how it looks | example hosts | +| -------------------------------- | ------------------------------------------------------------------------------------------ | --------------------------------- | +| **authentik forward auth (SSO)** | `auth_request` lines; unauthenticated request → **302** to `/outpost.goauthentik.io/start` | prowlarr, comfyui | +| **IP access list** | `allow`/`deny` lines; off-LAN request → **403** | scrutiny, ncmaster, nano, macnano | + +⚠️ **A 302 to `outpost.goauthentik.io` is PROTECTION, not exposure.** Do not read it as +"the service was served". Follow the `location` header before concluding anything. + +⚠️ **Public reachability here is intentional.** Services are meant to be usable from +outside the LAN; SSO is what secures them, including services that have no login of their +own. Do **not** attach IP access lists to SSO-protected hosts, and do not treat a +publicly-resolving hostname as a defect. An `access_list_id` set in the NPM database with +no `allow`/`deny` in the generated config is **not** evidence of a hole on an SSO host. + +Correct way to check what protects a host: + + id= + docker exec nginx-proxy-manager grep -cE '^[[:space:]]*(allow|deny) ' /data/nginx/proxy_host/$id.conf + docker exec nginx-proxy-manager grep -c 'auth_request' /data/nginx/proxy_host/$id.conf + +Zero for **both** would be a genuine finding. Non-zero for either is a protected host. + +### 13.10 The LB health monitor cannot validate either origin — it sends no SNI + +> **Moot for gbrown.org since 2026-08-13** — the LBs were deleted (§13.6) and the tunnel +> needs no health monitor. Kept because the no-SNI failure mode applies to any future +> monitor, and the pools still exist at the account level. + +**Do not attach a monitor to a pool expecting it to be a safe, traffic-neutral change.** +An earlier revision of this guide claimed attaching one "moves no traffic while home-pool +is up". **That is wrong** and was disproved in practice on 2026-08-12. + +What happened: monitor `Gib's Monitor` (HTTPS GET `/health`, port 443, expect 200, +`Host: health.gbrown.org`) was attached to both pools. After one check cycle **both** went +unhealthy: + + home-pool healthy: false "TLS unrecognized name error" + vps-pool healthy: false "TLS unrecognized name error" + +With every pool unhealthy the LB fell through to its **fallback pool (vps-pool)** and DNS +moved immediately — the apex, `sonarr`, and every unmatched name flipped to + on a 30s TTL. It was rolled back by detaching the monitor, and DNS +returned to . + +**Root cause, proved from the shell — the monitor is not sending SNI:** + + # WITH SNI -- both origins are fine + $ openssl s_client -connect 127.0.0.1:443 -servername health.gbrown.org + VPS: subject=CN=*.completeuphoria.club Verify return code: 0 (ok) + home: subject=CN=*.gbrown.org + + # WITHOUT SNI -- both origins reject, on both machines + $ openssl s_client -connect 127.0.0.1:443 + tlsv1 unrecognized name ... SSL alert number 112 + no peer certificate available + +So it is **not** "the `/health` path is missing" and **not** a certificate problem — +`health.gbrown.org` returns 200 and both certs verify. nginx has no default server with a +cert for a no-SNI handshake, so it aborts before HTTP is ever spoken. (Alert 112 again — +the same alert family as the ECH incident in §7.) + +**Options, in order of preference:** + +1. **Use an HTTP monitor on port 80** — no TLS, so no SNI to get wrong. Set expected codes + to include `301`, since NPM's Force-SSL will redirect. +2. Give both origins a default server with a valid certificate so no-SNI handshakes + succeed. +3. `allow_insecure` will **not** help — the handshake fails before certificate validation. + +⚠️ **Whatever you try, attach it to `vps-pool` ALONE first** and confirm it reports +healthy before home-pool is ever added. A monitor that fails on every pool does not +disable failover — it triggers a silent flip to the fallback pool. + +⚠️ **Corollary:** the earlier note that a "flapping monitor" was observed spontaneously +was wrong too. That DNS movement was this attach/detach, not an independent fault. + +### 13.11 Home's NPM database — collation fixed 2026-08-12 + +Home's `nginx-proxy-manager-db` ran `postgres:17` (glibc 2.41) on a cluster created under +glibc 2.36 — the mirror image of §13.4, and the same silent-wrong-results risk, on the +database that stores every proxy host and access list. + +Fixed properly rather than by silencing the warning: + + docker exec nginx-proxy-manager-db pg_dump -U npm -d npm > ~/npm-db-backup-.sql + docker exec nginx-proxy-manager-db psql -U npm -d npm -c "REINDEX DATABASE npm;" + docker exec nginx-proxy-manager-db psql -U npm -d npm -c "ALTER DATABASE npm REFRESH COLLATION VERSION;" + # repeated for the `postgres` database + +`NOTICE: changing version from 2.36 to 2.41`. Data intact (117 proxy_host rows), sites +serving afterwards. A 1.3 MB dump was taken first and is on the home server. + +⚠️ **Home runs Docker.** `docker exec`, not `podman`. + +### 13.12 Passwordless sudo was removed 2026-08-12 — both servers + +`/etc/sudoers.d/99-claude-temp` (`gib ALL=(ALL) NOPASSWD: ALL`) is **gone from the VPS and +the home server**. `gib` retains normal sudo via `%wheel ALL=(ALL) ALL`, now +password-prompted. + +**This does not affect running anything here.** Every container is rootless and every unit +is a `systemctl --user` unit, so day-to-day operation needs no sudo at all. Only host-level +work does — firewalld, SELinux relabeling, reading container-owned files under +`Gitea/volume`, and `docker` on the home server. + +⚠️ **Non-interactive sudo will now fail**, including over `ssh` without a TTY. An agent +running `sudo ...` in a pipeline gets `sudo: a password is required` and the surrounding +command may still report success — check for that string rather than trusting an exit code. +If host-level access is genuinely needed, ask the user rather than re-creating the drop-in. + +### 13.13 The `gbrown.org` apex proxy host on home NPM is LOAD-BEARING for the tunnel + +Home's cloudflared (`/etc/cloudflared/config.yml`) dials NPM with +`originServerName: gbrown.org` — that is the SNI for **every** request through the +tunnel, regardless of which site the visitor asked for (NPM then routes by the Host +header, which the tunnel preserves). Home NPM has no default TLS server, so a handshake +with an unrecognized SNI is aborted (alert 112 — same failure family as §13.10). + +Consequence: **the `gbrown.org` proxy host in home NPM (currently → Home Assistant, +added 2026-08-13) must exist for any tunnel-served site to work.** Deleting or renaming +it 525s the entire gbrown.org namespace, plus gibbyb.com / madelinemcguigan.com / +completeuphoria.club traffic riding the same tunnel. If the apex host must ever go away, +first change `originServerName` to another hostname NPM serves (e.g. +`nginx.gbrown.org`) and `sudo systemctl restart cloudflared`. + +### 13.14 VPS Actions runner — `cirunner` user (added 2026-08-13) + +CI redundancy: the three dockerized runners + one host runner all live on the **home +server**, so a home outage used to stop CI (and with it, deploys of VPS apps). A fourth +runner now runs on the VPS under a **dedicated `cirunner` user** (uid 1001, subuids +`589824:65536`, linger enabled) — NOT under gib, so a CI escape lands in an empty +account with no access to the production containers. + +Layout (all under `/home/cirunner`, systemd user unit `podman-runner.service`): + +| piece | value | +| --------------------------- | ------------------------------------------------------------------------------------------------------------ | +| runner name / labels | `cirunner-vps` — `ubuntu-latest`, `ubuntu-24.04`, `ubuntu-22.04`, **`vps`** | +| compose + act_runner config | `~cirunner/runner/{compose.yml,config.yaml}` | +| container storage | `/home/gib/Media/cirunner/storage` (196 GB volume; traverse-only ACL `u:cirunner:x` on `/home/gib{,/Media}`) | +| job execution | sibling containers via cirunner's podman API socket | + +Because it shares the `ubuntu-latest` label with the home runners, whichever runner +polls first takes the job — home down ⇒ the VPS keeps CI moving. `runs-on: vps` +forces a job here. + +**Two traps, both hit and fixed during setup — do not regress them:** + +1. **Socket path must be identical on host and in the runner container** + (`/run/user/1001/podman/podman.sock`, with `DOCKER_HOST` pointing at it). act_runner + passes its socket path to the daemon when mounting it into job containers; a + container-only path like `/var/run/docker.sock` fails with + `mkdir /var/run/docker.sock: permission denied`. +2. **SELinux blocks confined job containers from the socket** even when mounted — + `config.yaml` sets `container.options: "--security-opt label=disable"` (the §13.2/ + watchtower trade, contained by the cirunner userns). Symptom if lost: + `ls: /var/run/docker.sock: Permission denied` inside jobs. + +Smoke-test repo: `gib/ci-smoke` (private) — a `runs-on: vps` workflow that hard-checks +the docker socket inside a job and the watchtower hook at `192.168.2.2:8080`. Push any +commit to re-verify the runner end to end. Registration used the reusable instance +token (`gitea actions generate-runner-token`). + +### 13.15 Watchtower RETIRED 2026-08-17 — it took gitea down for three days + +**Do not reinstate watchtower on this host.** Updates are now handled by +`~/Server/update-containers.sh`, run nightly at 00:00 by `podman-update.timer`. + +#### What happened + +| time (EDT) | event | +| --------------- | ---------------------------------------------------------------------------------------------------------------- | +| Aug 14 04:30:03 | watchtower found a new `gitea:latest`, stopped the container, started a replacement, logged `failed=0 updated=1` | +| Aug 14 04:30:22 | Uptime Kuma monitor 17: **502**, gitea DOWN | +| Aug 17 12:39 | manual `systemctl --user restart podman-gitea.service` — UP | + +Gitea logged nothing at all between those points. Watchtower reported success, so +nothing escalated. + +#### Root cause + +podman-compose puts every project in a **pod** — 39 of 41 containers here. Watchtower +does not update through compose; it builds a replacement container from the image plus +the old container's config, and **the replacement lands outside the pod**, losing +whatever the pod provided. + +Proof: after the Aug 13 run, `portainer` was sitting podless, as was `agentchat`. Both +were restored by restarting their units. Check for this with: + + for c in $(podman ps --format '{{.Names}}'); do + [ -z "$(podman inspect $c --format '{{.Pod}}')" ] && echo "PODLESS: $c" + done + +Honest limit: `portainer` ran podless without visible breakage, so leaving the pod is +not automatically fatal, and the exact reason gitea's replacement returned 502 could not +be reconstructed — that container and its logs were removed by watchtower itself. What is +certain is the trigger, the timing, and that the replacement was not built by compose. + +#### The replacement + +`update-containers.sh` never touches containers. It pulls the image, then restarts the +**systemd unit**, so podman-compose rebuilds the project exactly as it does at boot. + + ./update-containers.sh # all services except SKIP (what the timer runs) + ./update-containers.sh gitea # one service — use this from CI + ./update-containers.sh --dry-run # report only + ./update-containers.sh --list # services and their resolved images + +Config lives in the two arrays at the top. `SKIP=(postgresql authentik watchtower)`. +Log: `~/Server/logs/update-containers.log`. + +Three non-obvious things it has to handle — do not "simplify" these away: + +1. **Compare the running container to the image on disk, not before-vs-pull.** A + `--dry-run` pulls the image, so a later real run would see before == after, decide + nothing changed, and never restart — leaving the service on the old image with the + new one on disk, permanently invisible. Anchoring on the container makes it + idempotent and self-healing. +2. **`podman inspect .Image`, never `podman ps --format {{.ImageID}}`.** The first is + the full 64-char ID matching `image inspect .Id`; the second is a 12-char short ID. + Comparing them is always unequal, and every service looks like it needs updating. +3. **Never restart an inactive unit.** `systemctl restart` _starts_ a stopped unit, so + without a guard a new n8n image would resurrect n8n months after it was deliberately + shut down. The script skips any unit that is not already `active`. + +Also: image tags like `${BACKEND_TAG:-latest}` are expanded from each service's `.env` +in a **subshell**, so those files (database passwords, API keys) never leak into the +script's environment. + +#### ⚠️ Push-to-deploy needs rewiring + +CI used to POST to watchtower's API at `192.168.2.2:8080/v1/update`. **That endpoint no +longer exists.** The equivalent is: + + ssh gib@192.168.2.2 '~/Server/update-containers.sh ' + +which needs an SSH key for the runner. Until that is done, pushing an image updates the +registry but does not deploy it — the nightly timer will pick it up at midnight. + +### 13.16 Instant deploys — SSH forced command (replaces watchtower's HTTP API) + +Watchtower's `192.168.2.2:8080/v1/update` is gone (§13.15). CI now triggers a deploy by +SSHing to `gib@192.168.2.2` with a key that can do exactly one thing. + +**Why SSH and not a small HTTP daemon:** the alternative was writing a network service +that executes `systemctl` based on request bodies. sshd with a forced command achieves +the same thing using software that already exists, is already exposed, and has decades of +review behind it. No new listening port either. + +**Why a bridge is needed at all:** the VPS runner is user `cirunner` (uid 1001) and the +other runners are on the home server. systemd user managers are per-user, so neither can +restart `gib`'s units. This crosses that boundary — and only that boundary. + +Pieces: + +| file | role | +| ----------------------------- | ------------------------------------------------------- | +| `~/Server/ci-deploy.sh` | the forced command; validates and dispatches | +| `~/Server/ci/ci_deploy_key` | private key — goes in the Gitea secret `DEPLOY_SSH_KEY` | +| `~/.ssh/authorized_keys` | pins the key to the forced command | +| §15 of this file | the workflow snippet to paste | +| `~/Server/logs/ci-deploy.log` | every request, allowed or denied | + +The authorized_keys entry: + + command="/home/gib/Server/ci-deploy.sh",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,no-user-rc ssh-ed25519 AAAA... + +⚠️ **`$SSH_ORIGINAL_COMMAND` is attacker-controlled if the key leaks. Never eval it, +never word-split it, never interpolate it into a shell string.** `ci-deploy.sh` matches +it against `^[a-z0-9][a-z0-9-]{0,40}$`, confirms a matching unit is _already active_, +rejects the frozen set, and passes it as a single quoted argument. + +Verified behaviour — re-run these after any change to the script or the key: + +| request | result | +| -------------------------- | ------------------- | +| `adminer` | deploys | +| _(bare shell)_ | denied — malformed | +| `cat .../PostgreSQL/.env` | denied — malformed | +| `adminer; cat /etc/passwd` | denied — malformed | +| `adminer$(id)` | denied — malformed | +| `postgresql` | denied — frozen | +| `n8n` | denied — not active | + +The "must already be active" check is doing double duty: it keeps the allowlist +self-maintaining as services come and go, and it means the key cannot start something +that was deliberately stopped. + +**Only two workflows ever called watchtower** — `gib/agentchat` and `gib/ci-smoke`. The +app stacks (`spoon`, `lashaddict`, `convex-monorepo`, `stpeteit-payload`, +`convex-monorepo-payload`) push images and wait for the nightly timer. Add the step from +the §15 step to any of them that should deploy immediately. + +⚠️ `~/.ssh/authorized_keys` was backed up to `authorized_keys.bak-` before the +key was appended. Four pre-existing keys were preserved — check `wc -l` is 5, not 1, if +you ever edit it. + +#### 13.16b Verified end-to-end 2026-08-19 + +First real push through the new path, `gib/agentchat` run 129: + + 10:55:01 ci-deploy.log: deploy requested from 192.168.1.2: agentchat + 10:55:02 update-containers.log: agentchat on 5d796ab4f297 -> new image 343d961d96ff + 10:55:09 update-containers.log: agentchat: RESTARTED ok + 10:55:09 update-containers.log: pruned: 5d796ab4f297... + 10:55:09 ci-deploy.log: deploy finished: agentchat (exit 0) + +Container came back **inside its pod** (`271d52b578ba`) on the new image, `chat.gbrown.org` 200. Roughly 8 seconds from trigger to live. `192.168.1.2` is the home server, so the +runner crossed the WireGuard tunnel to reach the VPS. + +**Pushing from the VPS.** Only `~/.ssh/id_rsa.pub` is registered with Gitea as +"Gib's VPS (Junior)". The desktop and home keys are passphrase-protected and unusable +non-interactively, and the key registered as "Gib's Server" matches neither machine — +it predates a rebuild. To push from here: + + GIT_SSH_COMMAND="ssh -p 2222 -i $HOME/.ssh/id_rsa" git clone ssh://git@127.0.0.1:2222/gib/.git + +⚠️ `scp` to `desktopw.gib` hangs, though plain `ssh` works. Pipe instead: +`ssh desktopw.gib 'cat > /tmp/f' < localfile`. + +⚠️ Do not build shell snippets for a remote host through nested SSH quoting — an earlier +attempt mangled `printf '%s\n'` into a literal newline and silently corrupted the +workflow. Write the file locally, pipe it over, and apply it with a script on the far end. + +**Remaining repos still without a deploy trigger** (they push an image and wait for the +midnight timer): `spoon`, `lashaddict`, `convex-monorepo`, `convex-monorepo-payload`, +`stpeteit-payload`, and `ci-smoke` (which still references the dead watchtower endpoint). +Add the step from §15 below, changing the service-name argument. + +--- + +### 13.17 LashAddict containers pin DNS to Cloudflare (2026-09-30) + +lashaddict moved from `*.lashaddict.gbrown.org` to `lashaddictstudio.com` +(`api.` → backend:3210, `convex.` → backend:3211, `dashboard.` → 6791, apex → next:3000, +`www` redirect host; cert NPM #44). Hetzner's resolvers (185.12.64.1/.2) kept a +cached NXDOMAIN for the brand-new hostnames for hours, and the Convex backend fetches +its own auth discovery URL (`https://convex.lashaddictstudio.com/.well-known/...`), so +every session failed (`AuthProviderDiscoveryFailed`). All three services in +`LashAddict/compose.yml` therefore set `dns: [1.1.1.1, 1.0.0.1]`; container names still +resolve through aardvark-dns. Do not remove the pin. The same trap applies to any new +hostname a container must reach soon after it is created: `resolvectl flush-caches` +fixes the host, not the Hetzner upstream. + +## 14. Restoring ~/Server from the home-server backup + +_Folded in from RESTORE.md, 2026-09-22. Read it before you need it._ + +Written 2026-09-17. Read this _before_ you need it. + +### Layout + + gib@server.gib:/home/gib/Public/Server_Backup/Containers/lilman.gib/current/ + +**One directory, overwritten daily.** History does not live in dated folders — it lives in +ZFS. `Media/Server_Backup` is its own dataset and `zfs-auto-snapshot` keeps **14 dailies**, +taken by `cron.daily` between roughly 03:22 and 03:52. The backup timer runs at **01:00** so +every snapshot captures a finished backup, never a half-written one. + +Older copies are reachable through the hidden snapshot directory (`snapdir=hidden`, so it +will not show in `ls`, but you can still `cd` into it): + + ls ~/Public/Server_Backup/.zfs/snapshot/ + ls ~/Public/Server_Backup/.zfs/snapshot/zfs-auto-snap_daily-2026-09-17-0722/Containers/lilman.gib/current/ + +ZFS snapshots are block-level, so a 1.7 GB database that changes a few pages costs a few +pages — not a new 1.7 GB copy. That is why this is a single mirror plus snapshots rather +than dated directories or `rsync --link-dest`, both of which copy whole files. + +The `03-17-2026`, `08-12-2026` and `09-11-2026` directories are leftovers from the old +scheme. They predate the ownership fix below and are **not reliable restore sources**. + +### What a backup contains + +| Path | What it is | +| ---------------------------------------- | -------------------------------------------------------- | +| `Adminer/`, `Gitea/`, `PostgreSQL/`, ... | Raw bind-mount data for each service | +| `_backup_state/postgres/all.sql.gz` | `pg_dumpall` of the live cluster — **authoritative** | +| `_backup_state/sqlite/` | Online `.backup` of each SQLite DB — **authoritative** | +| `_backup_state/files/` | Files gib cannot read off disk, pulled via `podman exec` | + +> **Use `_backup_state/` for databases, not the raw data dirs.** The raw dirs are copied +> while containers run, so they are crash-consistent at best. The dumps use the proper +> online-backup APIs and are what will actually restore cleanly. + +### Ownership, and why `--fake-super` is everywhere + +Most services run with `userns=private`, which maps the container's user back to `gib` — +their data is plain `gib`-owned on the host and needs nothing special. **Gitea is the +exception.** It runs _without_ `userns_mode` (s6 needs container-root, see AGENTS.md §3), +so its data is owned by subuid **525288:525290**, which `gib` cannot recreate on the home +server. The backup therefore stores the real mode/uid/gid in a `user.rsync.%stat` xattr +(`rsync --fake-super`), verified working on the home server's ZFS. + +**Consequence: a plain `rsync`/`scp`/`cp` of the backup gives you correct file contents with +every file owned by whoever ran it, and Gitea will fail on its own data dir.** Every restore +command below passes `-M--fake-super` so the sending side decodes those xattrs, and runs +under `sudo` locally so the ownership can actually be applied. + +(The old dated-directory backups were taken without this, so everything in them is uid 1000. +Restoring one means chown'ing Gitea's tree to 525288:525290 by hand afterwards.) + +### Restore in place (same machine) + +Stop everything first: + + systemctl --user stop 'podman-*.service' + +Pull it back. `sudo` has no SSH key of its own, so point it at gib's: + + sudo rsync -a --numeric-ids -M--fake-super --delete \ + -e "ssh -i /home/gib/.ssh/id_rsa -o UserKnownHostsFile=/home/gib/.ssh/known_hosts" \ + gib@server.gib:/home/gib/Public/Server_Backup/Containers/lilman.gib/current/ \ + /home/gib/Server/ + +To restore from an older day, swap the source for a snapshot path: + + .../.zfs/snapshot/zfs-auto-snap_daily-2026-09-15-0751/Containers/lilman.gib/current/ + +Verify Gitea's ownership came back — this must say `525288`, not `1000`: + + ls -ln /home/gib/Server/Gitea/volume/gitea/ | head + +Then restore databases (below) and start services: + + systemctl --user start 'podman-*.service' + +### Restoring the databases + +**Postgres** — start only the postgres container, then: + + gunzip -c /home/gib/Server/_backup_state/postgres/all.sql.gz \ + | podman exec -i postgresql psql -U npm -d postgres + +`pg_dumpall` includes roles and every database (npm, authentik, usesend, ...), so this +restores the whole cluster in one shot. + +**SQLite** — each dump sits at its original relative path. Stop the owning container, copy +the dump over the live file, restart. Gitea, which is the one needing a chown: + + systemctl --user stop podman-gitea.service + cp /home/gib/Server/_backup_state/sqlite/Gitea/volume/gitea/gitea.db \ + /home/gib/Server/Gitea/volume/gitea/gitea.db + sudo chown 525288:525290 /home/gib/Server/Gitea/volume/gitea/gitea.db + systemctl --user start podman-gitea.service + +For the others (uptime-kuma, vaultwarden, beszel, portainer, ...) the copy is enough — no +chown, since they are gib-owned already. + +**bridge.json** — `_backup_state/files/BillTracker--data--mcp--bridge.json` goes back to +`BillTracker/data/mcp/bridge.json`, owned `525288`, mode 0600. It holds the MCP bridge's +registered OAuth clients; without it those clients must re-register. + +### Migrating to a new server + +1. Install podman, `loginctl enable-linger gib`, and create `gib` with the **same uid (1000)** + and the **same `/etc/subuid` range**. If the subuid base differs, Gitea's restored files + land on the wrong owner and it will not start. +2. Restore as above, to the new box. +3. Restore the databases from `_backup_state/`. +4. Re-create the systemd units from each service's `podman-*.service` file — they are in the + backup, next to each `compose.yml`. +5. Re-point DNS, then re-issue certificates through Nginx Proxy Manager. + +> **If the new server is ARM (Hetzner CAX):** the data restores fine, it is +> architecture-independent, but every image must be rebuilt or re-pulled for `arm64`. +> Checked 2026-09-17: all third-party images used here publish arm64 except `minio`, which +> needs verifying. The `git.gbrown.org/gib/*` images rebuild natively via CI. + +### What is deliberately NOT backed up + +- `sessions/`, `issues.bleve` — ephemeral Gitea logins and a rebuildable search index. +- `dump.rdb` — redis/valkey cache dumps, regenerated on boot. +- `Gitea/ssh` — a 25-byte shell env file. **The Gitea SSH _host keys_ ARE backed up** + (`Gitea/volume/ssh`); losing those would give every client a host-key mismatch. +- `node_modules`, build output, `*.log`, media files. +- The podman image store and Gitea package registry — they live on the Hetzner volume at + `~/Media`, not `~/Server`. All rebuildable or re-pullable. + +### Checking it is actually working + + systemctl --user list-timers server-backup.timer + tail -40 ~/Server/logs/backup.log + ssh gib@server.gib 'ls -la ~/Public/Server_Backup/Containers/lilman.gib/current/ | head' + +A healthy run ends with `=== backup complete ===`. Things to watch for in the log: + +- `ZFS history OK` — the run confirmed snapshots are still being taken. If this ever says + **WARNING**, versioning is broken and `current/` is just a mirror: a mistake made here + would propagate with nothing to roll back to. Fix `zfs-auto-snapshot` before anything else. +- `rsync exit 24` is logged as a NOTE, not a failure. It only means a file changed + mid-transfer, which is normal on a live system. + +Optional: put an uptime-kuma push URL in `~/Server/.backup-heartbeat-url` and successful +runs will ping it, so a silent failure shows up as a down monitor. + +--- + +## 15. Instant deploys — the CI step + +_Folded in from ci/DEPLOY_STEP.md, 2026-09-22._ + +Watchtower was removed 2026-08-17 (AGENTS.md §13.15), so +`http://192.168.2.2:8080/v1/update` no longer answers. Deploys still happen, just at +midnight via `podman-update.timer`. To get them instant again, do the two steps below. + +--- + +### Step 1 — add the deploy key as a Gitea secret (once) + +The private key is on the VPS at: + + /home/gib/Server/ci/ci_deploy_key + +In Gitea → **Site Administration → Secrets** (or per-repo → Settings → Secrets), add: + +| name | value | +| ---------------- | --------------------------------------------------------------- | +| `DEPLOY_SSH_KEY` | the entire contents of that file, including the BEGIN/END lines | + +An org- or admin-level secret means every repo gets it without repeating yourself. + +**You can delete `secrets.WATCHTOWER_TOKEN` afterwards** — nothing consumes it now. + +### Step 2 — swap the deploy step in each workflow + +Replace this: + +```yaml +- name: Trigger immediate deploy via Watchtower API + run: | + curl -fsS -X POST -H "Authorization: Bearer ${{ secrets.WATCHTOWER_TOKEN }}" \ + http://192.168.2.2:8080/v1/update \ + || echo "::warning::watchtower trigger failed; nightly sweep will deploy instead" +``` + +with this — changing **`agentchat`** on the last line to the service being deployed: + +```yaml +- name: Trigger immediate deploy + run: | + install -m600 /dev/null "$HOME/.deploy_key" + printf '%s\n' "${{ secrets.DEPLOY_SSH_KEY }}" > "$HOME/.deploy_key" + ssh -i "$HOME/.deploy_key" \ + -o BatchMode=yes \ + -o StrictHostKeyChecking=accept-new \ + -o ConnectTimeout=15 \ + gib@192.168.2.2 agentchat \ + || echo "::warning::deploy trigger failed; nightly sweep will deploy instead" + rm -f "$HOME/.deploy_key" +``` + +Kept deliberately from the old step: + +- **`|| echo "::warning::"`** — a deploy hiccup warns instead of failing a build whose + image already pushed fine. The midnight timer is the backstop. +- **`192.168.2.2`** — the VPS's WireGuard address, so this works from the home runners + and the VPS runner alike, and never crosses the public internet. + +### Repo → service name + +The argument after `gib@192.168.2.2` is the **systemd service name**, not the repo name: + +| repo | argument | +| ------------------ | ---------------- | +| `agentchat` | `agentchat` | +| `spoon` | `spoon` | +| `lashaddict` | `lashaddict` | +| `convex-monorepo` | `convexmonorepo` | +| `stpeteit-payload` | `stpeteit` | + +Confirm any name with `~/Server/update-containers.sh --list`. + +--- + +### What the key can and cannot do + +It is pinned to a forced command in `~/.ssh/authorized_keys`: + + command="/home/gib/Server/ci-deploy.sh",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,no-user-rc ssh-ed25519 AAAA... + +sshd runs `ci-deploy.sh` no matter what the client asks for. Verified by testing: + +| attempt | result | +| ------------------------------------ | ---------------------------------------- | +| `ssh ... adminer` | deploys adminer | +| `ssh ...` (bare shell) | `denied: expected a single service name` | +| `ssh ... 'cat .../PostgreSQL/.env'` | `denied` | +| `ssh ... 'adminer; cat /etc/passwd'` | `denied` | +| `ssh ... 'adminer$(id)'` | `denied` | +| `ssh ... postgresql` | `denied: frozen` | +| `ssh ... n8n` | `denied: not an active service` | + +So a leaked key redeploys a service that is already running — it cannot open a shell, +read files, forward ports, start a stopped service, or touch postgres/authentik. + +### Optional hardening + +`StrictHostKeyChecking=accept-new` trusts the host key on first connection. To pin it +instead, add the VPS's key to a secret and write it to `known_hosts` in the job. The +current fingerprint is in `~/Server/ci/vps_host_key.pub` on the VPS. + +--- + +## 16. Adding a database and user to the shared Postgres + +_Folded in from PostgreSQL/add_db_and_user.md, 2026-09-22._ + +Here is how you can add a database & a user to postgres so you don't have 100 postgres containers for every little app that requires postgres. + +You can edit these values to quickly get started & copy this. +Database name: n8n +Database user: n8n_user + +```sql +-- 1. Create user +CREATE ROLE n8n_user + LOGIN + PASSWORD 'Overvalue-Unstuffed-Coconut-Gutless7-Prevent-Sharpie'; + +-- 2. Create database +CREATE DATABASE n8n + ENCODING 'UTF8'; + +-- 3. Allow connection +GRANT CONNECT ON DATABASE n8n TO n8n_user; + +-- 4. Switch to the database +\c n8n + +-- 5. Schema access +GRANT ALL PRIVILEGES ON SCHEMA public TO n8n_user; + +-- 6. Existing objects +GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO n8n_user; +GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO n8n_user; +GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public TO n8n_user; + +-- 7. Future objects +ALTER DEFAULT PRIVILEGES IN SCHEMA public +GRANT ALL PRIVILEGES ON TABLES TO n8n_user; + +ALTER DEFAULT PRIVILEGES IN SCHEMA public +GRANT ALL PRIVILEGES ON SEQUENCES TO n8n_user; + +ALTER DEFAULT PRIVILEGES IN SCHEMA public +GRANT ALL PRIVILEGES ON FUNCTIONS TO n8n_user; +``` diff --git a/server/vps/Authentik/podman-authentik.service b/server/vps/Authentik/podman-authentik.service index ddf4043..d21d446 100644 --- a/server/vps/Authentik/podman-authentik.service +++ b/server/vps/Authentik/podman-authentik.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: Authentik After=network-online.target podman.socket podman-postgresql.service Wants=network-online.target podman-postgresql.service diff --git a/server/vps/Beszel/podman-beszel.service b/server/vps/Beszel/podman-beszel.service index 36582ce..9e3a959 100644 --- a/server/vps/Beszel/podman-beszel.service +++ b/server/vps/Beszel/podman-beszel.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: Beszel After=network-online.target podman.socket Wants=network-online.target diff --git a/server/vps/BillTracker/compose.yml b/server/vps/BillTracker/compose.yml index 45c72d1..c443f7d 100644 --- a/server/vps/BillTracker/compose.yml +++ b/server/vps/BillTracker/compose.yml @@ -32,7 +32,7 @@ services: restart: unless-stopped bill-tracker-backend: - image: ${CONVEX_BACKEND_IMAGE:-ghcr.io/get-convex/convex-backend@sha256:1f2044e3eac463ac78973b136c0baf72d4ada602611d853d6f99f280e29e0a98} + image: ${CONVEX_BACKEND_IMAGE:-ghcr.io/convex/convex-backend@sha256:207cf95def42f01614b6e3af0af5c20d0dcaa5ff2adf258ee7bc9555576644fa} container_name: bill-tracker-backend hostname: bill-tracker-backend domainname: api.billtracker.gbrown.org @@ -56,7 +56,7 @@ services: stop_signal: SIGINT bill-tracker-dashboard: - image: ${CONVEX_DASHBOARD_IMAGE:-ghcr.io/get-convex/convex-dashboard@sha256:284a2638e0c1a4ec0c2327d8219776f3a426ca5824b81686ae4d9454dc0ce8ed} + image: ${CONVEX_DASHBOARD_IMAGE:-ghcr.io/convex/convex-dashboard@sha256:a55bf8abca1332f787bc4d4a79850182175027be7b70aa230be4389a1d3268e2} container_name: bill-tracker-dashboard hostname: bill-tracker-dashboard domainname: dashboard.billtracker.gbrown.org diff --git a/server/vps/ConvexMonorepo/compose.yml b/server/vps/ConvexMonorepo/compose.yml index a1837f0..4aa98ea 100644 --- a/server/vps/ConvexMonorepo/compose.yml +++ b/server/vps/ConvexMonorepo/compose.yml @@ -29,7 +29,7 @@ services: restart: unless-stopped convexmonorepo-backend: - image: ghcr.io/get-convex/convex-backend:${BACKEND_TAG:-latest} + image: ghcr.io/convex/convex-backend:${BACKEND_TAG:-latest} container_name: ${BACKEND_CONTAINER_NAME:-convex-backend} hostname: ${BACKEND_CONTAINER_NAME:-convex-backend} domainname: ${BACKEND_DOMAIN:-convex.gbrown.org} @@ -58,7 +58,7 @@ services: stop_signal: SIGINT convexmonorepo-dashboard: - image: ghcr.io/get-convex/convex-dashboard:${DASHBOARD_TAG:-latest} + image: ghcr.io/convex/convex-dashboard:${DASHBOARD_TAG:-latest} container_name: ${DASHBOARD_CONTAINER_NAME:-convex-dashboard} hostname: ${DASHBOARD_CONTAINER_NAME:-convex-dashboard} domainname: ${DASHBOARD_DOMAIN:-dashboard.${BACKEND_DOMAIN:-convex.gbrown.org}} diff --git a/server/vps/ISponsorBlockTV/podman-isponsorblocktv.service b/server/vps/ISponsorBlockTV/podman-isponsorblocktv.service index e466803..b490e11 100644 --- a/server/vps/ISponsorBlockTV/podman-isponsorblocktv.service +++ b/server/vps/ISponsorBlockTV/podman-isponsorblocktv.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: iSponsorBlockTV After=network-online.target podman.socket Wants=network-online.target diff --git a/server/vps/Infisical/podman-infisical.service b/server/vps/Infisical/podman-infisical.service index 94fcc27..82e1ffe 100644 --- a/server/vps/Infisical/podman-infisical.service +++ b/server/vps/Infisical/podman-infisical.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: Infisical After=network-online.target podman.socket podman-postgresql.service Wants=network-online.target podman-postgresql.service diff --git a/server/vps/LashAddict/compose.yml b/server/vps/LashAddict/compose.yml index 9ccbf34..16e8b22 100644 --- a/server/vps/LashAddict/compose.yml +++ b/server/vps/LashAddict/compose.yml @@ -19,7 +19,7 @@ services: - NODE_ENV=${NODE_ENV} - SENTRY_AUTH_TOKEN=${SENTRY_AUTH_TOKEN} - NEXT_PUBLIC_SITE_URL=${NEXT_PUBLIC_SITE_URL:-http://localhost:${NEXT_PORT:-3000}} - - NEXT_PUBLIC_CONVEX_URL=${NEXT_PUBLIC_CONVEX_URL:-http://${BACKEND_CONTAINER_NAME:-lashaddict-backend}:${BACKEND_PORT:-3210}} + - NEXT_PUBLIC_CONVEX_URL=${CONVEX_SERVER_URL:-http://${BACKEND_CONTAINER_NAME:-lashaddict-backend}:${BACKEND_PORT:-3210}} - NEXT_PUBLIC_PLAUSIBLE_URL=${NEXT_PUBLIC_PLAUSIBLE_URL:-https://plausible.gbrown.org} - NEXT_PUBLIC_SENTRY_DSN=${NEXT_PUBLIC_SENTRY_DSN} - NEXT_PUBLIC_SENTRY_URL=${NEXT_PUBLIC_SENTRY_URL} @@ -33,7 +33,7 @@ services: stdin_open: true restart: unless-stopped lashaddict-backend: - image: ghcr.io/get-convex/convex-backend:${BACKEND_TAG:-latest} + image: ghcr.io/convex/convex-backend:${BACKEND_TAG:-latest} container_name: ${BACKEND_CONTAINER_NAME:-lashaddict-backend} # Same Cloudflare DNS pin as lashaddict-next (see its comment): the backend # fetches its own auth discovery URL, which Hetzner resolvers failed. @@ -67,7 +67,7 @@ services: stop_grace_period: 10s stop_signal: SIGINT lashaddict-dashboard: - image: ghcr.io/get-convex/convex-dashboard:${DASHBOARD_TAG:-latest} + image: ghcr.io/convex/convex-dashboard:${DASHBOARD_TAG:-latest} container_name: ${DASHBOARD_CONTAINER_NAME:-lashaddict-dashboard} # Same Cloudflare DNS pin as lashaddict-next (see its comment): the backend # fetches its own auth discovery URL, which Hetzner resolvers failed. diff --git a/server/vps/N8n/podman-n8n.service b/server/vps/N8n/podman-n8n.service index 0f98cb2..3b6b46d 100644 --- a/server/vps/N8n/podman-n8n.service +++ b/server/vps/N8n/podman-n8n.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: N8n After=network-online.target podman-postgresql.service Wants=network-online.target diff --git a/server/vps/Nginx_Proxy_Manager/podman-nginx-proxy-manager.service b/server/vps/Nginx_Proxy_Manager/podman-nginx-proxy-manager.service index f249f9c..8bd09df 100644 --- a/server/vps/Nginx_Proxy_Manager/podman-nginx-proxy-manager.service +++ b/server/vps/Nginx_Proxy_Manager/podman-nginx-proxy-manager.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: Nginx Proxy Manager After=network-online.target podman-postgresql.service Wants=network-online.target diff --git a/server/vps/Portainer/podman-portainer.service b/server/vps/Portainer/podman-portainer.service index 1143219..837e52d 100644 --- a/server/vps/Portainer/podman-portainer.service +++ b/server/vps/Portainer/podman-portainer.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: Portainer After=network-online.target Wants=network-online.target diff --git a/server/vps/PostgreSQL/podman-postgresql.service b/server/vps/PostgreSQL/podman-postgresql.service index 59c839e..f27370a 100644 --- a/server/vps/PostgreSQL/podman-postgresql.service +++ b/server/vps/PostgreSQL/podman-postgresql.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: PostgreSQL After=network-online.target Wants=network-online.target diff --git a/server/vps/Rustdesk/podman-rustdesk.service b/server/vps/Rustdesk/podman-rustdesk.service index 61621e7..3e899a6 100644 --- a/server/vps/Rustdesk/podman-rustdesk.service +++ b/server/vps/Rustdesk/podman-rustdesk.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: Rustdesk After=network-online.target podman.socket Wants=network-online.target diff --git a/server/vps/Spoon/compose.yml b/server/vps/Spoon/compose.yml index 40c94e8..a3871cf 100644 --- a/server/vps/Spoon/compose.yml +++ b/server/vps/Spoon/compose.yml @@ -64,7 +64,7 @@ services: security_opt: - label:disable spoon-backend: - image: ghcr.io/get-convex/convex-backend:${BACKEND_TAG:-latest} + image: ghcr.io/convex/convex-backend:${BACKEND_TAG:-latest} container_name: ${BACKEND_CONTAINER_NAME:-spoon-backend} hostname: ${BACKEND_CONTAINER_NAME:-spoon-backend} domainname: ${BACKEND_DOMAIN:-convex.spoon.gbrown.org} @@ -93,7 +93,7 @@ services: stop_grace_period: 10s stop_signal: SIGINT spoon-dashboard: - image: ghcr.io/get-convex/convex-dashboard:${DASHBOARD_TAG:-latest} + image: ghcr.io/convex/convex-dashboard:${DASHBOARD_TAG:-latest} container_name: ${DASHBOARD_CONTAINER_NAME:-spoon-dashboard} hostname: ${DASHBOARD_CONTAINER_NAME:-spoon-dashboard} domainname: ${DASHBOARD_DOMAIN:-dashboard.${BACKEND_DOMAIN:-spoon.gbrown.org}} diff --git a/server/vps/StPeteIT/compose.yml b/server/vps/StPeteIT/compose.yml index b2669ab..afea7b5 100644 --- a/server/vps/StPeteIT/compose.yml +++ b/server/vps/StPeteIT/compose.yml @@ -29,7 +29,7 @@ services: restart: unless-stopped stpeteit-backend: - image: ghcr.io/get-convex/convex-backend:${BACKEND_TAG:-latest} + image: ghcr.io/convex/convex-backend:${BACKEND_TAG:-latest} container_name: ${BACKEND_CONTAINER_NAME:-stpeteit-backend} hostname: ${BACKEND_CONTAINER_NAME:-stpeteit-backend} domainname: ${BACKEND_DOMAIN:-convex.stpeteit.com} @@ -58,7 +58,7 @@ services: stop_signal: SIGINT stpeteit-dashboard: - image: ghcr.io/get-convex/convex-dashboard:${DASHBOARD_TAG:-latest} + image: ghcr.io/convex/convex-dashboard:${DASHBOARD_TAG:-latest} container_name: ${DASHBOARD_CONTAINER_NAME:-stpeteit-dashboard} hostname: ${DASHBOARD_CONTAINER_NAME:-stpeteit-dashboard} domainname: ${DASHBOARD_DOMAIN:-dashboard.${BACKEND_DOMAIN:-convex.stpete.com}} diff --git a/server/vps/Tools/podman-tools.service b/server/vps/Tools/podman-tools.service index d4362c3..169256d 100644 --- a/server/vps/Tools/podman-tools.service +++ b/server/vps/Tools/podman-tools.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: Tools After=network-online.target podman.socket Wants=network-online.target diff --git a/server/vps/Uptime/podman-uptime.service b/server/vps/Uptime/podman-uptime.service index 2436a6a..274780a 100644 --- a/server/vps/Uptime/podman-uptime.service +++ b/server/vps/Uptime/podman-uptime.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: Uptime After=network-online.target Wants=network-online.target diff --git a/server/vps/Vaultwarden/podman-vaultwarden.service b/server/vps/Vaultwarden/podman-vaultwarden.service index 25960a7..c373993 100644 --- a/server/vps/Vaultwarden/podman-vaultwarden.service +++ b/server/vps/Vaultwarden/podman-vaultwarden.service @@ -1,4 +1,5 @@ [Unit] +RequiresMountsFor=/home/gib/Media Description=Podman Compose: Vaultwarden After=network-online.target podman.socket Wants=network-online.target diff --git a/server/vps/backup b/server/vps/backup new file mode 100755 index 0000000..e97eae3 --- /dev/null +++ b/server/vps/backup @@ -0,0 +1,194 @@ +#!/usr/bin/env bash +# +# Daily backup of ~/Server to the home server, as hardlinked dated snapshots. +# +# WHAT THIS FIXES vs. the old version (2026-09-17): +# 1. OWNERSHIP. Rootless podman leaves most container data owned by subuids +# (gitea = 525288, postgres data = the userns root mapping). Plain `rsync -a` +# running as gib cannot chown on the far side, so every previous backup landed +# 100% owned by uid 1000. Restoring that gives you the right bytes with the wrong +# owners: postgres refuses to start unless PGDATA is owned correctly, and gitea +# fails on its own data dir. Fixed with `-M--fake-super`, which makes the REMOTE +# rsync stash mode/uid/gid in a user.rsync.%stat xattr (verified working on the +# home server's ZFS). Restore with --fake-super on the read side to put them back. +# 2. CONSISTENCY. 13 SQLite DBs and a live postgres were copied hot, mid-write. Now +# dumped properly first (sqlite3 .backup / pg_dumpall) into _backup_state/. +# 3. SPACE / HISTORY. Every run was a full ~19G copy into a new dated directory. +# Now it rsyncs into ONE stable directory, `current/`, and versioning is left to +# the home server: Media/Server_Backup is its own ZFS dataset with zfs-auto-snapshot +# keeping 14 dailies. ZFS snapshots are block-level, so a database that changes a +# few pages costs a few pages -- not a whole new copy the way dated dirs (or even +# rsync --link-dest hardlinks, which are per-file) would. +# +# Containers keep running throughout. That is safe *because* of the dumps above -- +# the dumps, not the raw data dirs, are the authoritative restore path. +# +# See RESTORE.md for how to bring this back on a new machine. + +set -uo pipefail + +REMOTE_HOST="gib@server.gib" +REMOTE_ROOT="/home/gib/Public/Server_Backup/Containers/lilman.gib" +SOURCE="$HOME/Server/" +STATE_DIR="$HOME/Server/_backup_state" +DEST_NAME="current" # One stable directory. ZFS snapshots provide history. +LOG_DIR="$HOME/Server/logs" +LOG="$LOG_DIR/backup.log" +LOCK="${XDG_RUNTIME_DIR:-/tmp}/server-backup.lock" + +# Versioning is ZFS's job, not ours -- see the header. We only sanity-check that it is +# actually still happening, so this can never silently degrade into a single mirror with +# no history. Warn if the newest snapshot of the dataset is older than this. +REMOTE_DATASET="Media/Server_Backup" +SNAPSHOT_MAX_AGE_HOURS=48 + +mkdir -p "$LOG_DIR" +log() { printf '%s %s\n' "$(date +'%Y-%m-%d %H:%M:%S')" "$*" | tee -a "$LOG"; } +die() { log "FATAL: $*"; exit 1; } + +# Refuse to overlap with a still-running backup rather than corrupting a snapshot. +exec 9>"$LOCK" || die "cannot open lock $LOCK" +flock -n 9 || die "another backup is already running" + +EXCLUSIONS=( + # --- build noise ------------------------------------------------------- + # NOTE: these are bare patterns, so they match at ANY depth. Today they only + # catch genuine scratch dirs (checked 2026-09-17). If you add a service whose + # real data lives in a dir called cache/build/dist/tmp, it will be silently + # skipped -- re-check this list when adding services. + "node_modules" ".git" "*.log" "*.tmp" ".cache" "cache" "tmp" "temp" + ".npm" ".yarn" "dist" "build" "target" ".venv" "venv" "__pycache__" + "*.pyc" ".DS_Store" "Thumbs.db" "vendor" "*.mp4" "*.mkv" + + # --- unreadable to gib, and genuinely disposable ------------------------ + # Redis/valkey cache dumps: mode 0600 under a subuid because the container + # chowns its own data. Pure cache (usesend + infisical), regenerated on boot. + "dump.rdb" + # Gitea runs WITHOUT userns_mode (s6 needs container-root -- see AGENTS.md §3), + # so these are subuid-owned and mode 0700. Excluding them is what lets this run + # without sudo. Nothing here is needed to restore gitea: + # sessions/ ephemeral web logins; worst case everyone logs in again + # issues.bleve search index, rebuilt from the DB on demand + # Gitea/ssh a 25-byte `environment` file for the git shell + # + # ⚠️ The SSH HOST KEYS are NOT here -- they live in Gitea/volume/ssh, are + # gib-owned, and DO get backed up. Losing those = host-key mismatch for every client. + "sessions" "issues.bleve" "Gitea/ssh" + + # Mode 0600 under subuid 525288, so unreadable to gib -- this single file is what + # made every previous run exit 23. It is NOT disposable (it holds the MCP bridge's + # registered OAuth clients), so it is captured via `podman exec` into + # _backup_state/files/ instead of being read off disk. + "BillTracker/data/mcp/bridge.json" + + # Stirling-PDF writes this 0600 under subuid 525287 (since its 2026-09-25 update), + # which made every run since exit 23. Excluded rather than captured: losing it only + # means re-entering any credentials stored in Stirling-PDF. + "pdf-tools/extraConfigs/credential-encryption.key" +) + +EXCLUDE_ARGS=() +for e in "${EXCLUSIONS[@]}"; do EXCLUDE_ARGS+=( "--exclude=$e" ); done + +# --------------------------------------------------------------------------- +# Phase 1: consistent dumps of everything transactional. +# --------------------------------------------------------------------------- +dump_state() { + log "Dumping databases to _backup_state/ ..." + rm -rf "$STATE_DIR" + mkdir -p "$STATE_DIR"/{postgres,sqlite,files} + printf 'Generated %s by ~/Server/backup. Authoritative restore source -- see RESTORE.md\n' \ + "$(date -Is)" > "$STATE_DIR/README" + + # Live postgres (serves npm, authentik, usesend, ... -- pg_dumpall gets all of them). + if podman ps --format '{{.Names}}' | grep -qx postgresql; then + if podman exec postgresql pg_dumpall -U npm 2>>"$LOG" | gzip -9 > "$STATE_DIR/postgres/all.sql.gz"; then + log " postgres: $(du -h "$STATE_DIR/postgres/all.sql.gz" | cut -f1)" + else + log " WARNING: pg_dumpall failed -- raw pg_data is still copied, but is a hot copy" + fi + else + log " WARNING: postgresql container not running, skipping pg_dumpall" + fi + + # SQLite: .backup uses the online backup API, so this is safe while containers write. + local n=0 + while IFS= read -r db; do + # *.db also matches stirling-pdf's H2 database, which is not SQLite. Check magic. + head -c 16 "$db" 2>/dev/null | grep -q "SQLite format 3" || continue + local rel="${db#"$HOME"/Server/}" out + out="$STATE_DIR/sqlite/$rel" + mkdir -p "$(dirname "$out")" + if sqlite3 -cmd ".timeout 30000" "$db" ".backup '$out'" 2>>"$LOG"; then n=$((n+1)); else log " WARNING: sqlite dump failed: $rel"; fi + done < <(find "$HOME/Server" -path "$STATE_DIR" -prune -o \ + -type f \( -name '*.db' -o -name '*.sqlite' -o -name '*.sqlite3' \) -print 2>/dev/null) + log " sqlite: $n databases" + + # Files gib cannot read off disk, pulled through their container instead. + if podman ps --format '{{.Names}}' | grep -qx bill-tracker-mcp; then + podman exec bill-tracker-mcp cat /data/bridge.json \ + > "$STATE_DIR/files/BillTracker--data--mcp--bridge.json" 2>>"$LOG" \ + && log " bridge.json captured" \ + || log " WARNING: could not capture bridge.json" + fi +} + +# --------------------------------------------------------------------------- +# Phase 2: confirm ZFS is still providing history. +# +# We deliberately overwrite one directory instead of keeping dated copies, which +# means ZFS snapshots are the ONLY thing standing between us and "a mirror of a +# mistake". If auto-snapshot ever stops, that must be loud, not silent. +# --------------------------------------------------------------------------- +check_zfs_history() { + local newest now age_h + newest=$(ssh -o BatchMode=yes "$REMOTE_HOST" \ + "zfs list -H -p -o creation -t snapshot -s creation -d 1 '$REMOTE_DATASET' 2>/dev/null | tail -1" || true) + if [ -z "$newest" ]; then + log " ⚠️ WARNING: no ZFS snapshots on $REMOTE_DATASET -- this backup has NO history." + log " It is a plain mirror right now. Check zfs-auto-snapshot on the home server." + return + fi + now=$(date +%s) + age_h=$(( (now - newest) / 3600 )) + if [ "$age_h" -gt "$SNAPSHOT_MAX_AGE_HOURS" ]; then + log " ⚠️ WARNING: newest ZFS snapshot is ${age_h}h old (limit ${SNAPSHOT_MAX_AGE_HOURS}h) -- versioning may be broken." + else + log " ZFS history OK (newest snapshot ${age_h}h old, 14 dailies retained)" + fi +} + +# --------------------------------------------------------------------------- +# Phase 3: mirror. +# --------------------------------------------------------------------------- +log "=== backup start -> $REMOTE_HOST:$REMOTE_ROOT/$DEST_NAME ===" +ssh -o BatchMode=yes -o ConnectTimeout=15 "$REMOTE_HOST" "mkdir -p '$REMOTE_ROOT/$DEST_NAME'" \ + || die "cannot reach $REMOTE_HOST" + +dump_state +check_zfs_history + +# --delete/--delete-excluded keep `current` an exact mirror of the source. Anything +# removed here still exists in yesterday's ZFS snapshot, so this is not destructive. +rsync -a --numeric-ids -M--fake-super --delete --delete-excluded \ + --human-readable --stats \ + "${EXCLUDE_ARGS[@]}" \ + --rsync-path="mkdir -p '$REMOTE_ROOT/$DEST_NAME' && rsync" \ + "$SOURCE" "$REMOTE_HOST:$REMOTE_ROOT/$DEST_NAME/" 2>&1 | tee -a "$LOG" +status=${PIPESTATUS[0]} + +# 24 = "source files vanished during transfer". Normal on a live system, not a failure. +if [ "$status" -ne 0 ] && [ "$status" -ne 24 ]; then + die "rsync failed with exit $status -- 'current' may be half-updated; yesterday's ZFS snapshot is still good" +fi +[ "$status" -eq 24 ] && log "NOTE: rsync exit 24 (files vanished mid-run) -- expected on a live system" + +# Optional heartbeat so a silent failure cannot go unnoticed. Create the file with an +# uptime-kuma push URL in it and this pings on success only; absent, nothing happens. +if [ -r "$HOME/Server/.backup-heartbeat-url" ]; then + url=$(< "$HOME/Server/.backup-heartbeat-url") + curl -fsS -m 20 "$url" >/dev/null 2>&1 \ + && log "Heartbeat sent" || log "NOTE: heartbeat ping failed (backup itself was fine)" +fi + +log "=== backup complete ===" diff --git a/server/vps/server-backup.service b/server/vps/server-backup.service new file mode 100644 index 0000000..21f61bb --- /dev/null +++ b/server/vps/server-backup.service @@ -0,0 +1,22 @@ +[Unit] +Description=Back up ~/Server to the home server +Documentation=file:///home/gib/Server/RESTORE.md +After=network-online.target +Wants=network-online.target +# The backup reads podman-managed data and shells into containers for dumps. +RequiresMountsFor=/home/gib/Media + +[Service] +Type=oneshot +ExecStart=%h/Server/backup +# A stalled rsync should fail loudly, not hang until the next timer fires. +TimeoutStartSec=6h +# This box is IO-bound before it is CPU-bound (~18%% avg60 pressure), so stay +# out of the way of the containers. best-effort/7 rather than idle, which can +# starve the transfer outright when the volume is busy. +Nice=10 +IOSchedulingClass=best-effort +IOSchedulingPriority=7 + +[Install] +WantedBy=default.target diff --git a/server/vps/server-backup.timer b/server/vps/server-backup.timer new file mode 100644 index 0000000..58e69fd --- /dev/null +++ b/server/vps/server-backup.timer @@ -0,0 +1,18 @@ +[Unit] +Description=Daily backup of ~/Server to the home server + +[Timer] +# MUST finish before the home server's cron.daily fires zfs-auto-snapshot, which has +# been landing between 03:22 and 03:52. If the snapshot catches a half-finished rsync, +# that day's history entry is a partial backup. 01:00 leaves >2h of margin; the first +# full run took well under that and daily deltas are far smaller. +OnCalendar=*-*-* 01:00:00 +# Small jitter only -- must not push us into the snapshot window. +RandomizedDelaySec=10m +# If the VPS was down at 01:00, run once after it comes back rather than +# silently skipping a day. +Persistent=true +Unit=server-backup.service + +[Install] +WantedBy=timers.target diff --git a/server/vps/verify-rootless b/server/vps/verify-rootless new file mode 100755 index 0000000..1bf023e --- /dev/null +++ b/server/vps/verify-rootless @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +# Post-reboot verification for the rootless podman stack. +# Run as gib. Compares against ~/premigration_backup_2026-08-10/post_migration_baseline.txt +set -uo pipefail + +fail=0 +ok() { printf ' \033[32mOK\033[0m %s\n' "$1"; } +bad() { printf ' \033[31mFAIL\033[0m %s\n' "$1"; fail=$((fail+1)); } + +echo "== linger ==" +[[ "$(loginctl show-user gib --property=Linger --value 2>/dev/null)" == "yes" ]] \ + && ok "Linger=yes" || bad "linger is OFF - user units will not start at boot" + +echo "== unprivileged port start ==" +p=$(sysctl -n net.ipv4.ip_unprivileged_port_start 2>/dev/null) +[[ "$p" == "80" ]] && ok "ip_unprivileged_port_start=80" \ + || bad "ip_unprivileged_port_start=$p (NPM cannot bind :80/:443)" + +echo "== user units ==" +for s in postgresql bang completeuphoria sierraandtyler uptime portainer beszel authentik tools isponsorblocktv vaultwarden infisical rustdesk nginx-proxy-manager gbrown-agent; do + st=$(systemctl --user is-active "podman-$s.service" 2>/dev/null) + [[ "$st" == "active" ]] && ok "$s" || bad "$s is $st" +done + +echo "== containers ==" +for c in postgresql nginx-proxy-manager uptime portainer bang completeuphoria sierraandtyler beszel beszel-agent authentik-server authentik-worker convertx it-tools pdf-tools bento-pdf iSponsorBlockTV vaultwarden infisical-redis infisical-backend rustdesk-server gbrown-agent; do + if podman ps --format '{{.Names}}' 2>/dev/null | grep -qx "$c"; then ok "$c"; else bad "$c not running"; fi +done + +echo "== postgres health ==" +h=$(podman inspect postgresql --format '{{.State.Health.Status}}' 2>/dev/null) +[[ "$h" == "healthy" ]] && ok "postgres healthy" || bad "postgres health=$h" + +echo "== endpoints ==" +chk() { local n=$1 exp=$2; shift 2; c=$(curl -sk -o /dev/null -w '%{http_code}' "$@" 2>/dev/null) + [[ "$c" == "$exp" ]] && ok "$n -> $c" || bad "$n -> $c (expected $exp)"; } +# Admin ports moved to the WireGuard address 2026-08-12 (AGENTS.md §13.2). +# They no longer listen on localhost, so probe 192.168.2.2. +chk "NPM :81" 200 http://192.168.2.2:81 +chk "NPM :80" 200 http://localhost:80 +chk "NPM :443" 200 --resolve nginx.gibbyb.com:443:127.0.0.1 https://nginx.gibbyb.com +chk "Uptime" 302 http://192.168.2.2:3001 +chk "Portainer" 200 https://192.168.2.2:9443 + +# ───────────────────────────────────────────────────────────────────────────── +# Boot-enablement check (added 2026-08-12) +# +# Five stacks were once found RUNNING but DISABLED — healthy in `podman ps`, yet they +# would have disappeared on the next reboot. Starting a stack and enabling it are +# separate actions; this catches the gap. See AGENTS.md §13.3. +# ───────────────────────────────────────────────────────────────────────────── +echo +echo "== boot-enablement ==" +# podman's own units are meant to stay disabled; n8n is intentionally down. +# podman's own units stay disabled by design; n8n is intentionally down; +# podman-update.service is `static` (the timer activates it, so it is never enabled); +# watchtower was RETIRED 2026-08-17 and must stay disabled (AGENTS.md §13.15). +SKIP='podman-auto-update|podman-restart|podman-clean-transient|podman-kube@|podman-user-wait-network-online|podman-n8n|podman-update|podman-watchtower' +bad=0 +for u in $(systemctl --user list-unit-files 'podman-*.service' --no-legend 2>/dev/null | awk '{print $1}'); do + echo "$u" | grep -qE "$SKIP" && continue + state=$(systemctl --user is-enabled "$u" 2>/dev/null) + if [ "$state" != "enabled" ]; then + echo " WILL NOT SURVIVE REBOOT: $u ($state) -> systemctl --user enable $u" + bad=$((bad+1)) + fi +done +[ "$bad" -eq 0 ] && echo " all service units enabled" + +echo +echo "== nightly image updates (replaces watchtower — AGENTS.md §13.15) ==" +if [ "$(systemctl --user is-enabled podman-update.timer 2>/dev/null)" = "enabled" ]; then + echo " timer enabled, next: $(systemctl --user list-timers podman-update.timer --no-pager 2>/dev/null | awk 'NR==2{print $1,$2,$3,$4}')" +else + echo " WARNING: podman-update.timer is NOT enabled — nothing will update" +fi +if [ "$(systemctl --user is-active podman-watchtower.service 2>/dev/null)" = "active" ]; then + echo " WARNING: watchtower is RUNNING — it recreates containers outside their pod (§13.13)" +fi +# A container outside its compose pod is the signature of a watchtower-style recreate. +podless="" +for c in $(podman ps --format '{{.Names}}' 2>/dev/null); do + [ -z "$(podman inspect "$c" --format '{{.Pod}}' 2>/dev/null)" ] && podless="$podless $c" +done +[ -n "$podless" ] && echo " PODLESS (rebuild with systemctl --user restart):$podless" || echo " all containers in their pod" + +echo +echo "== admin ports bound to 0.0.0.0 (should be 192.168.2.2 — AGENTS.md §13.2) ==" +podman ps --format '{{.Names}} {{.Ports}}' | grep -E '0\.0\.0\.0:(81|80-81|3001|9443|5432)(->|-)' \ + | sed 's/^/ EXPOSED: /' || echo " none" + +echo +if [[ $fail -eq 0 ]]; then + printf '\033[32mALL CHECKS PASSED\033[0m\n' +else + printf '\033[31m%d CHECK(S) FAILED\033[0m\n' "$fail" + echo "Investigate: journalctl --user -u podman-.service -n 50" +fi +exit $fail