Sync both server snapshots with what the live machines run, and keep their backup tooling

Hyprland 0.57 refuses a bind on a bare modifier and an empty input:scroll_method. The switcher's Super-release commit now binds Super_L and Super_R, which also brings back every bind below it in keybinds.lua. An empty scroll method is left unwritten, so libinput still picks per device.

.agents/skills/panama/SKILL.md

@@ -9,7 +9,7 @@ You are editing a running desktop, not a codebase that gets deployed later. Ever
 repository is symlinked into `~/.config`, so a save is live the moment it lands. That single fact
 drives every rule below.
 
-Read before large work, in this order:
+Before large work, read these in order:
 
 - `README.md` — layout, the `panama` command, how installing and updating work
 - `config/dot/hypr/README.md` — the compositor config is **Lua, not hyprlang**; read "The one
@@ -62,7 +62,8 @@ Every executable contract under `tests/` is classified in `tests/contracts.manif
   `PANAMA_TEST_TIMEOUT_SECONDS` value. Failures print the contract's captured stdout and stderr.
   Successful stdout stays quiet. Successful stderr is surfaced as a warning.
 - Contracts run directly too: `tests/setup/interview-contract`.
-- After changing `PreferenceSchema.qml` or `services/SettingsRoutes.qml`, regenerate:
+- After changing `PreferenceSchema.qml` or
+  `config/dot/quickshell/services/SettingsRoutes.qml`, regenerate:
   `config/dot/quickshell/scripts/panama-settings-docs` (writes `docs/settings.md`) and
   `config/dot/quickshell/scripts/panama-settings-commands` (writes the launcher deep links).
   Both take `--check`; `tests/quickshell/settings-docs-contract` fails when stale.

.claude/docs/specs/2026-08-25-server-role.md

@@ -183,8 +183,8 @@ currently parked (dir present, unit not active) — imported anyway, noted as
 parked in the catalog.
 
 **Excluded:** Watchtower (retired — replaced by the update timer after it took
-gitea down; its dir stays on junior until cleanup), Completeuphoria,
-sierraandtyler, Bang (source checkouts deployed by CI from their own repos —
+gitea down; its dir stays on junior until cleanup), the app source
+checkouts such as sierraandtyler and Bang (deployed by CI from their own repos —
 Panama owning a second copy of their compose would drift), `ci/`, `logs/`,
 and junior's loose scripts other than `update-containers.sh` (§4).
 

.claude/skills/panama

@@ -0,0 +1 @@
+../../.agents/skills/panama

.gitignore

@@ -3,15 +3,21 @@
 /config/bash/.env.lock
 # Server secrets and state never enter the repo: live .env files belong in
 # ~/Server/<Name>/ (only compose.yml is symlinked back here), and data/ is
-# the bind-mount convention. Ignored anyway as a seatbelt -- the repo is
-# public, and tests/server/compose-secrets-contract is the check that this
-# line still exists.
-/server/containers/**/.env
-/server/containers/**/data/
+# the bind-mount convention (the home server's older services use volume/ and
+# volumes/). Ignored anyway as a seatbelt -- the repo is public, and
+# tests/server/compose-secrets-contract is the check that these lines still exist.
+/server/**/.env
+/server/**/data/
+/server/**/volume/
+/server/**/volumes/
 # Personal espanso triggers (name, email), seeded per-machine by setup-identity.
 /config/dot/espanso/match/identity.yml
 # Ignore backups of old config files
 /config/old
+# MCP bearer tokens. The server list in user/agents/mcp/servers is tracked
+# and names which variable carries each token; the tokens themselves are
+# machine-local, because this repo is public.
+/user/agents/mcp/env
 # Ignore Wireguard config of course!
 /config/wg/**
 # Ignore Neovim lazy-lock file

README.md

@@ -102,13 +102,13 @@ in order, without stopping again:
 | `interview` | Every prompt, before anything is installed. Answers last one run and are never written to a durable path |
 | `install-packages` | Repos (RPM Fusion, Terra, Hyprland COPR), the package lists in `setup/packages/`, then whichever optional categories were chosen |
 | `link-dotfiles` | Symlinks `config/dot/<name>` → `~/.config/<name>`, and seeds the wallpaper, cursor theme and Firefox chrome |
-| `link-skills` | Links the agent skills in `skills/` into `~/.claude/skills`, one per skill. Every machine gets these; personal ones link after and win a name clash |
+| `link-skills` | Links the agent skills in `skills/` into `~/.agents/skills` and `~/.claude/skills`, one per skill. Every machine gets these; personal ones link after and win a name clash |
 | `link-user` | Links the personal content in `user/` — agent instructions, SSH host aliases — but only on a machine that answered yes. See [user/README.md](user/README.md) |
 | `change-settings` | Copies `config/copy/` over `/`, applies gsettings, enables user services |
 | `link-vicinae-scripts` | Publishes the Vicinae script commands |
 | `setup-identity` | git config, `gh auth login`, an SSH key — whichever were asked for |
 | `install-hardware` | NVIDIA, Secure Boot enrollment, Fedora's extras, firmware — each only if it was asked for. Last, because enrollment and firmware are consumed at the next boot |
-| `setup-server` | Server role only: linger, unprivileged ports from 80, firewalld (80/443/81), the `nginx-bridge` network, and the nightly image-update timer |
+| `setup-server` | Server role only: linger, unprivileged ports from 80, firewalld (80/443/81, opened only where nothing already grants them), the `nginx-bridge` network, and the nightly image-update timer |
 | `link-server` | Server role only: creates `~/Server` and refreshes the links of whatever services this machine has enabled |
 
 A server runs the shared stages plus its own two, and skips the rest —
@@ -235,10 +235,13 @@ server/         The server role: compose services (one directory per
                 service), the nightly image updater, and its units. See
                 server/README.md
 skills/         Agent skills for operating this desktop, linked into
-                ~/.claude/skills
+                ~/.agents/skills and ~/.claude/skills
 setup/
   apps/         Applications built from source, one file each
-  lib/          Shared by more than one stage; the extras catalog reader
+  keys/         Pinned signing keys, for publishers that ship no fetchable
+                one; setup/keys/README.md records where each came from
+  lib/          Shared by more than one stage; the extras catalog reader,
+                the machine role, the verified ChatGPT repository
   packages/     One package per line; extras/ holds the optional categories
   scripts/      Run in order by ./install
 tests/          Contracts. See below
@@ -247,7 +250,7 @@ docs/           Settings reference, and the design specs behind the work
 
 ## Tests
 
-188 of them, under `tests/`. `tests/contracts.manifest` classifies every
+193 of them, under `tests/`. `tests/contracts.manifest` classifies every
 contract by the capabilities it needs. Run the hermetic set, or grant a
 specific external capability when automation needs it:
 

bin/panama

@@ -134,7 +134,6 @@ ${BOLD}Examples:${RESET}
   $PROGRAM upgrade
   $PROGRAM apps
   $PROGRAM app
-  $PROGRAM app chatgpt-desktop
 EOF
 }
 
@@ -451,11 +450,16 @@ CONTRACT_CAPABILITIES=(hermetic live-host live-compositor live-desktop network p
 
 contract_paths() {
   local candidate
+  # The manifest is kept in byte order, so both the discovery sort and the
+  # comparison below have to be byte order too. A UTF-8 collation folds the
+  # punctuation away -- `calendar_agenda_bridge_test.py` sorts before
+  # `calendar-agenda-helper-contract` under en_US and after it under C -- and
+  # a gate that passes or fails on the machine's LANG is not a gate.
   while IFS= read -r candidate; do
     [[ -x "$candidate" || "$candidate" == *_test.py ]] || continue
     printf 'tests/%s\n' "${candidate#"$PANAMA_DIR/tests/"}"
   done < <(find "$PANAMA_DIR/tests" -type f \
-    -not -path '*/fixtures/*' -not -path '*__pycache__*' | sort)
+    -not -path '*/fixtures/*' -not -path '*__pycache__*' | LC_ALL=C sort)
 }
 
 contract_manifest_entries() {
@@ -478,6 +482,8 @@ validate_contract_manifest() {
   require_contract_manifest || return 1
 
   local manifest="$PANAMA_DIR/$CONTRACT_MANIFEST"
+  # Byte order, for the same reason contract_paths sorts in it.
+  local LC_ALL=C
   local line capabilities path extra previous_comment="" previous_was_comment=0
   local previous_path="" capability discovered
   local -a capability_list=() findings=()

bin/panama-server

@@ -2,10 +2,12 @@
 #
 # panama-server – the compose services a server machine runs.
 #
-# The repository carries the catalog: server/containers/<Name>/ holds a
+# The repository carries the catalog: server/vps/<Name>/ holds a
 # compose.yml, a podman-<name>.service user unit, and an .env.example naming
-# what the service needs told. The machine carries the rest: ~/Server/<Name>/
-# is a real directory owning the .env (secrets never sit inside the checkout,
+# what the service needs told. A service may also carry helper units (any
+# other *.service or *.timer, e.g. a cert refresh) and helper files (scripts,
+# a README) beside them. The machine carries the rest: ~/Server/<Name>/ is a
+# real directory owning the .env (secrets never sit inside the checkout,
 # gitignored or not) and the bind-mounted data. Only the tracked files are
 # symlinked, so a `git clean` in the repo can never reach a database.
 #
@@ -27,7 +29,7 @@ PANAMA_DIR=$(cd "$(dirname "$SCRIPT_PATH")/.." && pwd)
 
 # Overridable so the contract can point this at a fixture catalog and a
 # throwaway HOME rather than the machine's real services.
-SERVER_DIR="${PANAMA_SERVER_DIR:-$PANAMA_DIR/server/containers}"
+SERVER_DIR="${PANAMA_SERVER_DIR:-$PANAMA_DIR/server/vps}"
 TARGET_DIR="${PANAMA_SERVER_TARGET:-$HOME/Server}"
 UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user"
 STATE_FILE="${XDG_STATE_HOME:-$HOME/.local/state}/panama/server-definitions"
@@ -46,7 +48,7 @@ err()   { printf '%s✗%s %s\n'    "${RED}${BOLD}"    "$RESET" "$*" >&2; }
 
 usage() {
   cat <<EOF
-${BOLD}$PROGRAM${RESET} – manage the compose services in server/containers/
+${BOLD}$PROGRAM${RESET} – manage the compose services in server/vps/
 
 ${BOLD}Usage:${RESET}
   $PROGRAM list             Every service in the catalog, and its state here
@@ -83,12 +85,12 @@ require_service() {
   fi
 }
 
-# The one unit file a service directory carries. Its filename is its identity
-# -- podman-<name>.service, exactly what lands in the systemd user directory --
-# so nothing here invents a name that could drift from the file's.
+# The one compose unit a service directory carries. Its filename is its
+# identity -- podman-<name>.service, exactly what lands in the systemd user
+# directory -- so nothing here invents a name that could drift from the file's.
 unit_path_for() {
   local name="$1" unit
-  for unit in "$SERVER_DIR/$name"/*.service; do
+  for unit in "$SERVER_DIR/$name"/podman-*.service; do
     [[ -e "$unit" ]] || continue
     printf '%s' "$unit"
     return 0
@@ -96,6 +98,44 @@ unit_path_for() {
   return 1
 }
 
+# Every other unit in the directory: helper services and the timers that fire
+# them. They are linked beside the compose unit, and each timer is enabled.
+helper_units_for() {
+  local name="$1" unit
+  for unit in "$SERVER_DIR/$name"/*.service "$SERVER_DIR/$name"/*.timer; do
+    [[ -e "$unit" ]] || continue
+    [[ "$(basename "$unit")" == podman-*.service ]] && continue
+    printf '%s\n' "$unit"
+  done
+}
+
+# The files linked into ~/Server/<Name>/: every visible top-level file that is
+# not a unit. That is compose.yml plus any helper script or README; dotfiles
+# are skipped so .env.example (and a stray .env) never become links.
+live_files_for() {
+  local name="$1" file
+  for file in "$SERVER_DIR/$name"/*; do
+    [[ -f "$file" ]] || continue
+    case "$file" in *.service|*.timer) continue ;; esac
+    printf '%s\n' "$file"
+  done
+}
+
+link_service_files() {
+  local name="$1" file unit
+  mkdir -p "$TARGET_DIR/$name" "$UNIT_DIR"
+  while IFS= read -r file; do
+    link_file "$file" "$TARGET_DIR/$name/$(basename "$file")"
+  done < <(live_files_for "$name")
+  while IFS= read -r unit; do
+    link_file "$unit" "$UNIT_DIR/$(basename "$unit")"
+  done < <(helper_units_for "$name")
+}
+
+helper_timers_for() {
+  helper_units_for "$1" | grep '\.timer$' | xargs -r -n1 basename
+}
+
 unit_name_for() {
   local unit
   unit="$(unit_path_for "$1")" || return 1
@@ -111,10 +151,12 @@ unit_installed() {
   [[ -L "$UNIT_DIR/$unit_name" ]]
 }
 
+# Everything this machine links for the service, so a change to a helper
+# script or timer is reported by relink just like a compose change.
 definition_hash() {
-  local name="$1" unit
-  unit="$(unit_path_for "$name")" || unit=/dev/null
-  cat "$SERVER_DIR/$name/compose.yml" "$unit" 2>/dev/null | sha256sum | cut -d' ' -f1
+  local name="$1"
+  { live_files_for "$name"; unit_path_for "$name" && echo; helper_units_for "$name"; } \
+    | sort | xargs -r -d '\n' cat 2>/dev/null | sha256sum | cut -d' ' -f1
 }
 
 recorded_hash() {
@@ -198,6 +240,7 @@ cmd_enable() {
   fi
 
   mkdir -p "$UNIT_DIR"
+  link_service_files "$name"
   link_file "$unit_path" "$UNIT_DIR/$unit_name"
   systemctl --user daemon-reload
   if systemctl --user enable --now "$unit_name"; then
@@ -207,6 +250,10 @@ cmd_enable() {
     err "$unit_name did not start; see: systemctl --user status $unit_name"
     exit 1
   fi
+  local timer
+  for timer in $(helper_timers_for "$name"); do
+    systemctl --user enable --now "$timer" && ok "Enabled $timer"
+  done
 }
 
 cmd_disable() {
@@ -216,11 +263,15 @@ cmd_disable() {
   local unit_name
   unit_name="$(unit_name_for "$name")" || { err "$name has no unit file in the catalog"; exit 1; }
 
+  local timer unit
+  for timer in $(helper_timers_for "$name"); do
+    systemctl --user disable --now "$timer" 2>/dev/null || true
+  done
   systemctl --user disable --now "$unit_name" 2>/dev/null || true
-  if [[ -L "$UNIT_DIR/$unit_name" ]]; then
-    rm "$UNIT_DIR/$unit_name"
-    systemctl --user daemon-reload
-  fi
+  for unit in "$UNIT_DIR/$unit_name" $(helper_units_for "$name" | xargs -r -n1 basename | sed "s|^|$UNIT_DIR/|"); do
+    [[ -L "$unit" ]] && rm "$unit"
+  done
+  systemctl --user daemon-reload
   ok "$name disabled. ~/Server/$name, its .env and its data were not touched."
 }
 
@@ -267,8 +318,7 @@ cmd_relink() {
     unit_installed "$name" || continue
 
     link_file "$unit_path" "$UNIT_DIR/$unit_name"
-    mkdir -p "$TARGET_DIR/$name"
-    link_file "$SERVER_DIR/$name/compose.yml" "$TARGET_DIR/$name/compose.yml"
+    link_service_files "$name"
     refreshed=$((refreshed + 1))
 
     # Changed since the last time this machine looked, which is what makes a

config/bash/aliases

@@ -22,6 +22,7 @@ alias avante='nvim -c "lua vim.defer_fn(function()require(\"avante.api\").zen_mo
 alias nlc='nvim leetcode.nvim'
 alias clauded='claude --allow-dangerously-skip-permissions'
 alias claudedr='claude --allow-dangerously-skip-permissions --resume'
+alias klaude='CLAUDE_CONFIG_DIR="$HOME/.klaude" claude'
 alias commit-repo='git add -A && git commit -m "Update stuff" && git push'
 
 # Docker Shortcuts
@@ -36,9 +37,9 @@ alias fprint-on='sudo authselect enable-feature with-fingerprint'
 alias fprint-off='sudo authselect disable-feature with-fingerprint'
 
 # File system
-alias ls='eza -lh --group-directories-first --icons'
+alias ls='eza -lh --group-directories-first --icons=auto'
 alias lsa='ls -a'
-alias lt='eza --tree --level=2 --long --icons --git'
+alias lt='eza --tree --level=2 --long --icons=auto --git'
 alias lta='lt -a'
 # Fedora's bat installs /usr/bin/bat; batcat is the Debian name.
 alias ff="fzf --preview 'bat --style=numbers --color=always {}'"

config/bash/shell

@@ -16,8 +16,23 @@ unset PANAMA_ENV
 # Editor used by CLI
 export EDITOR="nvim"
 export SUDO_EDITOR="$EDITOR"
-export SSH_ASKPASS=/usr/bin/ksshaskpass
-export SSH_ASKPASS_REQUIRE=prefer
+# A graphical askpass, and only where there is a session to draw it in.
+# SSH_ASKPASS_REQUIRE=prefer means ssh and git reach for the dialog BEFORE the
+# terminal prompt, so on a headless machine the prompt that would have worked
+# is skipped in favour of one that cannot open: `git push` then fails reading
+# a username rather than asking for it. The desktop keeps the dialog; a
+# server, a serial console and a plain SSH session fall back to the terminal.
+if [ -n "${WAYLAND_DISPLAY:-}${DISPLAY:-}" ] && [ -x /usr/bin/ksshaskpass ]; then
+  export SSH_ASKPASS=/usr/bin/ksshaskpass
+  export SSH_ASKPASS_REQUIRE=prefer
+else
+  # Cleared, not merely left unset. An askpass inherited from a parent that
+  # did have a display -- an agent's shell, a forwarded environment, a login
+  # that predates this guard -- is exactly as unusable here as one this file
+  # set itself, and declining to add it does nothing about a value already in
+  # the environment.
+  unset SSH_ASKPASS SSH_ASKPASS_REQUIRE
+fi
 
 # Define Paths
 export CARGO_PATH="$HOME/.cargo"
@@ -32,8 +47,9 @@ export DOTNETPATH="$HOME/.dotnet/tools"
 # Podman
 # export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
 
-# Set complete path
-export PATH="$HOME/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PANAMA_PATH/bin:$BUN_INSTALL/bin:$CARGO_PATH/bin:$PNPM_HOME/bin:$PYENV_ROOT/bin:$HOME/.rbenv/bin:/usr/lib/ccache/bin/:$GOPATH/bin:$DOTNETPATH"
+# Set complete path. ~/.opencode/bin is OpenCode 2's own installer location
+# and sits ahead of Bun's bin, which can still hold an OpenCode 1 copy.
+export PATH="$HOME/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PANAMA_PATH/bin:$HOME/.opencode/bin:$BUN_INSTALL/bin:$CARGO_PATH/bin:$PNPM_HOME/bin:$PYENV_ROOT/bin:$HOME/.rbenv/bin:/usr/lib/ccache/bin/:$GOPATH/bin:$DOTNETPATH"
 
 # rustup writes this file, and initial-packages installs rustup rather than
 # running rustup-init -- so on a fresh machine it does not exist yet and an

config/dot/hypr/hyprland.lua

@@ -29,3 +29,9 @@ require("autostart")
 
 -- Per-machine overrides, loaded last so it wins. Ships effectively empty.
 require("overrides")
+
+-- T3 Code SnapShots. T3's setup wrote this line and finds it again by its exact
+-- shape (a bare hl.bind to a global), so keep it here and unwrapped. Going
+-- through keybinds.lua's bind() would make T3 add a second copy. Change or
+-- remove it from T3 Code under Settings > SnapShots.
+hl.bind("CTRL + SHIFT + 2", hl.dsp.global("com.t3tools.T3Code:capture-window"))

config/dot/hypr/input.lua

@@ -8,6 +8,14 @@
 local prefs = require("prefs")
 local actions = require("actions")
 
+-- Empty means "let libinput pick per device". Hyprland 0.57 rejects "" for
+-- this option, so empty is expressed by not writing it at all: nil drops the
+-- key from the table below and the option stays at its unset default.
+local scroll_method = prefs.get("scrollMethod", "")
+if scroll_method == "" then
+    scroll_method = nil
+end
+
 hl.config({
     input = {
         kb_layout  = prefs.get("keyboardLayout", "us"),
@@ -52,16 +60,12 @@ hl.config({
         scroll_factor  = prefs.get("scrollFactor", 1.0),
         left_handed    = prefs.get("leftHanded", false),
 
-        -- Empty is a value, not an omission: it is what `hyprctl getoption`
-        -- reports as "[[EMPTY]]" before anything writes the option, and it means
-        -- "let libinput pick per device" -- two fingers on a touchpad, the wheel
-        -- on a mouse. Writing it back explicitly is the same branch a stock
-        -- Hyprland takes, and it keeps the setting reversible: without an empty
-        -- choice in the schema there would be no way back from a scroll method
-        -- once one was picked. Settings offers 2fg / edge / on_button_down /
+        -- Empty (omitted, see above) is what `hyprctl getoption` reports as
+        -- "[[EMPTY]]": libinput picks per device -- two fingers on a touchpad,
+        -- the wheel on a mouse. Settings offers 2fg / edge / on_button_down /
         -- no_scroll alongside it, the four words the option's own description
         -- names (it publishes no map).
-        scroll_method = prefs.get("scrollMethod", ""),
+        scroll_method = scroll_method,
 
         -- Only consulted while scroll_method is on_button_down. 0 means the
         -- device's own middle button.

config/dot/hypr/keybinds.lua

@@ -377,15 +377,18 @@ bind(mod .. " + SHIFT + N", hl.dsp.window.resize({ x = 0, y = step, relative = t
 -- The gesture needs three binds, not two. Tab steps the selection, and the
 -- switch is only COMMITTED when the modifier is released -- which is the sole
 -- way the compositor can tell the gesture is finished. That release bind is on
--- the bare modifier, so it fires on EVERY Super release in the session; the
--- handler returns immediately when no switch is open, which is why this is
--- affordable.
+-- the Super keys themselves, so it fires on EVERY Super release in the
+-- session; the handler returns immediately when no switch is open, which is
+-- why this is affordable.
 --
--- The release bind carries no description on purpose: it is not a shortcut
--- anyone would look up or rebind, and the Shortcuts page lists what it finds.
+-- A modifier-only bind has to name the keysym (Super_L / Super_R). Hyprland
+-- 0.57 rejects a chord that is a bare modifier ("A bind requires a trigger"),
+-- and the error stops this file, costing every bind below it.
 bind(mod .. " + Tab", hl.dsp.exec_cmd(qs("switcher", "next")), { description = "Next window" })
 bind(mod .. " + SHIFT + Tab", hl.dsp.exec_cmd(qs("switcher", "previous")), { description = "Previous window" })
-bind(mod, hl.dsp.exec_cmd(qs("switcher", "commit")), { release = true, description = "Commit window switch" })
+for _, key in ipairs({ "Super_L", "Super_R" }) do
+    bind(mod .. " + " .. key, hl.dsp.exec_cmd(qs("switcher", "commit")), { release = true, description = "Commit window switch" })
+end
 -- Jump back to the previously focused window.
 bind(mod .. " + SHIFT + grave", hl.dsp.focus({ last = true }), { description = "Last window" })
 

config/dot/hypr/monitors.lua

@@ -244,10 +244,20 @@ local shipped_mode = "4500x3000@60"
 local shipped_scale = 1.5
 local shipped_transform = 0
 
--- 10-bit output. 4500x3000@60 at 10bpc is ~24 Gbps, right at the edge of DP 1.4
--- HBR3, so this relies on DSC. If the display fails to light up or falls back to
--- a lower mode, drop this to 8 first.
-local shipped_bitdepth = 10
+-- 8-bit output. 4500x3000@60 at 10bpc is ~24 Gbps, right at the edge of DP 1.4
+-- HBR3 and reliant on DSC, and this panel's link is marginal: every modeset
+-- retrains it and blanks the screen. 8bpc keeps headroom on the link.
+--
+-- Related: directScanoutPolicy is 0 in Panama settings (2026-09-13). With
+-- scanout on, a fullscreen game whose buffer depth differs from the desktop
+-- (games ship both 8- and 10-bit swapchains) makes Hyprland change the output
+-- format, and on amdgpu a format change is a full modeset. Compositing always
+-- keeps the format fixed, so the link never retrains mid-game.
+--
+-- vrrPolicy is also 0 there. VRR on this panel loses sync and blacks out
+-- (seen on GNOME in July 2026 and again here); a 60Hz panel gains little
+-- from it anyway.
+local shipped_bitdepth = 8
 
 -- "auto" = sRGB at 8bpc, wide gamut at 10bpc. Not HDR; see header.
 local shipped_cm = "auto"

config/dot/hypr/rules.lua

@@ -125,6 +125,16 @@ hl.window_rule({
     no_dim    = true,
 })
 
+-- Steam games never get true fullscreen. Everything Hyprland does only for a
+-- real fullscreen window (direct scanout, VRR, tearing, auto HDR) makes this
+-- panel's marginal DP link retrain and blank (2026-09-13). "1 2" keeps the
+-- window maximized internally while the game believes it is fullscreen, which
+-- is what borderless windowed looks like from the game's side.
+hl.window_rule({
+    match = { class = "^steam_app_\\d+$" },
+    fullscreen_state = "1 2",
+})
+
 -- Steam itself is a normal window, but its transient popups are a mess.
 hl.window_rule({
     match = { class = "^steam$", title = "^(Friends List|Steam Settings|Special Offer.*)$" },
@@ -269,16 +279,28 @@ hl.layer_rule({
     ignore_alpha = 0.2,
 })
 
--- Overview, capture and local screen-reading UI dim the desktop behind them.
+-- Overview and local screen-reading UI dim the desktop behind them.
 hl.layer_rule({
     name  = "qs-overlay",
-    match = { namespace = "^qs-(overview|capture|screen-intelligence)$" },
+    match = { namespace = "^qs-(overview|screen-intelligence)$" },
     blur         = true,
     ignore_alpha = 0.4,
     dim_around   = true,
     no_screen_share = true,
 })
 
+-- The screenshot picker paints an opaque frozen frame edge to edge, so blur and
+-- dim would be compositor work on pixels nobody sees. no_anim is load-bearing:
+-- services/Capture.qml unmaps the picker and runs grim 90ms later, and the
+-- 200ms layersOut fade would otherwise put the half-faded picker, dimming and
+-- all, into every screenshot.
+hl.layer_rule({
+    name  = "qs-capture",
+    match = { namespace = "^qs-capture$" },
+    no_anim         = true,
+    no_screen_share = true,
+})
+
 -- Notification toasts. Blurred like every other shell surface -- without this
 -- the cards are a near-transparent fill sitting directly on the wallpaper and
 -- read as washed out rather than as glass.

config/dot/quickshell/config/PreferenceSchema.qml

@@ -1805,7 +1805,7 @@ Singleton {
                 "org.mozilla.thunderbird_esr", "com.slack.Slack",
                 "app.bluebubbles.BlueBubbles", "rustdesk",
                 "io.podman_desktop.PodmanDesktop", "com.anthropic.Claude",
-                "codex-desktop", "md.obsidian.Obsidian",
+                "chatgpt", "md.obsidian.Obsidian",
                 "com.obsproject.Studio", "steam"
             ]
         },

config/dot/quickshell/modules/capture/CaptureOverlay.qml

@@ -27,9 +27,13 @@ PanelWindow {
         right: true
     }
 
-    // Fullscreen overlays must not reserve space, or every window on the
-    // workspace gets resized as the picker opens and closes.
-    exclusiveZone: 0
+    // Ignore, and nothing else: reserve no space (or every window on the
+    // workspace resizes as the picker opens and closes) and respect nobody's
+    // (or the window starts under the bar, the full-output freeze frame is
+    // squeezed into a shorter box, and every selection lands one bar-height
+    // off in the real capture). Never pair this with exclusiveZone; see
+    // tests/quickshell/exclusion-idiom-contract.
+    exclusionMode: ExclusionMode.Ignore
 
     WlrLayershell.namespace: "qs-capture" // matched by a layerrule in hypr/rules.lua
     WlrLayershell.layer: WlrLayer.Overlay

config/dot/quickshell/modules/cheatsheet/Cheatsheet.qml

@@ -57,7 +57,6 @@ PanelWindow {
 
     anchors { top: true; bottom: true; left: true; right: true }
     color: "transparent"
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
 
     // The `^qs-popover` prefix rule in hypr/rules.lua blurs what is behind

config/dot/quickshell/modules/dock/Dock.qml

@@ -55,7 +55,6 @@ PanelWindow {
     color: "transparent"
 
     // A dock that reserved space would not be intellihiding.
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
 
     // Matched by the `qs-dock` layer rule in hypr/rules.lua — do not rename.

config/dot/quickshell/modules/dock/DockPickOverlay.qml

@@ -40,7 +40,6 @@ PanelWindow {
 
     anchors { top: true; bottom: true; left: true; right: true }
     color: "transparent"
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
 
     // Blurred by the `^qs-popover` rule in hypr/rules.lua; the scrim is painted

config/dot/quickshell/modules/notifications/VisualBell.qml

@@ -44,7 +44,6 @@ PanelWindow {
     // Reserve nothing and respect nothing: the glow is drawn over the whole
     // output including under the bar and the dock, which is what makes it
     // visible from wherever the eyes happen to be.
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
     color: "transparent"
 

config/dot/quickshell/modules/osd/Osd.qml

@@ -29,7 +29,6 @@ PanelWindow {
     screen: root.modelData
     anchors.bottom: true
     margins.bottom: Theme.dockIconSize + Theme.dockPadding * 2 + Theme.barGap * 3
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
     implicitWidth: root.desiredWidth
     implicitHeight: 64

config/dot/quickshell/modules/overview/Overview.qml

@@ -22,7 +22,6 @@ PanelWindow {
     }
 
     color: "transparent"
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
 
     // Matched by the `qs-overlay` layer rule in hypr/rules.lua — do not rename.

config/dot/quickshell/modules/quicksettings/BrightnessControl.qml

@@ -34,11 +34,23 @@ Item {
     visible: root.rowCount > 0
     implicitHeight: rows.implicitHeight
 
-    // Probing I2C takes on the order of a second, so it waits until the panel
-    // is actually on screen rather than running at shell startup. Monitors do
-    // not come and go, so once is enough.
-    onVisibleChanged: if (visible && !Brightness.scanned) Brightness.refresh()
-    Component.onCompleted: if (root.visible && !Brightness.scanned) Brightness.refresh()
+    // Probing I2C takes on the order of a second, so it waits until the quick
+    // settings panel is actually on screen rather than running at shell
+    // startup. Monitors do not come and go, so once is enough.
+    //
+    // The trigger is the panel opening, not this item becoming visible: on a
+    // machine with no backlight, `visible` stays false until a scan has found
+    // a DDC display, so keying the scan off it would never scan at all.
+    Connections {
+        target: ShellState
+        function onQuickSettingsOpenChanged(): void { root.scanIfOpen(); }
+    }
+    Component.onCompleted: root.scanIfOpen()
+
+    function scanIfOpen(): void {
+        if (ShellState.quickSettingsOpen && !Brightness.scanned)
+            Brightness.refresh();
+    }
 
     // `-m` is the machine-readable form: name,class,current,percent,max
     Process {

config/dot/quickshell/modules/settings/DisplayIdentify.qml

@@ -22,7 +22,6 @@ Variants {
         implicitWidth: 260
         implicitHeight: 172
         color: "transparent"
-        exclusiveZone: 0
         exclusionMode: ExclusionMode.Ignore
         mask: Region {}
 

config/dot/quickshell/modules/settings/README.md

@@ -383,6 +383,13 @@ Lua-configured Hyprland, prints the refusal to stdout, and exits 0. `eval` exits
 0 on syntax and runtime errors too. The only trustworthy signal that a write
 landed is reading the value back.
 
+**`exclusionMode: ExclusionMode.Ignore` goes alone.** Quickshell's `exclusiveZone`
+setter flips `exclusionMode` back to `Normal` as a side effect, so a window that
+declares both is at the mercy of which property the QML engine applies last.
+The 2026-09-14 Qt update changed that order and every full-screen overlay slid
+under the bar. `tests/quickshell/exclusion-idiom-contract` fails any file that
+pairs them.
+
 **The Settings window is tiled.** `implicitWidth` is a hint; the layout decides,
 and it ranges from a half-screen split to the full display. `SliderRow` stacks
 its control under the label below 520px. Test narrow.

config/dot/quickshell/modules/welcome/Welcome.qml

@@ -78,7 +78,6 @@ PanelWindow {
 
     anchors { top: true; bottom: true; left: true; right: true }
     color: "transparent"
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
 
     WlrLayershell.namespace: "qs-popover-welcome"

config/dot/quickshell/services/Capture.qml

@@ -341,7 +341,10 @@ esac'
         property bool record: false
         property bool intelligence: false
         // A handful of frames at 60Hz, enough for the compositor to recomposite
-        // the output without the overlay on it.
+        // the output without the overlay on it. Only enough because the
+        // qs-capture layer rule in hypr/rules.lua sets no_anim: with the
+        // default 200ms layersOut fade the half-faded picker would still be
+        // on screen when grim reads it.
         interval: 90
         onTriggered: {
             const r = commitDelay.rect;

config/dot/quickshell/services/Keybinds.qml

@@ -487,11 +487,11 @@ Singleton {
             if ((bind.modmask & modifier.bit) !== 0)
                 parts.push(modifier.name.toUpperCase());
         }
-        // A modifier-only bind has no key at all -- the window switcher commits
-        // on Super RELEASE. Appending an empty string left a dangling "SUPER + "
-        // that matched neither the chord hypr/keybinds.lua binds nor the one an
-        // override would be keyed by, so that bind could never be rebound and
-        // never found its category.
+        // Guard against a bind with no key. Hyprland 0.57 refuses to create
+        // one (the switcher's Super-release commit now names Super_L/Super_R),
+        // but appending an empty string would leave a dangling "SUPER + " that
+        // matches neither the chord hypr/keybinds.lua binds nor the one an
+        // override would be keyed by.
         const key = String(bind.key ?? "");
         if (key !== "")
             parts.push(key);

config/zsh/.zshrc

@@ -0,0 +1,55 @@
+#!/usr/bin/env zsh
+# The Mac mini's shell. macOS stays on zsh: its bash is 3.2, and
+# config/bash/shell leans on Linux-only tools (stat -c, ksshaskpass, the nvm
+# profile script), so this is a small zsh counterpart rather than a port.
+# Linked to ~/.zshrc. Homebrew's PATH comes from ~/.zprofile (brew shellenv).
+export PANAMA_PATH="$HOME/.local/share/Panama"
+
+export EDITOR="nvim"
+export SUDO_EDITOR="$EDITOR"
+
+export BUN_INSTALL="$HOME/.bun"
+# ~/.opencode/bin is OpenCode 2's own installer location, ahead of Bun's bin.
+export PATH="$HOME/.local/bin:$HOME/.opencode/bin:$BUN_INSTALL/bin:$PANAMA_PATH/bin:$PATH"
+
+# History shared across sessions, so tmux panes see each other's commands.
+HISTFILE="$HOME/.zsh_history"
+HISTSIZE=50000
+SAVEHIST=50000
+setopt SHARE_HISTORY HIST_IGNORE_DUPS HIST_IGNORE_SPACE
+
+# Emacs keys, like bash's readline: ctrl-a/e/r work as they do on Linux.
+bindkey -e
+
+autoload -Uz compinit && compinit
+[ -s "$BUN_INSTALL/_bun" ] && source "$BUN_INSTALL/_bun"
+
+# Same SSH behaviour as config/bash/shell: an SSH login lands in a tmux
+# session that survives a dropped connection. PANAMA_SSH_TMUX=off opts out.
+if [[ -n "$SSH_CONNECTION" && -z "$TMUX" && -o interactive &&
+  "${PANAMA_SSH_TMUX:-on}" != "off" ]] && command -v tmux >/dev/null 2>&1; then
+  exec tmux new-session -A -s main
+fi
+
+alias c="clear"
+alias n="nvim"
+alias n.="nvim ."
+alias ..="cd .."
+alias ...="cd ../.."
+alias ....="cd ../../.."
+alias letscode="ranger ~/Documents/Documents/Code"
+alias commit-repo='git add -A && git commit -m "Update stuff" && git push'
+alias ls='eza -lh --group-directories-first --icons=auto'
+alias lsa='ls -a'
+alias lt='eza --tree --level=2 --long --icons=auto --git'
+alias ff="fzf --preview 'bat --style=numbers --color=always {}'"
+compress() { tar -czf "${1%/}.tar.gz" "${1%/}"; }
+alias decompress="tar -xzf"
+
+# Xcode cannot sign from an SSH session while the login keychain is locked
+# (errSecInternalComponent). Run this once per session before a build.
+alias unlock-keychain='security unlock-keychain ~/Library/Keychains/login.keychain-db'
+
+eval "$(zoxide init zsh)"
+source <(fzf --zsh)
+eval "$(oh-my-posh init zsh --config "$PANAMA_PATH/config/dot/ohmyposh/gib.omp.json")"

install

@@ -71,7 +71,7 @@ source "$PANAMA_PATH/bin/ascii"
 # time. On an upgrade it is worth running only when its package lists or
 # reviewed installer trust inputs changed, so this hashes them and remembers
 # the result. The framed, sorted stream includes top-level package files, the
-# package-stage adapter, the provenance helper, and regular provenance files;
+# package-stage adapter, every helper it sources, and regular provenance files;
 # both relative paths and bytes are part of the state.
 #
 # A content hash rather than a git range, because Panama is developed in place:
@@ -93,7 +93,10 @@ hash_packages() {
 
   for fixed_input in \
     "$PANAMA_PATH/setup/scripts/install-packages" \
-    "$PANAMA_PATH/setup/lib/artifact-provenance"; do
+    "$PANAMA_PATH/setup/lib/artifact-provenance" \
+    "$PANAMA_PATH/setup/lib/chatgpt-package" \
+    "$PANAMA_PATH/setup/lib/extras-catalog" \
+    "$PANAMA_PATH/setup/lib/machine-role"; do
     [[ -f "$fixed_input" && ! -L "$fixed_input" && -r "$fixed_input" ]] || return 1
   done
 
@@ -102,7 +105,10 @@ hash_packages() {
       find "$PANAMA_PATH/setup/packages" -maxdepth 1 -type f -print0 || exit 1
       printf '%s\0' \
         "$PANAMA_PATH/setup/scripts/install-packages" \
-        "$PANAMA_PATH/setup/lib/artifact-provenance" || exit 1
+        "$PANAMA_PATH/setup/lib/artifact-provenance" \
+        "$PANAMA_PATH/setup/lib/chatgpt-package" \
+        "$PANAMA_PATH/setup/lib/extras-catalog" \
+        "$PANAMA_PATH/setup/lib/machine-role" || exit 1
       find "$PANAMA_PATH/setup/provenance" -type f -print0 || exit 1
     } | LC_ALL=C sort -z | while IFS= read -r -d '' file; do
       relative="${file#"$PANAMA_PATH"/}"
@@ -116,12 +122,11 @@ hash_packages() {
 }
 
 packages_needed() {
-  local current_hash recorded_hash
+  local current_hash="$1" recorded_hash
 
   (( FORCE_PACKAGES )) && return 0
   (( UPGRADE )) || return 0
   [[ -r "$PACKAGES_HASH" ]] || return 0
-  current_hash="$(hash_packages)" || return 2
   recorded_hash="$(cat "$PACKAGES_HASH")" || return 2
   [[ "$current_hash" != "$recorded_hash" ]]
 }
@@ -130,10 +135,12 @@ packages_needed() {
 # documents for its markers: a step that did not complete has not happened, and
 # recording it as done hides it forever.
 record_packages_hash() {
-  local temporary_hash
+  local starting_hash="$1" current_hash temporary_hash
+  current_hash="$(hash_packages)" || return 1
+  [[ "$current_hash" == "$starting_hash" ]] || return 1
   mkdir -p "$STATE_DIR"
   temporary_hash="$(mktemp "$STATE_DIR/.packages-hash.XXXXXX")" || return 1
-  if hash_packages >"$temporary_hash"; then
+  if printf '%s\n' "$starting_hash" >"$temporary_hash"; then
     mv -f -- "$temporary_hash" "$PACKAGES_HASH"
   else
     rm -f -- "$temporary_hash"
@@ -142,20 +149,37 @@ record_packages_hash() {
 }
 
 # Repository trust is checked before the installer can reach its bootstrap DNF.
-# Status 78 is reserved for a trust-root failure and is propagated unchanged so
-# no later stage, especially install-hardware, can invoke DNF with that repo.
+# Status 78 is reserved for a trust-root failure. It suppresses every stage that
+# opens DNF -- install-hardware included, which would otherwise pull drivers
+# through the very repository in doubt -- and the run still exits 78 at the end.
+#
+# It suppresses nothing else. Linking dotfiles, skills and user content reads no
+# repository and installs no package, and a machine whose Terra is in question
+# still wants its configuration. Refusing the safe work because the unsafe work
+# is unavailable does not make the machine safer, it just leaves the machine
+# unconfigured with no way to fix itself. Exiting here instead meant link-skills
+# never ran on a machine whose Terra was merely unadopted, so ~/.claude/skills
+# stayed the whole-directory symlink it had been before skills were linked one
+# by one, and not one shipped skill was reachable.
 TERRA_TRUST_FAILURE_STATUS=78
+DNF_STAGES=(install-packages change-settings install-hardware)
+package_trust_refused=0
+
+stage_opens_dnf() {
+  local candidate="$1" dnf_stage
+  for dnf_stage in "${DNF_STAGES[@]}"; do
+    [[ "$candidate" == "$dnf_stage" ]] && return 0
+  done
+  return 1
+}
+
 trust_preflight="$PANAMA_PATH/setup/scripts/install-packages"
 if [[ ! -x "$trust_preflight" ]]; then
   printf 'install: package repository trust preflight is unavailable\n' >&2
-  exit "$TERRA_TRUST_FAILURE_STATUS"
-fi
-if "$trust_preflight" --trust-preflight; then
-  :
-else
-  trust_status=$?
+  package_trust_refused=1
+elif ! "$trust_preflight" --trust-preflight; then
   printf 'install: package repository trust preflight failed\n' >&2
-  exit "$trust_status"
+  package_trust_refused=1
 fi
 
 # ── The interview ────────────────────────────────────────────────────────────
@@ -175,7 +199,7 @@ fi
 # Gated exactly like the interview itself: under --upgrade no questions are
 # asked, so nothing here is used, and a machine that cannot install gum must
 # not have that stop an upgrade that never needed it.
-if (( ! UPGRADE )); then
+if (( ! UPGRADE && ! package_trust_refused )); then
   bootstrap=()
   command -v gum >/dev/null 2>&1 || bootstrap+=(gum)
   # The probe tools serve only the hardware questions, which a server is never
@@ -319,7 +343,7 @@ fi
 if [[ "$PANAMA_ROLE" == server ]]; then
   STAGES=(install-packages link-dotfiles link-user setup-server link-server setup-identity)
 else
-  STAGES=(install-packages link-dotfiles link-skills link-user change-settings link-vicinae-scripts setup-identity install-hardware)
+  STAGES=(install-packages link-dotfiles link-skills link-user link-mcp change-settings link-vicinae-scripts setup-identity install-hardware)
 fi
 
 # The two an upgrade drops. Both exist only to act on interview answers, and
@@ -342,9 +366,16 @@ for stage in "${STAGES[@]}"; do
   script="$PANAMA_PATH/setup/scripts/$stage"
   [[ -x "$script" ]] || continue
   printf '\n=== %s ===\n' "$stage"
+  if (( package_trust_refused )) && stage_opens_dnf "$stage"; then
+    echo "Skipped: the package repository trust check refused package work."
+    continue
+  fi
   if [[ "$stage" == install-packages ]]; then
     package_state_status=0
-    packages_needed || package_state_status=$?
+    package_start_hash="$(hash_packages)" || package_state_status=2
+    if (( package_state_status == 0 )); then
+      packages_needed "$package_start_hash" || package_state_status=$?
+    fi
     if (( package_state_status == 1 )); then
       echo "The package lists have not changed since the last run; skipping."
       echo "Run with --packages to install them anyway."
@@ -357,16 +388,19 @@ for stage in "${STAGES[@]}"; do
   fi
   if "$script"; then
     if [[ "$stage" == install-packages ]]; then
-      if ! record_packages_hash; then
+      if ! record_packages_hash "$package_start_hash"; then
         failed+=("$stage")
         printf '!!! %s could not record its tracked installation inputs\n' "$stage" >&2
       fi
     fi
   else
     stage_status=$?
+    # A configuration change between the preflight and this stage. Suppress the
+    # remaining DNF stages, keep the safe ones, and carry the status to the end.
     if [[ "$stage" == install-packages && "$stage_status" -eq "$TERRA_TRUST_FAILURE_STATUS" ]]; then
       printf '!!! %s stopped on an untrusted package repository\n' "$stage" >&2
-      exit "$stage_status"
+      package_trust_refused=1
+      continue
     fi
     failed+=("$stage")
     printf '!!! %s failed\n' "$stage" >&2
@@ -392,8 +426,16 @@ done
 # written for -- and baselining would skip every one of them forever. Every
 # migration is self-guarding and a no-op where it does not apply, so running
 # them is the safe direction.
+#
+# Held back when package work was refused. A migration is free to run a DNF
+# transaction -- the ChatGPT package replacement does exactly that -- so the
+# repositories have to be trustworthy before any of them is allowed to run.
+# They are not marked applied either, so the next run still has them pending.
 migrate="$PANAMA_PATH/bin/panama-migrate"
-if [[ -x "$migrate" ]]; then
+if (( package_trust_refused )) && [[ -x "$migrate" ]]; then
+  printf '\n=== migrations ===\n'
+  echo "Skipped: the package repository trust check refused package work."
+elif [[ -x "$migrate" ]]; then
   printf '\n=== migrations ===\n'
   if (( UPGRADE )) || [[ -d "$STATE_DIR/migrations" ]]; then
     "$migrate" run || failed+=(migrations)
@@ -441,6 +483,17 @@ else
   retry='./install'
 fi
 
+# Reported last and on its own, because it is not an ordinary stage failure:
+# everything safe did run, and what did not run is named rather than buried in a
+# list. The exit status stays 78 so a caller can still tell the two apart.
+if (( package_trust_refused )); then
+  printf 'Package work was refused: the Terra repository configuration on this\n' >&2
+  printf 'machine is not one Panama can verify. Skipped: %s\n' "${DNF_STAGES[*]}" >&2
+  printf 'Everything that touches no repository was still applied.\n' >&2
+  printf 'Inspect it with: panama diagnose\n' >&2
+  exit "$TERRA_TRUST_FAILURE_STATUS"
+fi
+
 if (( ${#failed[@]} == 0 )); then
   if (( UPGRADE )); then
     echo "Panama is up to date."

migrations/1787804505.sh

@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# replace the community ChatGPT Desktop build with the official OpenAI package
+#
+# `panama app chatgpt-desktop` used to build a community wrapper (codex-desktop)
+# from the upstream macOS disk image, complete with a local rebuild daemon.
+# OpenAI ships an official Linux RPM now, and the installer takes that instead;
+# this repairs machines still carrying the community build. The official
+# package goes on before the community one comes off, so a failure part-way
+# leaves the machine with an app, never without one.
+#
+# Rules, because the runner cannot enforce them:
+#
+#   * Safe to run twice. The marker records success, not intent.
+#   * Tolerant of the repair already being correct -- the user may have fixed
+#     it by hand, or a later ./install may have put it back.
+#   * Root work goes through `panama-sudo --reason "..."`, never bare sudo,
+#     so the password prompt names the repair.
+#   * Exit non-zero to be retried at the next login. Exit zero only when the
+#     machine is genuinely in the state this describes.
+
+set -euo pipefail
+
+PANAMA_PATH="${PANAMA_PATH:-$HOME/.local/share/Panama}"
+
+# Machines that never had the community build are already correct. The official
+# app is the installer's job, not this one's.
+rpm -q codex-desktop >/dev/null 2>&1 || exit 0
+
+# The same verified repository the installer establishes: the pinned signing
+# key, then a repository that names it, so dnf checks OpenAI's signature before
+# root installs anything. See setup/lib/chatgpt-package.
+# shellcheck source=../setup/lib/chatgpt-package
+source "$PANAMA_PATH/setup/lib/chatgpt-package"
+
+sudo_cmd=(sudo)
+if [[ -t 0 && -x "$PANAMA_PATH/bin/panama-sudo" ]]; then
+    sudo_cmd=(
+        "$PANAMA_PATH/bin/panama-sudo" --reason
+        "Replacing the community-built ChatGPT Desktop (codex-desktop) with the official OpenAI package"
+        --
+    )
+fi
+
+# The official package first, so the machine is never left without one.
+if ! rpm -q chatgpt >/dev/null 2>&1; then
+    chatgpt_install_repository "${sudo_cmd[@]}"
+    "${sudo_cmd[@]}" dnf install -y chatgpt
+fi
+
+# The community package's updater is a user unit; stop it before dnf removes
+# the unit file out from under it. Removal also takes the app in /opt, both
+# binaries, and the polkit policy the local rebuilds needed.
+systemctl --user disable --now codex-update-manager.service 2>/dev/null || true
+"${sudo_cmd[@]}" dnf remove -y codex-desktop
+systemctl --user daemon-reload 2>/dev/null || true
+
+# The rebuild state the updater kept; the official package needs none of it.
+rm -rf "${XDG_CACHE_HOME:-$HOME/.cache}/codex-update-manager" \
+       "${XDG_CACHE_HOME:-$HOME/.cache}/codex-runtimes" \
+       "${XDG_CONFIG_HOME:-$HOME/.config}/codex-update-manager" \
+       "${XDG_STATE_HOME:-$HOME/.local/state}/codex-update-manager"
+
+echo "Replaced the community codex-desktop build with the official chatgpt package."

server/README.md

@@ -6,18 +6,44 @@ screen, plus rootless podman compose services managed as systemd user units.
 
 ```
 server/
-  containers/     One directory per service: compose.yml, the
-                  podman-<name>.service user unit, and an .env.example
-                  naming what the service needs told
+  vps/            The catalog panama-server installs from, one directory
+                  per service: compose.yml, the podman-<name>.service user
+                  unit, and an .env.example naming what the service needs told.
+                  Loose files beside the directories are the VPS's own
+                  host tooling (see "Host tooling" below)
+  home/           The home server's services in the same shape (see below)
   scripts/        update-containers, the nightly image updater
   systemd/        The units behind it, linked by setup-server
 ```
 
+## The home server
+
+`server/home/` mirrors `~/Server` on the home server, where the services still
+run under Docker. Each directory has the same three files as the VPS catalog,
+and the unit is drafted for the eventual move to rootless podman. It has never
+run, and the composes still use Docker-only features (host networking, the
+GPU, the Docker socket) that need converting first. Nothing here is
+installable: panama-server reads `server/vps/` only, so a service moves into
+the catalog when it actually moves to podman.
+
+Paths keep their nesting (`server/home/Agents/Gitea` is `~/Server/Agents/Gitea`).
+`Media/compact-tv/` is the nightly TV re-encode job that runs beside the Media
+stack, with its own README.
+Three services are checkouts of upstream repositories. For those, only the
+local files are kept, and an `UPSTREAM` file names the repository and the exact
+commit the checkout sits at: `PostHog`, `Sentry` and `Plausible`.
+
+Both trees are snapshots of the live machines with every secret replaced by a
+`${VAR}` named in `.env.example`, or by `CHANGE_ME` in files that are not
+compose files. The live servers still carry some secrets inline, so a live
+compose and its copy here can differ on those lines alone.
+
 ## The contract with ~/Server
 
 The repository carries the definitions; the machine carries the state.
 `panama server enable <Name>` creates `~/Server/<Name>/` as a **real
-directory** and symlinks only `compose.yml` into it; the unit symlinks into
+directory** and symlinks the service's visible top-level files into it:
+`compose.yml`, plus any helper script or README. The units symlink into
 `~/.config/systemd/user/`. The `.env` (seeded once from `.env.example`, then
 yours) and the bind-mounted `data/` live in `~/Server/<Name>/` and are never
 inside the checkout — so nothing git can do, `clean -fdx` included, can reach
@@ -35,12 +61,18 @@ is what `panama server enable` refuses to start on.
 ## Conventions
 
 - Service directories are TitleCase, matching `~/Server` as it has always
-  looked: `server/containers/Gitea` becomes `~/Server/Gitea`.
+  looked: `server/vps/Gitea` becomes `~/Server/Gitea`.
 - Bind mounts live under `./data/` in new services. Imported services keep
   the volume path they were running with until their cutover, because the
   repo copy must render identically to the live one — renames happen with
   the service stopped, not in git.
-- Only 80, 443 and 81 are open (setup-server's doing). Everything else is
+- Only 80, 443 and 81 are open, and `setup-server` opens each one only where
+  nothing already grants it. A port reachable through an enabled service or an
+  accepting rich rule — a CDN-scoped grant, say — is left exactly as it is,
+  because turning a narrow grant into a public one during an unattended
+  `panama update` is not a decision an installer gets to make. A machine that
+  manages its own firewall opts out for good with
+  `echo manual > ~/.local/state/panama/firewall`. Everything else is
   reverse-proxied over the `nginx-bridge` network by container name, with no
   published ports of its own. A service that truly needs another host port
   says so in a README in its directory, and the port is opened by hand.
@@ -67,11 +99,44 @@ ones — and verify it came back up. One service at a time, quiet hours.
 
 ## Adding a service
 
-Add `server/containers/<Name>/` with the three files, modelled on any
+Add `server/vps/<Name>/` with the three files, modelled on any
 existing service. The unit is the standard oneshot `podman compose up -d`
 shape (`WorkingDirectory=%h/Server/<Name>`); the shape contract pins the
 details. Secrets go in `.env.example` as `KEY=CHANGE_ME` lines.
 
+A service that needs a scheduled job of its own carries it beside the
+compose unit: a helper `<job>.service` and a `<job>.timer` with the same
+name, plus the script it runs. Enable links all three and enables the timer.
+Disable stops the timer and unlinks the units. `UseSend` is the example: its
+`usesend-cert-refresh` timer keeps the SMTP proxy's certificate in step with
+Nginx Proxy Manager's.
+
+## Host tooling
+
+Each server's own scripts and runbooks sit as loose files at the top of its
+tree. They are reference copies for rebuilding a machine. Nothing installs
+them.
+
+- `vps/backup`, `vps/server-backup.service`, `vps/server-backup.timer`: the
+  VPS's nightly backup, run as a user unit. It dumps every database into
+  `~/Server/_backup_state/`, then rsyncs `~/Server` to the home server's ZFS
+  pool with `--fake-super`, so subuid ownership survives the trip.
+- `vps/verify-rootless`: the post-reboot health check for the rootless
+  stack. It covers linger, the unprivileged port floor, unit state and boot
+  enablement, postgres, endpoints, and admin ports bound to the WireGuard
+  address.
+- `vps/AGENTS.md`: the VPS runbook, with both public IPs redacted.
+- `home/RESTORE.md`: how to restore the home server's `~/Server` from the
+  ZFS backup.
+- `home/backup` and `home/systemd/`: the home server's backup trigger and its
+  root system units (`server-backup` nightly, `media-backup` weekly). The
+  scripts those units run live in `/usr/local/sbin` on that machine. They are
+  readable by root alone, so this tree has no copy of them.
+
+The VPS units carry `RequiresMountsFor=/home/gib/Media` because that host
+keeps its podman graphroot on an attached volume mounted there. The setting
+is harmless on a machine without the mount.
+
 ## Nightly updates
 
 `server/scripts/update-containers` pulls images and restarts changed

server/containers/Gitea/compose.yml

@@ -1,25 +0,0 @@
-networks:
-  nginx-bridge:
-    external: true
-services:
-  gitea:
-    image: docker.io/gitea/gitea:latest
-    container_name: gitea
-    hostname: gitea
-    domainname: git.gbrown.org
-    networks:
-    - nginx-bridge
-    ports:
-    - '2222:22'
-    environment:
-    - TZ=America/New_York
-    - USER_UID=1001
-    - USER_GID=1003
-    labels:
-      com.centurylinklabs.watchtower.enable: 'true'
-    volumes:
-    - ./volume:/data:Z
-    - /home/gib/Media/gitea-packages:/data/gitea/packages:Z
-    - ./ssh:/data/git/.ssh:Z
-    tty: true
-    restart: unless-stopped

server/containers/Penpot/.env.example

@@ -1,2 +0,0 @@
-PENPOT_SECRET_KEY=CHANGE_ME
-PENPOT_DATABASE_PASSWORD=CHANGE_ME

server/containers/Penpot/compose.yml

@@ -1,120 +0,0 @@
-# Penpot — VPS (ROOTLESS PODMAN). Ported from the home server 2026-08-12.
-#
-# ⚠️ NOT STARTED. No systemd unit is enabled. It has never run on either machine, so
-# there is no data to migrate. Bring it up with:
-#     systemctl --user enable --now podman-penpot.service
-#
-# BEFORE FIRST START, create its role and database on the SHARED postgres (this stack
-# no longer ships its own):
-#     podman exec -i postgresql psql -U npm <<'SQL'
-#     CREATE ROLE penpot LOGIN PASSWORD '<see this service .env on the machine>';
-#     CREATE DATABASE penpot OWNER penpot;
-#     SQL
-# Then add the same lines to PostgreSQL/initdb/00-roles-and-databases.sql.
-#
-# NOTE: the home version used postgres:15-alpine with --data-checksums. The shared
-# instance is postgres 17 without checksums. Penpot does not care, and since there is
-# no existing data this is not a migration — but if you ever DO need checksums, they
-# can only be set at initdb time for the whole cluster.
-#
-# Redis stays local to this stack: Penpot uses it for transient session and
-# notification state only, so it is not worth centralising.
-#
-# Rootless adaptations from the home version:
-#   - Dropped its private postgres:15-alpine; uses the shared instance (AGENTS.md §2).
-#   - PENPOT_DATABASE_URI repointed from penpot-db to the shared postgresql.
-#   - :Z on the assets bind mount. It is shared between frontend and backend, so it
-#     uses LOWERCASE :z -- uppercase would give each container a private MCS category
-#     and the second to start would lock the first out (AGENTS.md §6).
-#   - /etc/localtime mounts removed in favour of TZ (AGENTS.md §8).
-#
-# Registration is disabled by default (PENPOT_FLAGS disable-registration), so create
-# the first account from the backend container before you can log in.
-#
-# NPM: proxy penpot.gbrown.org -> http://penpot-frontend:8080
-
-networks:
-  nginx-bridge:
-    external: true
-
-services:
-  penpot-frontend:
-    image: docker.io/penpotapp/frontend:latest
-    container_name: penpot-frontend
-    hostname: penpot-frontend
-    domainname: penpot.gbrown.org
-    networks: ['nginx-bridge']
-    environment:
-      - TZ=America/New_York
-      - PENPOT_FLAGS=disable-registration enable-login-with-password disable-smtp
-    labels:
-      com.centurylinklabs.watchtower.enable: "true"
-    volumes:
-      - ./volumes/assets:/opt/data/assets:z
-    depends_on:
-      - penpot-backend
-      - penpot-exporter
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-
-  penpot-backend:
-    image: docker.io/penpotapp/backend:latest
-    container_name: penpot-backend
-    hostname: penpot-backend
-    domainname: penpot.gbrown.org
-    networks: ['nginx-bridge']
-    environment:
-      - TZ=America/New_York
-      - PENPOT_FLAGS=disable-registration enable-login-with-password disable-smtp enable-prepl-server
-      - PENPOT_PUBLIC_URI=https://penpot.gbrown.org
-      - PENPOT_SECRET_KEY=${PENPOT_SECRET_KEY}
-      # Shared postgres, not a private container
-      - PENPOT_DATABASE_URI=postgresql://postgresql/penpot
-      - PENPOT_DATABASE_USERNAME=penpot
-      - PENPOT_DATABASE_PASSWORD=${PENPOT_DATABASE_PASSWORD}
-      - PENPOT_REDIS_URI=redis://penpot-redis/0
-      - PENPOT_ASSETS_STORAGE_BACKEND=assets-fs
-      - PENPOT_STORAGE_ASSETS_FS_DIRECTORY=/opt/data/assets
-      - PENPOT_TELEMETRY_ENABLED=false
-    labels:
-      com.centurylinklabs.watchtower.enable: "true"
-    volumes:
-      - ./volumes/assets:/opt/data/assets:z
-    depends_on:
-      - penpot-redis
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-
-  penpot-exporter:
-    image: docker.io/penpotapp/exporter:latest
-    container_name: penpot-exporter
-    hostname: penpot-exporter
-    domainname: penpot.gbrown.org
-    networks: ['nginx-bridge']
-    environment:
-      - TZ=America/New_York
-      - PENPOT_PUBLIC_URI=http://penpot-frontend:8080
-      - PENPOT_REDIS_URI=redis://penpot-redis/0
-    labels:
-      com.centurylinklabs.watchtower.enable: "true"
-    depends_on:
-      - penpot-redis
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-
-  penpot-redis:
-    image: docker.io/library/redis:7-alpine
-    container_name: penpot-redis
-    hostname: penpot-redis
-    domainname: penpot.gbrown.org
-    networks: ['nginx-bridge']
-    environment:
-      - TZ=America/New_York
-    labels:
-      com.centurylinklabs.watchtower.enable: "true"
-    tty: true
-    stdin_open: true
-    restart: unless-stopped

server/containers/Rustdesk/.env.example

@@ -1 +0,0 @@
-RUSTDESK_API_RUSTDESK_JWT_KEY=CHANGE_ME

server/containers/UseSend/compose.yml

@@ -1,88 +0,0 @@
-name: usesend
-networks:
-  nginx-bridge:
-    external: true
-services:
-  redis:
-    image: redis:7
-    container_name: usesend-redis
-    hostname: usesend-redis
-    networks:
-    - nginx-bridge
-    env_file: .env
-    labels:
-      com.centurylinklabs.watchtower.enable: 'true'
-    volumes:
-    - ./volumes/redis:/data:Z
-    command:
-    - redis-server
-    - --maxmemory-policy
-    - noeviction
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-  minio:
-    image: minio/minio
-    container_name: usesend-storage
-    hostname: usesend-storage
-    networks:
-    - nginx-bridge
-    env_file: .env
-    environment:
-      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
-      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
-    labels:
-      com.centurylinklabs.watchtower.enable: 'true'
-    volumes:
-    - ./volumes/minio:/data:Z
-    entrypoint: sh
-    command: -c 'mkdir -p /data/unsend && minio server /data --console-address ':9001' --address ':9002''
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-  smtp-server:
-    image: usesend/smtp-proxy:v1.6.6
-    container_name: usesend-smtp-server
-    hostname: usesend-smtp
-    networks:
-    - nginx-bridge
-    env_file: .env
-    environment:
-    - SMTP_AUTH_USERNAME=${SMTP_USER:?err}
-    - USESEND_BASE_URL=${USESEND_URL:?err}
-    labels:
-      com.centurylinklabs.watchtower.enable: 'true'
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-  usesend:
-    image: usesend/usesend:v1.6.6
-    container_name: usesend
-    hostname: usesend
-    domainname: ${USESEND_URL:?err}
-    networks:
-    - nginx-bridge
-    env_file: .env
-    environment:
-    - PORT=${USESEND_PORT:-3000}
-    - DATABASE_URL=${DATABASE_URL:?err}
-    - NEXTAUTH_URL=${USESEND_URL:?err}
-    - NEXTAUTH_SECRET=${NEXTAUTH_SECRET:?err}
-    - AWS_ACCESS_KEY=${AWS_ACCESS_KEY:?err}
-    - AWS_SECRET_KEY=${AWS_SECRET_KEY:?err}
-    - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:?err}
-    - GITHUB_ID=${GITHUB_ID:?err}
-    - GITHUB_SECRET=${GITHUB_SECRET:?err}
-    - REDIS_URL=${REDIS_URL:?err}
-    - NEXT_PUBLIC_IS_CLOUD=${NEXT_PUBLIC_IS_CLOUD:-false}
-    - API_RATE_LIMIT=${API_RATE_LIMIT:-1}
-    - SMTP_HOST=${SMTP_HOST:-smtp.usesend.gbrown.org}
-    - SMTP_USER=${SMTP_USER:-usesend}
-    labels:
-      com.centurylinklabs.watchtower.enable: 'true'
-    depends_on:
-      redis:
-        condition: service_started
-    tty: true
-    stdin_open: true
-    restart: unless-stopped

server/containers/Vaultwarden/.env.example

@@ -1 +0,0 @@
-ADMIN_TOKEN=CHANGE_ME

server/containers/Vaultwarden/compose.yml

@@ -1,55 +0,0 @@
-# Vaultwarden — VPS (ROOTLESS PODMAN) port of the home server's ~/Server/vaultwarden,
-# staged 2026-08-12.
-#
-# ⚠️ STAGED, NOT AUTHORITATIVE. The home instance is still live and still serving
-# vault.gbrown.org. This copy holds a point-in-time snapshot of the SQLite database.
-# DO NOT log into this one and add credentials before cutover -- the two would diverge
-# and one side's changes would be lost.
-#
-# Cutover procedure (do it in this order, it matters):
-#   1. stop vaultwarden on the home server        (clean SQLite, no torn WAL)
-#   2. rsync -a server.gib:~/Server/vaultwarden/volume/ ~/Server/Vaultwarden/volume/
-#   3. systemctl --user restart podman-vaultwarden
-#   4. point vault.gbrown.org at the VPS, add the NPM proxy host
-#   5. verify a login + an item decrypts, THEN leave home stopped
-#
-# The SQLite copy taken while home was running is crash-consistent only. Step 1 is what
-# makes it clean -- don't skip it.
-#
-# Rootless adaptations:
-#   - :Z on the data mount (SELinux Enforcing).
-#   - /etc/localtime mount dropped in favour of TZ (AGENTS.md §8).
-#   - No published port; NPM proxies to http://vaultwarden:80 over nginx-bridge.
-#   - Image starts as root and drops privileges internally, so no userns_mode needed.
-#
-# NOTE: ADMIN_TOKEN is inline here, matching the home server's file verbatim. It grants
-# access to /admin. Worth moving to a mode-600 .env at some point -- compose files get
-# rsynced offsite by ~/Server/backup.
-
-networks:
-  nginx-bridge:
-    external: true
-
-services:
-  vaultwarden:
-    image: docker.io/vaultwarden/server:latest
-    container_name: vaultwarden
-    hostname: vaultwarden
-    domainname: vault.gbrown.org
-    networks:
-      - nginx-bridge
-    environment:
-      - TZ=America/New_York
-      - DOMAIN=https://vault.gbrown.org
-      - LOGIN_RATELIMIT_MAX_BURST=10
-      - LOGIN_RATELIMIT_SECONDS=60
-      - WEB_VAULT_ENABLED=true
-      - SIGNUPS_ALLOWED=false
-      - WEBSOCKET_ENABLED=true
-      - ADMIN_TOKEN=${ADMIN_TOKEN}
-    labels:
-      com.centurylinklabs.watchtower.enable: "true"
-    volumes:
-      - ./volume:/data:Z
-    tty: true
-    restart: unless-stopped

server/home/Agents/Beszel/compose.yml

@@ -0,0 +1,48 @@
+# Beszel AGENT ONLY — the hub now lives on the VPS.
+#
+# The hub (web UI + database) was moved to the VPS on 2026-08-11 and is reachable at
+# https://beszel.gibbyb.com. This host keeps only the agent, which collects metrics and
+# serves them to that hub over the WireGuard link (VPS 192.168.2.2 -> here 192.168.1.2:45876).
+#
+# KEY below is the VPS hub's public key. If the VPS hub is ever rebuilt its keypair
+# changes, and this value must be updated to match:
+#     ssh-keygen -y -f ~/Server/Beszel/volumes/hub/id_ed25519      (run on the VPS)
+#
+# Previous config, including the local hub service, is in compose.yml.bak-2026-08-11.
+#
+# The agent uses network_mode: host — that is the documented setup and is what lets it
+# read real host network interfaces rather than a container's veth. Docker forbids setting
+# hostname/domainname together with host networking, so those keys are absent.
+
+services:
+  beszel-agent:
+    image: henrygd/beszel-agent-nvidia:latest
+    container_name: beszel-agent
+    network_mode: host
+    # Give the agent read access to NVIDIA's management interface so Beszel can
+    # report GPU utilization, temperature, power draw, and VRAM usage.
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [utility]
+    environment:
+      - TZ=America/New_York
+      - PORT=45876
+      - KEY=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDBWNqxKGRltqXJZg+wlDOtICeLo19ftQ6P3P8+uJg8y
+      # /boot is a separate 974M partition (nvme1n1p1), not part of the LUKS
+      # root the agent finds on its own. It filling up is what broke a 1032
+      # package dnf transaction on 2026-08-09, so it is worth alerting on.
+      - EXTRA_FILESYSTEMS=/boot
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/agent:/var/lib/beszel-agent
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /boot:/boot:ro
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped

server/home/Agents/Beszel/podman-agents-beszel.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Agents/Beszel
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Agents/Beszel
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Agents/Gitea/.env.example

@@ -0,0 +1 @@
+GITEA_RUNNER_REGISTRATION_TOKEN=CHANGE_ME

server/home/Agents/Gitea/compose.yml

@@ -0,0 +1,153 @@
+# Gitea Actions runners — 3 parallel runners, each with its own dind sidecar.
+#
+# Each runner gets an isolated Docker daemon rather than the host socket, so a
+# CI job cannot reach the other ~180 containers on this box. Cost: each dind
+# keeps its own image cache under ./volumes/runner-N/dind.
+#
+# The pre-existing host runner (/home/gib/.local/bin/act_runner, name
+# "server-gib") is intentionally left running.
+#
+# Registration token generated with:
+#   docker exec -u git gitea gitea actions generate-runner-token
+# It is reusable — all three runners register with the same one.
+
+networks:
+  nginx-bridge:
+    external: true
+
+x-runner-env: &runner-env
+  TZ: America/New_York
+  GITEA_INSTANCE_URL: https://git.gbrown.org
+  GITEA_RUNNER_REGISTRATION_TOKEN: ${GITEA_RUNNER_REGISTRATION_TOKEN}
+  GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04,build:docker://docker.gitea.com/runner-images:ubuntu-latest"
+  DOCKER_TLS_VERIFY: "1"
+  DOCKER_CERT_PATH: /certs/client
+  CONFIG_FILE: /data/config.yml
+
+x-dind-env: &dind-env
+  TZ: America/New_York
+  DOCKER_TLS_CERTDIR: /certs
+
+services:
+  gitea-runner-1:
+    image: gitea/act_runner:latest
+    container_name: gitea-runner-1
+    hostname: gitea-runner-1
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    environment:
+      <<: *runner-env
+      GITEA_RUNNER_NAME: gitea-runner-1
+      DOCKER_HOST: tcp://gitea-runner-1-dind:2376
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-1/data:/data
+      - ./volumes/runner-1/certs:/certs
+      - /etc/localtime:/etc/localtime:ro
+    depends_on:
+      - gitea-runner-1-dind
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gitea-runner-1-dind:
+    image: docker:dind
+    container_name: gitea-runner-1-dind
+    hostname: gitea-runner-1-dind
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    privileged: true
+    environment:
+      <<: *dind-env
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-1/certs:/certs
+      - ./volumes/runner-1/dind:/var/lib/docker
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gitea-runner-2:
+    image: gitea/act_runner:latest
+    container_name: gitea-runner-2
+    hostname: gitea-runner-2
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    environment:
+      <<: *runner-env
+      GITEA_RUNNER_NAME: gitea-runner-2
+      DOCKER_HOST: tcp://gitea-runner-2-dind:2376
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-2/data:/data
+      - ./volumes/runner-2/certs:/certs
+      - /etc/localtime:/etc/localtime:ro
+    depends_on:
+      - gitea-runner-2-dind
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gitea-runner-2-dind:
+    image: docker:dind
+    container_name: gitea-runner-2-dind
+    hostname: gitea-runner-2-dind
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    privileged: true
+    environment:
+      <<: *dind-env
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-2/certs:/certs
+      - ./volumes/runner-2/dind:/var/lib/docker
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gitea-runner-3:
+    image: gitea/act_runner:latest
+    container_name: gitea-runner-3
+    hostname: gitea-runner-3
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    environment:
+      <<: *runner-env
+      GITEA_RUNNER_NAME: gitea-runner-3
+      DOCKER_HOST: tcp://gitea-runner-3-dind:2376
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-3/data:/data
+      - ./volumes/runner-3/certs:/certs
+      - /etc/localtime:/etc/localtime:ro
+    depends_on:
+      - gitea-runner-3-dind
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gitea-runner-3-dind:
+    image: docker:dind
+    container_name: gitea-runner-3-dind
+    hostname: gitea-runner-3-dind
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    privileged: true
+    environment:
+      <<: *dind-env
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-3/certs:/certs
+      - ./volumes/runner-3/dind:/var/lib/docker
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped

server/home/Agents/Gitea/podman-agents-gitea.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Agents/Gitea
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Agents/Gitea
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Agents/Portainer/compose.yml

@@ -0,0 +1,39 @@
+services:
+  # portainer:
+  #   image: portainer/portainer-ee:latest
+  #   container_name: portainer
+  #   hostname: portainer
+  #   domainname: port.gbrown.org
+  #   networks:
+  #     - nginx-bridge
+  #   labels:
+  #     com.centurylinklabs.watchtower.enable: "true"
+  #   ports:
+  #     - 9443:9443
+  #   volumes:
+  #     - ./volumes/data:/data
+  #     - /var/run/docker.sock:/var/run/docker.sock
+  #   tty: true
+  #   stdin_open: true
+  #   restart: unless-stopped
+
+  portainer-agent:
+    image: portainer/agent:latest
+    container_name: portainer-agent
+    hostname: portainer-agent
+    networks:
+      - nginx-bridge
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    ports: ["9001:9001"]
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - /var/lib/docker/volumes:/var/lib/docker/volumes
+      - /:/host
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+networks:
+  nginx-bridge:
+    external: true

server/home/Agents/Portainer/podman-agents-portainer.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Agents/Portainer
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Agents/Portainer
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Frigate/.env.example

@@ -0,0 +1,2 @@
+RTSP_PASSWORD=CHANGE_ME
+PLUS_API_KEY=CHANGE_ME

server/home/Frigate/compose.yml

@@ -0,0 +1,79 @@
+networks:
+  nginx-bridge:
+    external: true
+services:
+  frigate:
+    image: ghcr.io/blakeblackshear/frigate:0.17.1-tensorrt # For Nvidia GPU
+    container_name: frigate
+    hostname: frigate
+    domainname: frigate.gbrown.org
+    networks: [nginx-bridge]
+    ports:
+      #- "8971:8971"
+      - "5000:5000"
+      #- "8554:8554"
+      #- "8555:8555"
+    # Check host system's actual render/video/plugdev group IDs with 'getent group render', 'getent group video', and 'getent group plugdev'
+    # Must add these exact IDs in container's group_add section or OpenVINO GPU acceleration will fail
+    group_add:
+      - "105" # render
+      - "110" # render
+      - "39"  # video
+      - "46"  # plugdev
+    shm_size: "256mb"
+    env_file:
+      - .env
+    environment:
+      - TZ=America/New_York
+      - PGID=1000
+      - PUID=1000
+      - FRIGATE_RTSP_PASSWORD=${RTSP_PASSWORD}
+      #- HSA_OVERRIDE_GFX_VERSION=11.0.0
+      - PLUS_API_KEY=${PLUS_API_KEY}
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/frigate/config:/config:rw
+      - ./volumes/frigate/data:/usr/local/nginx/conf:rw
+      - /home/gib/Public/Server_Backup/frigate:/media/frigate
+      - type: tmpfs
+        target: /tmp/cache
+        tmpfs:
+          size: 1000000000
+      - /etc/localtime:/etc/localtime:ro
+      # - /dev/bus/usb:/dev/bus/usb # Uncomment for Google Coral USB
+    devices:
+      - /dev/dri:/dev/dri # for intel hwaccel, needs to be updated for your hardware
+      #- /dev/kfd:/dev/kfd
+    stop_grace_period: 30s
+    restart: unless-stopped
+    tty: true
+    security_opt: ['seccomp:unconfined','label:disable','systempaths=unconfined']
+    #privileged: true
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: 1
+              capabilities: [gpu]
+
+  mqtt:
+    image: eclipse-mosquitto:2
+    container_name: mqtt
+    hostname: mqtt
+    domainname: mqtt.gbrown.org
+    networks: [nginx-bridge]
+    ports: ["1883:1883"]
+    #env_file: [.env]
+    environment: [TZ=America/New_York]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/mosquitto/config:/mosquitto/config:rw
+      - ./volumes/mosquitto/data:/mosquitto/data:rw
+      - ./volumes/mosquitto/log:/mosquitto/log:rw
+    tty: true
+    security_opt: ['seccomp:unconfined','label:disable','systempaths=unconfined']
+    restart: unless-stopped

server/home/Frigate/podman-frigate.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Frigate
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Frigate
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Gbrown/.env.example

@@ -0,0 +1,16 @@
+AGENT_INGEST_SECRET=CHANGE_ME
+INSTANCE_NAME=CHANGE_ME
+INSTANCE_SECRET=CHANGE_ME
+NEXT_PUBLIC_CONVEX_URL=CHANGE_ME
+NEXT_PUBLIC_PLAUSIBLE_SRC=CHANGE_ME
+NEXT_PUBLIC_POSTHOG_HOST=CHANGE_ME
+NEXT_PUBLIC_POSTHOG_KEY=CHANGE_ME
+NEXT_PUBLIC_SENTRY_DSN=CHANGE_ME
+NEXT_PUBLIC_SENTRY_ORG=CHANGE_ME
+NEXT_PUBLIC_SENTRY_PROJECT_NAME=CHANGE_ME
+NEXT_PUBLIC_SENTRY_URL=CHANGE_ME
+NEXT_PUBLIC_SITE_URL=CHANGE_ME
+SENTRY_AUTH_TOKEN=CHANGE_ME
+UNIFI_API_KEY=CHANGE_ME
+UNIFI_CA=CHANGE_ME
+UNIFI_URL=CHANGE_ME

server/home/Gbrown/compose.yml

@@ -0,0 +1,219 @@
+# Production stack on the Home Server (server.gib, rootful Docker), deployed
+# as ~/Server/Gbrown/compose.yml. This file is the versioned source: edit it
+# here and copy it over, never the other way round.
+#
+# ~/Server/Gbrown holds only this file, .env and data/. Every bind mount lives
+# under data/ (data/convex, data/gibtv/channel, data/gibtv/nginx.conf, copied
+# from apps/gibtv/nginx.conf). No source or build context lives on the server:
+# app images come from the Gitea registry. CI redeploys at once through the
+# forced-command key behind ~/Server/ci-deploy.sh (`docker compose pull && up -d`).
+#
+# Secrets come from ~/Server/Gbrown/.env (mode 600), written from Infisical
+# `prod` by scripts/write-prod-env. Never commit or print it.
+#
+# NPM reaches everything over nginx-bridge, so nothing publishes a host port:
+#   api.gbrown.org              -> http://gbrown-backend:3210
+#   convex.gbrown.org           -> http://gbrown-backend:3211
+#   dashboard.gbrown.org        -> http://gbrown-dashboard:6791
+#   gbrown.org (host 119)       -> http://gbrown-next:3000   (at cutover)
+#   tv.gbrown.org               -> http://gbrown-gibtv-web:80 (phase 8)
+#
+# Watchtower (label-enabled, 04:00) updates the app images. The Convex backend,
+# its dashboard and the socket proxy are pinned and opt out: a backend upgrade
+# migrates its data, so it happens on purpose, not overnight.
+#
+# The `tools` profile keeps the one-off normalize pass out of a plain `up -d`
+# and `pull`. Run it with `docker compose run --rm gbrown-gibtv-normalize`.
+name: gbrown
+
+networks:
+  nginx-bridge:
+    external: true
+  # The agent and its socket proxy only. No route out.
+  agent-socket:
+    internal: true
+
+volumes:
+  # The live HLS window is about 10 MB and rewritten every 6 s. Keep it in RAM.
+  gibtv-hls:
+    driver_opts:
+      type: tmpfs
+      device: tmpfs
+      o: size=64m,uid=1000,gid=1000
+
+services:
+  gbrown-backend:
+    # Pinned to the build the VPS Convex stacks run (bumped 2026-10-28). Bump on purpose.
+    image: ghcr.io/convex/convex-backend@sha256:207cf95def42f01614b6e3af0af5c20d0dcaa5ff2adf258ee7bc9555576644fa
+    container_name: gbrown-backend
+    hostname: gbrown-backend
+    networks: ['nginx-bridge']
+    volumes: ['./data/convex:/convex/data']
+    # No TZ: the backend panics unless it runs in UTC.
+    environment:
+      - INSTANCE_NAME=${INSTANCE_NAME:?}
+      - INSTANCE_SECRET=${INSTANCE_SECRET:?}
+      - CONVEX_CLOUD_ORIGIN=https://api.gbrown.org
+      - CONVEX_SITE_ORIGIN=https://convex.gbrown.org
+      - DISABLE_BEACON=true
+      - REDACT_LOGS_TO_CLIENT=true
+    labels: ['com.centurylinklabs.watchtower.enable=false']
+    restart: unless-stopped
+    healthcheck:
+      test: ['CMD', 'curl', '-f', 'http://localhost:3210/version']
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 20s
+    stop_grace_period: 10s
+    stop_signal: SIGINT
+
+  gbrown-dashboard:
+    # Released alongside the backend; pinned with it.
+    image: ghcr.io/convex/convex-dashboard@sha256:a55bf8abca1332f787bc4d4a79850182175027be7b70aa230be4389a1d3268e2
+    container_name: gbrown-dashboard
+    hostname: gbrown-dashboard
+    networks: ['nginx-bridge']
+    environment:
+      # The browser talks to the backend directly, so this is the public URL.
+      - NEXT_PUBLIC_DEPLOYMENT_URL=https://api.gbrown.org
+    depends_on:
+      gbrown-backend:
+        condition: service_healthy
+    labels: ['com.centurylinklabs.watchtower.enable=false']
+    restart: unless-stopped
+    stop_grace_period: 10s
+    stop_signal: SIGINT
+
+  gbrown-next:
+    image: git.gbrown.org/gib/gbrown-next:latest
+    container_name: gbrown-next
+    hostname: gbrown-next
+    networks: ['nginx-bridge']
+    environment:
+      - TZ=America/New_York
+      - NODE_ENV=production
+      - SENTRY_AUTH_TOKEN=${SENTRY_AUTH_TOKEN}
+      - NEXT_PUBLIC_SITE_URL=${NEXT_PUBLIC_SITE_URL}
+      - NEXT_PUBLIC_CONVEX_URL=${NEXT_PUBLIC_CONVEX_URL}
+      - NEXT_PUBLIC_PLAUSIBLE_SRC=${NEXT_PUBLIC_PLAUSIBLE_SRC:-}
+      - NEXT_PUBLIC_POSTHOG_KEY=${NEXT_PUBLIC_POSTHOG_KEY:-}
+      - NEXT_PUBLIC_POSTHOG_HOST=${NEXT_PUBLIC_POSTHOG_HOST:-}
+      - NEXT_PUBLIC_SENTRY_DSN=${NEXT_PUBLIC_SENTRY_DSN}
+      - NEXT_PUBLIC_SENTRY_URL=${NEXT_PUBLIC_SENTRY_URL}
+      - NEXT_PUBLIC_SENTRY_ORG=${NEXT_PUBLIC_SENTRY_ORG}
+      - NEXT_PUBLIC_SENTRY_PROJECT_NAME=${NEXT_PUBLIC_SENTRY_PROJECT_NAME}
+    labels: ['com.centurylinklabs.watchtower.enable=true']
+    restart: unless-stopped
+    healthcheck:
+      test: ['CMD', 'wget', '-q', '-O', '/dev/null', 'http://127.0.0.1:3000/']
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 20s
+
+  # The agent never sees the Docker socket. The proxy answers only
+  # GET /containers/json, /events, /_ping and /version (with or without an API
+  # version prefix), refuses everything else, and only accepts connections from
+  # the gbrown-agent hostname on the internal agent-socket network.
+  # Docker here applies no SELinux labels to containers, so mounts need no :z.
+  gbrown-agent-socket-proxy:
+    image: docker.io/wollomatic/socket-proxy:1.13.1
+    container_name: gbrown-agent-socket-proxy
+    hostname: gbrown-agent-socket-proxy
+    networks: ['agent-socket']
+    command:
+      - '-loglevel=info'
+      - '-listenip=0.0.0.0'
+      - '-allowfrom=gbrown-agent'
+      - '-allowGET=/(v1\.[0-9]+/)?(containers/json|events|_ping|version)'
+      - '-allowhealthcheck'
+      # Exit if the socket breaks (a Docker upgrade) so Docker restarts it.
+      - '-watchdoginterval=3600'
+      - '-stoponwatchdog'
+      - '-shutdowngracetime=5'
+    # nobody, plus gid 963: the host's docker group, which owns the socket.
+    user: '65534:963'
+    volumes: ['/var/run/docker.sock:/var/run/docker.sock:ro']
+    read_only: true
+    mem_limit: 64M
+    cap_drop: [ALL]
+    security_opt: ['no-new-privileges:true']
+    healthcheck:
+      test: ['CMD', './healthcheck']
+      interval: 30s
+      timeout: 5s
+      retries: 2
+    # Pinned on purpose. Watchtower must not swap the gatekeeper on its own.
+    labels: ['com.centurylinklabs.watchtower.enable=false']
+    restart: unless-stopped
+
+  gbrown-agent:
+    image: git.gbrown.org/gib/gbrown-agent:latest
+    container_name: gbrown-agent
+    hostname: gbrown-agent
+    # nginx-bridge to reach gbrown-backend, agent-socket to reach the proxy.
+    networks: ['nginx-bridge', 'agent-socket']
+    environment:
+      - TZ=America/New_York
+      - HOST_ID=home
+      - DOCKER_HOST=tcp://gbrown-agent-socket-proxy:2375
+      # Straight to the backend's HTTP actions port, no proxy in between.
+      - INGEST_URL=http://gbrown-backend:3211/ingest
+      - AGENT_INGEST_SECRET=${AGENT_INGEST_SECRET:?}
+      # The UniFi gateway on the LAN. UNIFI_CA is its own certificate, the only
+      # one the agent trusts there. Leave all three unset to skip the network.
+      - UNIFI_URL=${UNIFI_URL:-}
+      - UNIFI_API_KEY=${UNIFI_API_KEY:-}
+      - UNIFI_CA=${UNIFI_CA:-}
+    volumes: ['/home/gib/Server/Nginx/volumes/data/logs:/npm-logs:ro']
+    depends_on:
+      gbrown-agent-socket-proxy:
+        condition: service_healthy
+    read_only: true
+    cap_drop: [ALL]
+    security_opt: ['no-new-privileges:true']
+    labels: ['com.centurylinklabs.watchtower.enable=true']
+    restart: unless-stopped
+    stop_grace_period: 10s
+
+  # gib.tv (apps/gibtv). Nothing here touches the media library except the
+  # normalize pass, which mounts it read-only. The streamer stream-copies, so
+  # CPU is near zero; each viewer pulls about 1 Mbps of home upload.
+  gbrown-gibtv:
+    image: git.gbrown.org/gib/gbrown-gibtv:latest
+    container_name: gbrown-gibtv
+    hostname: gbrown-gibtv
+    # No network needed: it only writes the HLS window to the shared volume.
+    network_mode: none
+    environment: ['TZ=America/New_York']
+    volumes:
+      - ./data/gibtv/channel:/media/channel:ro
+      - gibtv-hls:/srv/hls
+    cpus: 0.5
+    mem_limit: 256m
+    labels: ['com.centurylinklabs.watchtower.enable=true']
+    restart: unless-stopped
+
+  gbrown-gibtv-web:
+    image: docker.io/library/nginx:alpine
+    container_name: gbrown-gibtv-web
+    hostname: gbrown-gibtv-web
+    networks: ['nginx-bridge']
+    depends_on: [gbrown-gibtv]
+    volumes:
+      - gibtv-hls:/usr/share/nginx/html:ro
+      - ./data/gibtv/nginx.conf:/etc/nginx/conf.d/default.conf:ro
+    labels: ['com.centurylinklabs.watchtower.enable=true']
+    restart: unless-stopped
+
+  # One-off: `docker compose run --rm gbrown-gibtv-normalize`. Writes one
+  # 640x480 h264 file per episode, skips finished files, safe to resume.
+  gbrown-gibtv-normalize:
+    image: git.gbrown.org/gib/gbrown-gibtv:latest
+    profiles: ['tools']
+    command: ['nice', '-n', '19', 'bun', 'src/cli/normalize.ts']
+    network_mode: none
+    volumes:
+      - '/home/gib/Public/TV/Dragnet (1951):/media/source:ro'
+      - ./data/gibtv/channel:/media/channel

server/home/Gbrown/podman-gbrown.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Gbrown
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Gbrown
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Home/compose.yml

@@ -0,0 +1,24 @@
+services:
+  homeassistant:
+    image: homeassistant/home-assistant:latest
+    container_name: home
+    hostname: home
+    domainname: home.gbrown.org
+    network_mode: host
+    cap_add: ['NET_ADMIN','NET_RAW']
+    #networks:
+      #- nginx-bridge
+    #ports:
+      #- "8123:8123"
+    volumes:
+      - ./volume:/config
+      - /run/dbus:/run/dbus:ro
+      - /etc/localtime:/etc/localtime:ro
+    labels:
+      com.centurylinklabs.watchtower.enable: 'true'
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+#networks:
+  #nginx-bridge:
+    #external: true

server/home/Home/podman-home.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Home
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Home
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Immich/.env.example

@@ -0,0 +1,7 @@
+UPLOAD_LOCATION=CHANGE_ME
+DB_DATA_LOCATION=CHANGE_ME
+IMMICH_VERSION=CHANGE_ME
+DB_PASSWORD=CHANGE_ME
+DB_USERNAME=CHANGE_ME
+DB_DATABASE_NAME=CHANGE_ME
+TZ=CHANGE_ME

server/home/Immich/compose.yml

@@ -0,0 +1,86 @@
+name: immich
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  immich-server:
+    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
+    extends:
+      file: hwaccel.transcoding.yml
+      service: nvenc
+    container_name: immich-server
+    hostname: immich-server
+    domainname: immich.gbrown.org
+    networks: ['nginx-bridge']
+    #ports: ['2283:2283']
+    env_file: [.env]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ${UPLOAD_LOCATION}:/data
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    depends_on:
+      - redis
+      - database
+    healthcheck:
+      disable: false
+    privileged: true
+
+  immich-machine-learning:
+    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
+    extends:
+      file: hwaccel.transcoding.yml
+      service: nvenc
+    container_name: immich-machine-learning
+    hostname: immich-machine-learning
+    networks: ['nginx-bridge']
+    #ports: ['3003:3003']
+    env_file: [.env]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes: ['./volumes/model-cache:/cache:z']
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    healthcheck:
+      disable: false
+    privileged: true
+
+  redis:
+    image: docker.io/valkey/valkey:9@sha256:fb8d272e529ea567b9bf1302245796f21a2672b8368ca3fcb938ac334e613c8f
+    container_name: immich-redis
+    hostname: immich-redis
+    networks: ['nginx-bridge']
+    env_file: [.env]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    healthcheck:
+      test: redis-cli ping || exit 1
+
+  database:
+    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
+    container_name: immich-postgres
+    hostname: immich-postgres
+    networks: ['nginx-bridge']
+    #ports: ['5432:5432']
+    env_file: [.env]
+    environment:
+      POSTGRES_PASSWORD: ${DB_PASSWORD}
+      POSTGRES_USER: ${DB_USERNAME}
+      POSTGRES_DB: ${DB_DATABASE_NAME}
+      POSTGRES_INITDB_ARGS: '--data-checksums'
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    shm_size: 128mb

server/home/Immich/podman-immich.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Immich
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Immich
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Media/.env.example

@@ -0,0 +1,9 @@
+PLEX_CLAIM=CHANGE_ME
+PLEX_TOKEN=CHANGE_ME
+JELLYFIN_TOKEN=CHANGE_ME
+OPENVPN_PASSWORD=CHANGE_ME
+QBIT_PASS=CHANGE_ME
+SONARR_API_KEY=CHANGE_ME
+RADARR_API_KEY=CHANGE_ME
+UN_SONARR_0_API_KEY=CHANGE_ME
+UN_RADARR_0_API_KEY=CHANGE_ME

server/home/Media/compact-tv/README.md

@@ -0,0 +1,109 @@
+# compact-tv
+
+Sonarr and Radarr each have exactly four quality profiles. Two of them (**Standard** and
+**Mini**) also have their files shrunk by `compact-tv.py`, which runs every night.
+
+| Profile | Used for | What gets kept |
+|---|---|---|
+| **Mini** | shows I demote hard | TV: 720p at 800 kbps (~0.42 GB per hour). Movies: 720p at 1500 kbps |
+| **Standard** | **every request from other users (Seerr default)**, and anything I demote | TV: 1080p at 1200 kbps (~0.6 GB per hour). Movies: 1080p at 2500 kbps (~1.2 GB per hour) |
+| **HD** | my own library, unchanged | whatever downloads, up to 1080p |
+| **UHD** | only me, through Seerr's 4K servers | 4K |
+
+SD sources keep their resolution, at 600 kbps for TV and 900 kbps for movies. Movies get
+higher rates because dark scenes, grain and action break down first at low bitrates. Nothing is ever
+upscaled. Audio becomes Opus: 96 kbps for stereo, 160 for 5.1, 192 for 7.1. Subtitles,
+chapters and attachments are copied as they are. I picked the rates from sample encodes of
+Shameless S05E06 (VMAF ~91 for Standard, ~86 for Mini).
+
+## How the profiles work
+
+- **Standard and Mini** are copies of HD that upgrade only toward small releases (see below),
+  so Sonarr and Radarr never replace a file with a bigger one.
+- **Small downloads first.** Both profiles prefer small releases when they grab:
+  - The "Small Encode" format (MeGusta, PSA, ELiTE, GalaxyTV/RG, YTS/YIFY) scores +3000.
+  - LQ scores 0. In HD and UHD it scores -10000, which keeps those groups out of HD.
+  - Standard merges HDTV/WEB/Bluray at each resolution, so the format score decides between
+    sources rather than the source type.
+  - Mini merges 720p and 1080p into one group and adds +1000 for 720p.
+  - Many files therefore arrive already small, and the job skips them.
+- **Minimum sizes are shared by every profile.** For 720p I lowered them to 5 MB per minute
+  and for 1080p to 6 MB per minute (8 for Bluray-1080p), so small encodes are not rejected.
+  HD still never takes the LQ groups.
+- **Seerr:**
+  - The "Sonarr - HD" and "Radarr" servers default to Standard. Only my admin account sees
+    the profile picker, so other users always get Standard. I pick HD or UHD per request.
+  - Other users no longer have "Request 4K".
+- **Configarr is opt-in** (`docker compose --profile configarr up configarr`). Its templates
+  would recreate the old TRaSH profiles and size limits.
+- Backups from before all this: `backups/pre-four-profiles-20261028/`. That covers both
+  apps' profiles, custom formats and size definitions, which title was on which profile,
+  and Seerr's settings. `setup-small-profiles.py` reapplies the Standard and Mini settings.
+
+## How the job works
+
+The nightly run (00:30 to 08:00) has two phases.
+
+**1. Search first.** It asks Sonarr and Radarr to search titles on Standard or Mini that it
+hasn't searched before, biggest first: 5 shows and 20 movies a night, so downloads and
+indexer use stay spread out. Standard and Mini upgrade only toward small releases:
+- Every allowed quality sits in one group, so no resolution or source upgrade can happen.
+- An upgrade needs a custom-format score jump of at least 2500, and upgrades stop at 3000.
+  Only gaining "Small Encode" clears that bar.
+
+So a search replaces a big file only with a small encode, never with a big HEVC release.
+After that, the apps' regular RSS checks keep picking up small releases on their own.
+
+**2. Encode what's left.** A title becomes eligible for encoding 7 days after its search,
+and only its files that are still big get encoded. Files that arrive after the title's
+search work differently:
+- This covers new episodes of ongoing shows, and fresh requests.
+- The first night they're seen, they get their own search (one episode or movie at a time).
+- They're encoded only if they're still big 3 days later.
+
+Downloads never wait for any of this, so new episodes and requests land in the library as
+soon as they download. Only encoding waits. For each file, the job encodes to
+`~/.cache/compact-tv/work.mkv` on the NVMe and then checks the result:
+- the duration matches the original
+- the audio and subtitle track counts match
+- the file is smaller
+
+It then renames the new file over the original, keeping the same name with `.mkv`, and asks
+Sonarr or Radarr to rescan. Plex picks the change up through its folder watcher, and the
+episode keeps its Plex ID, so watch history stays. If a small release shows up later, the
+app may still replace an encoded file with it, which costs bandwidth, not GPU.
+
+Other details:
+- Each output carries a `COMPACT_TIER` tag. A file is skipped if re-encoding would save less
+  than 30%. Ruled-out files are cached by size in `state.json`, so they aren't probed every
+  night. Search dates and failures are kept there too.
+- A downloaded HEVC file no bigger than twice the target size counts as done. That is the
+  small release a search brought in (a MeGusta 1080p episode runs about 2.3 Mbps against
+  Standard's 1.2), and it is neither re-encoded nor searched again. The job's own tagged
+  outputs are exempt, so Standard to Mini still works.
+- **Skipped:** HDR and Dolby Vision sources (they need tone mapping), subtitle tracks ffprobe
+  can't identify, and files that have failed twice.
+- Speed is about 0.19 hours of GPU per hour of 1080p video (NVENC is the bottleneck). The job
+  stops starting new encodes at 08:00.
+- `cost-report.py` estimates, per title, the GPU hours the job would need against the size of
+  a fresh small download.
+- A `PAUSE` file in this folder makes the nightly run do nothing.
+
+## Going back up
+
+Moving down works: Standard to Mini re-encodes the Standard copy. Moving up does not
+restore quality. To get quality back, I set the title to HD and search, which redownloads
+it at full size.
+
+## Running it
+
+    ./compact-tv.py --dry-run                      # what tonight would do
+    ./compact-tv.py --search-all --limit 0         # search every unsearched title now, encode nothing
+    touch PAUSE                                    # skip nightly runs until PAUSE is deleted
+    ./compact-tv.py --series "Shameless (US)" --limit 1
+    systemctl status compact-tv.timer compact-tv.service
+    tail -f compact-tv.log
+
+The installed units are copies of `compact-tv.service` and `compact-tv.timer` from this
+folder. `ExecStart` goes through `/usr/bin/python3` because SELinux won't let systemd
+execute a file labeled as home content.

server/home/Media/compact-tv/compact-tv.py

@@ -0,0 +1,348 @@
+#!/usr/bin/env python3
+"""Shrink shows and movies that Sonarr/Radarr have on the "Standard" or "Mini" profile.
+
+Demoting something is just picking one of those profiles. Each night this job first asks
+Sonarr/Radarr to search a few titles it hasn't searched before. Both profiles upgrade only
+toward small releases, so the search swaps whatever big files have a small release for
+bandwidth instead of GPU. A week after a title's search, the job re-encodes whatever is
+still big to HEVC on the GPU (NVENC) at the tier's bitrate, verifies the result, swaps it in
+place of the original, and has Sonarr/Radarr rescan. Plex and Jellyfin pick up changes
+through their own folder watchers.
+
+Moving something back up means switching it to HD and searching, which redownloads it at
+full size. See README.md.
+
+Usage:
+  compact-tv.py [--until HH:MM] [--series TITLE] [--limit N] [--dry-run]
+
+--series matches a show or movie title. Runs as gib (files are gib-owned) from
+compact-tv.service, nightly.
+"""
+import argparse, datetime, fcntl, json, os, pathlib, re, shutil, subprocess, sys, time, urllib.request
+
+HERE = pathlib.Path(__file__).resolve().parent
+STATE = HERE / "state.json"
+LOG = HERE / "compact-tv.log"
+WORK = pathlib.Path.home() / ".cache" / "compact-tv"   # NVMe, so half-done encodes never land in a ZFS snapshot
+VOLUMES = pathlib.Path.home() / "Server/Media/volumes"
+
+# Per app: API port, container path -> host path, and how to list one title's files.
+APPS = {
+    "sonarr": {"port": 8989, "paths": ("/tv/", "/home/gib/Public/TV/"), "titles": "/series",
+               "files": lambda call, t: call(f"/episodefile?seriesId={t['id']}"),
+               "rescan": lambda t: {"name": "RescanSeries", "seriesId": t["id"]}},
+    "radarr": {"port": 7878, "paths": ("/movies/", "/home/gib/Public/Movies/"), "titles": "/movie",
+               "files": lambda call, t: [t["movieFile"]] if t.get("movieFile") else [],
+               "rescan": lambda t: {"name": "RescanMovie", "movieId": t["id"]}},
+}
+
+# Video bitrate (kbps) per output class, and the widest output each tier allows.
+# TV floors come from watching samples: 1200 kbps at 1080p, 800 at 720p, 600 at SD.
+# Movies get roughly double: dark scenes, grain and action break down first at low rates,
+# and a 2-hour movie at 2500 kbps is still ~2.3 GB (a quarter of a typical 1080p file).
+# That also leaves already-small releases (YTS-style, ~1.5-2 GB) under the 30% bar.
+# Nothing is ever upscaled: a 720p or SD source keeps its size and uses that class's rate.
+TIERS = {
+    "Standard": {"max_width": 1920, "kbps": {"sonarr": {"1080": 1200, "720": 1200, "sd": 600},
+                                             "radarr": {"1080": 2500, "720": 1800, "sd": 900}}},
+    "Mini":     {"max_width": 1280, "kbps": {"sonarr": {"1080": 800,  "720": 800,  "sd": 600},
+                                             "radarr": {"1080": 1500, "720": 1500, "sd": 900}}},
+}
+MIN_SAVING = 0.30            # leave a file alone unless re-encoding saves at least this much
+MAX_FAILURES = 2             # after this many failed attempts a file is skipped for good
+
+# Search first, encode later. Standard/Mini upgrade only toward small releases (see
+# setup-small-profiles.py), so asking the *arr to search a title swaps whatever big files
+# have a small release, for bandwidth instead of GPU. Each night searches a few titles not
+# searched before, biggest first, so downloads and indexer use stay spread out. A title's
+# files are only encoded once its search is SEARCH_GRACE_DAYS old.
+#
+# Files that arrive after their title's search (new episodes, fresh requests) were never
+# searched for a small copy, so the job searches that episode or movie on its own first,
+# and encodes it only if it is still big FILE_SEARCH_GRACE_DAYS later. Downloads are never
+# delayed by any of this; only encoding waits.
+SEARCHES_PER_NIGHT = {"sonarr": 5, "radarr": 20}
+SEARCH_GRACE_DAYS = 7
+FILE_SEARCH_GRACE_DAYS = 3        # small encodes usually appear within hours to a day; 3 leaves room
+HDR_TRANSFERS = {"smpte2084", "arib-std-b67"}
+
+
+def log(msg):
+    line = f"{datetime.datetime.now():%Y-%m-%d %H:%M:%S}  {msg}"
+    print(line, flush=True)
+    with open(LOG, "a") as f:
+        f.write(line + "\n")
+
+
+# ── Sonarr / Radarr ─────────────────────────────────────────────────────────
+def arr(app):
+    """Return a call(path, body=None) for the app's v3 API."""
+    key = re.search(r"<ApiKey>([^<]+)", (VOLUMES / app / "config.xml").read_text()).group(1)
+    ip = subprocess.run(["docker", "inspect", app, "--format",
+                         "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"],
+                        capture_output=True, text=True, check=True).stdout.split()[0]
+    base = f"http://{ip}:{APPS[app]['port']}/api/v3"
+
+    def call(path, body=None):
+        req = urllib.request.Request(base + path, headers={"X-Api-Key": key, "Content-Type": "application/json"},
+                                     data=json.dumps(body).encode() if body is not None else None)
+        with urllib.request.urlopen(req, timeout=60) as r:
+            return json.load(r)
+    return call
+
+
+# ── probing and planning ────────────────────────────────────────────────────
+def probe(path):
+    out = subprocess.run(["ffprobe", "-v", "error", "-print_format", "json", "-show_format", "-show_streams", str(path)],
+                         capture_output=True, text=True)
+    return json.loads(out.stdout) if out.returncode == 0 else None
+
+
+def subtitles(info):
+    """Subtitle tracks ffmpeg can carry over. Tracks ffprobe can't identify are dropped."""
+    return [s for s in info["streams"] if s["codec_type"] == "subtitle" and s.get("codec_name")]
+
+
+def audio_kbps(channels):
+    return 96 if channels <= 2 else 160 if channels <= 6 else 192
+
+
+def plan(info, tier, app):
+    """Return (ffmpeg args, estimated output bytes) for one file, or (None, reason) to skip it."""
+    t = TIERS[tier]
+    video = next((s for s in info["streams"] if s["codec_type"] == "video"
+                  and not s.get("disposition", {}).get("attached_pic")), None)
+    if not video:
+        return None, "no video stream"
+    if video.get("color_transfer") in HDR_TRANSFERS or any("Dolby Vision" in str(sd) for sd in video.get("side_data_list", [])):
+        return None, "HDR/Dolby Vision source (not handled yet)"
+    if (info["format"].get("tags") or {}).get("COMPACT_TIER") == tier:
+        return None, f"already {tier}"
+
+    src_w = int(video["width"])
+    out_w = min(src_w, t["max_width"])
+    cls = "1080" if out_w >= 1700 else "720" if out_w >= 1200 else "sd"
+    vkbps = t["kbps"][app][cls]
+    duration = float(info["format"]["duration"])
+    audios = [s for s in info["streams"] if s["codec_type"] == "audio"]
+    est = int((vkbps + sum(audio_kbps(int(a.get("channels") or 2)) for a in audios)) * 1000 / 8 * duration * 1.03)
+    if int(info["format"]["size"]) * (1 - MIN_SAVING) < est:
+        return None, f"already small ({int(info['format']['size']) / 1e9 * 3600 / duration:.2f} GB/h)"
+    # A downloaded HEVC file within 2x the target is a small release that already replaced a
+    # big one. Re-encoding it would cost GPU and quality for little space. Our own outputs
+    # (tagged) still go through, so moving a title from Standard to Mini works.
+    tags = info["format"].get("tags") or {}
+    if video.get("codec_name") == "hevc" and "COMPACT_TIER" not in tags and int(info["format"]["size"]) <= 2 * est:
+        return None, f"small HEVC release ({int(info['format']['size']) / 1e9 * 3600 / duration:.2f} GB/h)"
+
+    vf = []
+    if video.get("field_order") not in (None, "progressive", "unknown"):
+        vf.append("bwdif")
+    if out_w < src_w:
+        vf.append(f"scale={out_w}:-2:flags=lanczos")
+    args = ["-map", f"0:{video['index']}"]
+    if vf:
+        args += ["-filter:v", ",".join(vf)]
+    args += ["-c:v", "hevc_nvenc", "-preset", "p7", "-tune", "hq", "-profile:v", "main10", "-pix_fmt", "p010le",
+             "-rc", "vbr", "-multipass", "fullres", "-spatial-aq", "1", "-temporal-aq", "1", "-rc-lookahead", "32",
+             "-bf", "4", "-b_ref_mode", "middle",
+             "-b:v", f"{vkbps}k", "-maxrate", f"{vkbps * 2}k", "-bufsize", f"{vkbps * 4}k"]
+    for i, a in enumerate(audios):
+        ch = int(a.get("channels") or 2)
+        layout = {1: "mono", 2: "stereo", 6: "5.1", 8: "7.1"}.get(ch)
+        args += ["-map", f"0:{a['index']}", f"-c:a:{i}", "libopus", f"-b:a:{i}", f"{audio_kbps(ch)}k",
+                 f"-filter:a:{i}", f"aformat=channel_layouts={layout or 'stereo'}"]
+    args += ["-mapping_family", "1"]
+    for i, s in enumerate(subtitles(info)):
+        args += ["-map", f"0:{s['index']}", f"-c:s:{i}", "srt" if s["codec_name"] == "mov_text" else "copy"]
+    args += ["-map", "0:t?", "-c:t", "copy", "-metadata", f"COMPACT_TIER={tier}"]
+    return args, est
+
+
+# ── encode, verify, swap ────────────────────────────────────────────────────
+def encode(src, args, tier):
+    WORK.mkdir(parents=True, exist_ok=True)
+    tmp = WORK / "work.mkv"
+    tmp.unlink(missing_ok=True)
+    cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-hwaccel", "cuda", "-i", str(src),
+           *args, "-metadata", f"COMPACT_SOURCE={src.name} ({src.stat().st_size} bytes)", str(tmp)]
+    r = subprocess.run(cmd, capture_output=True, text=True)
+    if r.returncode != 0:
+        tmp.unlink(missing_ok=True)
+        raise RuntimeError("ffmpeg failed: " + r.stderr.strip()[-300:])
+    return tmp
+
+
+def verify(src_info, tmp):
+    new = probe(tmp)
+    if not new:
+        raise RuntimeError("output does not probe")
+    a, b = float(src_info["format"]["duration"]), float(new["format"]["duration"])
+    if abs(a - b) > max(2.0, a * 0.005):
+        raise RuntimeError(f"duration mismatch {a:.1f}s vs {b:.1f}s")
+    counts = lambda i: (sum(1 for s in i["streams"] if s["codec_type"] == "audio"), len(subtitles(i)))
+    if counts(src_info) != counts(new):
+        raise RuntimeError(f"audio/subtitle track counts {counts(src_info)} -> {counts(new)}")
+    if int(new["format"]["size"]) >= int(src_info["format"]["size"]):
+        raise RuntimeError("output is not smaller")
+    return int(new["format"]["size"])
+
+
+def swap(src, tmp):
+    """Copy the encode next to the original, then rename it over the original in one step."""
+    final = src.with_suffix(".mkv")
+    staged = src.with_name(f".{final.name}.compact-tmp")
+    shutil.copyfile(tmp, staged)
+    os.replace(staged, final)
+    if final != src:
+        src.unlink()
+    tmp.unlink(missing_ok=True)
+    return final
+
+
+# ── main ────────────────────────────────────────────────────────────────────
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--until", help="stop starting new encodes after this local time (HH:MM)")
+    ap.add_argument("--series", help="only this series title")
+    ap.add_argument("--limit", type=int, help="encode at most N files")
+    ap.add_argument("--dry-run", action="store_true", help="plan only, change nothing")
+    ap.add_argument("--search-all", action="store_true",
+                    help="search every not-yet-searched title now instead of a few per night")
+    opt = ap.parse_args()
+
+    if (HERE / "PAUSE").exists() and not opt.dry_run:
+        log("PAUSE file present, not encoding anything (delete it to resume)")
+        return
+
+    lock = open(HERE / ".lock", "w")
+    try:
+        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+    except BlockingIOError:
+        sys.exit("another compact-tv run is active")
+
+    deadline = None
+    if opt.until:
+        h, m = map(int, opt.until.split(":"))
+        now = datetime.datetime.now()
+        deadline = now.replace(hour=h, minute=m, second=0, microsecond=0)
+        if deadline <= now:
+            deadline += datetime.timedelta(days=1)
+
+    state = json.loads(STATE.read_text()) if STATE.exists() else {}
+    for k in ("failures", "searched", "skip"):
+        state.setdefault(k, {})
+    save = lambda: STATE.write_text(json.dumps(state, indent=1))
+    today = datetime.date.today()
+    work = []                                  # (app, call, title, tier); TV first, then movies
+    for app, a in APPS.items():
+        call = arr(app)
+        profiles = {p["id"]: p["name"] for p in call("/qualityprofile")}
+        work += [(app, call, t, profiles[t["qualityProfileId"]]) for t in call(a["titles"])
+                 if profiles.get(t["qualityProfileId"]) in TIERS and (not opt.series or t["title"] == opt.series)]
+    log(f"=== run start{' [dry run]' if opt.dry_run else ''}: {len(work)} title(s) on "
+        f"{'/'.join(TIERS)}{f', until {deadline:%H:%M}' if deadline else ''} ===")
+
+    # ── 1. search: ask the *arr for small replacements before spending GPU ──
+    size_of = lambda t: (t.get("statistics") or {}).get("sizeOnDisk") or t.get("sizeOnDisk") or 0
+    for app in APPS:
+        fresh = sorted((w for w in work if w[0] == app and f"{app}:{w[2]['id']}" not in state["searched"]
+                        and size_of(w[2]) > 0), key=lambda w: -size_of(w[2]))
+        fresh = fresh if opt.search_all else fresh[:SEARCHES_PER_NIGHT[app]]
+        for _, call, t, tier in fresh:
+            log(f"  {'would search' if opt.dry_run else 'search'} {t['title']} ({size_of(t) / 1e9:.0f} GB, {tier})")
+        if fresh and not opt.dry_run:
+            call = fresh[0][1]
+            if app == "sonarr":
+                for _, _, t, _ in fresh:
+                    call("/command", {"name": "SeriesSearch", "seriesId": t["id"]})
+            else:
+                call("/command", {"name": "MoviesSearch", "movieIds": [t["id"] for _, _, t, _ in fresh]})
+            for _, _, t, _ in fresh:
+                state["searched"][f"{app}:{t['id']}"] = today.isoformat()
+            save()
+
+    # ── 2. encode what is still big after its search ────────────────────────
+    age = lambda iso: (today - datetime.date.fromisoformat(iso)).days
+
+    def ready(app, ef, title_searched, to_search):
+        """True if this big file has had its chance to be replaced by a small download."""
+        if (ef.get("dateAdded") or "")[:10] <= title_searched:   # was on disk when the title was searched
+            return age(title_searched) >= SEARCH_GRACE_DAYS
+        key = f"{app}:file:{ef['id']}"                            # arrived later: search it on its own first
+        if key not in state["searched"]:
+            to_search.append(ef)
+            return False
+        return age(state["searched"][key]) >= FILE_SEARCH_GRACE_DAYS
+
+    def search_files(app, call, show, files):
+        if opt.dry_run:
+            log(f"  would search {len(files)} newer file(s) of {show['title']}"); return
+        if app == "sonarr":
+            ids = {f["id"] for f in files}
+            eps = [e["id"] for e in call(f"/episode?seriesId={show['id']}") if e.get("episodeFileId") in ids]
+            call("/command", {"name": "EpisodeSearch", "episodeIds": eps})
+        else:
+            call("/command", {"name": "MoviesSearch", "movieIds": [show["id"]]})
+        for f in files:
+            state["searched"][f"{app}:file:{f['id']}"] = today.isoformat()
+        log(f"  search {len(files)} newer file(s) of {show['title']} before encoding"); save()
+
+    done = saved = 0
+    for app, call, show, tier in work:
+        title_searched = state["searched"].get(f"{app}:{show['id']}")
+        if not title_searched:
+            continue                          # its title search comes first, on a later night
+        container, host = APPS[app]["paths"]
+        files = sorted(APPS[app]["files"](call, show), key=lambda f: f["path"])
+        changed, to_search = False, []
+        for ef in files:
+            if deadline and datetime.datetime.now() >= deadline:
+                log("deadline reached, stopping"); break
+            if opt.limit is not None and done >= opt.limit:
+                break
+            src = pathlib.Path(ef["path"].replace(container, host, 1))
+            if state["failures"].get(str(src), 0) >= MAX_FAILURES or not src.exists():
+                continue
+            # Files already ruled out at this tier and size need no re-probe (most of them, most nights).
+            seen = f"{tier}:{src.stat().st_size}"
+            if state["skip"].get(str(src)) == seen:
+                continue
+            info = probe(src)
+            if not info:
+                log(f"  SKIP {src.name}: does not probe"); continue
+            args, est = plan(info, tier, app)
+            if args is None:
+                state["skip"][str(src)] = seen  # reason in `est`; quiet, since most nights most files are done
+                continue
+            if not ready(app, ef, title_searched, to_search):
+                continue
+            old = int(info["format"]["size"])
+            if opt.dry_run:
+                log(f"  would encode {src.name}: {old / 1e6:.0f} MB -> ~{est / 1e6:.0f} MB ({tier})")
+                done += 1; continue
+            t0 = time.monotonic()
+            try:
+                tmp = encode(src, args, tier)
+                new = verify(info, tmp)
+                final = swap(src, tmp)
+            except Exception as e:
+                state["failures"][str(src)] = state["failures"].get(str(src), 0) + 1
+                save()
+                log(f"  FAIL {src.name}: {e}"); continue
+            done += 1; saved += old - new; changed = True
+            log(f"  {final.name}: {old / 1e6:.0f} MB -> {new / 1e6:.0f} MB "
+                f"(-{(1 - new / old) * 100:.0f}%, {time.monotonic() - t0:.0f}s, {tier})")
+        if to_search:
+            search_files(app, call, show, to_search)
+        if changed:
+            call("/command", APPS[app]["rescan"](show))
+        if (deadline and datetime.datetime.now() >= deadline) or (opt.limit is not None and done >= opt.limit):
+            break
+    if not opt.dry_run:
+        save()
+    log(f"=== run end: {done} file(s){' planned' if opt.dry_run else ''}, saved {saved / 1e9:.1f} GB ===")
+
+
+if __name__ == "__main__":
+    main()

server/home/Media/compact-tv/compact-tv.service

@@ -0,0 +1,17 @@
+# Installed to /etc/systemd/system/. Runs as gib because the TV files are gib-owned,
+# so the job needs no root. A system unit (rather than a user unit) because gib has
+# no lingering session on this headless box. ExecStart goes through python3 because
+# SELinux will not let systemd execute a file labeled as home content (user_home_t).
+[Unit]
+Description=Re-encode TV shows on Sonarr's Compact/Supercompact profiles
+Documentation=file:///home/gib/Server/Media/compact-tv/README.md
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+User=gib
+Group=gib
+ExecStart=/usr/bin/python3 /home/gib/Server/Media/compact-tv/compact-tv.py --until 08:00
+Nice=15
+IOSchedulingClass=idle

server/home/Media/compact-tv/compact-tv.timer

@@ -0,0 +1,11 @@
+# Starts after the 00:00 hour and stops starting new encodes at 08:00 (see --until in
+# the service), so Plex and Frigate keep the GPU's encoder during the day.
+[Unit]
+Description=Nightly TV re-encode for Compact/Supercompact shows
+
+[Timer]
+OnCalendar=*-*-* 00:30:00
+Persistent=false
+
+[Install]
+WantedBy=timers.target

server/home/Media/compact-tv/cost-report.py

@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+"""Per-title cost of shrinking everything on Standard/Mini: re-encode vs redownload.
+
+For each show or movie on those profiles, probes every file with compact-tv's own planner
+and reports what the nightly job would do (encode, skip as already small, skip as HDR),
+the space it would save, and the GPU time it would take. Alongside that it estimates what a
+fresh small download would cost instead, using typical small-encode sizes.
+
+GPU time is measured, not guessed: the Shameless pilot took 669 s for 57.5 minutes of
+1080p (about 0.19 h of GPU per hour of video). 4K sources decode slower; the 1.5x for them
+is an estimate until the first 4K night is logged.
+
+Usage: cost-report.py > report.tsv     (also prints a summary to stderr)
+"""
+import importlib.util, json, pathlib, sys
+
+HERE = pathlib.Path(__file__).resolve().parent
+spec = importlib.util.spec_from_file_location("ct", HERE / "compact-tv.py")
+ct = importlib.util.module_from_spec(spec); spec.loader.exec_module(ct)
+
+GPU_H_PER_H = 669 / 3450                 # pilot: hours of GPU per hour of 1080p video
+UHD_FACTOR = 1.5                         # estimate for 2160p sources
+SMALL_GB_PER_H = {"Standard": 0.85, "Mini": 0.57}   # MeGusta-class 1080p / 720p HEVC, measured on S.W.A.T.
+
+rows = []
+for app, a in ct.APPS.items():
+    call = ct.arr(app)
+    profiles = {p["id"]: p["name"] for p in call("/qualityprofile")}
+    container, host = a["paths"]
+    for t in call(a["titles"]):
+        tier = profiles.get(t["qualityProfileId"])
+        if tier not in ct.TIERS:
+            continue
+        r = {"app": app, "title": t["title"], "tier": tier, "files": 0, "hours": 0.0, "size": 0,
+             "enc_files": 0, "enc_size": 0, "enc_est": 0, "gpu_h": 0.0, "hdr_size": 0, "small_size": 0}
+        for f in a["files"](call, t):
+            p = pathlib.Path(f["path"].replace(container, host, 1))
+            info = ct.probe(p) if p.exists() else None
+            if not info:
+                continue
+            size, dur = int(info["format"]["size"]), float(info["format"]["duration"]) / 3600
+            r["files"] += 1; r["hours"] += dur; r["size"] += size
+            args, why = ct.plan(info, tier, app)
+            if args:
+                w = max((s.get("width") or 0) for s in info["streams"] if s["codec_type"] == "video")
+                r["enc_files"] += 1; r["enc_size"] += size; r["enc_est"] += why
+                r["gpu_h"] += dur * GPU_H_PER_H * (UHD_FACTOR if w > 1920 else 1)
+            elif "HDR" in str(why):
+                r["hdr_size"] += size
+            else:
+                r["small_size"] += size
+        if r["files"]:
+            r["redownload_gb"] = round(r["hours"] * SMALL_GB_PER_H[tier], 1)
+            rows.append(r)
+
+rows.sort(key=lambda r: -(r["enc_size"] - r["enc_est"]))
+cols = ["app", "title", "tier", "files", "hours", "size", "enc_files", "enc_size", "enc_est", "gpu_h", "hdr_size",
+        "small_size", "redownload_gb"]
+print("\t".join(cols))
+for r in rows:
+    print("\t".join(str(round(r[c], 2) if isinstance(r[c], float) else r[c]) for c in cols))
+tot = lambda k: sum(r[k] for r in rows)
+print(f"{len(rows)} titles; encode {tot('enc_files')} files {tot('enc_size')/1e12:.2f} TB -> ~{tot('enc_est')/1e12:.2f} TB, "
+      f"{tot('gpu_h'):.0f} GPU hours; HDR skipped {tot('hdr_size')/1e12:.2f} TB; already small {tot('small_size')/1e12:.2f} TB",
+      file=sys.stderr)

server/home/Media/compact-tv/setup-small-profiles.py

@@ -0,0 +1,99 @@
+#!/usr/bin/env python3
+"""Make Standard and Mini prefer small releases, and upgrade only toward small ones.
+
+Run once per app; it is idempotent, so re-running updates everything in place.
+
+- Adds "Small Encode" (release groups known for small HEVC/x264 encodes) and "720p" /
+  "1080p" resolution formats. They score 0 in every profile except Standard and Mini.
+- HD and UHD are otherwise untouched: they keep LQ at -10000, so they never pick these.
+- Standard and Mini:
+  - All allowed qualities share ONE group, so no "better resolution" or "better source"
+    upgrade can ever download a big file. Only the custom-format score decides.
+  - Scores: Small Encode +3000, x265 +2000 (so first grabs still prefer HEVC), LQ 0,
+    release-tier formats 0 (one big release must never "beat" another big one).
+    Standard adds +500 for 1080p and +250 for 720p; Mini adds +1000 for 720p and +500 for
+    1080p. Those preferences stay below the upgrade threshold, so they never cause churn.
+  - Upgrades are on, but an upgrade needs a score jump of at least 2500, and upgrading
+    stops at 3000. In practice only gaining "Small Encode" clears that bar: a big H.264
+    file gets replaced by a small encode, but never by a big HEVC one.
+
+Usage: setup-small-profiles.py <sonarr|radarr>
+"""
+import json, re, subprocess, sys, urllib.request, pathlib
+
+APP = sys.argv[1]
+cfg = pathlib.Path.home() / f"Server/Media/volumes/{APP}/config.xml"
+key = re.search(r"<ApiKey>([^<]+)", cfg.read_text()).group(1)
+ip = subprocess.run(["docker", "inspect", APP, "--format", "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"],
+                    capture_output=True, text=True, check=True).stdout.split()[0]
+base = f"http://{ip}:{8989 if APP == 'sonarr' else 7878}/api/v3"
+
+def call(path, body=None, method=None):
+    req = urllib.request.Request(base + path, method=method, headers={"X-Api-Key": key, "Content-Type": "application/json"},
+                                 data=json.dumps(body).encode() if body is not None else None)
+    with urllib.request.urlopen(req, timeout=60) as r:
+        return json.load(r)
+
+GROUPS = r"^(MeGusta|PSA|ELiTE|GalaxyTV|GalaxyRG|GalaxyRG265|YIFY|YTS(\.(MX|LT|AG))?)$"
+resolution = lambda r: [{"name": f"{r}p", "implementation": "ResolutionSpecification",
+                         "negate": False, "required": True, "fields": [{"name": "value", "value": r}]}]
+WANTED = {
+    "Small Encode": [{"name": "Small encode groups", "implementation": "ReleaseGroupSpecification",
+                      "negate": False, "required": False, "fields": [{"name": "value", "value": GROUPS}]}],
+    "720p": resolution(720),
+    "1080p": resolution(1080),
+}
+TIERS = {
+    "Standard": {"Small Encode": 3000, "x265 (HD)": 2000, "LQ": 0, "LQ (Release Title)": 0, "1080p": 500, "720p": 250},
+    "Mini":     {"Small Encode": 3000, "x265 (HD)": 2000, "LQ": 0, "LQ (Release Title)": 0, "720p": 1000, "1080p": 500},
+}
+RELEASE_TIER = re.compile(r"Tier \d+|WEB Scene")
+MIN_UPGRADE_JUMP, UPGRADE_UNTIL = 2500, 3000
+
+cfs = {c["name"]: c for c in call("/customformat")}
+for name, specs in WANTED.items():
+    body = {"name": name, "includeCustomFormatWhenRenaming": False, "specifications": specs}
+    if name in cfs:
+        body["id"] = cfs[name]["id"]; call(f"/customformat/{body['id']}", body, "PUT")
+    else:
+        call("/customformat", body)
+cfs = {c["name"]: c["id"] for c in call("/customformat")}
+
+
+def flatten(items):
+    """Every quality in the profile with whether it is allowed, groups expanded, lowest rank first."""
+    for it in items:
+        for sub in (it["items"] or [it]):
+            yield sub["quality"], it["allowed"]
+
+
+def one_group(profile):
+    """Put every allowed quality into a single top-ranked group; leave disallowed ones as they are."""
+    flat = list(flatten(profile["items"]))
+    group = {"id": 1100, "name": "Any allowed", "allowed": True,
+             "items": [{"quality": q, "items": [], "allowed": True} for q, ok in flat if ok]}
+    profile["items"] = [{"quality": q, "items": [], "allowed": False} for q, ok in flat if not ok] + [group]
+    profile["cutoff"] = group["id"]
+    return profile
+
+
+for prof in call("/qualityprofile"):
+    have = {f["format"] for f in prof["formatItems"]}
+    for name in WANTED:
+        if cfs[name] not in have:
+            prof["formatItems"].append({"format": cfs[name], "name": name, "score": 0})
+    scores = TIERS.get(prof["name"])
+    if scores:
+        prof = one_group(prof)
+        prof.update(upgradeAllowed=True, minUpgradeFormatScore=MIN_UPGRADE_JUMP, cutoffFormatScore=UPGRADE_UNTIL)
+        for f in prof["formatItems"]:
+            if f["name"] in scores:
+                f["score"] = scores[f["name"]]
+            elif RELEASE_TIER.search(f["name"]):
+                f["score"] = 0
+    call(f"/qualityprofile/{prof['id']}", prof, "PUT")
+    if scores:
+        allowed = [q["quality"]["name"] for it in prof["items"] if it["allowed"] for q in it["items"]]
+        print(f"{APP} {prof['name']}: one group of {len(allowed)} qualities ({allowed[0]} .. {allowed[-1]}); "
+              f"upgrades on, jump >= {MIN_UPGRADE_JUMP}, until {UPGRADE_UNTIL}")
+        print("   nonzero scores: " + ", ".join(f"{f['name']}={f['score']}" for f in prof["formatItems"] if f["score"]))

server/home/Media/compose.yml

@@ -0,0 +1,434 @@
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  plex:
+    image: plexinc/pms-docker:latest
+    container_name: plex
+    hostname: plex
+    domainname: plex.gbrown.org
+    networks: [nginx-bridge]
+    group_add: [105, 39]
+    ports:
+      - 32400:32400/tcp # REQUIRED
+      #- 5353:5353/udp # For older Bonjour/Avahi network discovery
+      #- 8324:8324/tcp # For Plex Roku App
+      - 32410:32410/udp # For network discovery
+      - 32412:32412/udp # For network discovery
+      - 32413:32413/udp # For network discovery
+      - 32414:32414/udp # For network discovery
+      #- 1900:1900/udp # For DLNA
+      - 32469:32469/tcp # For DLNA
+    environment:
+      - TZ=America/New_York
+      - PLEX_UID=1000
+      - PLEX_GID=1000
+      - PLEX_CLAIM=${PLEX_CLAIM}
+      - ADVERTISE_IP=http://192.168.0.43
+      - NVIDIA_VISIBLE_DEVICES=all
+      - NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
+    labels: ["com.centurylinklabs.watchtower.enable=true"]
+    volumes:
+      - /run:/run
+      - ./volumes/plex/config:/config
+      - ./volumes/plex/transcode:/transcode
+      - /home/gib/Public/Movies:/data/movies
+      - /home/gib/Public/TV:/data/tvshows
+      - /home/gib/Public/Music:/data/music
+      - /home/gib/Public/Audiobooks:/data/audiobooks
+      - /home/gib/Public/Podcasts:/data/podcasts
+    devices: [/dev/dri:/dev/dri]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    security_opt: ["seccomp:unconfined"]
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  jellyfin:
+    image: jellyfin/jellyfin
+    container_name: jellyfin
+    hostname: jellyfin
+    domainname: jellyfin.gbrown.org
+    networks: [nginx-bridge]
+    group_add: [105, 39]
+    ports: [7359:7359/udp]
+    user: 1000:1000
+    environment:
+      - JELLYFIN_PublishedServerUrl=https://jellyfin.gbrown.org
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    volumes:
+      - ./volumes/jellyfin/config:/config
+      - ./volumes/jellyfin/cache:/cache
+      - /home/gib/Public/Movies:/data/movies
+      - /home/gib/Public/TV:/data/tvshows
+      - /home/gib/Public/Music:/data/music
+      - /home/gib/Public/Audiobooks:/data/audiobooks
+      - /home/gib/Public/Podcasts:/data/podcasts
+      - /home/gib/.local/share/fonts:/usr/local/share/fonts/custom:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    devices: [/dev/dri:/dev/dri]
+    security_opt: [seccomp:unconfined]
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  audiobookshelf:
+    image: ghcr.io/advplyr/audiobookshelf:latest
+    container_name: audiobookshelf
+    hostname: audiobookshelf
+    domainname: audio.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [80:80]
+    environment: [TZ=America/New_York, PUID=1000, PGID=1000]
+    labels: ["com.centurylinklabs.watchtower.enable=true"]
+    volumes:
+      - ./volumes/audiobookshelf/config:/config
+      - ./volumes/audiobookshelf/metadata:/metadata
+      - /home/gib/Public/Audiobooks:/audiobooks
+      - /home/gib/Public/Podcasts:/podcasts
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  jellyplex-watched:
+    image: luigi311/jellyplex-watched:latest
+    container_name: jellyplex-watched
+    hostname: jellyplex-watched
+    networks: [nginx-bridge]
+    environment:
+      DRYRUN: False
+      DEBUG_LEVEL: INFO # Or debug for verbose logging
+      RUN_ONLY_ONCE: False
+      SLEEP_DURATION: 3600
+      REQUEST_TIMEOUT: 300
+      MAX_THREADS: 2
+      GENERATE_GUIDS: False
+      GENERATE_LOCATIONS: True
+      USER_MAPPING: '{ "gib": "gibby_b", "adrian": "Adrian"}'
+      LIBRARY_MAPPING: '{ "TV Shows": "Shows"}'
+      PLEX_BASEURL: http://plex:32400
+      PLEX_TOKEN: ${PLEX_TOKEN}
+      SSL_BYPASS: False
+      JELLYFIN_BASEURL: http://jellyfin:8096
+      JELLYFIN_TOKEN: ${JELLYFIN_TOKEN}
+      SYNC_FROM_PLEX_TO_JELLYFIN: True
+      SYNC_FROM_JELLYFIN_TO_PLEX: True
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gluetun:
+    image: qmcgaw/gluetun:latest
+    container_name: gluetun
+    hostname: gluetun
+    networks: [nginx-bridge]
+    cap_add: [NET_ADMIN]
+    devices: [/dev/net/tun:/dev/net/tun]
+    #ports:
+    #- 8000:8000 # HTTP control server
+    #- 8080:8080 # For SABnzbd
+    #- 6969:6969 # For qBittorrent
+    #- 6881:6881 # For torrent connectivity
+    #- 6881:6881/udp # For torrent connectivity
+    environment:
+      - VPN_SERVICE_PROVIDER=private internet access
+      - VPN_TYPE=openvpn
+      - SERVER_REGIONS=US East
+      - OPENVPN_USER=p7671057
+      - OPENVPN_PASSWORD=${OPENVPN_PASSWORD}
+      - TZ=America/New_York
+      - PUID=1000
+      - PGID=100
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    restart: unless-stopped
+    tty: true
+    stdin_open: true
+
+  sabnzbd:
+    image: lscr.io/linuxserver/sabnzbd:latest
+    container_name: sabnzbd
+    domainname: sabnzbd.gbrown.org
+    network_mode: service:gluetun
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    volumes:
+      - ./volumes/sabnzbd:/config
+      - /home/gib/Public/Downloads:/downloads
+      - /home/gib/Public/Downloads/incomplete:/incomplete-downloads
+    depends_on: [gluetun]
+    restart: unless-stopped
+    tty: true
+    stdin_open: true
+
+  qbittorrent:
+    image: linuxserver/qbittorrent:latest
+    container_name: qbittorrent
+    network_mode: service:gluetun
+    environment:
+      - PUID=1000
+      - PGID=1000
+      - TZ=America/New_York
+      - WEBUI_PORT=6969
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    volumes:
+      - ./volumes/qbittorrent:/config
+      - /home/gib/Public/Downloads:/downloads
+    depends_on: [gluetun]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  qbit-guard:
+    image: ghcr.io/gengines/qbit-guard:latest
+    container_name: qbitguard
+    hostname: qbitguard
+    networks: [nginx-bridge]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    environment:
+      - QBIT_HOST=http://gluetun:6969
+      - QBIT_USER=gib
+      - QBIT_PASS=${QBIT_PASS}
+      - QBIT_ALLOWED_CATEGORIES=tv-sonarr,radarr
+      - ENABLE_PREAIR_CHECK=1
+      - SONARR_URL=http://sonarr:8989
+      - SONARR_API_KEY=${SONARR_API_KEY}
+      - SONARR_CATEGORIES=tv-sonarr
+      - RADARR_URL=http://radarr:7878
+      - RADARR_API_KEY=${RADARR_API_KEY}
+      - RADARR_CATEGORIES=radarr
+      - ENABLE_ISO_CHECK=1
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  seerr:
+    image: ghcr.io/seerr-team/seerr:latest
+    container_name: seerr
+    init: true
+    hostname: seerr
+    domainname: seerr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [5055:5055]
+    environment: [LOG_LEVEL=debug, TZ=America/New_York]
+    volumes: [./volumes/seerr:/app/config]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    depends_on: [radarr, sonarr]
+    healthcheck:
+      test: wget --no-verbose --tries=1 --spider http://localhost:5055/api/v1/status || exit 1
+      start_period: 20s
+      timeout: 3s
+      interval: 15s
+      retries: 3
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  prowlarr:
+    image: lscr.io/linuxserver/prowlarr:latest
+    container_name: prowlarr
+    hostname: prowlarr
+    domainname: prowlarr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [9696:9696]
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    volumes: [./volumes/prowlarr:/config]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    depends_on: [qbittorrent, sabnzbd]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  lidarr:
+    image: lscr.io/linuxserver/lidarr:latest
+    container_name: lidarr
+    hostname: lidarr
+    domainname: lidarr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [8686:8686]
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    volumes:
+      - ./volumes/lidarr:/config
+      - /home/gib/Public/Music:/music
+      - /home/gib/Public/Downloads:/downloads
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    depends_on: [prowlarr, qbittorrent, sabnzbd]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  radarr:
+    image: ghcr.io/linuxserver/radarr:latest
+    container_name: radarr
+    hostname: radarr
+    domainname: radarr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [7878:7878]
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    volumes:
+      - ./volumes/radarr:/config
+      - /home/gib/Public/Movies:/movies
+      - /home/gib/Public/Downloads:/downloads
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    depends_on: [prowlarr, qbittorrent, sabnzbd]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  sonarr:
+    image: ghcr.io/linuxserver/sonarr:latest
+    container_name: sonarr
+    hostname: sonarr
+    domainname: sonarr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [8989:8989]
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    volumes:
+      - ./volumes/sonarr:/config
+      - /home/gib/Public/TV:/tv
+      - /home/gib/Public/Downloads:/downloads
+    depends_on: [prowlarr, qbittorrent, sabnzbd]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  bazarr:
+    image: lscr.io/linuxserver/bazarr:latest
+    container_name: bazarr
+    hostname: bazarr
+    domainname: bazarr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [6767:6767]
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    volumes:
+      - ./volumes/bazarr:/config
+      - /home/gib/Public/Movies:/movies
+      - /home/gib/Public/TV:/tv
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  whisperasr:
+    image: onerahmet/openai-whisper-asr-webservice:latest-gpu
+    container_name: whisperasr
+    hostname: whisperasr
+    networks: [nginx-bridge]
+    #ports: [9000:9000]
+    environment: [TZ=America/New_York, PUID=1000, PGID=1000, ASR_MODEL=small]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    devices: [/dev/dri:/dev/dri]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    security_opt: ["seccomp:unconfined"]
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  unpackerr:
+    image: golift/unpackerr:latest
+    container_name: unpackerr
+    hostname: unpackerr
+    networks: [nginx-bridge]
+    volumes:
+      - /home/gib/Public/Downloads:/downloads
+      - /home/gib/Server/Media/volumes/unpackerr:/config
+    environment:
+      - TZ=America/New_York
+      # log lives in its own volume, NOT in /downloads where the *arr apps scan
+      - UN_LOG_FILE=/config/unpackerr.log
+      - UN_LOG_FILES=4
+      - UN_LOG_FILE_MB=5
+      - UN_SONARR_0_URL=http://sonarr:8989
+      - UN_SONARR_0_API_KEY=${UN_SONARR_0_API_KEY}
+      - UN_RADARR_0_URL=http://radarr:7878
+      - UN_RADARR_0_API_KEY=${UN_RADARR_0_API_KEY}
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    tty: true
+    stdin_open: true
+    restart: always
+
+  metube:
+    image: ghcr.io/alexta69/metube:latest
+    container_name: metube
+    hostname: metube
+    domainname: metube.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [8081:8081]
+    environment:
+      - TZ=America/New_York
+      - PUID=1000
+      - PGID=1000
+      - YTDL_OPTIONS={"cookiefile":"/cookies/cookies.txt"}
+    volumes:
+      - ./volumes/metube:/cookies:rw
+      - /home/gib/Public/Downloads:/downloads:rw
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  pinchflat:
+    image: ghcr.io/kieraneglin/pinchflat:latest
+    container_name: pinchflat
+    hostname: pinchflat
+    domainname: pinchflat.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [8945:8945]
+    environment: [TZ=America/New_York, PUID=1000, PGID=1000]
+    volumes:
+      - ./volumes/pinchflat:/config:rw
+      - /home/gib/Public/Downloads:/downloads:rw
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  configarr:
+    image: ghcr.io/raydak-labs/configarr:latest
+    container_name: configarr
+    hostname: configarr
+    domainname: configarr.gbrown.org
+    networks: [nginx-bridge]
+    environment: [TZ=America/New_York, PUID=1000, PGID=1000]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    volumes:
+      - ./volumes/configarr/config:/app/config
+      - ./volumes/configarr/repos:/app/repos
+      - ./volumes/configarr/cfs:/app/cfs
+      - ./volumes/configarr/templates:/app/templates
+    depends_on: [radarr, sonarr]
+    tty: true
+    stdin_open: true
+    # Run-to-completion: configarr syncs TRaSH guides into Sonarr/Radarr,
+    # prints an execution summary, and exits 0. It has no built-in scheduler
+    # (upstream points at Ofelia or cron for that), so `unless-stopped` here
+    # meant a completed sync was immediately restarted -- a hot loop of ~9
+    # full syncs every 3 minutes. `no` lets it run once per `up` and stop.
+    restart: "no"
+    # Opt-in only (`docker compose --profile configarr up configarr`). Its templates
+    # create the TRaSH profiles and size limits that were removed on 2026-10-28 in
+    # favour of four hand-managed profiles (Mini/Standard/HD/UHD), so a plain `up`
+    # must never run it. See Media/compact-tv/README.md.
+    profiles: [configarr]

server/home/Media/podman-media.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Media
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Media
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Minecraft/Bedrock/Adrian/compose.yml

@@ -0,0 +1,28 @@
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  bedrock-minecraft:
+    image: itzg/minecraft-bedrock-server
+    container_name: mc-bedrock-adrian
+    network_mode: host
+    #hostname: mc-bedrock-adrian
+    #domainname: mc-bedrock-adrian.gbrown.org
+    #networks: ['nginx-bridge']
+    #ports:
+      #- 19132:19132
+      #- 19132:19132/udp
+    environment:
+     - EULA=TRUE
+     #- UID=1000
+     #- GID=1000
+     - TZ=America/New_York
+    #user: 1000:1000
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - './data:/data'
+    tty: true
+    stdin_open: true
+    restart: unless-stopped

server/home/Minecraft/Bedrock/Adrian/podman-minecraft-bedrock-adrian.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Minecraft/Bedrock/Adrian
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Minecraft/Bedrock/Adrian
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Minecraft/Java/compose.yml

@@ -0,0 +1,104 @@
+# Java Edition Minecraft behind mc-router. Live since 2026-10-06 at mc.gbrown.org.
+#
+# mc-router owns the one public port (25565) and routes each player by the hostname
+# they typed: Java clients send it in their first packet, so several servers can
+# share one port and one IP. Bedrock clients do not send it, which is why the
+# Bedrock server in ../Bedrock stays separate on its own UDP port.
+#
+# Start:    docker compose up -d        (from this directory)
+#
+# Each hostname needs a DNS-only (grey cloud) A record in Cloudflare pointing at the
+# home IP. The proxied *.gbrown.org wildcard can't carry Minecraft traffic, and a
+# specific record overrides it. UniFi forwards 25565/tcp to this host. Home
+# Assistant's Cloudflare integration keeps these records on the current home IP;
+# select each new record there too.
+#
+# Add a server:
+#   1. mkdir servers/<name>, then copy the "gib" service below and point it there
+#   2. add "<hostname>=<service>:25565" to MAPPING
+#   3. add the DNS-only record for <hostname>
+# Servers publish no ports of their own; only mc-router is reachable from outside.
+
+services:
+  mc-router:
+    image: itzg/mc-router:latest
+    container_name: mc-router
+    environment:
+      # <hostname players type>=<service name>:<port inside the network>
+      MAPPING: |
+        mc.gbrown.org=gib:25565
+    ports: ["25565:25565"]
+    networks: [minecraft]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    restart: unless-stopped
+
+  gib:
+    image: itzg/minecraft-server:latest
+    container_name: mc-gib
+    environment:
+      # 26.1.2 matches the newest OptiFine (SEUS shaders) on the desktop. The 1.20.4
+      # world is in backups/; upgrades rewrite the world and cannot be undone.
+      - VERSION=26.1.2
+      - DIFFICULTY=normal
+      - MAX_PLAYERS=30
+      # Paper since 2026-10-13. The last vanilla state is backups/gib-26.1.2-vanilla-*.
+      - TYPE=PAPER
+      # DonutSMP-style plugin set, pulled from Modrinth on each start. Every entry
+      # needs a 26.1.2 Paper build; check Modrinth before adding or bumping VERSION.
+      # Config lives in servers/gib/plugins/<Plugin>/. GrimAC only publishes alpha builds.
+      - MODRINTH_DOWNLOAD_DEPENDENCIES=required
+      - |
+        MODRINTH_PROJECTS=
+        luckperms
+        vaultunlocked
+        essentialsx
+        essentialsx-spawn
+        placeholderapi
+        playerpoints
+        zmenu
+        zauctionhouse
+        ultimateshop
+        smartspawner
+        wildrtp
+        ultimate-teams
+        asbounty
+        axafkzone
+        tab-was-taken
+        deluxemenus
+        coreprotect
+        worldedit
+        worldguard
+        grimac:alpha
+        chunky
+      - MEMORY=12G
+      - SIMULATION_DISTANCE=16
+      - VIEW_DISTANCE=16
+      - ALLOW_FLIGHT=true
+      - ENABLE_COMMAND_BLOCK=true
+      - ENABLE_WHITELIST=true
+      - UID=1000
+      - GID=1000
+      - EULA=true
+      - TZ=America/New_York
+      - ENABLE_ROLLING_LOGS=true
+      - LANG=en_US.UTF-8
+      - LANGUAGE=en_US:en
+      - LC_ALL=en_US.UTF-8
+      - SERVER_NAME=Gib's Minecraft Server
+      - SNOOPER_ENABLED=false
+      - WHITELIST_FILE=/data/whitelist.json
+      #- ENABLE_AUTOPAUSE=true
+      # Don't change MOTD it was really hard to make it look good.
+      - MOTD=           \u00A7aGib's Minecraft Server\u00A7r                                            \u00A74Welcome to HECK
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./servers/gib:/data
+    networks: [minecraft]
+    stdin_open: true
+    tty: true
+    restart: unless-stopped
+
+networks:
+  minecraft: {}

server/home/Minecraft/Java/podman-minecraft-java.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Minecraft/Java
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Minecraft/Java
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Nginx/.env.example

@@ -0,0 +1,2 @@
+DB_POSTGRES_PASSWORD=CHANGE_ME
+POSTGRES_PASSWORD=CHANGE_ME

server/home/Nginx/compose.yml

@@ -0,0 +1,55 @@
+services:
+  app:
+    image: 'jc21/nginx-proxy-manager:latest'
+    container_name: nginx-proxy-manager
+    hostname: nginx-proxy-manager
+    domainname: nginx.gbrown.org
+    networks:
+      - nginx-bridge
+    restart: unless-stopped
+    ports:
+      - '80:80'
+      - '443:443'
+      - '81:81' # Admin Web Port
+      #- '21:21' # FTP
+      #- '22:22' # SSH
+      #- '25565:25565' # Minecraft
+    environment:
+      PUID: 1000
+      PGID: 1000
+      DB_POSTGRES_HOST: 'db'
+      DB_POSTGRES_PORT: '5432'
+      DB_POSTGRES_USER: 'npm'
+      DB_POSTGRES_PASSWORD: ${DB_POSTGRES_PASSWORD}
+      DB_POSTGRES_NAME: 'npm'
+    labels:
+      com.centurylinklabs.watchtower.enable: 'true'
+    volumes:
+      - ./volumes/data:/data
+      - ./volumes/letsencrypt:/etc/letsencrypt
+    depends_on:
+      - db
+
+  db:
+    image: postgres:17
+    container_name: nginx-proxy-manager-db
+    hostname: nginx-proxy-manager-db
+    networks:
+      - nginx-bridge
+    environment:
+      POSTGRES_USER: 'npm'
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
+      POSTGRES_DB: 'npm'
+    labels:
+      com.centurylinklabs.watchtower.enable: 'true'
+    volumes:
+      - ./volumes/postgres:/var/lib/postgresql/data
+    restart: unless-stopped
+
+networks:
+  nginx-bridge:
+    driver: bridge
+    ipam:
+      config:
+        - subnet: 172.25.0.0/24
+    name: nginx-bridge

server/home/Nginx/podman-nginx.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Nginx
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Nginx
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/OpenWebUI/.env.example

@@ -0,0 +1,4 @@
+BW_SESSION=CHANGE_ME
+OPENAI_API_KEY=CHANGE_ME
+OAUTH_CLIENT_ID=CHANGE_ME
+OAUTH_CLIENT_SECRET=CHANGE_ME

server/home/OpenWebUI/compose.yml

@@ -0,0 +1,147 @@
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  openwebui:
+    image: ghcr.io/open-webui/open-webui:main
+    #image: ghcr.io/open-webui/open-webui:ollama
+    container_name: openwebui
+    hostname: openwebui
+    domainname: chat.gbrown.org
+    networks:
+      - nginx-bridge
+    #ports:
+    #- 8080:8080
+    environment:
+      - WEBUI_NAME=Gib's Chat
+      - WEBUI_URL=https://chat.gbrown.org
+      - OLLAMA_BASE_URL=http://ollama:11434
+      - [email protected]
+      - DEFAULT_MODELS=llama3.1
+      - TASK_MODEL=llama3.1
+      - ENABLE_AUTOCOMPLETE_GENERATION=false
+      - OPENAI_API_KEY=${OPENAI_API_KEY}
+      - AUTOMATIC1111_BASE_URL=http://192.168.0.43:7860
+      - COMFYUI_BASE_URL=http://comfyui:8188
+      - WHISPER_MODEL_AUTO_UPDATE=True
+      - PDF_EXTRACT_IMAGES=True
+      - ENABLE_RAG_WEB_SEARCH=True
+      - RAG_WEB_SEARCH_ENGINE=searxng
+      - SEARXNG_QUERY_URL=https://search.gbrown.org/search?q=<query>
+      - ENABLE_IMAGE_GENERATION=True
+      - IMAGE_GENERATION_ENGINE=ComfyUI
+      - IMAGE_SIZE=1216x832
+      #- USE_OLLAMA_DOCKER=True
+      #- USE_CUDA_DOCKER=True
+      - DO_NOT_TRACK=True
+      - ENABLE_OAUTH_SIGNUP=true
+      - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
+      #- ENABLE_OAUTH_ROLE_MANAGEMENT=true
+      #- OAUTH_ROLES_CLAIM=roles
+      #- OAUTH_ADMIN_ROLES=admin,internal,superuser
+      - OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID}
+      - OAUTH_CLIENT_SECRET=${OAUTH_CLIENT_SECRET}
+      - OPENID_PROVIDER_URL=https://auth.gbrown.org/application/o/openwebui/.well-known/openid-configuration
+      - OPENID_REDIRECT_URI=https://chat.gbrown.org/oauth/oidc/callback
+      - OAUTH_PROVIDER_NAME=Gib's Auth
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/openwebui:/app/backend/data
+      #- /home/gib/Public/Server_Backup/Ollama_Models:/root/.ollama
+    #devices:
+    #- /dev/dri:/dev/dri
+    tty: true
+    restart: unless-stopped
+    #security_opt: ['seccomp:unconfined','label:disable']
+    #deploy:
+    #resources:
+    #reservations:
+    #devices:
+    #- driver: nvidia
+    #count: 1
+    #capabilities: [gpu]
+
+  ollama:
+    #image: ollama/ollama:rocm
+    image: ollama/ollama:latest
+    container_name: ollama
+    hostname: ollama
+    domainname: ollama.gbrown.org
+    networks:
+      - nginx-bridge
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    ports:
+      - 11434:11434
+    volumes:
+      - /home/gib/Public/Server_Backup/Ollama_Models:/root/.ollama
+    devices:
+      - /dev/dri:/dev/dri
+    tty: true
+    restart: unless-stopped
+    security_opt: ["seccomp:unconfined", "label:disable"]
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  comfyui:
+    #image: yanwk/comfyui-boot:rocm
+    image: yanwk/comfyui-boot:cu128-megapak
+    init: true
+    container_name: comfyui
+    hostname: comfyui
+    domainname: comfyui.gbrown.org
+    networks:
+      - nginx-bridge
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    #ports: [8188:8188]
+    volumes:
+      - /home/gib/Public/Server_Backup/ComfyUI/:/root
+    #environment:
+    #- HSA_OVERRIDE_GFX_VERSION=11.0.0
+    #- CLI_ARGS=''
+    devices:
+      - /dev/dri:/dev/dri
+      #- /dev/kfd:/dev/kfd
+    tty: true
+    security_opt: ["seccomp:unconfined", "label:disable"]
+    restart: unless-stopped
+    group_add: [video]
+    #cap_add: [SYS_PTRACE]
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  mcpo:
+    image: ghcr.io/open-webui/mcpo:main
+    container_name: mcpo
+    hostname: mcpo.gbrown.org
+    domainname: mcpo.gbrown.org
+    command: 'mcpo --port 8010 --config /app/config.json --api-key "Tragedy8-Unfazed-Craftily-Hedging-Gem-Velcro"'
+    restart: unless-stopped
+    tty: true
+    networks:
+      - nginx-bridge
+    ports:
+      - 8010:8010
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    env_file:
+      - .env
+    volumes:
+      - ./volumes/mcpo/openapi.json:/app/config.json
+      - ./volumes/mcpo/bin/gitea-mcp:/usr/local/bin/gitea-mcp
+      - ./volumes/mcpo/bin/bw:/usr/local/bin/bw

server/home/OpenWebUI/podman-openwebui.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: OpenWebUI
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/OpenWebUI
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Plausible/.env.example

@@ -0,0 +1,8 @@
+BASE_URL=CHANGE_ME
+SECRET_KEY_BASE=CHANGE_ME
+HTTP_PORT=CHANGE_ME
+HTTPS_PORT=CHANGE_ME
+SMTP_USER_NAME=CHANGE_ME
+SMTP_USER_PWD=CHANGE_ME
+POSTGRES_PASSWORD=CHANGE_ME
+TOTP_VAULT_KEY=CHANGE_ME

server/home/Plausible/UPSTREAM

@@ -0,0 +1,6 @@
+checkout: ~/Server/Plausible/
+repo:     https://github.com/plausible/community-edition
+ref:      tag v3.0.1
+commit:   20385b05aa8e5645c88fc1a526360608e0fba324
+
+compose.yml here is the local, modified copy of the upstream file.

server/home/Plausible/compose.yml

@@ -0,0 +1,131 @@
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  plausible_db:
+    image: postgres:16-alpine
+    container_name: plausible_db
+    hostname: plausible_db
+    networks: [nginx-bridge]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - db-data:/var/lib/postgresql/data
+    environment:
+      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      start_period: 1m
+    restart: always
+    tty: true
+
+  plausible_events_db:
+    image: clickhouse/clickhouse-server:24.12-alpine
+    container_name: plausible_events_db
+    hostname: plausible_events_db
+    networks: [nginx-bridge]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - event-data:/var/lib/clickhouse
+      - event-logs:/var/log/clickhouse-server
+      - ./clickhouse/logs.xml:/etc/clickhouse-server/config.d/logs.xml:ro
+      # This makes ClickHouse bind to IPv4 only, since Docker doesn't enable IPv6 in bridge networks by default.
+      # Fixes "Listen [::]:9000 failed: Address family for hostname not supported" warnings.
+      - ./clickhouse/ipv4-only.xml:/etc/clickhouse-server/config.d/ipv4-only.xml:ro
+      # This makes ClickHouse consume less resources, which is useful for small setups.
+      # https://clickhouse.com/docs/en/operations/tips#using-less-than-16gb-of-ram
+      #- ./clickhouse/low-resources.xml:/etc/clickhouse-server/config.d/low-resources.xml:ro
+    ulimits:
+      nofile:
+        soft: 262144
+        hard: 262144
+    environment:
+      - CLICKHOUSE_SKIP_USER_SETUP=1
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          "wget --no-verbose --tries=1 -O - http://127.0.0.1:8123/ping || exit 1",
+        ]
+      start_period: 1m
+    restart: always
+    tty: true
+
+  plausible:
+    image: ghcr.io/plausible/community-edition:v3.2.1
+    container_name: plausible
+    hostname: plausible
+    networks: [nginx-bridge]
+    #ports:
+    #- ${HTTP_PORT}:8000
+    #- ${HTTPS_PORT}:443
+    command: sh -c "/entrypoint.sh db createdb && /entrypoint.sh db migrate && /entrypoint.sh run"
+    depends_on:
+      plausible_db:
+        condition: service_healthy
+      plausible_events_db:
+        condition: service_healthy
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - plausible-data:/var/lib/plausible
+    ulimits:
+      nofile:
+        soft: 65535
+        hard: 65535
+    extra_hosts:
+      # Pin the relay to the WireGuard address. Docker containers do NOT read the host
+      # /etc/hosts, and the public *.gbrown.org wildcard still returns Cloudflare AAAA
+      # records alongside the UniFi A record -- so without this, delivery depends on
+      # resolver address-ordering. The name (not the IP) is used so the wildcard cert
+      # still validates by hostname.
+      - "smtp.usesend.gbrown.org:192.168.2.2"
+    environment:
+      - TMPDIR=/var/lib/plausible/tmp
+      # required: https://github.com/plausible/community-edition/wiki/configuration#required
+      - BASE_URL=${BASE_URL}
+      - SECRET_KEY_BASE=${SECRET_KEY_BASE}
+      # optional: https://github.com/plausible/community-edition/wiki/configuration#optional
+      # registration: https://github.com/plausible/community-edition/wiki/configuration#registration
+      - TOTP_VAULT_KEY=${TOTP_VAULT_KEY}
+      - DISABLE_REGISTRATION=invite_only
+      - ENABLE_EMAIL_VERIFICATION=true
+      # web: https://github.com/plausible/community-edition/wiki/configuration#web
+      - HTTP_PORT=${HTTP_PORT}
+      - HTTPS_PORT=${HTTPS_PORT}
+      # databases: https://github.com/plausible/community-edition/wiki/configuration#database
+      - DATABASE_URL
+      - CLICKHOUSE_DATABASE_URL
+      # Google: https://github.com/plausible/community-edition/wiki/configuration#google
+      - GOOGLE_CLIENT_ID
+      - GOOGLE_CLIENT_SECRET
+      # geolocation: https://github.com/plausible/community-edition/wiki/configuration#ip-geolocation
+      - IP_GEOLOCATION_DB
+      - GEONAMES_SOURCE_FILE
+      - MAXMIND_LICENSE_KEY
+      - MAXMIND_EDITION
+      # email: https://github.com/plausible/community-edition/wiki/configuration#email
+      - MAILER_ADAPTER
+      - [email protected]
+      - MAILER_NAME=Gib's Plausible Analytics
+      - SMTP_HOST_ADDR=smtp.usesend.gbrown.org
+      - SMTP_HOST_PORT=587
+      - SMTP_USER_NAME=${SMTP_USER_NAME}
+      - SMTP_USER_PWD=${SMTP_USER_PWD}
+      - SMTP_HOST_SSL_ENABLED=false
+      - POSTMARK_API_KEY
+      - MAILGUN_API_KEY
+      - MAILGUN_DOMAIN
+      - MAILGUN_BASE_URI
+      - MANDRILL_API_KEY
+      - SENDGRID_API_KEY
+    restart: always
+    tty: true
+
+volumes:
+  db-data:
+  event-data:
+  event-logs:
+  plausible-data:

server/home/Plausible/podman-plausible.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Plausible
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Plausible
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/PostHog/.env.example

@@ -0,0 +1,29 @@
+POSTHOG_SECRET=CHANGE_ME
+ENCRYPTION_SALT_KEYS=CHANGE_ME
+BROWSERLESS_SECRET=CHANGE_ME
+DOMAIN=CHANGE_ME
+REGISTRY_URL=CHANGE_ME
+POSTHOG_APP_TAG=CHANGE_ME
+TLS_BLOCK=CHANGE_ME
+CADDY_TLS_BLOCK=CHANGE_ME
+CADDY_HOST=CHANGE_ME
+POSTHOG_HTTP_PORT=CHANGE_ME
+ANTHROPIC_API_KEY=CHANGE_ME
+OPENAI_API_KEY=CHANGE_ME
+USESEND_API_KEY=CHANGE_ME
+POSTHOG_GITLAB_LOGIN_KEY=CHANGE_ME
+POSTHOG_GITLAB_LOGIN_SECRET=CHANGE_ME
+SECRET_KEY=CHANGE_ME
+OBJECT_STORAGE_ACCESS_KEY_ID=CHANGE_ME
+OBJECT_STORAGE_SECRET_ACCESS_KEY=CHANGE_ME
+SESSION_RECORDING_V2_S3_ACCESS_KEY_ID=CHANGE_ME
+SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY=CHANGE_ME
+INTERNAL_API_SECRET=CHANGE_ME
+CSP_REPORT_ENDPOINT=CHANGE_ME
+DOCKER_REGISTRY_PREFIX=CHANGE_ME
+ELAPSED=CHANGE_ME
+HEATMAP_BROWSERLESS_SCREENSHOT_COOKIES_ENABLED=CHANGE_ME
+OPT_OUT_CAPTURE=CHANGE_ME
+POSTHOG_NODE_TAG=CHANGE_ME
+SEAWEEDFS_DOCKER_NAME=CHANGE_ME
+TIMEOUT=CHANGE_ME

server/home/PostHog/README.md

@@ -0,0 +1,391 @@
+# PostHog (self-hosted) on Fedora
+
+PostHog's official installer (`bin/deploy-hobby`) assumes Ubuntu/Debian and a
+bare host. This directory is that install done by hand, adapted to this server.
+
+## Why the official installer doesn't work here
+
+| What `deploy-hobby` does | Why it fails / what we did instead |
+|---|---|
+| `apt update`, `apt install git brotli curl` | Not Debian. All three are already installed via dnf. |
+| Installs Docker from `download.docker.com/linux/ubuntu` | Docker 29.8 is already installed. |
+| Downloads the v1 `docker-compose` binary to `/usr/local/bin` | We use the Compose v5 plugin (`docker compose`). |
+| Runs Caddy on host ports `80:80` / `443:443` and gets its own Let's Encrypt certs | nginx-proxy-manager owns 80/443. Caddy now serves plain HTTP on `127.0.0.1:8002` and NPM terminates TLS in front of it. |
+| Publishes Temporal on `0.0.0.0:7233` and its UI on `0.0.0.0:8081` | Rebound to `127.0.0.1`. |
+| Phones home to PostHog Cloud with your domain | Skipped. |
+| Generates `compose/start` calling `./compose/wait` | Upstream bug, fixed here. WORKDIR is `/code` but the mount is at `/compose`, so the relative path never resolves. Harmless (`bin/migrate` retries 10x) but it meant the readiness check never ran. Ours calls `/compose/wait`. |
+
+Note: Docker here runs *without* SELinux labelling (`docker info` lists only
+seccomp and cgroupns), so the bind mounts need no `:z` / `:Z` relabelling even
+though the host is SELinux-enforcing.
+
+Caddy is kept rather than replaced by NPM because it does path-based routing
+that NPM can't express in one proxy host: `/e`, `/i/v0`, `/batch`, `/capture`
+go to `capture`; `/s` to `replay-capture`; `/i/v1/logs` to `capture-logs`;
+`/livestream` to `livestream`; `/posthog/*` to object storage; everything else
+to `web`. NPM just forwards the whole hostname to Caddy.
+
+## Layout
+
+    compose.yml               docker-compose.hobby.yml + local patches
+    compose.override.yml      additive local settings, auto-merged by compose
+    docker-compose.base.yml   verbatim from the repo (compose.yml `extends` it)
+    .env                      secrets + domain  (chmod 600, never regenerate)
+    .env.services             verbatim from the repo
+    posthog/                  upstream checkout, only used for config bind mounts
+    share/                    GeoLite2-City.mmdb
+    compose/                  entrypoint scripts the web/worker containers run,
+                              plus local-patches.py (see "Multiple projects" below)
+    bin/upgrade               Fedora-native replacement for bin/upgrade-hobby
+    bin/apply-local-patches.py  replays our compose.yml deltas after an upgrade
+
+## nginx-proxy-manager
+
+Add a proxy host:
+
+- Domain: `posthog.gbrown.org`
+- Scheme: `http`, Forward host: `posthog`, Forward port: `80`
+- **Websockets: ON** (required — the `/livestream` endpoint)
+- Request an SSL cert, force SSL
+
+`posthog` is a network alias on `nginx-bridge` for the Caddy container.
+
+Also worth raising NPM's `client_max_body_size` for this host (Advanced tab:
+`client_max_body_size 50m;`) — session-recording batches and source-map
+uploads exceed the 1 MB default.
+
+## Two config files, two reasons
+
+`compose.yml` holds changes that *replace* upstream values -- the proxy's ports
+and networks. These have to be patched into the file directly, because Compose
+merges list-valued keys like `ports` by appending: an override file saying
+`8002:80` would leave you with 80, 443 *and* 8002. `bin/apply-local-patches.py`
+replays them after an upgrade overwrites the file.
+
+`compose.override.yml` holds purely *additive* settings (currently
+`TRUST_ALL_PROXIES` for the three Django services). Compose merges it
+automatically and no upgrade ever touches it, so nothing needs replaying.
+
+Rule of thumb: if you're adding an env var, put it in the override file. If
+you're changing a port or a network, patch compose.yml and add it to
+`bin/apply-local-patches.py`. `command` is the exception: Compose replaces it
+on merge rather than appending, so the worker's command lives in the override.
+
+## Operating it
+
+    docker compose up -d           # start
+    docker compose stop            # stop (keeps volumes)
+    docker compose logs -f web     # watch migrations / app
+    docker compose ps              # status
+    bin/upgrade                    # upgrade (runs async migration checks first)
+
+First boot runs Django + ClickHouse migrations and takes 5-15 minutes before
+`/_health` returns 200. The first person to visit the site creates the admin
+account, so log in promptly.
+
+## Multiple projects (source patches)
+
+Without a license, upstream PostHog allows one project per org and hides the
+"New project" button. PostHog no longer sells self-hosted licenses, so
+`compose/local-patches.py` patches two files in `/code` when `web`, `worker`
+and `temporal-django-worker` start:
+
+- `posthog/utils.py`: `can_create_org` follows `MULTI_ORG_ENABLED` alone, which
+  shows the button.
+- `posthog/models/organization.py`: the org keeps the `organizations_projects`
+  feature, which the project API checks. Without this, the hourly
+  `sync_all_organization_available_product_features` task resets the org's
+  features to `[]` and the API refuses a second project.
+
+Both are gated on `MULTI_ORG_ENABLED` in `compose.override.yml`. Because they run
+at container start, recreating a container no longer loses them. An earlier
+version patched running containers by hand and was wiped when the containers
+were recreated on 2026-09-22.
+
+If upstream changes the code, the script logs `PATTERN NOT FOUND` and the app
+starts unpatched. Check after upgrades (`bin/upgrade` prints it):
+
+    docker compose logs web worker | grep LOCAL-PATCH
+
+## Rebooting the host
+
+A host boot is not `docker compose up`. Dockerd starts containers from each one's
+restart policy alone, in no order, and never reads `depends_on`, so the dependency
+ordering the stack is built around does not exist at boot time.
+
+Upstream ships `restart: no` or `on-failure` on nearly every service, and
+`on-failure` does not retry a clean exit. Several services exit *zero* on purpose:
+`hypercache-server` when it loses the race to `redis7` on a cold boot, and
+`capture` when it loses Kafka (on 2026-09-27 a host memory spike stalled Kafka,
+capture shut down gracefully, and it stayed down for three days). So
+`compose.override.yml` sets `restart: unless-stopped` on every long-running
+service. A manual `docker compose stop` is still respected.
+
+Expect `hypercache-server` to flap a few times early in boot until `redis7`
+answers. That is the retry working, not a fault.
+
+`kafka-init` sitting at `Exited (0)` is correct. It is a one-shot init job and
+must keep `restart: no`.
+
+The web UI serves fine while these four are missing, so check
+`docker ps -a --filter name=posthog` for anything not `Up` before assuming a
+reboot went cleanly. To bring back anything stopped, in dependency order, without
+recreating running containers:
+
+    docker compose up -d --no-recreate
+
+## Watchtower is deliberately not enabled
+
+Every other stack here carries `com.centurylinklabs.watchtower.enable=true`.
+PostHog does not, on purpose: its upgrades run schema migrations that must be
+checked with `asyncmigrationscheck` before the app restarts, and a silent image
+swap underneath a running ClickHouse/Kafka pipeline can corrupt data. Use
+`bin/upgrade` instead.
+
+## Resource notes
+
+This stack is 37 containers (ClickHouse, Kafka, Zookeeper, Postgres, Redis,
+Valkey, Temporal, Elasticsearch, SeaweedFS, Browserless, and ~12 Rust/Node
+consumers). PostHog asks for 8 GB minimum; with Sentry already running here,
+keep an eye on `free -h`.
+
+If memory gets tight, these can be commented out of `compose.yml` with little
+loss on a personal instance:
+
+- `elasticsearch` — Temporal is configured with `ENABLE_ES=false`, so it is
+  only there for Temporal's advanced visibility. Saves ~1 GB.
+- `temporal-ui` — debugging UI for Temporal workflows only.
+- `cymbal` / `cymbal-resolution` — error-tracking stack-trace symbolication.
+  Skip if you're keeping errors in Sentry rather than PostHog.
+
+## Ad blockers and the hostname
+
+The host is `analytics.gbrown.org`. It was originally `posthog.gbrown.org`, which
+uBlock Origin blocked, leaving the dashboard a blank white page.
+
+The culprit is EasyPrivacy line 2964 (EasyPrivacy is enabled by default in uBO):
+
+    ://posthog.$script,domain=~posthog.com
+
+That is an *unanchored substring* pattern, not a domain rule, so any URL
+containing `://posthog.` matches -- including `https://posthog.gbrown.org/...`.
+The `$script` option is why the symptom was a white page rather than a failed
+connection: the HTML document is a `document` request and loaded fine, but every
+`/static/*.js` was blocked, so React never mounted.
+
+Renaming fixes it. Checked against all five lists uBO enables by default
+(EasyPrivacy, EasyList, uBO filters/privacy/badware, ~160k rules), with controls
+validating in both directions:
+
+    posthog.gbrown.org      BLOCKED (rule above)
+    analytics.gbrown.org    clean
+    stats/insights/metrics  clean
+    bills.gbrown.org        clean (control)
+
+Also clean on the new host: `/static/array.js`, `/static/recorder.js`,
+`/static/recorder-v2.js`, `/e/`, `/i/v0/e/`, `/batch/`, `/capture/`, `/decide/`.
+
+Generic words like "analytics" survive because the lists anchor to specific
+vendor domains (`||google-analytics.com`) rather than bare words -- blocking the
+word outright would break the web. The `://posthog.` rule is the unusual one.
+
+### To change the hostname again
+
+1. Edit `DOMAIN` in `.env`.
+2. Add the proxy host in nginx-proxy-manager: http -> `posthog` -> 80,
+   websockets on, `client_max_body_size 50m;`.
+3. `docker compose up -d`.
+
+Caddy listens on `:80` regardless of hostname, so nothing inside the stack needs
+touching -- that is the payoff of terminating TLS at NPM instead of in Caddy.
+Note that `SITE_URL` feeds `CSRF_TRUSTED_ORIGINS`, so existing browser sessions
+may need a fresh login after a rename. Capture/API traffic is unaffected.
+
+Still worth proxying capture through the app's own origin (a Next.js `/ingest`
+rewrite): it is same-origin, so no CORS and no preflight, and it survives future
+filter-list changes regardless of what this host is called.
+
+## Capture settings for the bill-tracker app (decided 2026-09-17)
+
+The bill tracker (Next + Expo + Convex) sends to this instance. Gib asked for
+everything on, so both clients ship with session replay, autocapture, heatmaps,
+console log capture and exception capture ENABLED, on posthog-js defaults:
+
+    maskAllInputs: true        (default — form inputs are masked)
+    rendered text: NOT masked  (default)
+
+The tradeoff, recorded deliberately rather than discovered later: this is a
+finance app, so bill amounts, balances, payee names and due dates are rendered
+*text*, not inputs, and therefore land in session replays verbatim. Autocapture
+also stores the text of clicked elements, so a button reading "Pay $1,247.33 to
+Chase" becomes an event property. Console log capture ships whatever the app
+logs, which in dev is often a whole transaction object.
+
+This is considered acceptable because the instance is self-hosted on Gib's own
+hardware and serves his own household — the data never leaves the house.
+
+To tighten later:
+
+    maskTextSelector: '*'                  // or scope to the money elements
+    class="ph-no-capture"                  // on balance / amount / payee nodes
+
+One asymmetry worth remembering: tightening does NOT retroactively clean stored
+replays. Anything recorded before the change stays recorded, so the cheap moment
+to decide is before real household data flows, not after.
+
+Client config lives in the bill-tracker repo, not here. The host is read from
+NEXT_PUBLIC_POSTHOG_HOST / EXPO_PUBLIC_POSTHOG_HOST (Infisical), and web routes
+capture through a same-origin /ingest rewrite rather than hitting this host
+directly.
+
+## Blocked asset filenames — why the client requests ext-a.js / ext-b.js
+
+Renaming the host to `analytics.gbrown.org` fixed the dashboard, but two of
+PostHog's own script filenames are blocked by rules that match the FILENAME, not
+the host, so a reverse proxy cannot dodge them:
+
+    easyprivacy.txt line 1906:   /posthog-recorder.js
+    easyprivacy.txt line  809:   /dead-clicks-autocapture.js
+
+Both are unanchored substrings with no type or domain options, so they match on
+any host, any request type, first-party included. Confirmed blocked both direct
+and through the app's /ingest proxy. Symptom: session replay and dead-click
+capture silently never start for anyone running uBlock.
+
+The bill-tracker app works around this with posthog-js's
+`prepare_external_dependency_script` hook, serving them under inert names:
+
+    posthog-recorder.js        -> ext-a.js
+    dead-clicks-autocapture.js -> ext-b.js
+
+(`strict_script_versioning` off, so only the plain `/static/<name>.js?v=` form is
+requested.) The Next rewrite maps the disguised names back. Verified: replay
+blobs land in `posthog.session_replay_events`, which was stuck at 0 before.
+
+Everything else PostHog serves is clean on both paths — `array.js`,
+`array.full.js`, `recorder.js`, `recorder-v2.js`, `surveys.js`,
+`exception-autocapture.js`, `web-vitals-with-attribution.js`,
+`tracing-headers.js`, `toolbar.js`, `external-scripts-loader.js`, `/e/`,
+`/flags/`. Note the legacy `recorder.js` and `recorder-v2.js` names are NOT
+listed; only the newer `posthog-recorder.js` is.
+
+One more, shape-dependent: `/s/array/<token>/config` is blocked by
+ubo-privacy.txt line 3159 (`$script,xhr,1p` — first-party too), but
+`/array/<token>/config.js` is clean. posthog-js 1.433.10 requests the latter, so
+it does not bite today. Worth re-checking after a major posthog-js upgrade.
+
+### Expected absences (not bugs)
+
+- `$pageleave` does NOT fire on client-side SPA navigation. posthog-js hangs it
+  off the real unload/visibilitychange lifecycle, so it tracks actual page exits
+  only. Verified against every pageleave in the instance.
+- `$web_vitals` flushes on page hide; `$dead_click` needs a click that resolves
+  to nothing. Both stay absent until those conditions occur.
+
+## nginx-proxy-manager buffer sizes (required, or replay breaks)
+
+Symptom: the Session Replay list page shows "Error while trying to load
+recordings". `GET /api/environments/<id>/session_recordings` returns 502 from
+openresty in ~130ms, while a SINGLE recording
+(`.../session_recordings/<session_id>`) returns 200 and `/events` returns 200.
+
+Cause is nginx, not PostHog. From `/data/logs/proxy-host-121_error.log`:
+
+    upstream sent too big header while reading response header from upstream
+
+nginx's default `proxy_buffer_size` is 4-8KB. PostHog's session_recordings LIST
+endpoint sends response headers larger than that, so nginx aborts before reading
+the body and returns 502. The request never reaches Django, which is why nothing
+appears in the web container logs and Caddy logs no 5xx at all. The single
+recording endpoint sends normal-sized headers, hence the asymmetry.
+
+Required in the NPM Advanced tab for this proxy host:
+
+    client_max_body_size 50m;
+    proxy_buffering on;
+    proxy_buffer_size 128k;
+    proxy_buffers 4 256k;
+
+The Sentry proxy host (1.conf) already carried the same three buffer directives —
+same nginx default, same fix, just never applied to PostHog's host.
+
+Verified in `/data/logs/proxy-host-121_access.log`: 502 at 22:46:06 before the
+change, 200 at 22:46:36 after, no further "too big header" entries.
+
+Do NOT edit the generated `.conf` files directly — NPM regenerates them from its
+own database, so a direct edit reverts on the next save.
+
+### Verifying replay from the server side
+
+ClickHouse `session_replay_events` proves events ARRIVED; it does not prove they
+can be PLAYED BACK. Those are different paths (ingestion -> Kafka -> object
+storage vs. API -> proxy -> browser). To check the storage side:
+
+    docker compose exec seaweedfs sh -c \
+      "echo 'fs.ls -l /buckets/posthog/session_recordings/5y' | weed shell -master=localhost:9333"
+
+Blobs live under `session_recordings/<retention>/`, named `<epoch_ms>-<hash>`.
+
+## "No recordings found" with replay working fine
+
+The Session Replay list ships a DEFAULT filter of `> 5 active seconds`. Any
+session with 5 or fewer *active* seconds is hidden, even though it recorded
+perfectly and plays back fine.
+
+Active time is not wall-clock time. The first verified session here ran 458s
+(7.6 min) but logged only 643ms of activity — one click and a little mouse
+movement in an otherwise idle tab. PostHog counts interaction, not duration,
+deliberately, to suppress bounces and accidental loads.
+
+That default makes quick dev checks vanish and looks identical to "replay is
+broken". To see them: open the duration chip on /replay/home and set the
+condition to `> 0 active seconds`, or save a filter preset. Date range and the
+"filter out internal and test users" toggle are usually NOT the culprit.
+
+Check the real numbers server-side before assuming anything is missing:
+
+    docker compose exec clickhouse clickhouse-client -q \
+      "select session_id, sum(active_milliseconds) active_ms,
+              dateDiff('second', min(min_first_timestamp), max(max_last_timestamp)) duration_s,
+              sum(click_count) clicks, sum(event_count) evts
+       from posthog.session_replay_events group by session_id"
+
+Verified 2026-09-17: session 01a0b177-d35c-7292-ac59-021af608d29a recorded,
+listed once the threshold was lowered, and played back in the browser.
+
+## Real client IPs through NPM -> Caddy
+
+By default Django logged `x_forwarded_for=172.25.0.8` — nginx-proxy-manager's own
+container IP — so every event's `$ip` resolved to the proxy and GeoIP failed on a
+private address.
+
+NPM was not at fault; it sends `X-Forwarded-For $proxy_add_x_forwarded_for` and
+`X-Real-IP $remote_addr` from its global proxy include. The loss was in Caddy:
+its global options block was empty, so `trusted_proxies` was unset, and Caddy's
+default is to distrust an incoming X-Forwarded-For and OVERWRITE it with the
+direct peer. Proven by sending `X-Forwarded-For: 9.9.9.9` to Caddy and watching
+Django log `172.25.0.1`.
+
+Fixed in `.env` — no compose patch, because `CADDY_TLS_BLOCK` is substituted
+directly into Caddy's global options block:
+
+    CADDY_TLS_BLOCK="servers {
+    	trusted_proxies static 172.25.0.0/24
+    }"
+
+Multi-line values work in `.env`; `docker compose config` resolves it correctly
+and `caddy validate` accepts it. Apply with `docker compose up -d proxy`.
+
+Verified after: a spoofed `X-Forwarded-For: 9.9.9.9` arrives as `ip=9.9.9.9`, and
+a real request through NPM arrives as the true client address.
+
+TRADEOFF: `/24` trusts every container on nginx-bridge (~42 of them), any of
+which could forge an X-Forwarded-For that Caddy now believes — compounded by
+`TRUST_ALL_PROXIES=true` on Django (see compose.override.yml). Acceptable for a
+LAN-only household instance; tighten to NPM's address if this is ever exposed to
+the internet. Pinning `172.25.0.8/32` is narrower but breaks silently whenever
+Docker reassigns NPM's IP.
+
+Note this does NOT produce geolocation: LAN clients are 192.168.x.x, which is not
+in the MaxMind database, so GeoIP still errors. What it buys is the ability to
+tell devices apart.

server/home/PostHog/UPSTREAM

@@ -0,0 +1,6 @@
+checkout: ~/Server/PostHog/posthog/
+repo:     https://github.com/PostHog/posthog
+ref:      master at this commit
+commit:   ddfaf8843fca8e796bf1f795976842d2d1adb380
+
+docker-compose.base.yml is the upstream file at that commit, unchanged.

server/home/PostHog/bin/apply-local-patches.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+"""Re-apply this server's deltas to a freshly-copied docker-compose.hobby.yml.
+
+Upstream's bin/upgrade-hobby overwrites compose.yml from the repo on every
+upgrade, so these three changes have to be replayed afterwards. Idempotent:
+running it on an already-patched file is a no-op.
+
+Usage: bin/apply-local-patches.py [compose.yml]
+"""
+import sys
+
+path = sys.argv[1] if len(sys.argv) > 1 else 'compose.yml'
+s = open(path).read()
+changed = []
+
+# 1) nginx-proxy-manager owns host 80/443. Publish Caddy on loopback instead and
+#    put it on nginx-bridge so NPM can reach it by name.
+old = """        ports:
+            - '80:80'
+            - '443:443'
+"""
+new = """        ports:
+            # nginx-proxy-manager owns host 80/443. Loopback only; NPM reaches
+            # this container over nginx-bridge as "posthog".
+            - '127.0.0.1:${POSTHOG_HTTP_PORT:-8002}:80'
+        networks:
+            default:
+            nginx-bridge:
+                aliases:
+                    - posthog
+"""
+if old in s:
+    s = s.replace(old, new)
+    changed.append('proxy ports + nginx-bridge')
+
+# 2) TLS terminates at NPM, so Caddy must not try to get its own certs. Take the
+#    site address from .env (CADDY_HOST=:80) instead of the hardcoded domain.
+old = "            CADDY_HOST: '$DOMAIN, http://, https://'\n"
+if old in s:
+    s = s.replace(old, "            CADDY_HOST: $CADDY_HOST\n")
+    changed.append('CADDY_HOST from .env')
+
+# 3) Temporal gRPC + UI on loopback, not every interface.
+for a, b in (("            - 7233:7233\n", "            - 127.0.0.1:7233:7233\n"),
+             ("            - 8081:8080\n", "            - 127.0.0.1:8081:8080\n")):
+    if a in s:
+        s = s.replace(a, b)
+        changed.append(a.strip())
+
+# 4) nginx-bridge is created and owned by ~/Server/Nginx.
+old = "networks:\n    otel_network:\n"
+if old in s and 'nginx-bridge:\n        external: true' not in s:
+    s = s.replace(old, "networks:\n    otel_network:\n    nginx-bridge:\n        external: true\n")
+    changed.append('nginx-bridge external')
+
+open(path, 'w').write(s)
+print(f"{path}: " + (", ".join(changed) if changed else "already patched, no changes"))

server/home/PostHog/bin/upgrade

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+#
+# Fedora-native replacement for posthog/bin/upgrade-hobby.
+#
+# Upstream's script is unusable here: it calls apt-get, re-downloads the v1
+# `docker-compose` binary, and overwrites compose.yml from the repo (throwing
+# away this server's nginx-proxy-manager and port changes). This does the same
+# work using dnf-era tooling and replays the local patches afterwards.
+#
+set -euo pipefail
+cd "$(dirname "$0")/.."
+
+echo "==> Refreshing the PostHog checkout"
+git -C posthog fetch origin
+git -C posthog reset --hard origin/master
+echo "    now on $(git -C posthog rev-parse --short HEAD)"
+
+echo "==> Refreshing compose files and re-applying local patches"
+cp posthog/docker-compose.base.yml docker-compose.base.yml
+cp posthog/docker-compose.hobby.yml compose.yml
+cp posthog/.env.services .env.services
+bin/apply-local-patches.py compose.yml
+docker compose config >/dev/null && echo "    compose config OK"
+
+echo "==> Refreshing the GeoIP database"
+curl -sL 'https://mmdbcdn.posthog.net/' --http1.1 | brotli --decompress > share/GeoLite2-City.mmdb
+echo "{\"date\": \"$(date +%F)\"}" > share/GeoLite2-City.json
+chmod 644 share/GeoLite2-City.mmdb share/GeoLite2-City.json
+
+echo "==> Pulling images"
+docker compose pull
+
+echo "==> Checking async migrations against the new version"
+docker compose up -d db redis7 clickhouse
+docker compose run --rm asyncmigrationscheck
+
+echo "==> Restarting the stack"
+docker compose up -d --remove-orphans
+
+# web, worker and temporal-django-worker apply compose/local-patches.py on start.
+# Surface the result so an upstream change that breaks a patch is not missed.
+echo "==> Local source patches"
+sleep 15
+docker compose logs --since 5m web worker temporal-django-worker 2>&1 | grep LOCAL-PATCH || \
+    echo "    no LOCAL-PATCH lines yet, check: docker compose logs web worker | grep LOCAL-PATCH"
+
+echo
+echo "Done. Watch the migrations with:  docker compose logs -f web"

server/home/PostHog/compose.override.yml

@@ -0,0 +1,99 @@
+services:
+  web: &id001
+    restart: unless-stopped
+    environment:
+      TRUST_ALL_PROXIES: 'true'
+      MULTI_ORG_ENABLED: 'true'
+      # SSO: the "GitLab" login button is really Authentik. NPM on the VPS maps GitLab's
+      # /oauth/* and /api/v4/user paths on auth.gbrown.org to Authentik, the same shim
+      # Infisical uses. GitLab login needs no license here; only Google is gated.
+      SOCIAL_AUTH_GITLAB_API_URL: https://auth.gbrown.org
+      SOCIAL_AUTH_GITLAB_KEY: ${POSTHOG_GITLAB_LOGIN_KEY:?set POSTHOG_GITLAB_LOGIN_KEY in .env}
+      SOCIAL_AUTH_GITLAB_SECRET: ${POSTHOG_GITLAB_LOGIN_SECRET:?set POSTHOG_GITLAB_LOGIN_SECRET in .env}
+      PERSONHOG_ADDR: personhog-router:50052
+      PERSISTED_FEATURE_FLAGS: sql-editor-vim-mode,cdp-hog-sources,customer-analytics-roadmap,marketing-analytics,web-experiments,dashboard-widgets,twig,tracing,metrics
+      EMAIL_ENABLED: 'true'
+      EMAIL_HOST: smtp.usesend.gbrown.org
+      EMAIL_PORT: '587'
+      EMAIL_HOST_USER: usesend
+      EMAIL_HOST_PASSWORD: ${USESEND_API_KEY:?set USESEND_API_KEY in .env}
+      EMAIL_USE_TLS: 'true'
+      EMAIL_USE_SSL: 'false'
+      EMAIL_DEFAULT_FROM: PostHog <[email protected]>
+      EMAIL_REPLY_TO: [email protected]
+  worker:
+    <<: *id001
+    # compose/worker applies compose/local-patches.py, then runs upstream's
+    # command. `command` is replaced on merge (not appended), so it is safe here.
+    command: /compose/worker
+    volumes:
+      - ./compose:/compose
+  temporal-django-worker: *id001
+  # Every long-running service restarts regardless of exit code. Upstream ships
+  # `on-failure` (or `no`), which never retries a clean exit, and these services
+  # exit 0 on purpose: capture shuts down gracefully when it loses Kafka
+  # (2026-09-27, down 3 days), hypercache-server exits 0 when it beats redis7 at
+  # boot. web/worker/temporal-django-worker get this via the anchor above.
+  # kafka-init and asyncmigrationscheck are one-shots and must stay `no`.
+  browserless:
+    restart: unless-stopped
+  capture:
+    restart: unless-stopped
+  capture-logs:
+    restart: unless-stopped
+  clickhouse:
+    restart: unless-stopped
+  cymbal:
+    restart: unless-stopped
+  cymbal-resolution:
+    restart: unless-stopped
+  db:
+    restart: unless-stopped
+  elasticsearch:
+    restart: unless-stopped
+  feature-flags:
+    restart: unless-stopped
+  hypercache-server:
+    restart: unless-stopped
+  ingestion-error-tracking:
+    restart: unless-stopped
+  ingestion-general:
+    restart: unless-stopped
+  ingestion-logs:
+    restart: unless-stopped
+  ingestion-sessionreplay:
+    restart: unless-stopped
+  ingestion-traces:
+    restart: unless-stopped
+  kafka:
+    restart: unless-stopped
+  livestream:
+    restart: unless-stopped
+  objectstorage:
+    restart: unless-stopped
+  personhog-replica:
+    restart: unless-stopped
+  personhog-router:
+    restart: unless-stopped
+  plugins:
+    restart: unless-stopped
+  property-defs-rs:
+    restart: unless-stopped
+  recording-api:
+    restart: unless-stopped
+  redis7:
+    restart: unless-stopped
+  replay-capture:
+    restart: unless-stopped
+  seaweedfs:
+    restart: unless-stopped
+  temporal:
+    restart: unless-stopped
+  temporal-admin-tools:
+    restart: unless-stopped
+  temporal-ui:
+    restart: unless-stopped
+  valkey:
+    restart: unless-stopped
+  zookeeper:
+    restart: unless-stopped

server/home/PostHog/compose.yml

@@ -0,0 +1,767 @@
+#
+# `docker-compose` file used ONLY for hobby deployments.
+#
+# Please take a look at https://posthog.com/docs/self-host/deploy/hobby
+# for more info.
+#
+# PostHog has sunset support for self-hosted K8s deployments.
+# See: https://posthog.com/blog/sunsetting-helm-support-posthog
+#
+
+x-logging: &default-logging
+    driver: json-file
+    options:
+        max-size: '50m'
+        max-file: '3'
+
+services:
+    db:
+        extends:
+            file: docker-compose.base.yml
+            service: db
+        image: ${DOCKER_REGISTRY_PREFIX:-}postgres:15.12-alpine
+        logging: *default-logging
+        volumes:
+            - postgres-data:/var/lib/postgresql/data
+
+    redis7:
+        extends:
+            file: docker-compose.base.yml
+            service: redis7
+        logging: *default-logging
+        volumes:
+            - redis7-data:/data
+
+    valkey:
+        extends:
+            file: docker-compose.base.yml
+            service: valkey
+        logging: *default-logging
+
+    clickhouse:
+        extends:
+            file: docker-compose.base.yml
+            service: clickhouse
+        restart: on-failure
+        logging: *default-logging
+        depends_on:
+            - kafka
+            - zookeeper
+        volumes:
+            - ./posthog/posthog/idl:/idl
+            - ./posthog/docker/clickhouse/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
+            - ./posthog/docker/clickhouse/config.xml:/etc/clickhouse-server/config.xml
+            - ./posthog/docker/clickhouse/config.d/default.xml:/etc/clickhouse-server/config.d/default.xml
+            - ./posthog/docker/clickhouse/users.xml:/etc/clickhouse-server/users.xml
+            - ./posthog/docker/clickhouse/user_defined_function.xml:/etc/clickhouse-server/user_defined_function.xml
+            - ./posthog/posthog/user_scripts:/var/lib/clickhouse/user_scripts
+            - clickhouse-data:/var/lib/clickhouse
+
+    zookeeper:
+        extends:
+            file: docker-compose.base.yml
+            service: zookeeper
+        logging: *default-logging
+        environment:
+            ZOO_AUTOPURGE_PURGEINTERVAL: 1
+            ZOO_AUTOPURGE_SNAPRETAINCOUNT: 3
+        volumes:
+            - zookeeper-datalog:/datalog
+            - zookeeper-data:/data
+            - zookeeper-logs:/logs
+    kafka:
+        extends:
+            file: docker-compose.base.yml
+            service: kafka
+        logging: *default-logging
+        depends_on:
+            - zookeeper
+        environment:
+            KAFKA_LOG_RETENTION_MS: 3600000
+            KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 300000
+            KAFKA_LOG_RETENTION_HOURS: 1
+        volumes:
+            - kafka-data:/bitnami/kafka
+
+    worker:
+        extends:
+            file: docker-compose.base.yml
+            service: worker
+        logging: *default-logging
+        environment:
+            SITE_URL: https://$DOMAIN
+            SECRET_KEY: ${SECRET_KEY}
+            OBJECT_STORAGE_ACCESS_KEY_ID: ${OBJECT_STORAGE_ACCESS_KEY_ID}
+            OBJECT_STORAGE_SECRET_ACCESS_KEY: ${OBJECT_STORAGE_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENDPOINT: http://objectstorage:19000
+            OBJECT_STORAGE_PUBLIC_ENDPOINT: https://$DOMAIN
+            SESSION_RECORDING_V2_S3_ENDPOINT: http://seaweedfs:8333
+            SESSION_RECORDING_V2_S3_ACCESS_KEY_ID: ${SESSION_RECORDING_V2_S3_ACCESS_KEY_ID}
+            SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY: ${SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENABLED: true
+            RECORDING_API_URL: http://recording-api:6738
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET:-$POSTHOG_SECRET}
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            POSTHOG_SKIP_MIGRATION_CHECKS: '1'
+            PERSONHOG_ADDR: 'personhog-router:50052'
+            PERSONHOG_ENABLED: 'true'
+            BROWSERLESS_CDP_URL: 'ws://browserless:3000'
+            BROWSERLESS_TOKEN: ${BROWSERLESS_SECRET:-$POSTHOG_SECRET}
+            HEATMAP_BROWSERLESS_URL: 'http://browserless:3000'
+            HEATMAP_BROWSERLESS_TOKEN: ${BROWSERLESS_SECRET:-$POSTHOG_SECRET}
+            HEATMAP_BROWSERLESS_SCREENSHOT_COOKIES_ENABLED: ${HEATMAP_BROWSERLESS_SCREENSHOT_COOKIES_ENABLED:-false}
+            # Point Django at the Rust feature-flags container; the setting
+            # otherwise defaults to localhost:3001.
+            FEATURE_FLAGS_SERVICE_URL: 'http://feature-flags:3001'
+            # The logs product uses a dedicated ClickHouse cluster config (separate
+            # from CLICKHOUSE_HOST) whose host defaults to localhost and whose SECURE
+            # defaults to true outside DEBUG. Point it at the stack's ClickHouse over
+            # plaintext; the other logs-cluster settings already default correctly
+            # (native port 9000, user "default", empty password).
+            CLICKHOUSE_LOGS_CLUSTER_HOST: 'clickhouse'
+            CLICKHOUSE_LOGS_CLUSTER_SECURE: 'false'
+            # AI provider keys for PostHog's AI features. Blank unless set in .env.
+            ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
+            OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+        image: $REGISTRY_URL:$POSTHOG_APP_TAG
+        depends_on:
+            db:
+                condition: service_healthy
+            redis7:
+                condition: service_healthy
+            clickhouse:
+                condition: service_started
+            kafka:
+                condition: service_healthy
+            web:
+                condition: service_started
+            personhog-router:
+                condition: service_started
+            browserless:
+                condition: service_started
+    web:
+        extends:
+            file: docker-compose.base.yml
+            service: web
+        command: /compose/start
+        logging: *default-logging
+        volumes:
+            - ./compose:/compose
+        image: $REGISTRY_URL:$POSTHOG_APP_TAG
+        environment:
+            SITE_URL: https://$DOMAIN
+            LIVESTREAM_HOST: 'https://${DOMAIN}/livestream'
+            SECRET_KEY: ${SECRET_KEY}
+            OBJECT_STORAGE_ACCESS_KEY_ID: ${OBJECT_STORAGE_ACCESS_KEY_ID}
+            OBJECT_STORAGE_SECRET_ACCESS_KEY: ${OBJECT_STORAGE_SECRET_ACCESS_KEY}
+            SESSION_RECORDING_V2_S3_ACCESS_KEY_ID: ${SESSION_RECORDING_V2_S3_ACCESS_KEY_ID}
+            SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY: ${SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENDPOINT: http://objectstorage:19000
+            OBJECT_STORAGE_PUBLIC_ENDPOINT: https://$DOMAIN
+            SESSION_RECORDING_V2_S3_ENDPOINT: http://seaweedfs:8333
+            OBJECT_STORAGE_ENABLED: true
+            RECORDING_API_URL: http://recording-api:6738
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET:-$POSTHOG_SECRET}
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            OTEL_SERVICE_NAME: 'posthog'
+            OTEL_EXPORTER_OTLP_ENDPOINT: ''
+            OTEL_SDK_DISABLED: 'true'
+            OPT_OUT_CAPTURE: ${OPT_OUT_CAPTURE:-false}
+            # Blank sends no CSP reports. Point it at this instance's own /report/ endpoint to
+            # collect them yourself.
+            CSP_REPORT_ENDPOINT: ${CSP_REPORT_ENDPOINT:-}
+            DEPLOYMENT: 'hobby'
+            PERSONHOG_ADDR: 'personhog-router:50052'
+            PERSONHOG_ENABLED: 'true'
+            # Point Django at the Rust feature-flags container; the setting
+            # otherwise defaults to localhost:3001.
+            FEATURE_FLAGS_SERVICE_URL: 'http://feature-flags:3001'
+            # The logs product uses a dedicated ClickHouse cluster config (separate
+            # from CLICKHOUSE_HOST) whose host defaults to localhost and whose SECURE
+            # defaults to true outside DEBUG. Point it at the stack's ClickHouse over
+            # plaintext; the other logs-cluster settings already default correctly
+            # (native port 9000, user "default", empty password).
+            CLICKHOUSE_LOGS_CLUSTER_HOST: 'clickhouse'
+            CLICKHOUSE_LOGS_CLUSTER_SECURE: 'false'
+            # AI provider keys for PostHog's AI features. Blank unless set in .env.
+            ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
+            OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+        depends_on:
+            - db
+            - redis7
+            - clickhouse
+            - kafka
+            - objectstorage
+            - seaweedfs
+            - personhog-router
+
+    plugins:
+        extends:
+            file: docker-compose.base.yml
+            service: plugins
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            SITE_URL: https://$DOMAIN
+            SECRET_KEY: ${SECRET_KEY}
+            OBJECT_STORAGE_ACCESS_KEY_ID: ${OBJECT_STORAGE_ACCESS_KEY_ID}
+            OBJECT_STORAGE_SECRET_ACCESS_KEY: ${OBJECT_STORAGE_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENDPOINT: http://objectstorage:19000
+            OBJECT_STORAGE_PUBLIC_ENDPOINT: https://$DOMAIN
+            OBJECT_STORAGE_ENABLED: true
+            CDP_REDIS_HOST: redis7
+            CDP_REDIS_PORT: 6379
+            LOGS_REDIS_HOST: redis7
+            LOGS_REDIS_PORT: 6379
+            LOGS_REDIS_TLS: 'false'
+            TRACES_REDIS_HOST: redis7
+            TRACES_REDIS_PORT: 6379
+            TRACES_REDIS_TLS: 'false'
+            # Every CDP process dual-writes to Valkey and refuses to start without a host.
+            CDP_VALKEY_HOST: valkey
+            CDP_VALKEY_PORT: 6379
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            PERSONS_DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            BEHAVIORAL_COHORTS_DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            OTEL_EXPORTER_OTLP_ENDPOINT: ''
+        volumes:
+            - ./share:/share
+        depends_on:
+            - db
+            - redis7
+            - valkey
+            - clickhouse
+            - kafka
+            - objectstorage
+            - seaweedfs
+
+    ingestion-general:
+        extends:
+            file: docker-compose.base.yml
+            service: ingestion-general
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            PERSONS_DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            BEHAVIORAL_COHORTS_DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            # The combined-mode AI consumer reads persons through PersonHog
+            PERSONHOG_ADDR: 'personhog-router:50052'
+            PERSONHOG_ENABLED: 'true'
+        depends_on:
+            - db
+            - redis7
+            - clickhouse
+            - kafka
+            # The combined AI consumer's blob store healthchecks the ai-blobs bucket at startup
+            - objectstorage
+            - personhog-router
+
+    ingestion-sessionreplay:
+        extends:
+            file: docker-compose.base.yml
+            service: ingestion-sessionreplay
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            SESSION_RECORDING_V2_S3_ACCESS_KEY_ID: ${SESSION_RECORDING_V2_S3_ACCESS_KEY_ID}
+            SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY: ${SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY}
+            SESSION_RECORDING_V2_S3_TIMEOUT_MS: 120000
+            SESSION_RECORDING_V2_S3_ENDPOINT: http://seaweedfs:8333
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+        depends_on:
+            - db
+            - redis7
+            - kafka
+            - seaweedfs
+
+    recording-api:
+        extends:
+            file: docker-compose.base.yml
+            service: recording-api
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            # Read side of the session recording V2 blob store. Must point at the same
+            # SeaweedFS the ingestion-sessionreplay writer uses — without these the
+            # service falls back to its localhost:8333 default, so every playback
+            # request 500s with ECONNREFUSED ("We're unable to play this recording").
+            SESSION_RECORDING_V2_S3_ACCESS_KEY_ID: ${SESSION_RECORDING_V2_S3_ACCESS_KEY_ID}
+            SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY: ${SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY}
+            SESSION_RECORDING_V2_S3_ENDPOINT: http://seaweedfs:8333
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            # Must resolve to the same value as the web and worker services, which send it as
+            # X-Internal-Api-Secret. The JWT signing secret is deliberately left unset here, so hobby
+            # stays on the legacy shared secret until that scheme is rolled out.
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET:-$POSTHOG_SECRET}
+        depends_on:
+            - db
+            - redis7
+            - clickhouse
+            - seaweedfs
+
+    ingestion-error-tracking:
+        extends:
+            file: docker-compose.base.yml
+            service: ingestion-error-tracking
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            ERROR_TRACKING_CYMBAL_BASE_URL: 'http://cymbal:3302'
+        depends_on:
+            - db
+            - redis7
+            - kafka
+
+    ingestion-logs:
+        extends:
+            file: docker-compose.base.yml
+            service: ingestion-logs
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            LOGS_REDIS_TLS: 'false'
+        depends_on:
+            - db
+            - redis7
+            - kafka
+
+    ingestion-traces:
+        extends:
+            file: docker-compose.base.yml
+            service: ingestion-traces
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            TRACES_REDIS_TLS: 'false'
+        depends_on:
+            - db
+            - redis7
+            - kafka
+
+    proxy:
+        extends:
+            file: docker-compose.base.yml
+            service: proxy
+        logging: *default-logging
+        ports:
+            # nginx-proxy-manager owns host 80/443. Loopback only; NPM reaches
+            # this container over nginx-bridge as "posthog".
+            - '127.0.0.1:${POSTHOG_HTTP_PORT:-8002}:80'
+        networks:
+            default:
+            nginx-bridge:
+                aliases:
+                    - posthog
+        volumes:
+            - caddy-data:/data
+            - caddy-config:/config
+        environment:
+            CADDY_TLS_BLOCK: $TLS_BLOCK
+            CADDY_HOST: $CADDY_HOST
+        depends_on:
+            - web
+            - livestream
+    objectstorage:
+        # SeaweedFS on the same objectstorage:19000 endpoint and credentials this service
+        # served when it ran MinIO, so no other service in this file changes.
+        extends:
+            file: docker-compose.base.yml
+            service: objectstorage
+        restart: on-failure
+        logging: *default-logging
+        # The dev/CI bootstrap this replaces also registers an `anonymous` identity with
+        # Admin rights. Hobby must not, because Caddy proxies /posthog/* straight to this
+        # service: an anonymous identity opens every object to the internet. With only the
+        # posthog identity, SeaweedFS rejects unsigned requests and the presigned URLs
+        # Django hands out keep working. There is no duckgres here, so there is no
+        # ducklake-dev bucket either. The loop holds no shell variables on purpose:
+        # bin/upgrade-hobby pipes this file through envsubst, which collapses a
+        # doubled-dollar variable reference to a bare dollar sign.
+        entrypoint:
+            - /bin/sh
+            - -c
+            - |
+                (
+                    while true; do
+                        printf 's3.configure -access_key=object_storage_root_user -secret_key=object_storage_root_password -user=posthog -actions=Admin -apply\ns3.bucket.create -name posthog\ns3.bucket.create -name ai-blobs\n' \
+                                | /usr/bin/weed shell -master=localhost:19001 >/dev/null 2>&1 || true
+                        if echo 's3.bucket.list' | /usr/bin/weed shell -master=localhost:19001 2>/dev/null | grep -q ai-blobs \
+                                && echo 's3.configure' | /usr/bin/weed shell -master=localhost:19001 2>/dev/null | grep -q object_storage_root_user; then
+                            touch /tmp/objectstorage-ready
+                            echo "Objectstorage SeaweedFS ready with required buckets and identities"
+                            break
+                        fi
+                        sleep 1
+                    done
+                ) &
+                exec /usr/bin/weed "$$@"
+            - --
+        # A new volume. SeaweedFS cannot read the MinIO layout, so the old objects stay on
+        # the `objectstorage` volume until the operator copies them across with
+        # bin/migrate-storage-hobby or deletes the volume.
+        volumes:
+            - objectstorage-data:/data
+        ports:
+            # Localhost only. MinIO published this port on every interface.
+            - '127.0.0.1:19000:19000'
+
+    seaweedfs:
+        extends:
+            file: docker-compose.base.yml
+            service: seaweedfs
+        container_name: '${SEAWEEDFS_DOCKER_NAME:-seaweedfs-main}'
+        restart: on-failure
+        logging: *default-logging
+        ports:
+            - '127.0.0.1:8333:8333' # S3 API
+            - '127.0.0.1:9333:9333' # Master server
+        volumes:
+            - seaweedfs:/data
+        healthcheck:
+            test:
+                [
+                    'CMD',
+                    'sh',
+                    '-c',
+                    "echo 's3.bucket.list' | /usr/bin/weed shell -master=localhost:9333 2>&1 | grep -q posthog",
+                ]
+            interval: 5s
+            timeout: 10s
+            retries: 30
+            start_period: 10s
+
+    browserless:
+        # Renders image exports and heatmap screenshots (no browser ships in the PostHog image).
+        image: ghcr.io/browserless/chromium:v2.51.2
+        restart: on-failure
+        logging: *default-logging
+        environment:
+            TOKEN: ${BROWSERLESS_SECRET:-$POSTHOG_SECRET}
+            LOG_LEVEL: warn
+            DEBUG: '-*'
+
+    asyncmigrationscheck:
+        extends:
+            file: docker-compose.base.yml
+            service: asyncmigrationscheck
+        image: $REGISTRY_URL:$POSTHOG_APP_TAG
+        logging: *default-logging
+        environment:
+            SITE_URL: https://$DOMAIN
+            SECRET_KEY: ${SECRET_KEY}
+            SKIP_ASYNC_MIGRATIONS_SETUP: 0
+
+    # Temporal containers
+    temporal:
+        extends:
+            file: docker-compose.base.yml
+            service: temporal
+        logging: *default-logging
+        environment:
+            - ENABLE_ES=false
+        ports:
+            - 127.0.0.1:7233:7233
+        volumes:
+            - ./posthog/docker/temporal/dynamicconfig:/etc/temporal/config/dynamicconfig
+    elasticsearch:
+        extends:
+            file: docker-compose.base.yml
+            service: elasticsearch
+        logging: *default-logging
+    temporal-admin-tools:
+        extends:
+            file: docker-compose.base.yml
+            service: temporal-admin-tools
+        logging: *default-logging
+        depends_on:
+            - temporal
+    temporal-ui:
+        extends:
+            file: docker-compose.base.yml
+            service: temporal-ui
+        logging: *default-logging
+        ports:
+            - 127.0.0.1:8081:8080
+        depends_on:
+            temporal:
+                condition: service_started
+            db:
+                condition: service_healthy
+    temporal-django-worker:
+        command: /compose/temporal-django-worker
+        extends:
+            file: docker-compose.base.yml
+            service: temporal-django-worker
+        logging: *default-logging
+        volumes:
+            - ./compose:/compose
+        image: $REGISTRY_URL:$POSTHOG_APP_TAG
+        environment:
+            SITE_URL: https://$DOMAIN
+            SECRET_KEY: ${SECRET_KEY}
+            BROWSERLESS_CDP_URL: 'ws://browserless:3000'
+            BROWSERLESS_TOKEN: ${BROWSERLESS_SECRET:-$POSTHOG_SECRET}
+            HEATMAP_BROWSERLESS_URL: 'http://browserless:3000'
+            HEATMAP_BROWSERLESS_TOKEN: ${BROWSERLESS_SECRET:-$POSTHOG_SECRET}
+            HEATMAP_BROWSERLESS_SCREENSHOT_COOKIES_ENABLED: ${HEATMAP_BROWSERLESS_SCREENSHOT_COOKIES_ENABLED:-false}
+            # This worker runs the recording deletion Temporal activities, which call
+            # recording-api. Without these two the calls go out unauthenticated and
+            # recording-api rejects them with a 401.
+            RECORDING_API_URL: http://recording-api:6738
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET:-$POSTHOG_SECRET}
+            # Point Django at the Rust feature-flags container; the setting
+            # otherwise defaults to localhost:3001.
+            FEATURE_FLAGS_SERVICE_URL: 'http://feature-flags:3001'
+            # The logs product uses a dedicated ClickHouse cluster config (separate
+            # from CLICKHOUSE_HOST) whose host defaults to localhost and whose SECURE
+            # defaults to true outside DEBUG. Point it at the stack's ClickHouse over
+            # plaintext; the other logs-cluster settings already default correctly
+            # (native port 9000, user "default", empty password). This worker runs
+            # the logs alert-check Temporal activities, which query ClickHouse under
+            # Workload.LOGS.
+            CLICKHOUSE_LOGS_CLUSTER_HOST: 'clickhouse'
+            CLICKHOUSE_LOGS_CLUSTER_SECURE: 'false'
+            # AI provider keys for PostHog's AI features. Blank unless set in .env.
+            ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
+            OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+        depends_on:
+            - db
+            - redis7
+            - clickhouse
+            - kafka
+            - objectstorage
+            - seaweedfs
+            - temporal
+            - browserless
+
+    capture:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: capture
+        logging: *default-logging
+
+    capture-logs:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: capture-logs
+        logging: *default-logging
+
+    replay-capture:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: replay-capture
+        logging: *default-logging
+
+    property-defs-rs:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: property-defs-rs
+        logging: *default-logging
+        depends_on:
+            kafka-init:
+                condition: service_completed_successfully
+            db:
+                condition: service_healthy
+
+    livestream:
+        extends:
+            file: docker-compose.base.yml
+            service: livestream
+        logging: *default-logging
+        environment:
+            - LIVESTREAM_JWT_SECRET=${POSTHOG_SECRET}
+        volumes:
+            - ./posthog/docker/livestream/configs-hobby.yml:/configs/configs.yml
+
+    personhog-replica:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: personhog-replica
+        logging: *default-logging
+        environment:
+            PRIMARY_DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+        depends_on:
+            db:
+                condition: service_healthy
+
+    personhog-router:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: personhog-router
+        logging: *default-logging
+        depends_on:
+            - personhog-replica
+
+    feature-flags:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: feature-flags
+        logging: *default-logging
+        depends_on:
+            - db
+            - redis7
+
+    hypercache-server:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: hypercache-server
+        logging: *default-logging
+        depends_on:
+            - redis7
+
+    kafka-init:
+        image: docker.io/redpandadata/redpanda:v25.1.9
+        entrypoint: /bin/sh
+        logging: *default-logging
+        command:
+            - -c
+            - |
+                set -x
+                echo "Waiting for Kafka broker to accept connections..."
+                TIMEOUT=60
+                ELAPSED=0
+                # Use rpk topic list as the readiness check - it's a simple operation
+                # that confirms the broker is accepting Kafka protocol requests
+                until rpk topic list --brokers kafka:9092 2>/dev/null; do
+                    echo "Kafka broker not ready yet (elapsed: $${ELAPSED}s)..."
+                    sleep 2
+                    ELAPSED=$$((ELAPSED + 2))
+                    if [ $$ELAPSED -ge $$TIMEOUT ]; then
+                        echo "Timeout waiting for Kafka broker after $${TIMEOUT}s"
+                        echo "Final attempt to list topics:"
+                        rpk topic list --brokers kafka:9092 || true
+                        exit 1
+                    fi
+                done
+                echo "Kafka broker is accepting requests, creating topics..."
+                # The topic list is docker/kafka/topics.txt, mounted below, shared with
+                # docker-compose.dev.yml so the two stacks cannot drift apart. That header
+                # explains what needs pre-creating.
+                # Note: $$ escapes $ for docker-compose variable substitution
+                TOPICS=$$(grep -vE '^[[:space:]]*#|^[[:space:]]*$$' /topics.txt)
+                for topic in $$TOPICS; do
+                    if rpk topic create "$$topic" --brokers kafka:9092 -p 1 -r 1 2>&1; then
+                        echo "Topic $$topic created successfully"
+                    else
+                        if rpk topic list --brokers kafka:9092 | awk '{print $$1}' | grep -qx "$$topic"; then
+                            echo "Topic $$topic already exists, continuing"
+                        else
+                            echo "Failed to create topic $$topic"
+                            exit 1
+                        fi
+                    fi
+                done
+                echo "Final topic list:"
+                rpk topic list --brokers kafka:9092
+                echo "Topics ready"
+        volumes:
+            - ./posthog/docker/kafka/topics.txt:/topics.txt:ro
+        depends_on:
+            kafka:
+                condition: service_healthy
+
+    cymbal:
+        image: ghcr.io/posthog/posthog/cymbal:master
+        build:
+            context: ./posthog/rust
+            args:
+                BIN: cymbal
+        restart: on-failure
+        logging: *default-logging
+        volumes:
+            - ./share:/share
+        environment:
+            KAFKA_HOSTS: 'kafka:9092'
+            OBJECT_STORAGE_BUCKET: posthog
+            OBJECT_STORAGE_ACCESS_KEY_ID: ${OBJECT_STORAGE_ACCESS_KEY_ID}
+            OBJECT_STORAGE_SECRET_ACCESS_KEY: ${OBJECT_STORAGE_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENDPOINT: 'http://seaweedfs:8333'
+            OBJECT_STORAGE_FORCE_PATH_STYLE: 'true'
+            BIND_HOST: '0.0.0.0'
+            BIND_PORT: '3302'
+            DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            PERSONS_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            MAXMIND_DB_PATH: '/share/GeoLite2-City.mmdb'
+            REDIS_URL: 'redis://redis7:6379/'
+            ISSUE_BUCKETS_REDIS_URL: 'redis://redis7:6379/'
+            RUST_LOG: 'info'
+            # Processing offloads all symbol resolution to cymbal-resolution over gRPC. Both are
+            # mandatory: startup aborts when either is empty, so omitting them crash-loops cymbal.
+            CYMBAL_REMOTE_RESOLUTION_HOST: 'cymbal-resolution'
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET}
+        depends_on:
+            kafka-init:
+                condition: service_completed_successfully
+            seaweedfs:
+                condition: service_healthy
+            db:
+                condition: service_healthy
+            redis7:
+                condition: service_healthy
+            cymbal-resolution:
+                condition: service_started
+
+    # The cymbal binary in resolution mode: the gRPC symbol-resolution service that cymbal's
+    # processing mode calls. Needs only Postgres and object storage — no Kafka or Redis — hence
+    # the narrower env than cymbal above.
+    cymbal-resolution:
+        image: ghcr.io/posthog/posthog/cymbal:master
+        build:
+            context: ./posthog/rust
+            args:
+                BIN: cymbal
+        restart: on-failure
+        logging: *default-logging
+        environment:
+            CYMBAL_MODE: resolution
+            GRPC_ADDRESS: '0.0.0.0:50061'
+            OBJECT_STORAGE_BUCKET: posthog
+            OBJECT_STORAGE_ACCESS_KEY_ID: ${OBJECT_STORAGE_ACCESS_KEY_ID}
+            OBJECT_STORAGE_SECRET_ACCESS_KEY: ${OBJECT_STORAGE_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENDPOINT: 'http://seaweedfs:8333'
+            OBJECT_STORAGE_FORCE_PATH_STYLE: 'true'
+            DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET}
+            RUST_LOG: 'info'
+        depends_on:
+            seaweedfs:
+                condition: service_healthy
+            db:
+                condition: service_healthy
+
+networks:
+    otel_network:
+    nginx-bridge:
+        external: true
+
+volumes:
+    zookeeper-data:
+    zookeeper-datalog:
+    zookeeper-logs:
+    objectstorage-data:
+    seaweedfs:
+    postgres-data:
+    clickhouse-data:
+    caddy-data:
+    caddy-config:
+    redis7-data:
+    kafka-data:
+    redpanda-data:

server/home/PostHog/compose/local-patches.py

@@ -0,0 +1,72 @@
+"""Local source patches, applied to /code at container start.
+
+Called from compose/start, compose/worker and compose/temporal-django-worker
+before the app boots, so every recreated container comes back patched. (The old
+approach patched running containers by hand and silently vanished on the next
+`docker compose up`.)
+
+Both patches exist because PostHog no longer sells self-hosted licenses, so the
+license branches below are unreachable here. Both are gated on
+MULTI_ORG_ENABLED (set in compose.override.yml), so unsetting it restores
+upstream behaviour without un-patching. Only posthog/ files are touched, which
+is MIT-licensed; ee/ is not involved.
+
+Idempotent. If upstream changes shape, this warns loudly and lets the app start
+unpatched rather than taking PostHog down. Check with:
+    docker compose logs web worker | grep LOCAL-PATCH
+"""
+
+import sys
+
+PATCHES = [
+    (
+        # Shows the "New project" / "New organization" buttons. Upstream gates
+        # them on preflight's can_create_org, which demands a license.
+        "can_create_org",
+        "/code/posthog/utils.py",
+        """    if settings.MULTI_ORG_ENABLED:
+        license = get_cached_instance_license()
+        if license is not None and AvailableFeature.ZAPIER in license.available_features:
+            return True
+        else:
+            logger.warning("You have configured MULTI_ORG_ENABLED, but not the required premium PostHog plan!")
+""",
+        """    if settings.MULTI_ORG_ENABLED:
+        # Local patch, see compose/local-patches.py
+        return True
+""",
+    ),
+    (
+        # Lifts the 1-project-per-org limit in the project API. Without a
+        # license, every org save and the periodic feature sync reset the org's
+        # features to [], and an org without `organizations_projects` may only
+        # have one project. With it and no `limit`, the cap is upstream's
+        # MAX_ALLOWED_PROJECTS_PER_ORG (2000).
+        "organizations_projects",
+        "/code/posthog/models/organization.py",
+        """        return self.available_product_features
+
+    def get_available_feature(""",
+        """        # Local patch, see compose/local-patches.py
+        if settings.MULTI_ORG_ENABLED and not self.get_available_feature("organizations_projects"):
+            self.available_product_features = [
+                *self.available_product_features,
+                {"key": "organizations_projects", "name": "Organizations projects"},
+            ]
+
+        return self.available_product_features
+
+    def get_available_feature(""",
+    ),
+]
+
+for name, path, old, new in PATCHES:
+    src = open(path).read()
+    if new in src:
+        status = "already applied"
+    elif src.count(old) == 1:
+        open(path, "w").write(src.replace(old, new))
+        status = "applied"
+    else:
+        status = "PATTERN NOT FOUND, upstream changed, running UNPATCHED"
+    print(f"LOCAL-PATCH {name}: {status}", file=sys.stderr, flush=True)

server/home/PostHog/compose/start

@@ -0,0 +1,5 @@
+#!/bin/bash
+python /compose/local-patches.py
+/compose/wait
+./bin/migrate
+./bin/docker-server

server/home/PostHog/compose/temporal-django-worker

@@ -0,0 +1,3 @@
+#!/bin/bash
+python /compose/local-patches.py
+./bin/temporal-django-worker

server/home/PostHog/compose/wait

@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+
+import socket
+import time
+
+def loop():
+    print("Waiting for ClickHouse and Postgres to be ready")
+    try:
+        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+            s.connect(('clickhouse', 9000))
+        print("Clickhouse is ready")
+        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+            s.connect(('db', 5432))
+        print("Postgres is ready")
+    except ConnectionRefusedError as e:
+        time.sleep(5)
+        loop()
+
+loop()

server/home/PostHog/compose/worker

@@ -0,0 +1,3 @@
+#!/bin/bash
+python /compose/local-patches.py
+exec ./bin/docker-worker-celery --with-scheduler

server/home/PostHog/podman-posthog.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: PostHog
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/PostHog
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/RESTORE.md

@@ -0,0 +1,140 @@
+# Restoring ~/Server on server.gib from the ZFS-pool backup
+
+Written 2026-09-17. Read this *before* you need it.
+
+## What this protects against
+
+The NVMe/LUKS btrfs volume (`/`, `/home`, and `/var/lib/docker`) dying. The backup lives on
+the ZFS pool in the same machine, so it does **not** protect against fire, theft, or the
+whole box failing. It is a disk-failure backup, not an offsite one.
+
+## Layout
+
+    /home/gib/Public/Server_Backup/Containers/server.gib/current/
+
+**One directory, overwritten daily at 02:00** by `server-backup.timer`. History lives in ZFS:
+`Media/Server_Backup` has `zfs-auto-snapshot` keeping **14 dailies**, taken by `cron.daily`
+between roughly 03:22 and 03:52. The 02:00 schedule is deliberate — it must finish before
+that window, or a snapshot captures a half-written backup as that day's history. It also
+staggers against the VPS backup, which pushes into this same pool at 01:00.
+
+Older copies are under the hidden snapshot dir (`snapdir=hidden`, so `ls` will not show it,
+but you can `cd` in):
+
+    ls /home/gib/Public/Server_Backup/.zfs/snapshot/
+    ls /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
+
+The `03-16-2026` and `03-31-2026` directories are leftovers from the old manual script.
+
+## Why this runs as root
+
+This box runs Docker, so most container data under `~/Server` is root-owned. The old script
+used `sudo rsync`, which worked but meant it could never be automated without a sudoers
+rule. The replacement runs as root via `server-backup.service`, a systemd **system** unit —
+already root, so ownership is preserved natively and **no sudoers entry is needed at
+runtime**. If you see `/etc/sudoers.d/99-claude-setup` on this machine, it was temporary
+setup scaffolding and should be deleted.
+
+(This differs from the VPS, which is rootless podman and needs `rsync --fake-super` to carry
+subuid ownership through. Nothing like that is needed here.)
+
+The script lives at `/usr/local/sbin/server-backup`, deliberately **not** in `~/Server` — a
+root-executed script inside a user-writable directory is a footgun. `~/Server/backup` is a
+thin wrapper that triggers the systemd unit so manual runs behave identically to nightly ones.
+
+## What is in a backup
+
+| Path | What it is |
+|---|---|
+| `Media/`, `Frigate/`, `Home/`, `OpenWebUI/`, ... | Raw bind-mount data per service |
+| `_backup_state/sqlite/<original path>` | Online `.backup` of every SQLite DB — **authoritative** |
+| `_backup_state/convex/<container>.zip` | `npx convex export` snapshot of each SQLite-backed Convex deployment (gbrown) — **authoritative** |
+
+> **Use `_backup_state/` for databases, not the raw copies.** Containers keep running during
+> the backup, so the raw files are crash-consistent at best. The dumps use SQLite's online
+> backup API and are what will actually restore cleanly. That covers the ones that matter:
+> Plex's `com.plexapp.plugins.library.db` (watch history, ratings), Jellyfin, Frigate, Home
+> Assistant, OpenWebUI, Audiobookshelf, and every *arr config.
+>
+> **Convex is the exception.** It writes to its SQLite file constantly, so a raw `.backup`
+> never completes. Its `_backup_state/convex/` export is the restore source, and the raw
+> `db.sqlite3` is not dumped at all.
+
+## What is deliberately NOT backed up
+
+- **`Agents/Gitea/volumes/runner-{1,2,3}/dind` — ~203 GB.** Docker-in-Docker containerd
+  image layers for the CI runners. Pure cache that churns constantly; a rebuilt runner just
+  re-pulls it. The parts that matter — `runner-N/data/.runner` (the 648-byte registration
+  token) and `certs/` — **are** backed up.
+  > Measured as root. A plain `du` as `gib` reports these as ~13 MB because gib cannot
+  > descend into them. Do not trust non-root sizes here.
+- Docker **named volumes** (`/var/lib/docker/volumes`, 47 in use, ~61 GB): Sentry (13) and
+  Plausible (4) were consciously excluded — Sentry's event data is not worth restoring and
+  Plausible had nothing in it. Romm looked like a gap but is not: its anonymous `/romm`
+  volume is only a parent directory, with `assets/`, `config/` and `resources/` bind-mounted
+  from `~/Server` and `library/` from the pool.
+- **Nextcloud AIO** — has its own borg backup at `~/Public/Nextcloud/Backups`, run by
+  hand from the AIO admin UI. Last run 2026-09-17 (finished 09-18). Runs are slow and
+  AIO stops the Nextcloud containers while one is going.
+- `node_modules`, build output, `*.log`, `cache`/`tmp` dirs, `*.mp4`/`*.mkv`.
+
+## Restoring
+
+Stop the stack first so nothing overwrites what you restore:
+
+    cd ~/Server/<service> && docker compose down     # per service, or stop docker entirely
+
+Pull it back — as root, so ownership is restored:
+
+    sudo rsync -a --numeric-ids --delete \
+      /home/gib/Public/Server_Backup/Containers/server.gib/current/ \
+      /home/gib/Server/
+
+For an older day, use a snapshot path instead:
+
+    /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
+
+Then put the databases back from the dumps. For each one, stop the owning container, copy the
+dump over the live file, and match the original ownership. Plex, for example:
+
+    docker compose -f ~/Server/Media/compose.yml stop plex
+    D=~/Server/_backup_state/sqlite/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
+    L=~/Server/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
+    sudo cp "$D/com.plexapp.plugins.library.db" "$L/"
+    sudo chown --reference="$L/com.plexapp.dlna.db" "$L/com.plexapp.plugins.library.db"
+    docker compose -f ~/Server/Media/compose.yml start plex
+
+`chown --reference` copies ownership from a neighbouring file, which saves looking up what
+uid each container expects.
+
+For Convex, start the backend (it can be empty) and import the export through it. This
+replaces every table with the snapshot:
+
+    Z=~/Server/_backup_state/convex/gbrown-backend.zip
+    KEY=$(docker exec gbrown-backend ./generate_admin_key.sh | tail -1)
+    docker run --rm -it --network nginx-bridge -v "$Z:/backup.zip:ro" \
+      -e CONVEX_SELF_HOSTED_URL=http://gbrown-backend:3210 -e CONVEX_SELF_HOSTED_ADMIN_KEY="$KEY" \
+      docker.io/library/node:22-alpine sh -ec 'mkdir /w && cd /w && echo "{}" > package.json &&
+        npm install --silent convex && npx convex import --replace-all /backup.zip'
+
+Functions and schema are not in the export. Redeploy them from the app's repo as usual.
+
+After a full restore the CI runners will have no `dind` storage. That is fine — they re-pull
+images on the next job. The registration in `runner-N/data/.runner` is intact, so they
+reconnect to Gitea without re-registering.
+
+## Checking it is working
+
+    systemctl list-timers server-backup.timer
+    systemctl status server-backup.service
+    tail -40 ~/Server/logs/backup.log
+
+A healthy run ends with `=== backup complete ===`. Watch for:
+
+- `ZFS history OK` — confirms snapshots are still being taken. If this ever says **WARNING**,
+  versioning is broken and `current/` is just a mirror: a mistake would propagate with
+  nothing to roll back to. Fix `zfs-auto-snapshot` before anything else.
+- `rsync exit 24` is a NOTE, not a failure — a file changed mid-transfer, normal on a live box.
+
+Optional: put an uptime-kuma push URL in `~/Server/.backup-heartbeat-url` and successful runs
+will ping it, turning a silent failure into a down monitor.

server/home/Romm/.env.example

@@ -0,0 +1,10 @@
+DB_PASSWD=CHANGE_ME
+ROMM_AUTH_SECRET_KEY=CHANGE_ME
+IGDB_CLIENT_ID=CHANGE_ME
+IGDB_CLIENT_SECRET=CHANGE_ME
+RETROACHIEVEMENTS_API_KEY=CHANGE_ME
+STEAMGRIDDB_API_KEY=CHANGE_ME
+OIDC_CLIENT_ID=CHANGE_ME
+OIDC_CLIENT_SECRET=CHANGE_ME
+MARIADB_ROOT_PASSWORD=CHANGE_ME
+MARIADB_PASSWORD=CHANGE_ME

server/home/Romm/compose.yml

@@ -0,0 +1,66 @@
+networks:
+  nginx-bridge:
+    external: true
+services:
+  romm:
+    image: rommapp/romm:latest
+    container_name: romm
+    hostname: romm
+    domainname: romm.gbrown.org
+    networks: [nginx-bridge]
+    restart: unless-stopped
+    stdin_open: true
+    tty: true
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    environment:
+      - TZ=America/New_York
+      - DB_HOST=romm-db
+      - DB_NAME=romm
+      - DB_USER=gib
+      - DB_PASSWD=${DB_PASSWD}
+      - ROMM_AUTH_SECRET_KEY=${ROMM_AUTH_SECRET_KEY}
+      - IGDB_CLIENT_ID=${IGDB_CLIENT_ID}
+      - IGDB_CLIENT_SECRET=${IGDB_CLIENT_SECRET}
+      - RETROACHIEVEMENTS_API_KEY=${RETROACHIEVEMENTS_API_KEY}
+      - STEAMGRIDDB_API_KEY=${STEAMGRIDDB_API_KEY}
+      - HASHEOUS_API_ENABLED=true
+      - PLAYMATCH_API_ENABLED=true
+      - OIDC_ENABLED=true
+      - OIDC_PROVIDER=Gibs Auth
+      - OIDC_CLIENT_ID=${OIDC_CLIENT_ID}
+      - OIDC_CLIENT_SECRET=${OIDC_CLIENT_SECRET}
+      - OIDC_REDIRECT_URI=https://romm.gbrown.org/api/oauth/openid
+      - OIDC_SERVER_APPLICATION_URL=https://auth.gbrown.org/application/o/romm
+      #- ENABLE_RESCAN_ON_FILESYSTEM_CHANGE=true
+    volumes:
+      - ./volumes/romm/resources:/romm/resources # Resources fetched from IGDB (covers, screenshots, etc.)
+      - ./volumes/romm/redis_data:/redis-data # Cached data for background tasks
+      - /home/gib/Public/Server_Backup/ROMs:/romm/library # Your game library
+      - ./volumes/romm/assests:/romm/assets # Uploaded saves, states, etc.
+      - ./volumes/romm/config:/romm/config # Path where config.yml is stored
+    #ports: [8080:8080]
+    depends_on:
+      romm-db:
+        condition: service_healthy
+        restart: true
+
+  romm-db:
+    image: mariadb:latest
+    container_name: romm-db
+    hostname: romm-db
+    networks: [nginx-bridge]
+    restart: unless-stopped
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    environment:
+      - MARIADB_ROOT_PASSWORD=${MARIADB_ROOT_PASSWORD}
+      - MARIADB_DATABASE=romm
+      - MARIADB_USER=gib
+      - MARIADB_PASSWORD=${MARIADB_PASSWORD}
+    volumes: [./volumes/mysql:/var/lib/mysql]
+    healthcheck:
+      test: [CMD, healthcheck.sh, --connect, --innodb_initialized]
+      start_period: 30s
+      start_interval: 10s
+      interval: 10s
+      timeout: 5s
+      retries: 5

server/home/Romm/podman-romm.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Romm
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Romm
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Scrutiny/compose.yml

@@ -0,0 +1,59 @@
+# Scrutiny — SMART disk health monitoring with historical trending.
+#
+# Omnibus image bundles the web UI, InfluxDB and the metrics collector in one
+# container, which suits a single host.
+#
+# Needs raw device access: SYS_RAWIO for SMART ioctls on the SATA disks, and
+# SYS_ADMIN additionally for NVMe. /run/udev is mounted read-only so scrutiny
+# can resolve device metadata (model, serial, WWN).
+#
+# Devices listed explicitly rather than using privileged mode:
+#   sda-sdd   4x WDC WD40EFPX  (4TB)
+#   sde-sdg   3x TOSHIBA MG08ACA14TE (14TB)
+#   sdh       1x WUH721414ALE601 (14TB)
+#   nvme0n1   WDS100T3X0C (1TB)
+#   nvme1n1   Samsung 990 PRO 2TB  (root/LUKS)
+# sdi (USB Mass Storage) is deliberately omitted.
+#
+# If a disk is added or replaced, add it to the devices: list and recreate.
+
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  scrutiny:
+    image: ghcr.io/analogj/scrutiny:master-omnibus
+    container_name: scrutiny
+    hostname: scrutiny
+    domainname: scrutiny.gbrown.org
+    networks: ['nginx-bridge']
+    #ports:
+      #- "8080:8080"
+    environment:
+      - TZ=America/New_York
+      - SCRUTINY_WEB_INFLUXDB_RETENTION_POLICY=true
+    cap_add:
+      - SYS_RAWIO
+      - SYS_ADMIN
+    devices:
+      - /dev/sda:/dev/sda
+      - /dev/sdb:/dev/sdb
+      - /dev/sdc:/dev/sdc
+      - /dev/sdd:/dev/sdd
+      - /dev/sde:/dev/sde
+      - /dev/sdf:/dev/sdf
+      - /dev/sdg:/dev/sdg
+      - /dev/sdh:/dev/sdh
+      - /dev/nvme0n1:/dev/nvme0n1
+      - /dev/nvme1n1:/dev/nvme1n1
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/config:/opt/scrutiny/config
+      - ./volumes/influxdb:/opt/scrutiny/influxdb
+      - /run/udev:/run/udev:ro
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped

server/home/Scrutiny/podman-scrutiny.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Scrutiny
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Scrutiny
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Sentry/.env.example

@@ -0,0 +1,24 @@
+COMPOSE_PROJECT_NAME=CHANGE_ME
+COMPOSE_PROFILES=CHANGE_ME
+SENTRY_EVENT_RETENTION_DAYS=CHANGE_ME
+LAUNCHPAD_RPC_SHARED_SECRET=CHANGE_ME
+SENTRY_BIND=CHANGE_ME
+SENTRY_TASKWORKER_CONCURRENCY=CHANGE_ME
+SENTRY_IMAGE=CHANGE_ME
+SNUBA_IMAGE=CHANGE_ME
+RELAY_IMAGE=CHANGE_ME
+SYMBOLICATOR_IMAGE=CHANGE_ME
+TASKBROKER_IMAGE=CHANGE_ME
+VROOM_IMAGE=CHANGE_ME
+UPTIME_CHECKER_IMAGE=CHANGE_ME
+LAUNCHPAD_IMAGE=CHANGE_ME
+HEALTHCHECK_INTERVAL=CHANGE_ME
+HEALTHCHECK_TIMEOUT=CHANGE_ME
+HEALTHCHECK_RETRIES=CHANGE_ME
+HEALTHCHECK_START_PERIOD=CHANGE_ME
+HEALTHCHECK_FILE_INTERVAL=CHANGE_ME
+HEALTHCHECK_FILE_TIMEOUT=CHANGE_ME
+HEALTHCHECK_FILE_RETRIES=CHANGE_ME
+HEALTHCHECK_FILE_START_PERIOD=CHANGE_ME
+SENTRY_MAIL_HOST=CHANGE_ME
+SENTRY_SYSTEM_SECRET_KEY=CHANGE_ME