Show mocks in T3 Code threads, and drop the t3 MCP servers that T3 already provides

Hyprland 0.57 refuses a bind on a bare modifier and an empty input:scroll_method. The switcher's Super-release commit now binds Super_L and Super_R, which also brings back every bind below it in keybinds.lua. An empty scroll method is left unwritten, so libinput still picks per device.

.agents/skills/panama/SKILL.md

@@ -9,7 +9,7 @@ You are editing a running desktop, not a codebase that gets deployed later. Ever
 repository is symlinked into `~/.config`, so a save is live the moment it lands. That single fact
 drives every rule below.
 
-Read before large work, in this order:
+Before large work, read these in order:
 
 - `README.md` — layout, the `panama` command, how installing and updating work
 - `config/dot/hypr/README.md` — the compositor config is **Lua, not hyprlang**; read "The one
@@ -62,7 +62,8 @@ Every executable contract under `tests/` is classified in `tests/contracts.manif
   `PANAMA_TEST_TIMEOUT_SECONDS` value. Failures print the contract's captured stdout and stderr.
   Successful stdout stays quiet. Successful stderr is surfaced as a warning.
 - Contracts run directly too: `tests/setup/interview-contract`.
-- After changing `PreferenceSchema.qml` or `services/SettingsRoutes.qml`, regenerate:
+- After changing `PreferenceSchema.qml` or
+  `config/dot/quickshell/services/SettingsRoutes.qml`, regenerate:
   `config/dot/quickshell/scripts/panama-settings-docs` (writes `docs/settings.md`) and
   `config/dot/quickshell/scripts/panama-settings-commands` (writes the launcher deep links).
   Both take `--check`; `tests/quickshell/settings-docs-contract` fails when stale.

.claude/docs/specs/2026-08-25-server-role.md

@@ -183,8 +183,8 @@ currently parked (dir present, unit not active) — imported anyway, noted as
 parked in the catalog.
 
 **Excluded:** Watchtower (retired — replaced by the update timer after it took
-gitea down; its dir stays on junior until cleanup), Completeuphoria,
-sierraandtyler, Bang (source checkouts deployed by CI from their own repos —
+gitea down; its dir stays on junior until cleanup), the app source
+checkouts such as sierraandtyler and Bang (deployed by CI from their own repos —
 Panama owning a second copy of their compose would drift), `ci/`, `logs/`,
 and junior's loose scripts other than `update-containers.sh` (§4).
 

.claude/skills/panama

@@ -0,0 +1 @@
+../../.agents/skills/panama

.gitignore

@@ -3,15 +3,21 @@
 /config/bash/.env.lock
 # Server secrets and state never enter the repo: live .env files belong in
 # ~/Server/<Name>/ (only compose.yml is symlinked back here), and data/ is
-# the bind-mount convention. Ignored anyway as a seatbelt -- the repo is
-# public, and tests/server/compose-secrets-contract is the check that this
-# line still exists.
-/server/containers/**/.env
-/server/containers/**/data/
+# the bind-mount convention (the home server's older services use volume/ and
+# volumes/). Ignored anyway as a seatbelt -- the repo is public, and
+# tests/server/compose-secrets-contract is the check that these lines still exist.
+/server/**/.env
+/server/**/data/
+/server/**/volume/
+/server/**/volumes/
 # Personal espanso triggers (name, email), seeded per-machine by setup-identity.
 /config/dot/espanso/match/identity.yml
 # Ignore backups of old config files
 /config/old
+# MCP bearer tokens. The server list in user/agents/mcp/servers is tracked
+# and names which variable carries each token; the tokens themselves are
+# machine-local, because this repo is public.
+/user/agents/mcp/env
 # Ignore Wireguard config of course!
 /config/wg/**
 # Ignore Neovim lazy-lock file

README.md

@@ -102,13 +102,13 @@ in order, without stopping again:
 | `interview` | Every prompt, before anything is installed. Answers last one run and are never written to a durable path |
 | `install-packages` | Repos (RPM Fusion, Terra, Hyprland COPR), the package lists in `setup/packages/`, then whichever optional categories were chosen |
 | `link-dotfiles` | Symlinks `config/dot/<name>` → `~/.config/<name>`, and seeds the wallpaper, cursor theme and Firefox chrome |
-| `link-skills` | Links the agent skills in `skills/` into `~/.claude/skills`, one per skill. Every machine gets these; personal ones link after and win a name clash |
+| `link-skills` | Links the agent skills in `skills/` into `~/.agents/skills` and `~/.claude/skills`, one per skill. Every machine gets these; personal ones link after and win a name clash |
 | `link-user` | Links the personal content in `user/` — agent instructions, SSH host aliases — but only on a machine that answered yes. See [user/README.md](user/README.md) |
 | `change-settings` | Copies `config/copy/` over `/`, applies gsettings, enables user services |
 | `link-vicinae-scripts` | Publishes the Vicinae script commands |
 | `setup-identity` | git config, `gh auth login`, an SSH key — whichever were asked for |
 | `install-hardware` | NVIDIA, Secure Boot enrollment, Fedora's extras, firmware — each only if it was asked for. Last, because enrollment and firmware are consumed at the next boot |
-| `setup-server` | Server role only: linger, unprivileged ports from 80, firewalld (80/443/81), the `nginx-bridge` network, and the nightly image-update timer |
+| `setup-server` | Server role only: linger, unprivileged ports from 80, firewalld (80/443/81, opened only where nothing already grants them), the `nginx-bridge` network, and the nightly image-update timer |
 | `link-server` | Server role only: creates `~/Server` and refreshes the links of whatever services this machine has enabled |
 
 A server runs the shared stages plus its own two, and skips the rest —
@@ -235,10 +235,13 @@ server/         The server role: compose services (one directory per
                 service), the nightly image updater, and its units. See
                 server/README.md
 skills/         Agent skills for operating this desktop, linked into
-                ~/.claude/skills
+                ~/.agents/skills and ~/.claude/skills
 setup/
   apps/         Applications built from source, one file each
-  lib/          Shared by more than one stage; the extras catalog reader
+  keys/         Pinned signing keys, for publishers that ship no fetchable
+                one; setup/keys/README.md records where each came from
+  lib/          Shared by more than one stage; the extras catalog reader,
+                the machine role, the verified ChatGPT repository
   packages/     One package per line; extras/ holds the optional categories
   scripts/      Run in order by ./install
 tests/          Contracts. See below
@@ -247,7 +250,7 @@ docs/           Settings reference, and the design specs behind the work
 
 ## Tests
 
-188 of them, under `tests/`. `tests/contracts.manifest` classifies every
+193 of them, under `tests/`. `tests/contracts.manifest` classifies every
 contract by the capabilities it needs. Run the hermetic set, or grant a
 specific external capability when automation needs it:
 

bin/panama

@@ -134,7 +134,6 @@ ${BOLD}Examples:${RESET}
   $PROGRAM upgrade
   $PROGRAM apps
   $PROGRAM app
-  $PROGRAM app chatgpt-desktop
 EOF
 }
 
@@ -451,11 +450,16 @@ CONTRACT_CAPABILITIES=(hermetic live-host live-compositor live-desktop network p
 
 contract_paths() {
   local candidate
+  # The manifest is kept in byte order, so both the discovery sort and the
+  # comparison below have to be byte order too. A UTF-8 collation folds the
+  # punctuation away -- `calendar_agenda_bridge_test.py` sorts before
+  # `calendar-agenda-helper-contract` under en_US and after it under C -- and
+  # a gate that passes or fails on the machine's LANG is not a gate.
   while IFS= read -r candidate; do
     [[ -x "$candidate" || "$candidate" == *_test.py ]] || continue
     printf 'tests/%s\n' "${candidate#"$PANAMA_DIR/tests/"}"
   done < <(find "$PANAMA_DIR/tests" -type f \
-    -not -path '*/fixtures/*' -not -path '*__pycache__*' | sort)
+    -not -path '*/fixtures/*' -not -path '*__pycache__*' | LC_ALL=C sort)
 }
 
 contract_manifest_entries() {
@@ -478,6 +482,8 @@ validate_contract_manifest() {
   require_contract_manifest || return 1
 
   local manifest="$PANAMA_DIR/$CONTRACT_MANIFEST"
+  # Byte order, for the same reason contract_paths sorts in it.
+  local LC_ALL=C
   local line capabilities path extra previous_comment="" previous_was_comment=0
   local previous_path="" capability discovered
   local -a capability_list=() findings=()

bin/panama-agent

@@ -10,9 +10,16 @@
 # start time, so choosing an agent in Settings takes effect on the next crash
 # without restarting anything:
 #
-#   preferredAgent    none | claude | codex   ("none" is the default: silence)
+#   preferredAgent    none | claude | codex | t3   ("none" is the default: silence)
 #   agentAutoApprove  true -> the agent starts in its own "don't stop to ask"
 #                     mode; false -> its normal prompting mode, untouched.
+#                     T3 Code ignores it: a T3 thread runs in the mode its
+#                     project is set to, which is a choice made inside T3.
+#
+# claude and codex open in a kitty window. t3 starts a thread in the Panama
+# project on the running T3 Code server and brings it up in the desktop app,
+# so the investigation lives beside every other thread instead of in a
+# terminal of its own.
 #
 # "none" exits 0 without a word. It is not an error to have no agent; it is the
 # shipped state, and a rung that shouted about it would be a rung that gets
@@ -95,8 +102,9 @@ usage() {
     cat <<'EOF'
 Usage: panama-agent [--prompt "text"]
 
-Opens the agent named by preferredAgent in a terminal, in the Panama checkout.
-With no agent chosen, exits silently: choose one on Settings > System > Agents.
+Opens the agent named by preferredAgent in the Panama checkout: a terminal for
+claude and codex, a thread in the T3 Code app for t3. With no agent chosen,
+exits silently: choose one on Settings > System > Agents.
 EOF
 }
 
@@ -139,7 +147,7 @@ agent="$(read_setting preferredAgent none)"
 auto_approve="$(read_setting agentAutoApprove true)"
 
 case "$agent" in
-    claude | codex) ;;
+    claude | codex | t3) ;;
     *)
         printf 'panama-agent: unsupported preferredAgent: %s\n' "$agent" >&2
         printf 'Choose one on Settings > System > Agents.\n' >&2
@@ -148,7 +156,8 @@ case "$agent" in
 esac
 
 # Resolved to a path before argv is built, so kitty is never asked to repeat a
-# PATH lookup this script has already done more carefully than kitty could.
+# PATH lookup this script has already done more carefully than kitty could. T3
+# Code's CLI is `t3`, which the desktop app writes into ~/.local/bin.
 if ! agent_bin="$(resolve_agent "$agent")"; then
     printf 'panama-agent: %s is not installed.\n' "$agent" >&2
     printf 'Looked on PATH and in %s.\n' "$HOME/.local/bin" >&2
@@ -156,6 +165,32 @@ if ! agent_bin="$(resolve_agent "$agent")"; then
     exit 1
 fi
 
+# T3 Code: no terminal. The thread starts on the running server in the
+# project whose root is the checkout, with the project's default model, and
+# the t3code:// link opens it in the app. `project add` refuses a root that is
+# already a project, which is the common case, so its failure is not one. The
+# prompt goes over stdin ("-"), so a prompt that begins with a dash is never
+# read as a flag. With no prompt there is nothing to start, so the app just
+# opens on the project.
+if [[ "$agent" == "t3" ]]; then
+    repair_path
+    "$agent_bin" project add "$PANAMA_PATH" >/dev/null 2>&1 || true
+    if [[ -z "$prompt" ]]; then
+        exec setsid "$agent_bin" app "$PANAMA_PATH"
+    fi
+    if ! thread="$(printf '%s' "$prompt" | "$agent_bin" thread start "$PANAMA_PATH" -)"; then
+        printf 'panama-agent: T3 Code could not start a thread.\n' >&2
+        printf 'Is the T3 Code server running? Check: systemctl --user status t3code\n' >&2
+        exit 1
+    fi
+    environment_file="${T3CODE_HOME:-$HOME/.t3}/userdata/environment-id"
+    environment="$(cat "$environment_file" 2>/dev/null || true)"
+    if [[ -n "$environment" ]]; then
+        exec setsid xdg-open "t3code://threads/$environment/$thread"
+    fi
+    exec setsid "$agent_bin" app "$PANAMA_PATH"
+fi
+
 # The per-agent launch table. VERIFIED against the installed binaries' --help on
 # 2026-08-25 (claude 2.1.245, codex-cli 0.149.1) -- these flags are not the same
 # from release to release, so re-run --help before changing them.

bin/panama-server

@@ -2,10 +2,12 @@
 #
 # panama-server – the compose services a server machine runs.
 #
-# The repository carries the catalog: server/containers/<Name>/ holds a
+# The repository carries the catalog: server/vps/<Name>/ holds a
 # compose.yml, a podman-<name>.service user unit, and an .env.example naming
-# what the service needs told. The machine carries the rest: ~/Server/<Name>/
-# is a real directory owning the .env (secrets never sit inside the checkout,
+# what the service needs told. A service may also carry helper units (any
+# other *.service or *.timer, e.g. a cert refresh) and helper files (scripts,
+# a README) beside them. The machine carries the rest: ~/Server/<Name>/ is a
+# real directory owning the .env (secrets never sit inside the checkout,
 # gitignored or not) and the bind-mounted data. Only the tracked files are
 # symlinked, so a `git clean` in the repo can never reach a database.
 #
@@ -27,7 +29,7 @@ PANAMA_DIR=$(cd "$(dirname "$SCRIPT_PATH")/.." && pwd)
 
 # Overridable so the contract can point this at a fixture catalog and a
 # throwaway HOME rather than the machine's real services.
-SERVER_DIR="${PANAMA_SERVER_DIR:-$PANAMA_DIR/server/containers}"
+SERVER_DIR="${PANAMA_SERVER_DIR:-$PANAMA_DIR/server/vps}"
 TARGET_DIR="${PANAMA_SERVER_TARGET:-$HOME/Server}"
 UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user"
 STATE_FILE="${XDG_STATE_HOME:-$HOME/.local/state}/panama/server-definitions"
@@ -46,7 +48,7 @@ err()   { printf '%s✗%s %s\n'    "${RED}${BOLD}"    "$RESET" "$*" >&2; }
 
 usage() {
   cat <<EOF
-${BOLD}$PROGRAM${RESET} – manage the compose services in server/containers/
+${BOLD}$PROGRAM${RESET} – manage the compose services in server/vps/
 
 ${BOLD}Usage:${RESET}
   $PROGRAM list             Every service in the catalog, and its state here
@@ -83,12 +85,12 @@ require_service() {
   fi
 }
 
-# The one unit file a service directory carries. Its filename is its identity
-# -- podman-<name>.service, exactly what lands in the systemd user directory --
-# so nothing here invents a name that could drift from the file's.
+# The one compose unit a service directory carries. Its filename is its
+# identity -- podman-<name>.service, exactly what lands in the systemd user
+# directory -- so nothing here invents a name that could drift from the file's.
 unit_path_for() {
   local name="$1" unit
-  for unit in "$SERVER_DIR/$name"/*.service; do
+  for unit in "$SERVER_DIR/$name"/podman-*.service; do
     [[ -e "$unit" ]] || continue
     printf '%s' "$unit"
     return 0
@@ -96,6 +98,44 @@ unit_path_for() {
   return 1
 }
 
+# Every other unit in the directory: helper services and the timers that fire
+# them. They are linked beside the compose unit, and each timer is enabled.
+helper_units_for() {
+  local name="$1" unit
+  for unit in "$SERVER_DIR/$name"/*.service "$SERVER_DIR/$name"/*.timer; do
+    [[ -e "$unit" ]] || continue
+    [[ "$(basename "$unit")" == podman-*.service ]] && continue
+    printf '%s\n' "$unit"
+  done
+}
+
+# The files linked into ~/Server/<Name>/: every visible top-level file that is
+# not a unit. That is compose.yml plus any helper script or README; dotfiles
+# are skipped so .env.example (and a stray .env) never become links.
+live_files_for() {
+  local name="$1" file
+  for file in "$SERVER_DIR/$name"/*; do
+    [[ -f "$file" ]] || continue
+    case "$file" in *.service|*.timer) continue ;; esac
+    printf '%s\n' "$file"
+  done
+}
+
+link_service_files() {
+  local name="$1" file unit
+  mkdir -p "$TARGET_DIR/$name" "$UNIT_DIR"
+  while IFS= read -r file; do
+    link_file "$file" "$TARGET_DIR/$name/$(basename "$file")"
+  done < <(live_files_for "$name")
+  while IFS= read -r unit; do
+    link_file "$unit" "$UNIT_DIR/$(basename "$unit")"
+  done < <(helper_units_for "$name")
+}
+
+helper_timers_for() {
+  helper_units_for "$1" | grep '\.timer$' | xargs -r -n1 basename
+}
+
 unit_name_for() {
   local unit
   unit="$(unit_path_for "$1")" || return 1
@@ -111,10 +151,12 @@ unit_installed() {
   [[ -L "$UNIT_DIR/$unit_name" ]]
 }
 
+# Everything this machine links for the service, so a change to a helper
+# script or timer is reported by relink just like a compose change.
 definition_hash() {
-  local name="$1" unit
-  unit="$(unit_path_for "$name")" || unit=/dev/null
-  cat "$SERVER_DIR/$name/compose.yml" "$unit" 2>/dev/null | sha256sum | cut -d' ' -f1
+  local name="$1"
+  { live_files_for "$name"; unit_path_for "$name" && echo; helper_units_for "$name"; } \
+    | sort | xargs -r -d '\n' cat 2>/dev/null | sha256sum | cut -d' ' -f1
 }
 
 recorded_hash() {
@@ -198,6 +240,7 @@ cmd_enable() {
   fi
 
   mkdir -p "$UNIT_DIR"
+  link_service_files "$name"
   link_file "$unit_path" "$UNIT_DIR/$unit_name"
   systemctl --user daemon-reload
   if systemctl --user enable --now "$unit_name"; then
@@ -207,6 +250,10 @@ cmd_enable() {
     err "$unit_name did not start; see: systemctl --user status $unit_name"
     exit 1
   fi
+  local timer
+  for timer in $(helper_timers_for "$name"); do
+    systemctl --user enable --now "$timer" && ok "Enabled $timer"
+  done
 }
 
 cmd_disable() {
@@ -216,11 +263,15 @@ cmd_disable() {
   local unit_name
   unit_name="$(unit_name_for "$name")" || { err "$name has no unit file in the catalog"; exit 1; }
 
+  local timer unit
+  for timer in $(helper_timers_for "$name"); do
+    systemctl --user disable --now "$timer" 2>/dev/null || true
+  done
   systemctl --user disable --now "$unit_name" 2>/dev/null || true
-  if [[ -L "$UNIT_DIR/$unit_name" ]]; then
-    rm "$UNIT_DIR/$unit_name"
-    systemctl --user daemon-reload
-  fi
+  for unit in "$UNIT_DIR/$unit_name" $(helper_units_for "$name" | xargs -r -n1 basename | sed "s|^|$UNIT_DIR/|"); do
+    [[ -L "$unit" ]] && rm "$unit"
+  done
+  systemctl --user daemon-reload
   ok "$name disabled. ~/Server/$name, its .env and its data were not touched."
 }
 
@@ -267,8 +318,7 @@ cmd_relink() {
     unit_installed "$name" || continue
 
     link_file "$unit_path" "$UNIT_DIR/$unit_name"
-    mkdir -p "$TARGET_DIR/$name"
-    link_file "$SERVER_DIR/$name/compose.yml" "$TARGET_DIR/$name/compose.yml"
+    link_service_files "$name"
     refreshed=$((refreshed + 1))
 
     # Changed since the last time this machine looked, which is what makes a

boot

@@ -45,65 +45,6 @@ checkout_command() {
   fi
 }
 
-# Git's index hints are performance promises, not trust evidence. In
-# particular, assume-unchanged and skip-worktree can make porcelain status
-# report a clean checkout whose files no longer match HEAD. Compare every
-# tracked blob and Git mode with the verified commit before handing control to
-# any file in the worktree.
-checkout_matches_verified_commit() (
-  local checkout="$1" listing="" entry metadata mode type expected path actual
-  local link_target_with_sentinel link_target
-
-  trap '[[ -z "$listing" ]] || rm -f -- "$listing"' EXIT
-  trap 'exit 130' INT
-  trap 'exit 143' TERM
-  listing="$(mktemp -u -t panama-boot-tree.XXXXXX)" || exit 1
-  umask 077
-  if ! (set -o noclobber; : >"$listing") 2>/dev/null; then
-    listing=""
-    exit 1
-  fi
-
-  checkout_command git -C "$checkout" ls-tree -rz --full-tree \
-    "$PANAMA_BOOT_REVISION" >"$listing" || exit 1
-  while IFS= read -r -d '' entry; do
-    [[ "$entry" == *$'\t'* ]] || exit 1
-    metadata="${entry%%$'\t'*}"
-    path="${entry#*$'\t'}"
-    read -r mode type expected <<<"$metadata"
-    [[ "$type" == blob && -n "$path" && "$path" != /* ]] || exit 1
-
-    case "$mode" in
-      100644) [[ -f "$checkout/$path" && ! -L "$checkout/$path" \
-        && ! -x "$checkout/$path" ]] || exit 1 ;;
-      100755) [[ -f "$checkout/$path" && ! -L "$checkout/$path" \
-        && -x "$checkout/$path" ]] || exit 1 ;;
-      120000)
-        [[ -L "$checkout/$path" ]] || exit 1
-        # hash-object given a pathname follows a symlink. Git's 120000 blob is
-        # the link text itself, including any trailing newlines, so preserve
-        # those bytes with a sentinel and hash stdin instead.
-        link_target_with_sentinel="$(
-          readlink -n -- "$checkout/$path" && printf .
-        )" || exit 1
-        [[ "$link_target_with_sentinel" == *. ]] || exit 1
-        link_target="${link_target_with_sentinel%.}"
-        actual="$(
-          printf '%s' "$link_target" \
-            | checkout_command git -C "$checkout" hash-object --stdin
-        )" || exit 1
-        [[ "$actual" == "$expected" ]] || exit 1
-        continue
-        ;;
-      *) exit 1 ;;
-    esac
-
-    actual="$(checkout_command git -C "$checkout" hash-object --no-filters -- "$path")" \
-      || exit 1
-    [[ "$actual" == "$expected" ]] || exit 1
-  done <"$listing"
-)
-
 prepare_panama_checkout() {
   local checkout="$1" actual_head checkout_status
 
@@ -169,30 +110,6 @@ for arg in "$@"; do
   esac
 done
 
-# Keep the worktree comparison at the last possible boundary. Checkout
-# preparation may invoke several commands and return to the caller; performing
-# the byte/mode/link check here ensures a change in that interval is rejected
-# before any tracked file is executed.
-verified_install_handoff() {
-  local use_tty="$1"
-  if ! checkout_matches_verified_commit "$PANAMA_PATH"; then
-    echo "boot: checkout files do not match PANAMA_BOOT_REVISION" >&2
-    return 1
-  fi
-  if [[ -n "$BOOTSTRAP_USER" ]]; then
-    if (( use_tty )); then
-      exec runuser -u "$BOOTSTRAP_USER" -- env PANAMA_PATH="$PANAMA_PATH" \
-        "$PANAMA_PATH/install" ${INSTALL_ARGS[@]+"${INSTALL_ARGS[@]}"} </dev/tty
-    fi
-    exec runuser -u "$BOOTSTRAP_USER" -- env PANAMA_PATH="$PANAMA_PATH" \
-      "$PANAMA_PATH/install" ${INSTALL_ARGS[@]+"${INSTALL_ARGS[@]}"}
-  fi
-  if (( use_tty )); then
-    exec "$PANAMA_PATH/install" ${INSTALL_ARGS[@]+"${INSTALL_ARGS[@]}"} </dev/tty
-  fi
-  exec "$PANAMA_PATH/install" ${INSTALL_ARGS[@]+"${INSTALL_ARGS[@]}"}
-}
-
 # The public bootstrap contract runs this branch as an ordinary user with a
 # stubbed root identity. Keep its filesystem adapter unavailable to a real root
 # shell so it cannot redirect a real installation by accident.
@@ -554,7 +471,7 @@ if [[ "$(id -u)" -eq 0 ]]; then
 
   if ! command -v git >/dev/null 2>&1; then
     echo "Installing git, which the clone needs"
-    dnf install -y --repo=fedora --repo=updates --from-repo=fedora,updates git
+    dnf install -y git
   fi
 
   # Create or advance the checkout as the target user. A root-owned .git in a
@@ -564,14 +481,15 @@ if [[ "$(id -u)" -eq 0 ]]; then
   prepare_panama_checkout "$PANAMA_PATH"
 
   echo "Handing off to install as $username"
-  verified_install_handoff 1
+  exec runuser -u "$username" -- env PANAMA_PATH="$PANAMA_PATH" \
+    "$PANAMA_PATH/install" --server </dev/tty
 fi
 
 # git is the one dependency the clone itself needs. Everything else -- gum
 # included -- is bootstrapped by `install`.
 if ! command -v git >/dev/null 2>&1; then
   echo "Installing git, which the clone needs"
-  sudo dnf install -y --repo=fedora --repo=updates --from-repo=fedora,updates git
+  sudo dnf install -y git
 fi
 
 prepare_panama_checkout "$PANAMA_PATH"
@@ -582,8 +500,7 @@ prepare_panama_checkout "$PANAMA_PATH"
 # so itself.
 # The probe actually opens /dev/tty rather than testing -r: a process with no
 # controlling terminal passes -r and then fails the redirect.
-handoff_tty=0
 if [[ ! -t 0 ]] && (exec </dev/tty) 2>/dev/null; then
-  handoff_tty=1
+  exec "$PANAMA_PATH/install" ${INSTALL_ARGS[@]+"${INSTALL_ARGS[@]}"} </dev/tty
 fi
-verified_install_handoff "$handoff_tty"
+exec "$PANAMA_PATH/install" ${INSTALL_ARGS[@]+"${INSTALL_ARGS[@]}"}

config/bash/aliases

@@ -22,6 +22,7 @@ alias avante='nvim -c "lua vim.defer_fn(function()require(\"avante.api\").zen_mo
 alias nlc='nvim leetcode.nvim'
 alias clauded='claude --allow-dangerously-skip-permissions'
 alias claudedr='claude --allow-dangerously-skip-permissions --resume'
+alias klaude='CLAUDE_CONFIG_DIR="$HOME/.klaude" claude'
 alias commit-repo='git add -A && git commit -m "Update stuff" && git push'
 
 # Docker Shortcuts
@@ -36,9 +37,9 @@ alias fprint-on='sudo authselect enable-feature with-fingerprint'
 alias fprint-off='sudo authselect disable-feature with-fingerprint'
 
 # File system
-alias ls='eza -lh --group-directories-first --icons'
+alias ls='eza -lh --group-directories-first --icons=auto'
 alias lsa='ls -a'
-alias lt='eza --tree --level=2 --long --icons --git'
+alias lt='eza --tree --level=2 --long --icons=auto --git'
 alias lta='lt -a'
 # Fedora's bat installs /usr/bin/bat; batcat is the Debian name.
 alias ff="fzf --preview 'bat --style=numbers --color=always {}'"

config/bash/shell

@@ -16,8 +16,23 @@ unset PANAMA_ENV
 # Editor used by CLI
 export EDITOR="nvim"
 export SUDO_EDITOR="$EDITOR"
-export SSH_ASKPASS=/usr/bin/ksshaskpass
-export SSH_ASKPASS_REQUIRE=prefer
+# A graphical askpass, and only where there is a session to draw it in.
+# SSH_ASKPASS_REQUIRE=prefer means ssh and git reach for the dialog BEFORE the
+# terminal prompt, so on a headless machine the prompt that would have worked
+# is skipped in favour of one that cannot open: `git push` then fails reading
+# a username rather than asking for it. The desktop keeps the dialog; a
+# server, a serial console and a plain SSH session fall back to the terminal.
+if [ -n "${WAYLAND_DISPLAY:-}${DISPLAY:-}" ] && [ -x /usr/bin/ksshaskpass ]; then
+  export SSH_ASKPASS=/usr/bin/ksshaskpass
+  export SSH_ASKPASS_REQUIRE=prefer
+else
+  # Cleared, not merely left unset. An askpass inherited from a parent that
+  # did have a display -- an agent's shell, a forwarded environment, a login
+  # that predates this guard -- is exactly as unusable here as one this file
+  # set itself, and declining to add it does nothing about a value already in
+  # the environment.
+  unset SSH_ASKPASS SSH_ASKPASS_REQUIRE
+fi
 
 # Define Paths
 export CARGO_PATH="$HOME/.cargo"
@@ -32,8 +47,9 @@ export DOTNETPATH="$HOME/.dotnet/tools"
 # Podman
 # export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
 
-# Set complete path
-export PATH="$HOME/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PANAMA_PATH/bin:$BUN_INSTALL/bin:$CARGO_PATH/bin:$PNPM_HOME/bin:$PYENV_ROOT/bin:$HOME/.rbenv/bin:/usr/lib/ccache/bin/:$GOPATH/bin:$DOTNETPATH"
+# Set complete path. ~/.opencode/bin is OpenCode 2's own installer location
+# and sits ahead of Bun's bin, which can still hold an OpenCode 1 copy.
+export PATH="$HOME/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$PANAMA_PATH/bin:$HOME/.opencode/bin:$BUN_INSTALL/bin:$CARGO_PATH/bin:$PNPM_HOME/bin:$PYENV_ROOT/bin:$HOME/.rbenv/bin:/usr/lib/ccache/bin/:$GOPATH/bin:$DOTNETPATH"
 
 # rustup writes this file, and initial-packages installs rustup rather than
 # running rustup-init -- so on a fresh machine it does not exist yet and an

config/dot/hypr/hyprland.lua

@@ -29,3 +29,9 @@ require("autostart")
 
 -- Per-machine overrides, loaded last so it wins. Ships effectively empty.
 require("overrides")
+
+-- T3 Code SnapShots. T3's setup wrote this line and finds it again by its exact
+-- shape (a bare hl.bind to a global), so keep it here and unwrapped. Going
+-- through keybinds.lua's bind() would make T3 add a second copy. Change or
+-- remove it from T3 Code under Settings > SnapShots.
+hl.bind("CTRL + SHIFT + 2", hl.dsp.global("com.t3tools.T3Code:capture-window"))

config/dot/hypr/input.lua

@@ -8,6 +8,14 @@
 local prefs = require("prefs")
 local actions = require("actions")
 
+-- Empty means "let libinput pick per device". Hyprland 0.57 rejects "" for
+-- this option, so empty is expressed by not writing it at all: nil drops the
+-- key from the table below and the option stays at its unset default.
+local scroll_method = prefs.get("scrollMethod", "")
+if scroll_method == "" then
+    scroll_method = nil
+end
+
 hl.config({
     input = {
         kb_layout  = prefs.get("keyboardLayout", "us"),
@@ -52,16 +60,12 @@ hl.config({
         scroll_factor  = prefs.get("scrollFactor", 1.0),
         left_handed    = prefs.get("leftHanded", false),
 
-        -- Empty is a value, not an omission: it is what `hyprctl getoption`
-        -- reports as "[[EMPTY]]" before anything writes the option, and it means
-        -- "let libinput pick per device" -- two fingers on a touchpad, the wheel
-        -- on a mouse. Writing it back explicitly is the same branch a stock
-        -- Hyprland takes, and it keeps the setting reversible: without an empty
-        -- choice in the schema there would be no way back from a scroll method
-        -- once one was picked. Settings offers 2fg / edge / on_button_down /
+        -- Empty (omitted, see above) is what `hyprctl getoption` reports as
+        -- "[[EMPTY]]": libinput picks per device -- two fingers on a touchpad,
+        -- the wheel on a mouse. Settings offers 2fg / edge / on_button_down /
         -- no_scroll alongside it, the four words the option's own description
         -- names (it publishes no map).
-        scroll_method = prefs.get("scrollMethod", ""),
+        scroll_method = scroll_method,
 
         -- Only consulted while scroll_method is on_button_down. 0 means the
         -- device's own middle button.

config/dot/hypr/keybinds.lua

@@ -377,15 +377,18 @@ bind(mod .. " + SHIFT + N", hl.dsp.window.resize({ x = 0, y = step, relative = t
 -- The gesture needs three binds, not two. Tab steps the selection, and the
 -- switch is only COMMITTED when the modifier is released -- which is the sole
 -- way the compositor can tell the gesture is finished. That release bind is on
--- the bare modifier, so it fires on EVERY Super release in the session; the
--- handler returns immediately when no switch is open, which is why this is
--- affordable.
+-- the Super keys themselves, so it fires on EVERY Super release in the
+-- session; the handler returns immediately when no switch is open, which is
+-- why this is affordable.
 --
--- The release bind carries no description on purpose: it is not a shortcut
--- anyone would look up or rebind, and the Shortcuts page lists what it finds.
+-- A modifier-only bind has to name the keysym (Super_L / Super_R). Hyprland
+-- 0.57 rejects a chord that is a bare modifier ("A bind requires a trigger"),
+-- and the error stops this file, costing every bind below it.
 bind(mod .. " + Tab", hl.dsp.exec_cmd(qs("switcher", "next")), { description = "Next window" })
 bind(mod .. " + SHIFT + Tab", hl.dsp.exec_cmd(qs("switcher", "previous")), { description = "Previous window" })
-bind(mod, hl.dsp.exec_cmd(qs("switcher", "commit")), { release = true, description = "Commit window switch" })
+for _, key in ipairs({ "Super_L", "Super_R" }) do
+    bind(mod .. " + " .. key, hl.dsp.exec_cmd(qs("switcher", "commit")), { release = true, description = "Commit window switch" })
+end
 -- Jump back to the previously focused window.
 bind(mod .. " + SHIFT + grave", hl.dsp.focus({ last = true }), { description = "Last window" })
 

config/dot/hypr/monitors.lua

@@ -244,10 +244,20 @@ local shipped_mode = "4500x3000@60"
 local shipped_scale = 1.5
 local shipped_transform = 0
 
--- 10-bit output. 4500x3000@60 at 10bpc is ~24 Gbps, right at the edge of DP 1.4
--- HBR3, so this relies on DSC. If the display fails to light up or falls back to
--- a lower mode, drop this to 8 first.
-local shipped_bitdepth = 10
+-- 8-bit output. 4500x3000@60 at 10bpc is ~24 Gbps, right at the edge of DP 1.4
+-- HBR3 and reliant on DSC, and this panel's link is marginal: every modeset
+-- retrains it and blanks the screen. 8bpc keeps headroom on the link.
+--
+-- Related: directScanoutPolicy is 0 in Panama settings (2026-09-13). With
+-- scanout on, a fullscreen game whose buffer depth differs from the desktop
+-- (games ship both 8- and 10-bit swapchains) makes Hyprland change the output
+-- format, and on amdgpu a format change is a full modeset. Compositing always
+-- keeps the format fixed, so the link never retrains mid-game.
+--
+-- vrrPolicy is also 0 there. VRR on this panel loses sync and blacks out
+-- (seen on GNOME in July 2026 and again here); a 60Hz panel gains little
+-- from it anyway.
+local shipped_bitdepth = 8
 
 -- "auto" = sRGB at 8bpc, wide gamut at 10bpc. Not HDR; see header.
 local shipped_cm = "auto"

config/dot/hypr/rules.lua

@@ -125,6 +125,16 @@ hl.window_rule({
     no_dim    = true,
 })
 
+-- Steam games never get true fullscreen. Everything Hyprland does only for a
+-- real fullscreen window (direct scanout, VRR, tearing, auto HDR) makes this
+-- panel's marginal DP link retrain and blank (2026-09-13). "1 2" keeps the
+-- window maximized internally while the game believes it is fullscreen, which
+-- is what borderless windowed looks like from the game's side.
+hl.window_rule({
+    match = { class = "^steam_app_\\d+$" },
+    fullscreen_state = "1 2",
+})
+
 -- Steam itself is a normal window, but its transient popups are a mess.
 hl.window_rule({
     match = { class = "^steam$", title = "^(Friends List|Steam Settings|Special Offer.*)$" },
@@ -269,16 +279,28 @@ hl.layer_rule({
     ignore_alpha = 0.2,
 })
 
--- Overview, capture and local screen-reading UI dim the desktop behind them.
+-- Overview and local screen-reading UI dim the desktop behind them.
 hl.layer_rule({
     name  = "qs-overlay",
-    match = { namespace = "^qs-(overview|capture|screen-intelligence)$" },
+    match = { namespace = "^qs-(overview|screen-intelligence)$" },
     blur         = true,
     ignore_alpha = 0.4,
     dim_around   = true,
     no_screen_share = true,
 })
 
+-- The screenshot picker paints an opaque frozen frame edge to edge, so blur and
+-- dim would be compositor work on pixels nobody sees. no_anim is load-bearing:
+-- services/Capture.qml unmaps the picker and runs grim 90ms later, and the
+-- 200ms layersOut fade would otherwise put the half-faded picker, dimming and
+-- all, into every screenshot.
+hl.layer_rule({
+    name  = "qs-capture",
+    match = { namespace = "^qs-capture$" },
+    no_anim         = true,
+    no_screen_share = true,
+})
+
 -- Notification toasts. Blurred like every other shell surface -- without this
 -- the cards are a near-transparent fill sitting directly on the wallpaper and
 -- read as washed out rather than as glass.

config/dot/quickshell/config/PreferenceSchema.qml

@@ -184,7 +184,8 @@ Singleton {
             options: [
                 { value: "none", label: "None" },
                 { value: "claude", label: "Claude Code" },
-                { value: "codex", label: "Codex" }
+                { value: "codex", label: "Codex" },
+                { value: "t3", label: "T3 Code" }
             ]
         },
         {
@@ -205,7 +206,7 @@ Singleton {
         {
             key: "agentAutoApprove", type: "bool", def: true, group: "agents",
             label: "Launched agents approve their own tools",
-            detail: "Investigations run without permission prompts. The diagnose skill still holds agents to reading rather than fixing, and root still goes through panama-sudo, reason and all"
+            detail: "Investigations run without permission prompts. T3 Code threads use the mode set on the Panama project in T3 instead. The diagnose skill still holds agents to reading rather than fixing, and root still goes through panama-sudo, reason and all"
         },
         {
             key: "agentUsageClaude", type: "bool", def: true, group: "agents",
@@ -1805,7 +1806,7 @@ Singleton {
                 "org.mozilla.thunderbird_esr", "com.slack.Slack",
                 "app.bluebubbles.BlueBubbles", "rustdesk",
                 "io.podman_desktop.PodmanDesktop", "com.anthropic.Claude",
-                "codex-desktop", "md.obsidian.Obsidian",
+                "chatgpt", "md.obsidian.Obsidian",
                 "com.obsproject.Studio", "steam"
             ]
         },

config/dot/quickshell/modules/capture/CaptureOverlay.qml

@@ -27,9 +27,13 @@ PanelWindow {
         right: true
     }
 
-    // Fullscreen overlays must not reserve space, or every window on the
-    // workspace gets resized as the picker opens and closes.
-    exclusiveZone: 0
+    // Ignore, and nothing else: reserve no space (or every window on the
+    // workspace resizes as the picker opens and closes) and respect nobody's
+    // (or the window starts under the bar, the full-output freeze frame is
+    // squeezed into a shorter box, and every selection lands one bar-height
+    // off in the real capture). Never pair this with exclusiveZone; see
+    // tests/quickshell/exclusion-idiom-contract.
+    exclusionMode: ExclusionMode.Ignore
 
     WlrLayershell.namespace: "qs-capture" // matched by a layerrule in hypr/rules.lua
     WlrLayershell.layer: WlrLayer.Overlay

config/dot/quickshell/modules/cheatsheet/Cheatsheet.qml

@@ -57,7 +57,6 @@ PanelWindow {
 
     anchors { top: true; bottom: true; left: true; right: true }
     color: "transparent"
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
 
     // The `^qs-popover` prefix rule in hypr/rules.lua blurs what is behind

config/dot/quickshell/modules/dock/Dock.qml

@@ -55,7 +55,6 @@ PanelWindow {
     color: "transparent"
 
     // A dock that reserved space would not be intellihiding.
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
 
     // Matched by the `qs-dock` layer rule in hypr/rules.lua — do not rename.

config/dot/quickshell/modules/dock/DockPickOverlay.qml

@@ -40,7 +40,6 @@ PanelWindow {
 
     anchors { top: true; bottom: true; left: true; right: true }
     color: "transparent"
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
 
     // Blurred by the `^qs-popover` rule in hypr/rules.lua; the scrim is painted

config/dot/quickshell/modules/notifications/VisualBell.qml

@@ -44,7 +44,6 @@ PanelWindow {
     // Reserve nothing and respect nothing: the glow is drawn over the whole
     // output including under the bar and the dock, which is what makes it
     // visible from wherever the eyes happen to be.
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
     color: "transparent"
 

config/dot/quickshell/modules/osd/Osd.qml

@@ -29,7 +29,6 @@ PanelWindow {
     screen: root.modelData
     anchors.bottom: true
     margins.bottom: Theme.dockIconSize + Theme.dockPadding * 2 + Theme.barGap * 3
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
     implicitWidth: root.desiredWidth
     implicitHeight: 64

config/dot/quickshell/modules/overview/Overview.qml

@@ -22,7 +22,6 @@ PanelWindow {
     }
 
     color: "transparent"
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
 
     // Matched by the `qs-overlay` layer rule in hypr/rules.lua — do not rename.

config/dot/quickshell/modules/quicksettings/BrightnessControl.qml

@@ -34,11 +34,23 @@ Item {
     visible: root.rowCount > 0
     implicitHeight: rows.implicitHeight
 
-    // Probing I2C takes on the order of a second, so it waits until the panel
-    // is actually on screen rather than running at shell startup. Monitors do
-    // not come and go, so once is enough.
-    onVisibleChanged: if (visible && !Brightness.scanned) Brightness.refresh()
-    Component.onCompleted: if (root.visible && !Brightness.scanned) Brightness.refresh()
+    // Probing I2C takes on the order of a second, so it waits until the quick
+    // settings panel is actually on screen rather than running at shell
+    // startup. Monitors do not come and go, so once is enough.
+    //
+    // The trigger is the panel opening, not this item becoming visible: on a
+    // machine with no backlight, `visible` stays false until a scan has found
+    // a DDC display, so keying the scan off it would never scan at all.
+    Connections {
+        target: ShellState
+        function onQuickSettingsOpenChanged(): void { root.scanIfOpen(); }
+    }
+    Component.onCompleted: root.scanIfOpen()
+
+    function scanIfOpen(): void {
+        if (ShellState.quickSettingsOpen && !Brightness.scanned)
+            Brightness.refresh();
+    }
 
     // `-m` is the machine-readable form: name,class,current,percent,max
     Process {

config/dot/quickshell/modules/settings/AgentsPage.qml

@@ -37,8 +37,8 @@ SettingsPage {
     // them not to believe the page.
     //
     // Probed through a LOGIN shell rather than this one. Quickshell is started
-    // by systemd, whose PATH does not include ~/.local/bin -- where both of
-    // these usually land -- and a login shell is the environment the launcher
+    // by systemd, whose PATH does not include ~/.local/bin -- where all three of
+    // these usually land (T3 Code's CLI is `t3`) -- and a login shell is the environment the launcher
     // hands the agent when it spawns a terminal. Asking with the shell's own
     // PATH would report "not installed" for an agent that starts perfectly.
     property var installed: ({})
@@ -60,7 +60,7 @@ SettingsPage {
 
         running: true
         command: ["bash", "-lc",
-            "for agent in claude codex; do command -v \"$agent\" >/dev/null 2>&1 && printf '%s\\n' \"$agent\"; done"]
+            "for agent in claude codex t3; do command -v \"$agent\" >/dev/null 2>&1 && printf '%s\\n' \"$agent\"; done"]
         stdout: StdioCollector {
             onStreamFinished: root.absorbProbe(this.text)
         }
@@ -70,6 +70,7 @@ SettingsPage {
         switch (value) {
         case "claude": return "starred-symbolic";
         case "codex": return "utilities-terminal-symbolic";
+        case "t3": return "view-dual-symbolic";
         default: return "notifications-disabled-symbolic";
         }
     }

config/dot/quickshell/modules/settings/DisplayIdentify.qml

@@ -22,7 +22,6 @@ Variants {
         implicitWidth: 260
         implicitHeight: 172
         color: "transparent"
-        exclusiveZone: 0
         exclusionMode: ExclusionMode.Ignore
         mask: Region {}
 

config/dot/quickshell/modules/settings/README.md

@@ -383,6 +383,13 @@ Lua-configured Hyprland, prints the refusal to stdout, and exits 0. `eval` exits
 0 on syntax and runtime errors too. The only trustworthy signal that a write
 landed is reading the value back.
 
+**`exclusionMode: ExclusionMode.Ignore` goes alone.** Quickshell's `exclusiveZone`
+setter flips `exclusionMode` back to `Normal` as a side effect, so a window that
+declares both is at the mercy of which property the QML engine applies last.
+The 2026-09-14 Qt update changed that order and every full-screen overlay slid
+under the bar. `tests/quickshell/exclusion-idiom-contract` fails any file that
+pairs them.
+
 **The Settings window is tiled.** `implicitWidth` is a hint; the layout decides,
 and it ranges from a half-screen split to the full display. `SliderRow` stacks
 its control under the label below 520px. Test narrow.

config/dot/quickshell/modules/welcome/Welcome.qml

@@ -78,7 +78,6 @@ PanelWindow {
 
     anchors { top: true; bottom: true; left: true; right: true }
     color: "transparent"
-    exclusiveZone: 0
     exclusionMode: ExclusionMode.Ignore
 
     WlrLayershell.namespace: "qs-popover-welcome"

config/dot/quickshell/services/Capture.qml

@@ -341,7 +341,10 @@ esac'
         property bool record: false
         property bool intelligence: false
         // A handful of frames at 60Hz, enough for the compositor to recomposite
-        // the output without the overlay on it.
+        // the output without the overlay on it. Only enough because the
+        // qs-capture layer rule in hypr/rules.lua sets no_anim: with the
+        // default 200ms layersOut fade the half-faded picker would still be
+        // on screen when grim reads it.
         interval: 90
         onTriggered: {
             const r = commitDelay.rect;

config/dot/quickshell/services/Keybinds.qml

@@ -487,11 +487,11 @@ Singleton {
             if ((bind.modmask & modifier.bit) !== 0)
                 parts.push(modifier.name.toUpperCase());
         }
-        // A modifier-only bind has no key at all -- the window switcher commits
-        // on Super RELEASE. Appending an empty string left a dangling "SUPER + "
-        // that matched neither the chord hypr/keybinds.lua binds nor the one an
-        // override would be keyed by, so that bind could never be rebound and
-        // never found its category.
+        // Guard against a bind with no key. Hyprland 0.57 refuses to create
+        // one (the switcher's Super-release commit now names Super_L/Super_R),
+        // but appending an empty string would leave a dangling "SUPER + " that
+        // matches neither the chord hypr/keybinds.lua binds nor the one an
+        // override would be keyed by.
         const key = String(bind.key ?? "");
         if (key !== "")
             parts.push(key);

config/dot/quickshell/services/SettingsSearch.qml

@@ -356,6 +356,7 @@ Singleton {
         { label: "AI assistant", detail: "Choose the agent this desktop hands crashes, failed reloads and red health checks to", page: "agents" },
         { label: "Claude Code", detail: "Use Claude Code as the agent the desktop escalates to, and watch its usage", page: "agents" },
         { label: "Codex", detail: "Use Codex as the agent the desktop escalates to, and watch its usage", page: "agents" },
+        { label: "T3 Code", detail: "Open a T3 Code thread in the Panama project when the desktop escalates something", page: "agents" },
         { label: "Diagnose crashes", detail: "When a program dumps core, the notification carries a click that opens your AI assistant with the crash details", page: "agents" },
         { label: "Ask the agent", detail: "A red System Health check with no repair left hands its snapshot to your AI assistant", page: "agents" },
         { label: "Agent usage panel", detail: "Limits, resets and tokens for each agent, opened from the bar", page: "agents" },

config/local/share/vicinae/scripts/settings-agents

@@ -5,6 +5,6 @@
 # @vicinae.mode silent
 # @vicinae.icon ../../icons/hicolor/scalable/apps/panama-settings.svg
 # @vicinae.description Open Agents in Settings.
-# @vicinae.keywords ["settings", "preferred agent", "offer to diagnose crashes", "offer help when the shell fails to reload", "system health hands off unrepairable checks", "launched agents approve their own tools", "collect claude code usage", "collect codex usage", "refresh interval", "ai assistant", "claude code", "codex", "diagnose crashes"]
+# @vicinae.keywords ["settings", "preferred agent", "offer to diagnose crashes", "offer help when the shell fails to reload", "system health hands off unrepairable checks", "launched agents approve their own tools", "collect claude code usage", "collect codex usage", "refresh interval", "ai assistant", "claude code", "codex", "t3 code"]
 
 exec "$HOME/.config/quickshell/scripts/panama-action" settings-page agents

config/zsh/.zshrc

@@ -0,0 +1,55 @@
+#!/usr/bin/env zsh
+# The Mac mini's shell. macOS stays on zsh: its bash is 3.2, and
+# config/bash/shell leans on Linux-only tools (stat -c, ksshaskpass, the nvm
+# profile script), so this is a small zsh counterpart rather than a port.
+# Linked to ~/.zshrc. Homebrew's PATH comes from ~/.zprofile (brew shellenv).
+export PANAMA_PATH="$HOME/.local/share/Panama"
+
+export EDITOR="nvim"
+export SUDO_EDITOR="$EDITOR"
+
+export BUN_INSTALL="$HOME/.bun"
+# ~/.opencode/bin is OpenCode 2's own installer location, ahead of Bun's bin.
+export PATH="$HOME/.local/bin:$HOME/.opencode/bin:$BUN_INSTALL/bin:$PANAMA_PATH/bin:$PATH"
+
+# History shared across sessions, so tmux panes see each other's commands.
+HISTFILE="$HOME/.zsh_history"
+HISTSIZE=50000
+SAVEHIST=50000
+setopt SHARE_HISTORY HIST_IGNORE_DUPS HIST_IGNORE_SPACE
+
+# Emacs keys, like bash's readline: ctrl-a/e/r work as they do on Linux.
+bindkey -e
+
+autoload -Uz compinit && compinit
+[ -s "$BUN_INSTALL/_bun" ] && source "$BUN_INSTALL/_bun"
+
+# Same SSH behaviour as config/bash/shell: an SSH login lands in a tmux
+# session that survives a dropped connection. PANAMA_SSH_TMUX=off opts out.
+if [[ -n "$SSH_CONNECTION" && -z "$TMUX" && -o interactive &&
+  "${PANAMA_SSH_TMUX:-on}" != "off" ]] && command -v tmux >/dev/null 2>&1; then
+  exec tmux new-session -A -s main
+fi
+
+alias c="clear"
+alias n="nvim"
+alias n.="nvim ."
+alias ..="cd .."
+alias ...="cd ../.."
+alias ....="cd ../../.."
+alias letscode="ranger ~/Documents/Documents/Code"
+alias commit-repo='git add -A && git commit -m "Update stuff" && git push'
+alias ls='eza -lh --group-directories-first --icons=auto'
+alias lsa='ls -a'
+alias lt='eza --tree --level=2 --long --icons=auto --git'
+alias ff="fzf --preview 'bat --style=numbers --color=always {}'"
+compress() { tar -czf "${1%/}.tar.gz" "${1%/}"; }
+alias decompress="tar -xzf"
+
+# Xcode cannot sign from an SSH session while the login keychain is locked
+# (errSecInternalComponent). Run this once per session before a build.
+alias unlock-keychain='security unlock-keychain ~/Library/Keychains/login.keychain-db'
+
+eval "$(zoxide init zsh)"
+source <(fzf --zsh)
+eval "$(oh-my-posh init zsh --config "$PANAMA_PATH/config/dot/ohmyposh/gib.omp.json")"

docs/settings.md

@@ -28,11 +28,11 @@ Found on **System › Agents**.
 
 | Setting | Default | What it does |
 |---|---|---|
-| **Preferred agent**<br>`preferredAgent` | none | Who answers when the desktop offers to investigate something Choices: None, Claude Code, Codex. |
+| **Preferred agent**<br>`preferredAgent` | none | Who answers when the desktop offers to investigate something Choices: None, Claude Code, Codex, T3 Code. |
 | **Offer to diagnose crashes**<br>`crashDiagnoseOffer` | true | When a program dumps core, the notification carries a click that opens the preferred agent mid-investigation with the crash details in hand |
 | **Offer help when the shell fails to reload**<br>`reloadFailureOffer` | true | A broken change to the shell's own configuration offers the failing log to the agent |
 | **System Health hands off unrepairable checks**<br>`healthAgentHandoff` | true | A red check with no repair, or whose repair failed, grows an Ask-the-agent button carrying the check's snapshot |
-| **Launched agents approve their own tools**<br>`agentAutoApprove` | true | Investigations run without permission prompts. The diagnose skill still holds agents to reading rather than fixing, and root still goes through panama-sudo, reason and all |
+| **Launched agents approve their own tools**<br>`agentAutoApprove` | true | Investigations run without permission prompts. T3 Code threads use the mode set on the Panama project in T3 instead. The diagnose skill still holds agents to reading rather than fixing, and root still goes through panama-sudo, reason and all |
 | **Collect Claude Code usage**<br>`agentUsageClaude` | true | Limits from Anthropic's usage endpoint, tokens from the local transcripts |
 | **Collect Codex usage**<br>`agentUsageCodex` | true | Limits over the Codex app-server, sessions from its local files |
 | **Refresh interval**<br>`agentUsageRefreshMinutes` | 15  min | How often the usage collectors ask for fresh numbers, in minutes. Range 5–60. |

install

@@ -71,7 +71,7 @@ source "$PANAMA_PATH/bin/ascii"
 # time. On an upgrade it is worth running only when its package lists or
 # reviewed installer trust inputs changed, so this hashes them and remembers
 # the result. The framed, sorted stream includes top-level package files, the
-# package-stage adapter, the provenance helper, and regular provenance files;
+# package-stage adapter, every helper it sources, and regular provenance files;
 # both relative paths and bytes are part of the state.
 #
 # A content hash rather than a git range, because Panama is developed in place:
@@ -88,75 +88,42 @@ source "$PANAMA_PATH/bin/ascii"
 STATE_DIR="${XDG_STATE_HOME:-$HOME/.local/state}/panama"
 PACKAGES_HASH="$STATE_DIR/packages-hash"
 
-_collect_package_inputs() {
-  local destination="$1" raw="${1}.raw"
-  [[ -d "$PANAMA_PATH/setup/packages" \
-      && ! -L "$PANAMA_PATH/setup/packages" \
-      && -d "$PANAMA_PATH/setup/provenance" \
-      && ! -L "$PANAMA_PATH/setup/provenance" ]] || return 1
-  {
-    printf '%s\0' \
-      "$PANAMA_PATH/setup/scripts/install-packages" \
-      "$PANAMA_PATH/setup/lib/artifact-provenance" \
-      "$PANAMA_PATH/setup/lib/extras-catalog" \
-      "$PANAMA_PATH/setup/lib/machine-role" || exit 1
-    # extras/ is deliberately excluded. A symlink or other non-directory
-    # object at this level is still an input error, not something discovery may
-    # silently omit.
-    find "$PANAMA_PATH/setup/packages" -mindepth 1 -maxdepth 1 \
-      ! -type d -print0 || exit 1
-    find "$PANAMA_PATH/setup/provenance" -mindepth 1 \
-      ! -type d -print0 || exit 1
-  } >"$raw" || return 1
-  LC_ALL=C sort -z "$raw" >"$destination"
+hash_packages() {
+  local file relative size fixed_input digest
+
+  for fixed_input in \
+    "$PANAMA_PATH/setup/scripts/install-packages" \
+    "$PANAMA_PATH/setup/lib/artifact-provenance" \
+    "$PANAMA_PATH/setup/lib/chatgpt-package" \
+    "$PANAMA_PATH/setup/lib/extras-catalog" \
+    "$PANAMA_PATH/setup/lib/machine-role"; do
+    [[ -f "$fixed_input" && ! -L "$fixed_input" && -r "$fixed_input" ]] || return 1
+  done
+
+  digest="$(
+    {
+      find "$PANAMA_PATH/setup/packages" -maxdepth 1 -type f -print0 || exit 1
+      printf '%s\0' \
+        "$PANAMA_PATH/setup/scripts/install-packages" \
+        "$PANAMA_PATH/setup/lib/artifact-provenance" \
+        "$PANAMA_PATH/setup/lib/chatgpt-package" \
+        "$PANAMA_PATH/setup/lib/extras-catalog" \
+        "$PANAMA_PATH/setup/lib/machine-role" || exit 1
+      find "$PANAMA_PATH/setup/provenance" -type f -print0 || exit 1
+    } | LC_ALL=C sort -z | while IFS= read -r -d '' file; do
+      relative="${file#"$PANAMA_PATH"/}"
+      size="$(wc -c <"$file")" || exit 1
+      printf '%s\0%s\0' "$relative" "$size" || exit 1
+      cat -- "$file" || exit 1
+      printf '\0' || exit 1
+    done | sha256sum | cut -d' ' -f1
+  )" || return 1
+  printf '%s\n' "$digest"
 }
 
-_write_package_manifest() {
-  local inputs="$1" destination="$2" file relative digest
-  : >"$destination" || return 1
-  while IFS= read -r -d '' file; do
-    [[ -f "$file" && ! -L "$file" && -r "$file" ]] || return 1
-    relative="${file#"$PANAMA_PATH"/}"
-    [[ "$relative" != "$file" ]] || return 1
-    digest="$(sha256sum -- "$file" | awk '{ print $1 }')" || return 1
-    [[ "$digest" =~ ^[0-9a-f]{64}$ ]] || return 1
-    printf '%s\0%s\0' "$relative" "$digest" >>"$destination" || return 1
-  done <"$inputs"
-}
-
-# Read every input twice from the same enumerated set. A file or path that
-# changes while the snapshot is built cannot produce a receipt.
-hash_packages() (
-  local work="" inputs_before inputs_after manifest_before manifest_after
-  trap '[[ -z "$work" ]] || rm -rf -- "$work"' EXIT
-  trap 'exit 130' INT
-  trap 'exit 143' TERM
-  work="$(mktemp -u -d -t panama-packages-hash.XXXXXX)" || exit 1
-  if ! mkdir -m 700 -- "$work"; then
-    work=""
-    exit 1
-  fi
-  inputs_before="$work/inputs-before"
-  inputs_after="$work/inputs-after"
-  manifest_before="$work/manifest-before"
-  manifest_after="$work/manifest-after"
-
-  _collect_package_inputs "$inputs_before" || exit 1
-  _write_package_manifest "$inputs_before" "$manifest_before" || exit 1
-  _collect_package_inputs "$inputs_after" || exit 1
-  cmp -s -- "$inputs_before" "$inputs_after" || exit 1
-  _write_package_manifest "$inputs_after" "$manifest_after" || exit 1
-  cmp -s -- "$manifest_before" "$manifest_after" || exit 1
-  sha256sum -- "$manifest_before" | awk '{ print $1 }'
-)
-
-PACKAGE_START_HASH=""
-
 packages_needed() {
-  local current_hash recorded_hash
+  local current_hash="$1" recorded_hash
 
-  current_hash="$(hash_packages)" || return 2
-  PACKAGE_START_HASH="$current_hash"
   (( FORCE_PACKAGES )) && return 0
   (( UPGRADE )) || return 0
   [[ -r "$PACKAGES_HASH" ]] || return 0
@@ -167,41 +134,52 @@ packages_needed() {
 # Written only after the stage succeeds, mirroring the rule panama-migrate
 # documents for its markers: a step that did not complete has not happened, and
 # recording it as done hides it forever.
-record_packages_hash() (
-  local expected_hash="$1" current_hash temporary_hash=""
-  trap '[[ -z "$temporary_hash" ]] || rm -f -- "$temporary_hash"' EXIT
-  trap 'exit 130' INT
-  trap 'exit 143' TERM
-  [[ "$expected_hash" =~ ^[0-9a-f]{64}$ ]] || return 1
+record_packages_hash() {
+  local starting_hash="$1" current_hash temporary_hash
   current_hash="$(hash_packages)" || return 1
-  [[ "$current_hash" == "$expected_hash" ]] || return 1
+  [[ "$current_hash" == "$starting_hash" ]] || return 1
   mkdir -p "$STATE_DIR"
-  temporary_hash="$(mktemp -u "$STATE_DIR/.packages-hash.XXXXXX")" || return 1
-  umask 077
-  if ! (set -o noclobber; : >"$temporary_hash") 2>/dev/null; then
-    temporary_hash=""
+  temporary_hash="$(mktemp "$STATE_DIR/.packages-hash.XXXXXX")" || return 1
+  if printf '%s\n' "$starting_hash" >"$temporary_hash"; then
+    mv -f -- "$temporary_hash" "$PACKAGES_HASH"
+  else
+    rm -f -- "$temporary_hash"
     return 1
   fi
-  printf '%s\n' "$expected_hash" >"$temporary_hash" || return 1
-  mv -f -- "$temporary_hash" "$PACKAGES_HASH" || return 1
-  temporary_hash=""
-)
+}
 
 # Repository trust is checked before the installer can reach its bootstrap DNF.
-# Status 78 is reserved for a trust-root failure and is propagated unchanged so
-# no later stage, especially install-hardware, can invoke DNF with that repo.
+# Status 78 is reserved for a trust-root failure. It suppresses every stage that
+# opens DNF -- install-hardware included, which would otherwise pull drivers
+# through the very repository in doubt -- and the run still exits 78 at the end.
+#
+# It suppresses nothing else. Linking dotfiles, skills and user content reads no
+# repository and installs no package, and a machine whose Terra is in question
+# still wants its configuration. Refusing the safe work because the unsafe work
+# is unavailable does not make the machine safer, it just leaves the machine
+# unconfigured with no way to fix itself. Exiting here instead meant link-skills
+# never ran on a machine whose Terra was merely unadopted, so ~/.claude/skills
+# stayed the whole-directory symlink it had been before skills were linked one
+# by one, and not one shipped skill was reachable.
 TERRA_TRUST_FAILURE_STATUS=78
+DNF_STAGES=(install-packages change-settings install-hardware)
+package_trust_refused=0
+
+stage_opens_dnf() {
+  local candidate="$1" dnf_stage
+  for dnf_stage in "${DNF_STAGES[@]}"; do
+    [[ "$candidate" == "$dnf_stage" ]] && return 0
+  done
+  return 1
+}
+
 trust_preflight="$PANAMA_PATH/setup/scripts/install-packages"
 if [[ ! -x "$trust_preflight" ]]; then
   printf 'install: package repository trust preflight is unavailable\n' >&2
-  exit "$TERRA_TRUST_FAILURE_STATUS"
-fi
-if "$trust_preflight" --trust-preflight; then
-  :
-else
-  trust_status=$?
+  package_trust_refused=1
+elif ! "$trust_preflight" --trust-preflight; then
   printf 'install: package repository trust preflight failed\n' >&2
-  exit "$trust_status"
+  package_trust_refused=1
 fi
 
 # ── The interview ────────────────────────────────────────────────────────────
@@ -221,7 +199,7 @@ fi
 # Gated exactly like the interview itself: under --upgrade no questions are
 # asked, so nothing here is used, and a machine that cannot install gum must
 # not have that stop an upgrade that never needed it.
-if (( ! UPGRADE )); then
+if (( ! UPGRADE && ! package_trust_refused )); then
   bootstrap=()
   command -v gum >/dev/null 2>&1 || bootstrap+=(gum)
   # The probe tools serve only the hardware questions, which a server is never
@@ -236,8 +214,7 @@ if (( ! UPGRADE )); then
   fi
   if (( ${#bootstrap[@]} > 0 )); then
     echo "Installing what the setup questions are built on: ${bootstrap[*]}"
-    sudo dnf install -y --repo=fedora --repo=updates \
-      --from-repo=fedora,updates "${bootstrap[@]}" >/dev/null || {
+    sudo dnf install -y "${bootstrap[@]}" >/dev/null || {
       echo "Could not install ${bootstrap[*]}, so the setup questions cannot be asked." >&2
       exit 1
     }
@@ -300,12 +277,7 @@ gsettings set org.gnome.desktop.session idle-delay 0 2>/dev/null || true
 # is unset and each stage takes the empty-answer path it already documents --
 # which is why this is a flag rather than a rewrite of seven stage scripts.
 if (( ! UPGRADE )); then
-  PANAMA_ANSWERS="$(mktemp -u -t panama-answers.XXXXXX)" || exit 1
-  umask 077
-  if ! (set -o noclobber; : >"$PANAMA_ANSWERS") 2>/dev/null; then
-    PANAMA_ANSWERS=""
-    exit 1
-  fi
+  PANAMA_ANSWERS="$(mktemp -t panama-answers.XXXXXX)"
   export PANAMA_ANSWERS
 
   if ! PANAMA_ROLE_PRESET="$ROLE_PRESET" "$PANAMA_PATH/setup/scripts/interview"; then
@@ -371,7 +343,7 @@ fi
 if [[ "$PANAMA_ROLE" == server ]]; then
   STAGES=(install-packages link-dotfiles link-user setup-server link-server setup-identity)
 else
-  STAGES=(install-packages link-dotfiles link-skills link-user change-settings link-vicinae-scripts setup-identity install-hardware)
+  STAGES=(install-packages link-dotfiles link-skills link-user link-mcp change-settings link-vicinae-scripts setup-identity install-hardware)
 fi
 
 # The two an upgrade drops. Both exist only to act on interview answers, and
@@ -394,11 +366,16 @@ for stage in "${STAGES[@]}"; do
   script="$PANAMA_PATH/setup/scripts/$stage"
   [[ -x "$script" ]] || continue
   printf '\n=== %s ===\n' "$stage"
+  if (( package_trust_refused )) && stage_opens_dnf "$stage"; then
+    echo "Skipped: the package repository trust check refused package work."
+    continue
+  fi
   if [[ "$stage" == install-packages ]]; then
-    package_start_hash=""
     package_state_status=0
-    packages_needed || package_state_status=$?
-    package_start_hash="$PACKAGE_START_HASH"
+    package_start_hash="$(hash_packages)" || package_state_status=2
+    if (( package_state_status == 0 )); then
+      packages_needed "$package_start_hash" || package_state_status=$?
+    fi
     if (( package_state_status == 1 )); then
       echo "The package lists have not changed since the last run; skipping."
       echo "Run with --packages to install them anyway."
@@ -418,9 +395,12 @@ for stage in "${STAGES[@]}"; do
     fi
   else
     stage_status=$?
+    # A configuration change between the preflight and this stage. Suppress the
+    # remaining DNF stages, keep the safe ones, and carry the status to the end.
     if [[ "$stage" == install-packages && "$stage_status" -eq "$TERRA_TRUST_FAILURE_STATUS" ]]; then
       printf '!!! %s stopped on an untrusted package repository\n' "$stage" >&2
-      exit "$stage_status"
+      package_trust_refused=1
+      continue
     fi
     failed+=("$stage")
     printf '!!! %s failed\n' "$stage" >&2
@@ -446,8 +426,16 @@ done
 # written for -- and baselining would skip every one of them forever. Every
 # migration is self-guarding and a no-op where it does not apply, so running
 # them is the safe direction.
+#
+# Held back when package work was refused. A migration is free to run a DNF
+# transaction -- the ChatGPT package replacement does exactly that -- so the
+# repositories have to be trustworthy before any of them is allowed to run.
+# They are not marked applied either, so the next run still has them pending.
 migrate="$PANAMA_PATH/bin/panama-migrate"
-if [[ -x "$migrate" ]]; then
+if (( package_trust_refused )) && [[ -x "$migrate" ]]; then
+  printf '\n=== migrations ===\n'
+  echo "Skipped: the package repository trust check refused package work."
+elif [[ -x "$migrate" ]]; then
   printf '\n=== migrations ===\n'
   if (( UPGRADE )) || [[ -d "$STATE_DIR/migrations" ]]; then
     "$migrate" run || failed+=(migrations)
@@ -495,6 +483,17 @@ else
   retry='./install'
 fi
 
+# Reported last and on its own, because it is not an ordinary stage failure:
+# everything safe did run, and what did not run is named rather than buried in a
+# list. The exit status stays 78 so a caller can still tell the two apart.
+if (( package_trust_refused )); then
+  printf 'Package work was refused: the Terra repository configuration on this\n' >&2
+  printf 'machine is not one Panama can verify. Skipped: %s\n' "${DNF_STAGES[*]}" >&2
+  printf 'Everything that touches no repository was still applied.\n' >&2
+  printf 'Inspect it with: panama diagnose\n' >&2
+  exit "$TERRA_TRUST_FAILURE_STATUS"
+fi
+
 if (( ${#failed[@]} == 0 )); then
   if (( UPGRADE )); then
     echo "Panama is up to date."

migrations/1787804505.sh

@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+# replace the community ChatGPT Desktop build with the official OpenAI package
+#
+# `panama app chatgpt-desktop` used to build a community wrapper (codex-desktop)
+# from the upstream macOS disk image, complete with a local rebuild daemon.
+# OpenAI ships an official Linux RPM now, and the installer takes that instead;
+# this repairs machines still carrying the community build. The official
+# package goes on before the community one comes off, so a failure part-way
+# leaves the machine with an app, never without one.
+#
+# Rules, because the runner cannot enforce them:
+#
+#   * Safe to run twice. The marker records success, not intent.
+#   * Tolerant of the repair already being correct -- the user may have fixed
+#     it by hand, or a later ./install may have put it back.
+#   * Root work goes through `panama-sudo --reason "..."`, never bare sudo,
+#     so the password prompt names the repair.
+#   * Exit non-zero to be retried at the next login. Exit zero only when the
+#     machine is genuinely in the state this describes.
+
+set -euo pipefail
+
+PANAMA_PATH="${PANAMA_PATH:-$HOME/.local/share/Panama}"
+
+# Machines that never had the community build are already correct. The official
+# app is the installer's job, not this one's.
+rpm -q codex-desktop >/dev/null 2>&1 || exit 0
+
+# The same verified repository the installer establishes: the pinned signing
+# key, then a repository that names it, so dnf checks OpenAI's signature before
+# root installs anything. See setup/lib/chatgpt-package.
+# shellcheck source=../setup/lib/chatgpt-package
+source "$PANAMA_PATH/setup/lib/chatgpt-package"
+
+sudo_cmd=(sudo)
+if [[ -t 0 && -x "$PANAMA_PATH/bin/panama-sudo" ]]; then
+    sudo_cmd=(
+        "$PANAMA_PATH/bin/panama-sudo" --reason
+        "Replacing the community-built ChatGPT Desktop (codex-desktop) with the official OpenAI package"
+        --
+    )
+fi
+
+# The official package first, so the machine is never left without one.
+if ! rpm -q chatgpt >/dev/null 2>&1; then
+    chatgpt_install_repository "${sudo_cmd[@]}"
+    "${sudo_cmd[@]}" dnf install -y chatgpt
+fi
+
+# The community package's updater is a user unit; stop it before dnf removes
+# the unit file out from under it. Removal also takes the app in /opt, both
+# binaries, and the polkit policy the local rebuilds needed.
+systemctl --user disable --now codex-update-manager.service 2>/dev/null || true
+"${sudo_cmd[@]}" dnf remove -y codex-desktop
+systemctl --user daemon-reload 2>/dev/null || true
+
+# The rebuild state the updater kept; the official package needs none of it.
+rm -rf "${XDG_CACHE_HOME:-$HOME/.cache}/codex-update-manager" \
+       "${XDG_CACHE_HOME:-$HOME/.cache}/codex-runtimes" \
+       "${XDG_CONFIG_HOME:-$HOME/.config}/codex-update-manager" \
+       "${XDG_STATE_HOME:-$HOME/.local/state}/codex-update-manager"
+
+echo "Replaced the community codex-desktop build with the official chatgpt package."

server/README.md

@@ -6,18 +6,44 @@ screen, plus rootless podman compose services managed as systemd user units.
 
 ```
 server/
-  containers/     One directory per service: compose.yml, the
-                  podman-<name>.service user unit, and an .env.example
-                  naming what the service needs told
+  vps/            The catalog panama-server installs from, one directory
+                  per service: compose.yml, the podman-<name>.service user
+                  unit, and an .env.example naming what the service needs told.
+                  Loose files beside the directories are the VPS's own
+                  host tooling (see "Host tooling" below)
+  home/           The home server's services in the same shape (see below)
   scripts/        update-containers, the nightly image updater
   systemd/        The units behind it, linked by setup-server
 ```
 
+## The home server
+
+`server/home/` mirrors `~/Server` on the home server, where the services still
+run under Docker. Each directory has the same three files as the VPS catalog,
+and the unit is drafted for the eventual move to rootless podman. It has never
+run, and the composes still use Docker-only features (host networking, the
+GPU, the Docker socket) that need converting first. Nothing here is
+installable: panama-server reads `server/vps/` only, so a service moves into
+the catalog when it actually moves to podman.
+
+Paths keep their nesting (`server/home/Agents/Gitea` is `~/Server/Agents/Gitea`).
+`Media/compact-tv/` is the nightly TV re-encode job that runs beside the Media
+stack, with its own README.
+Three services are checkouts of upstream repositories. For those, only the
+local files are kept, and an `UPSTREAM` file names the repository and the exact
+commit the checkout sits at: `PostHog`, `Sentry` and `Plausible`.
+
+Both trees are snapshots of the live machines with every secret replaced by a
+`${VAR}` named in `.env.example`, or by `CHANGE_ME` in files that are not
+compose files. The live servers still carry some secrets inline, so a live
+compose and its copy here can differ on those lines alone.
+
 ## The contract with ~/Server
 
 The repository carries the definitions; the machine carries the state.
 `panama server enable <Name>` creates `~/Server/<Name>/` as a **real
-directory** and symlinks only `compose.yml` into it; the unit symlinks into
+directory** and symlinks the service's visible top-level files into it:
+`compose.yml`, plus any helper script or README. The units symlink into
 `~/.config/systemd/user/`. The `.env` (seeded once from `.env.example`, then
 yours) and the bind-mounted `data/` live in `~/Server/<Name>/` and are never
 inside the checkout — so nothing git can do, `clean -fdx` included, can reach
@@ -35,12 +61,18 @@ is what `panama server enable` refuses to start on.
 ## Conventions
 
 - Service directories are TitleCase, matching `~/Server` as it has always
-  looked: `server/containers/Gitea` becomes `~/Server/Gitea`.
+  looked: `server/vps/Gitea` becomes `~/Server/Gitea`.
 - Bind mounts live under `./data/` in new services. Imported services keep
   the volume path they were running with until their cutover, because the
   repo copy must render identically to the live one — renames happen with
   the service stopped, not in git.
-- Only 80, 443 and 81 are open (setup-server's doing). Everything else is
+- Only 80, 443 and 81 are open, and `setup-server` opens each one only where
+  nothing already grants it. A port reachable through an enabled service or an
+  accepting rich rule — a CDN-scoped grant, say — is left exactly as it is,
+  because turning a narrow grant into a public one during an unattended
+  `panama update` is not a decision an installer gets to make. A machine that
+  manages its own firewall opts out for good with
+  `echo manual > ~/.local/state/panama/firewall`. Everything else is
   reverse-proxied over the `nginx-bridge` network by container name, with no
   published ports of its own. A service that truly needs another host port
   says so in a README in its directory, and the port is opened by hand.
@@ -67,11 +99,46 @@ ones — and verify it came back up. One service at a time, quiet hours.
 
 ## Adding a service
 
-Add `server/containers/<Name>/` with the three files, modelled on any
+Add `server/vps/<Name>/` with the three files, modelled on any
 existing service. The unit is the standard oneshot `podman compose up -d`
 shape (`WorkingDirectory=%h/Server/<Name>`); the shape contract pins the
 details. Secrets go in `.env.example` as `KEY=CHANGE_ME` lines.
 
+A service that needs a scheduled job of its own carries it beside the
+compose unit: a helper `<job>.service` and a `<job>.timer` with the same
+name, plus the script it runs. Enable links all three and enables the timer.
+Disable stops the timer and unlinks the units. `UseSend` is the example: its
+`usesend-cert-refresh` timer keeps the SMTP proxy's certificate in step with
+Nginx Proxy Manager's.
+
+## Host tooling
+
+Each server's own scripts and runbooks sit as loose files at the top of its
+tree. They are reference copies for rebuilding a machine. Nothing installs
+them.
+
+- `vps/backup`, `vps/server-backup.service`, `vps/server-backup.timer`: the
+  VPS's nightly backup, run as a user unit. It dumps every database into
+  `~/Server/_backup_state/`, then rsyncs `~/Server` to the home server's ZFS
+  pool with `--fake-super`, so subuid ownership survives the trip.
+- `vps/verify-rootless`: the post-reboot health check for the rootless
+  stack. It covers linger, the unprivileged port floor, unit state and boot
+  enablement, postgres, endpoints, and admin ports bound to the WireGuard
+  address.
+- `vps/AGENTS.md`: the VPS runbook, with both public IPs redacted.
+- `home/RESTORE.md`: how to restore the home server's `~/Server` from the
+  ZFS backup.
+- `home/backup` and `home/systemd/`: the home server's backup trigger and its
+  root system units (`server-backup` nightly, `media-backup` weekly).
+- `home/server-backup` and `home/media-backup`: the scripts those units run.
+  On that machine they live in `/usr/local/sbin`, root-owned, because a
+  root-run script must not sit in a directory the user can write to. Restore
+  them there, not into `~/Server`.
+
+The VPS units carry `RequiresMountsFor=/home/gib/Media` because that host
+keeps its podman graphroot on an attached volume mounted there. The setting
+is harmless on a machine without the mount.
+
 ## Nightly updates
 
 `server/scripts/update-containers` pulls images and restarts changed

server/containers/Gitea/compose.yml

@@ -1,25 +0,0 @@
-networks:
-  nginx-bridge:
-    external: true
-services:
-  gitea:
-    image: docker.io/gitea/gitea:latest
-    container_name: gitea
-    hostname: gitea
-    domainname: git.gbrown.org
-    networks:
-    - nginx-bridge
-    ports:
-    - '2222:22'
-    environment:
-    - TZ=America/New_York
-    - USER_UID=1001
-    - USER_GID=1003
-    labels:
-      com.centurylinklabs.watchtower.enable: 'true'
-    volumes:
-    - ./volume:/data:Z
-    - /home/gib/Media/gitea-packages:/data/gitea/packages:Z
-    - ./ssh:/data/git/.ssh:Z
-    tty: true
-    restart: unless-stopped

server/containers/Penpot/.env.example

@@ -1,2 +0,0 @@
-PENPOT_SECRET_KEY=CHANGE_ME
-PENPOT_DATABASE_PASSWORD=CHANGE_ME

server/containers/Penpot/compose.yml

@@ -1,120 +0,0 @@
-# Penpot — VPS (ROOTLESS PODMAN). Ported from the home server 2026-08-12.
-#
-# ⚠️ NOT STARTED. No systemd unit is enabled. It has never run on either machine, so
-# there is no data to migrate. Bring it up with:
-#     systemctl --user enable --now podman-penpot.service
-#
-# BEFORE FIRST START, create its role and database on the SHARED postgres (this stack
-# no longer ships its own):
-#     podman exec -i postgresql psql -U npm <<'SQL'
-#     CREATE ROLE penpot LOGIN PASSWORD '<see this service .env on the machine>';
-#     CREATE DATABASE penpot OWNER penpot;
-#     SQL
-# Then add the same lines to PostgreSQL/initdb/00-roles-and-databases.sql.
-#
-# NOTE: the home version used postgres:15-alpine with --data-checksums. The shared
-# instance is postgres 17 without checksums. Penpot does not care, and since there is
-# no existing data this is not a migration — but if you ever DO need checksums, they
-# can only be set at initdb time for the whole cluster.
-#
-# Redis stays local to this stack: Penpot uses it for transient session and
-# notification state only, so it is not worth centralising.
-#
-# Rootless adaptations from the home version:
-#   - Dropped its private postgres:15-alpine; uses the shared instance (AGENTS.md §2).
-#   - PENPOT_DATABASE_URI repointed from penpot-db to the shared postgresql.
-#   - :Z on the assets bind mount. It is shared between frontend and backend, so it
-#     uses LOWERCASE :z -- uppercase would give each container a private MCS category
-#     and the second to start would lock the first out (AGENTS.md §6).
-#   - /etc/localtime mounts removed in favour of TZ (AGENTS.md §8).
-#
-# Registration is disabled by default (PENPOT_FLAGS disable-registration), so create
-# the first account from the backend container before you can log in.
-#
-# NPM: proxy penpot.gbrown.org -> http://penpot-frontend:8080
-
-networks:
-  nginx-bridge:
-    external: true
-
-services:
-  penpot-frontend:
-    image: docker.io/penpotapp/frontend:latest
-    container_name: penpot-frontend
-    hostname: penpot-frontend
-    domainname: penpot.gbrown.org
-    networks: ['nginx-bridge']
-    environment:
-      - TZ=America/New_York
-      - PENPOT_FLAGS=disable-registration enable-login-with-password disable-smtp
-    labels:
-      com.centurylinklabs.watchtower.enable: "true"
-    volumes:
-      - ./volumes/assets:/opt/data/assets:z
-    depends_on:
-      - penpot-backend
-      - penpot-exporter
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-
-  penpot-backend:
-    image: docker.io/penpotapp/backend:latest
-    container_name: penpot-backend
-    hostname: penpot-backend
-    domainname: penpot.gbrown.org
-    networks: ['nginx-bridge']
-    environment:
-      - TZ=America/New_York
-      - PENPOT_FLAGS=disable-registration enable-login-with-password disable-smtp enable-prepl-server
-      - PENPOT_PUBLIC_URI=https://penpot.gbrown.org
-      - PENPOT_SECRET_KEY=${PENPOT_SECRET_KEY}
-      # Shared postgres, not a private container
-      - PENPOT_DATABASE_URI=postgresql://postgresql/penpot
-      - PENPOT_DATABASE_USERNAME=penpot
-      - PENPOT_DATABASE_PASSWORD=${PENPOT_DATABASE_PASSWORD}
-      - PENPOT_REDIS_URI=redis://penpot-redis/0
-      - PENPOT_ASSETS_STORAGE_BACKEND=assets-fs
-      - PENPOT_STORAGE_ASSETS_FS_DIRECTORY=/opt/data/assets
-      - PENPOT_TELEMETRY_ENABLED=false
-    labels:
-      com.centurylinklabs.watchtower.enable: "true"
-    volumes:
-      - ./volumes/assets:/opt/data/assets:z
-    depends_on:
-      - penpot-redis
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-
-  penpot-exporter:
-    image: docker.io/penpotapp/exporter:latest
-    container_name: penpot-exporter
-    hostname: penpot-exporter
-    domainname: penpot.gbrown.org
-    networks: ['nginx-bridge']
-    environment:
-      - TZ=America/New_York
-      - PENPOT_PUBLIC_URI=http://penpot-frontend:8080
-      - PENPOT_REDIS_URI=redis://penpot-redis/0
-    labels:
-      com.centurylinklabs.watchtower.enable: "true"
-    depends_on:
-      - penpot-redis
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-
-  penpot-redis:
-    image: docker.io/library/redis:7-alpine
-    container_name: penpot-redis
-    hostname: penpot-redis
-    domainname: penpot.gbrown.org
-    networks: ['nginx-bridge']
-    environment:
-      - TZ=America/New_York
-    labels:
-      com.centurylinklabs.watchtower.enable: "true"
-    tty: true
-    stdin_open: true
-    restart: unless-stopped

server/containers/Rustdesk/.env.example

@@ -1 +0,0 @@
-RUSTDESK_API_RUSTDESK_JWT_KEY=CHANGE_ME

server/containers/UseSend/compose.yml

@@ -1,88 +0,0 @@
-name: usesend
-networks:
-  nginx-bridge:
-    external: true
-services:
-  redis:
-    image: redis:7
-    container_name: usesend-redis
-    hostname: usesend-redis
-    networks:
-    - nginx-bridge
-    env_file: .env
-    labels:
-      com.centurylinklabs.watchtower.enable: 'true'
-    volumes:
-    - ./volumes/redis:/data:Z
-    command:
-    - redis-server
-    - --maxmemory-policy
-    - noeviction
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-  minio:
-    image: minio/minio
-    container_name: usesend-storage
-    hostname: usesend-storage
-    networks:
-    - nginx-bridge
-    env_file: .env
-    environment:
-      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
-      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
-    labels:
-      com.centurylinklabs.watchtower.enable: 'true'
-    volumes:
-    - ./volumes/minio:/data:Z
-    entrypoint: sh
-    command: -c 'mkdir -p /data/unsend && minio server /data --console-address ':9001' --address ':9002''
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-  smtp-server:
-    image: usesend/smtp-proxy:v1.6.6
-    container_name: usesend-smtp-server
-    hostname: usesend-smtp
-    networks:
-    - nginx-bridge
-    env_file: .env
-    environment:
-    - SMTP_AUTH_USERNAME=${SMTP_USER:?err}
-    - USESEND_BASE_URL=${USESEND_URL:?err}
-    labels:
-      com.centurylinklabs.watchtower.enable: 'true'
-    tty: true
-    stdin_open: true
-    restart: unless-stopped
-  usesend:
-    image: usesend/usesend:v1.6.6
-    container_name: usesend
-    hostname: usesend
-    domainname: ${USESEND_URL:?err}
-    networks:
-    - nginx-bridge
-    env_file: .env
-    environment:
-    - PORT=${USESEND_PORT:-3000}
-    - DATABASE_URL=${DATABASE_URL:?err}
-    - NEXTAUTH_URL=${USESEND_URL:?err}
-    - NEXTAUTH_SECRET=${NEXTAUTH_SECRET:?err}
-    - AWS_ACCESS_KEY=${AWS_ACCESS_KEY:?err}
-    - AWS_SECRET_KEY=${AWS_SECRET_KEY:?err}
-    - AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:?err}
-    - GITHUB_ID=${GITHUB_ID:?err}
-    - GITHUB_SECRET=${GITHUB_SECRET:?err}
-    - REDIS_URL=${REDIS_URL:?err}
-    - NEXT_PUBLIC_IS_CLOUD=${NEXT_PUBLIC_IS_CLOUD:-false}
-    - API_RATE_LIMIT=${API_RATE_LIMIT:-1}
-    - SMTP_HOST=${SMTP_HOST:-smtp.usesend.gbrown.org}
-    - SMTP_USER=${SMTP_USER:-usesend}
-    labels:
-      com.centurylinklabs.watchtower.enable: 'true'
-    depends_on:
-      redis:
-        condition: service_started
-    tty: true
-    stdin_open: true
-    restart: unless-stopped

server/containers/Vaultwarden/.env.example

@@ -1 +0,0 @@
-ADMIN_TOKEN=CHANGE_ME

server/containers/Vaultwarden/compose.yml

@@ -1,55 +0,0 @@
-# Vaultwarden — VPS (ROOTLESS PODMAN) port of the home server's ~/Server/vaultwarden,
-# staged 2026-08-12.
-#
-# ⚠️ STAGED, NOT AUTHORITATIVE. The home instance is still live and still serving
-# vault.gbrown.org. This copy holds a point-in-time snapshot of the SQLite database.
-# DO NOT log into this one and add credentials before cutover -- the two would diverge
-# and one side's changes would be lost.
-#
-# Cutover procedure (do it in this order, it matters):
-#   1. stop vaultwarden on the home server        (clean SQLite, no torn WAL)
-#   2. rsync -a server.gib:~/Server/vaultwarden/volume/ ~/Server/Vaultwarden/volume/
-#   3. systemctl --user restart podman-vaultwarden
-#   4. point vault.gbrown.org at the VPS, add the NPM proxy host
-#   5. verify a login + an item decrypts, THEN leave home stopped
-#
-# The SQLite copy taken while home was running is crash-consistent only. Step 1 is what
-# makes it clean -- don't skip it.
-#
-# Rootless adaptations:
-#   - :Z on the data mount (SELinux Enforcing).
-#   - /etc/localtime mount dropped in favour of TZ (AGENTS.md §8).
-#   - No published port; NPM proxies to http://vaultwarden:80 over nginx-bridge.
-#   - Image starts as root and drops privileges internally, so no userns_mode needed.
-#
-# NOTE: ADMIN_TOKEN is inline here, matching the home server's file verbatim. It grants
-# access to /admin. Worth moving to a mode-600 .env at some point -- compose files get
-# rsynced offsite by ~/Server/backup.
-
-networks:
-  nginx-bridge:
-    external: true
-
-services:
-  vaultwarden:
-    image: docker.io/vaultwarden/server:latest
-    container_name: vaultwarden
-    hostname: vaultwarden
-    domainname: vault.gbrown.org
-    networks:
-      - nginx-bridge
-    environment:
-      - TZ=America/New_York
-      - DOMAIN=https://vault.gbrown.org
-      - LOGIN_RATELIMIT_MAX_BURST=10
-      - LOGIN_RATELIMIT_SECONDS=60
-      - WEB_VAULT_ENABLED=true
-      - SIGNUPS_ALLOWED=false
-      - WEBSOCKET_ENABLED=true
-      - ADMIN_TOKEN=${ADMIN_TOKEN}
-    labels:
-      com.centurylinklabs.watchtower.enable: "true"
-    volumes:
-      - ./volume:/data:Z
-    tty: true
-    restart: unless-stopped

server/home/Agents/Beszel/compose.yml

@@ -0,0 +1,48 @@
+# Beszel AGENT ONLY — the hub now lives on the VPS.
+#
+# The hub (web UI + database) was moved to the VPS on 2026-08-11 and is reachable at
+# https://beszel.gibbyb.com. This host keeps only the agent, which collects metrics and
+# serves them to that hub over the WireGuard link (VPS 192.168.2.2 -> here 192.168.1.2:45876).
+#
+# KEY below is the VPS hub's public key. If the VPS hub is ever rebuilt its keypair
+# changes, and this value must be updated to match:
+#     ssh-keygen -y -f ~/Server/Beszel/volumes/hub/id_ed25519      (run on the VPS)
+#
+# Previous config, including the local hub service, is in compose.yml.bak-2026-08-11.
+#
+# The agent uses network_mode: host — that is the documented setup and is what lets it
+# read real host network interfaces rather than a container's veth. Docker forbids setting
+# hostname/domainname together with host networking, so those keys are absent.
+
+services:
+  beszel-agent:
+    image: henrygd/beszel-agent-nvidia:latest
+    container_name: beszel-agent
+    network_mode: host
+    # Give the agent read access to NVIDIA's management interface so Beszel can
+    # report GPU utilization, temperature, power draw, and VRAM usage.
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [utility]
+    environment:
+      - TZ=America/New_York
+      - PORT=45876
+      - KEY=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDBWNqxKGRltqXJZg+wlDOtICeLo19ftQ6P3P8+uJg8y
+      # /boot is a separate 974M partition (nvme1n1p1), not part of the LUKS
+      # root the agent finds on its own. It filling up is what broke a 1032
+      # package dnf transaction on 2026-08-09, so it is worth alerting on.
+      - EXTRA_FILESYSTEMS=/boot
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/agent:/var/lib/beszel-agent
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /boot:/boot:ro
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped

server/home/Agents/Beszel/podman-agents-beszel.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Agents/Beszel
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Agents/Beszel
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Agents/Gitea/.env.example

@@ -0,0 +1 @@
+GITEA_RUNNER_REGISTRATION_TOKEN=CHANGE_ME

server/home/Agents/Gitea/compose.yml

@@ -0,0 +1,153 @@
+# Gitea Actions runners — 3 parallel runners, each with its own dind sidecar.
+#
+# Each runner gets an isolated Docker daemon rather than the host socket, so a
+# CI job cannot reach the other ~180 containers on this box. Cost: each dind
+# keeps its own image cache under ./volumes/runner-N/dind.
+#
+# The pre-existing host runner (/home/gib/.local/bin/act_runner, name
+# "server-gib") is intentionally left running.
+#
+# Registration token generated with:
+#   docker exec -u git gitea gitea actions generate-runner-token
+# It is reusable — all three runners register with the same one.
+
+networks:
+  nginx-bridge:
+    external: true
+
+x-runner-env: &runner-env
+  TZ: America/New_York
+  GITEA_INSTANCE_URL: https://git.gbrown.org
+  GITEA_RUNNER_REGISTRATION_TOKEN: ${GITEA_RUNNER_REGISTRATION_TOKEN}
+  GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04,build:docker://docker.gitea.com/runner-images:ubuntu-latest"
+  DOCKER_TLS_VERIFY: "1"
+  DOCKER_CERT_PATH: /certs/client
+  CONFIG_FILE: /data/config.yml
+
+x-dind-env: &dind-env
+  TZ: America/New_York
+  DOCKER_TLS_CERTDIR: /certs
+
+services:
+  gitea-runner-1:
+    image: gitea/act_runner:latest
+    container_name: gitea-runner-1
+    hostname: gitea-runner-1
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    environment:
+      <<: *runner-env
+      GITEA_RUNNER_NAME: gitea-runner-1
+      DOCKER_HOST: tcp://gitea-runner-1-dind:2376
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-1/data:/data
+      - ./volumes/runner-1/certs:/certs
+      - /etc/localtime:/etc/localtime:ro
+    depends_on:
+      - gitea-runner-1-dind
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gitea-runner-1-dind:
+    image: docker:dind
+    container_name: gitea-runner-1-dind
+    hostname: gitea-runner-1-dind
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    privileged: true
+    environment:
+      <<: *dind-env
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-1/certs:/certs
+      - ./volumes/runner-1/dind:/var/lib/docker
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gitea-runner-2:
+    image: gitea/act_runner:latest
+    container_name: gitea-runner-2
+    hostname: gitea-runner-2
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    environment:
+      <<: *runner-env
+      GITEA_RUNNER_NAME: gitea-runner-2
+      DOCKER_HOST: tcp://gitea-runner-2-dind:2376
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-2/data:/data
+      - ./volumes/runner-2/certs:/certs
+      - /etc/localtime:/etc/localtime:ro
+    depends_on:
+      - gitea-runner-2-dind
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gitea-runner-2-dind:
+    image: docker:dind
+    container_name: gitea-runner-2-dind
+    hostname: gitea-runner-2-dind
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    privileged: true
+    environment:
+      <<: *dind-env
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-2/certs:/certs
+      - ./volumes/runner-2/dind:/var/lib/docker
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gitea-runner-3:
+    image: gitea/act_runner:latest
+    container_name: gitea-runner-3
+    hostname: gitea-runner-3
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    environment:
+      <<: *runner-env
+      GITEA_RUNNER_NAME: gitea-runner-3
+      DOCKER_HOST: tcp://gitea-runner-3-dind:2376
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-3/data:/data
+      - ./volumes/runner-3/certs:/certs
+      - /etc/localtime:/etc/localtime:ro
+    depends_on:
+      - gitea-runner-3-dind
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gitea-runner-3-dind:
+    image: docker:dind
+    container_name: gitea-runner-3-dind
+    hostname: gitea-runner-3-dind
+    domainname: git.gbrown.org
+    networks: ['nginx-bridge']
+    privileged: true
+    environment:
+      <<: *dind-env
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/runner-3/certs:/certs
+      - ./volumes/runner-3/dind:/var/lib/docker
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped

server/home/Agents/Gitea/podman-agents-gitea.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Agents/Gitea
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Agents/Gitea
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Agents/Portainer/compose.yml

@@ -0,0 +1,39 @@
+services:
+  # portainer:
+  #   image: portainer/portainer-ee:latest
+  #   container_name: portainer
+  #   hostname: portainer
+  #   domainname: port.gbrown.org
+  #   networks:
+  #     - nginx-bridge
+  #   labels:
+  #     com.centurylinklabs.watchtower.enable: "true"
+  #   ports:
+  #     - 9443:9443
+  #   volumes:
+  #     - ./volumes/data:/data
+  #     - /var/run/docker.sock:/var/run/docker.sock
+  #   tty: true
+  #   stdin_open: true
+  #   restart: unless-stopped
+
+  portainer-agent:
+    image: portainer/agent:latest
+    container_name: portainer-agent
+    hostname: portainer-agent
+    networks:
+      - nginx-bridge
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    ports: ["9001:9001"]
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - /var/lib/docker/volumes:/var/lib/docker/volumes
+      - /:/host
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+networks:
+  nginx-bridge:
+    external: true

server/home/Agents/Portainer/podman-agents-portainer.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Agents/Portainer
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Agents/Portainer
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Frigate/.env.example

@@ -0,0 +1,2 @@
+RTSP_PASSWORD=CHANGE_ME
+PLUS_API_KEY=CHANGE_ME

server/home/Frigate/compose.yml

@@ -0,0 +1,79 @@
+networks:
+  nginx-bridge:
+    external: true
+services:
+  frigate:
+    image: ghcr.io/blakeblackshear/frigate:0.17.1-tensorrt # For Nvidia GPU
+    container_name: frigate
+    hostname: frigate
+    domainname: frigate.gbrown.org
+    networks: [nginx-bridge]
+    ports:
+      #- "8971:8971"
+      - "5000:5000"
+      #- "8554:8554"
+      #- "8555:8555"
+    # Check host system's actual render/video/plugdev group IDs with 'getent group render', 'getent group video', and 'getent group plugdev'
+    # Must add these exact IDs in container's group_add section or OpenVINO GPU acceleration will fail
+    group_add:
+      - "105" # render
+      - "110" # render
+      - "39"  # video
+      - "46"  # plugdev
+    shm_size: "256mb"
+    env_file:
+      - .env
+    environment:
+      - TZ=America/New_York
+      - PGID=1000
+      - PUID=1000
+      - FRIGATE_RTSP_PASSWORD=${RTSP_PASSWORD}
+      #- HSA_OVERRIDE_GFX_VERSION=11.0.0
+      - PLUS_API_KEY=${PLUS_API_KEY}
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/frigate/config:/config:rw
+      - ./volumes/frigate/data:/usr/local/nginx/conf:rw
+      - /home/gib/Public/Server_Backup/frigate:/media/frigate
+      - type: tmpfs
+        target: /tmp/cache
+        tmpfs:
+          size: 1000000000
+      - /etc/localtime:/etc/localtime:ro
+      # - /dev/bus/usb:/dev/bus/usb # Uncomment for Google Coral USB
+    devices:
+      - /dev/dri:/dev/dri # for intel hwaccel, needs to be updated for your hardware
+      #- /dev/kfd:/dev/kfd
+    stop_grace_period: 30s
+    restart: unless-stopped
+    tty: true
+    security_opt: ['seccomp:unconfined','label:disable','systempaths=unconfined']
+    #privileged: true
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: 1
+              capabilities: [gpu]
+
+  mqtt:
+    image: eclipse-mosquitto:2
+    container_name: mqtt
+    hostname: mqtt
+    domainname: mqtt.gbrown.org
+    networks: [nginx-bridge]
+    ports: ["1883:1883"]
+    #env_file: [.env]
+    environment: [TZ=America/New_York]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/mosquitto/config:/mosquitto/config:rw
+      - ./volumes/mosquitto/data:/mosquitto/data:rw
+      - ./volumes/mosquitto/log:/mosquitto/log:rw
+    tty: true
+    security_opt: ['seccomp:unconfined','label:disable','systempaths=unconfined']
+    restart: unless-stopped

server/home/Frigate/podman-frigate.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Frigate
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Frigate
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Gbrown/.env.example

@@ -0,0 +1,16 @@
+AGENT_INGEST_SECRET=CHANGE_ME
+INSTANCE_NAME=CHANGE_ME
+INSTANCE_SECRET=CHANGE_ME
+NEXT_PUBLIC_CONVEX_URL=CHANGE_ME
+NEXT_PUBLIC_PLAUSIBLE_SRC=CHANGE_ME
+NEXT_PUBLIC_POSTHOG_HOST=CHANGE_ME
+NEXT_PUBLIC_POSTHOG_KEY=CHANGE_ME
+NEXT_PUBLIC_SENTRY_DSN=CHANGE_ME
+NEXT_PUBLIC_SENTRY_ORG=CHANGE_ME
+NEXT_PUBLIC_SENTRY_PROJECT_NAME=CHANGE_ME
+NEXT_PUBLIC_SENTRY_URL=CHANGE_ME
+NEXT_PUBLIC_SITE_URL=CHANGE_ME
+SENTRY_AUTH_TOKEN=CHANGE_ME
+UNIFI_API_KEY=CHANGE_ME
+UNIFI_CA=CHANGE_ME
+UNIFI_URL=CHANGE_ME

server/home/Gbrown/compose.yml

@@ -0,0 +1,219 @@
+# Production stack on the Home Server (server.gib, rootful Docker), deployed
+# as ~/Server/Gbrown/compose.yml. This file is the versioned source: edit it
+# here and copy it over, never the other way round.
+#
+# ~/Server/Gbrown holds only this file, .env and data/. Every bind mount lives
+# under data/ (data/convex, data/gibtv/channel, data/gibtv/nginx.conf, copied
+# from apps/gibtv/nginx.conf). No source or build context lives on the server:
+# app images come from the Gitea registry. CI redeploys at once through the
+# forced-command key behind ~/Server/ci-deploy.sh (`docker compose pull && up -d`).
+#
+# Secrets come from ~/Server/Gbrown/.env (mode 600), written from Infisical
+# `prod` by scripts/write-prod-env. Never commit or print it.
+#
+# NPM reaches everything over nginx-bridge, so nothing publishes a host port:
+#   api.gbrown.org              -> http://gbrown-backend:3210
+#   convex.gbrown.org           -> http://gbrown-backend:3211
+#   dashboard.gbrown.org        -> http://gbrown-dashboard:6791
+#   gbrown.org (host 119)       -> http://gbrown-next:3000   (at cutover)
+#   tv.gbrown.org               -> http://gbrown-gibtv-web:80 (phase 8)
+#
+# Watchtower (label-enabled, 04:00) updates the app images. The Convex backend,
+# its dashboard and the socket proxy are pinned and opt out: a backend upgrade
+# migrates its data, so it happens on purpose, not overnight.
+#
+# The `tools` profile keeps the one-off normalize pass out of a plain `up -d`
+# and `pull`. Run it with `docker compose run --rm gbrown-gibtv-normalize`.
+name: gbrown
+
+networks:
+  nginx-bridge:
+    external: true
+  # The agent and its socket proxy only. No route out.
+  agent-socket:
+    internal: true
+
+volumes:
+  # The live HLS window is about 10 MB and rewritten every 6 s. Keep it in RAM.
+  gibtv-hls:
+    driver_opts:
+      type: tmpfs
+      device: tmpfs
+      o: size=64m,uid=1000,gid=1000
+
+services:
+  gbrown-backend:
+    # Pinned to the build the VPS Convex stacks run (bumped 2026-10-28). Bump on purpose.
+    image: ghcr.io/convex/convex-backend@sha256:207cf95def42f01614b6e3af0af5c20d0dcaa5ff2adf258ee7bc9555576644fa
+    container_name: gbrown-backend
+    hostname: gbrown-backend
+    networks: ['nginx-bridge']
+    volumes: ['./data/convex:/convex/data']
+    # No TZ: the backend panics unless it runs in UTC.
+    environment:
+      - INSTANCE_NAME=${INSTANCE_NAME:?}
+      - INSTANCE_SECRET=${INSTANCE_SECRET:?}
+      - CONVEX_CLOUD_ORIGIN=https://api.gbrown.org
+      - CONVEX_SITE_ORIGIN=https://convex.gbrown.org
+      - DISABLE_BEACON=true
+      - REDACT_LOGS_TO_CLIENT=true
+    labels: ['com.centurylinklabs.watchtower.enable=false']
+    restart: unless-stopped
+    healthcheck:
+      test: ['CMD', 'curl', '-f', 'http://localhost:3210/version']
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 20s
+    stop_grace_period: 10s
+    stop_signal: SIGINT
+
+  gbrown-dashboard:
+    # Released alongside the backend; pinned with it.
+    image: ghcr.io/convex/convex-dashboard@sha256:a55bf8abca1332f787bc4d4a79850182175027be7b70aa230be4389a1d3268e2
+    container_name: gbrown-dashboard
+    hostname: gbrown-dashboard
+    networks: ['nginx-bridge']
+    environment:
+      # The browser talks to the backend directly, so this is the public URL.
+      - NEXT_PUBLIC_DEPLOYMENT_URL=https://api.gbrown.org
+    depends_on:
+      gbrown-backend:
+        condition: service_healthy
+    labels: ['com.centurylinklabs.watchtower.enable=false']
+    restart: unless-stopped
+    stop_grace_period: 10s
+    stop_signal: SIGINT
+
+  gbrown-next:
+    image: git.gbrown.org/gib/gbrown-next:latest
+    container_name: gbrown-next
+    hostname: gbrown-next
+    networks: ['nginx-bridge']
+    environment:
+      - TZ=America/New_York
+      - NODE_ENV=production
+      - SENTRY_AUTH_TOKEN=${SENTRY_AUTH_TOKEN}
+      - NEXT_PUBLIC_SITE_URL=${NEXT_PUBLIC_SITE_URL}
+      - NEXT_PUBLIC_CONVEX_URL=${NEXT_PUBLIC_CONVEX_URL}
+      - NEXT_PUBLIC_PLAUSIBLE_SRC=${NEXT_PUBLIC_PLAUSIBLE_SRC:-}
+      - NEXT_PUBLIC_POSTHOG_KEY=${NEXT_PUBLIC_POSTHOG_KEY:-}
+      - NEXT_PUBLIC_POSTHOG_HOST=${NEXT_PUBLIC_POSTHOG_HOST:-}
+      - NEXT_PUBLIC_SENTRY_DSN=${NEXT_PUBLIC_SENTRY_DSN}
+      - NEXT_PUBLIC_SENTRY_URL=${NEXT_PUBLIC_SENTRY_URL}
+      - NEXT_PUBLIC_SENTRY_ORG=${NEXT_PUBLIC_SENTRY_ORG}
+      - NEXT_PUBLIC_SENTRY_PROJECT_NAME=${NEXT_PUBLIC_SENTRY_PROJECT_NAME}
+    labels: ['com.centurylinklabs.watchtower.enable=true']
+    restart: unless-stopped
+    healthcheck:
+      test: ['CMD', 'wget', '-q', '-O', '/dev/null', 'http://127.0.0.1:3000/']
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 20s
+
+  # The agent never sees the Docker socket. The proxy answers only
+  # GET /containers/json, /events, /_ping and /version (with or without an API
+  # version prefix), refuses everything else, and only accepts connections from
+  # the gbrown-agent hostname on the internal agent-socket network.
+  # Docker here applies no SELinux labels to containers, so mounts need no :z.
+  gbrown-agent-socket-proxy:
+    image: docker.io/wollomatic/socket-proxy:1.13.1
+    container_name: gbrown-agent-socket-proxy
+    hostname: gbrown-agent-socket-proxy
+    networks: ['agent-socket']
+    command:
+      - '-loglevel=info'
+      - '-listenip=0.0.0.0'
+      - '-allowfrom=gbrown-agent'
+      - '-allowGET=/(v1\.[0-9]+/)?(containers/json|events|_ping|version)'
+      - '-allowhealthcheck'
+      # Exit if the socket breaks (a Docker upgrade) so Docker restarts it.
+      - '-watchdoginterval=3600'
+      - '-stoponwatchdog'
+      - '-shutdowngracetime=5'
+    # nobody, plus gid 963: the host's docker group, which owns the socket.
+    user: '65534:963'
+    volumes: ['/var/run/docker.sock:/var/run/docker.sock:ro']
+    read_only: true
+    mem_limit: 64M
+    cap_drop: [ALL]
+    security_opt: ['no-new-privileges:true']
+    healthcheck:
+      test: ['CMD', './healthcheck']
+      interval: 30s
+      timeout: 5s
+      retries: 2
+    # Pinned on purpose. Watchtower must not swap the gatekeeper on its own.
+    labels: ['com.centurylinklabs.watchtower.enable=false']
+    restart: unless-stopped
+
+  gbrown-agent:
+    image: git.gbrown.org/gib/gbrown-agent:latest
+    container_name: gbrown-agent
+    hostname: gbrown-agent
+    # nginx-bridge to reach gbrown-backend, agent-socket to reach the proxy.
+    networks: ['nginx-bridge', 'agent-socket']
+    environment:
+      - TZ=America/New_York
+      - HOST_ID=home
+      - DOCKER_HOST=tcp://gbrown-agent-socket-proxy:2375
+      # Straight to the backend's HTTP actions port, no proxy in between.
+      - INGEST_URL=http://gbrown-backend:3211/ingest
+      - AGENT_INGEST_SECRET=${AGENT_INGEST_SECRET:?}
+      # The UniFi gateway on the LAN. UNIFI_CA is its own certificate, the only
+      # one the agent trusts there. Leave all three unset to skip the network.
+      - UNIFI_URL=${UNIFI_URL:-}
+      - UNIFI_API_KEY=${UNIFI_API_KEY:-}
+      - UNIFI_CA=${UNIFI_CA:-}
+    volumes: ['/home/gib/Server/Nginx/volumes/data/logs:/npm-logs:ro']
+    depends_on:
+      gbrown-agent-socket-proxy:
+        condition: service_healthy
+    read_only: true
+    cap_drop: [ALL]
+    security_opt: ['no-new-privileges:true']
+    labels: ['com.centurylinklabs.watchtower.enable=true']
+    restart: unless-stopped
+    stop_grace_period: 10s
+
+  # gib.tv (apps/gibtv). Nothing here touches the media library except the
+  # normalize pass, which mounts it read-only. The streamer stream-copies, so
+  # CPU is near zero; each viewer pulls about 1 Mbps of home upload.
+  gbrown-gibtv:
+    image: git.gbrown.org/gib/gbrown-gibtv:latest
+    container_name: gbrown-gibtv
+    hostname: gbrown-gibtv
+    # No network needed: it only writes the HLS window to the shared volume.
+    network_mode: none
+    environment: ['TZ=America/New_York']
+    volumes:
+      - ./data/gibtv/channel:/media/channel:ro
+      - gibtv-hls:/srv/hls
+    cpus: 0.5
+    mem_limit: 256m
+    labels: ['com.centurylinklabs.watchtower.enable=true']
+    restart: unless-stopped
+
+  gbrown-gibtv-web:
+    image: docker.io/library/nginx:alpine
+    container_name: gbrown-gibtv-web
+    hostname: gbrown-gibtv-web
+    networks: ['nginx-bridge']
+    depends_on: [gbrown-gibtv]
+    volumes:
+      - gibtv-hls:/usr/share/nginx/html:ro
+      - ./data/gibtv/nginx.conf:/etc/nginx/conf.d/default.conf:ro
+    labels: ['com.centurylinklabs.watchtower.enable=true']
+    restart: unless-stopped
+
+  # One-off: `docker compose run --rm gbrown-gibtv-normalize`. Writes one
+  # 640x480 h264 file per episode, skips finished files, safe to resume.
+  gbrown-gibtv-normalize:
+    image: git.gbrown.org/gib/gbrown-gibtv:latest
+    profiles: ['tools']
+    command: ['nice', '-n', '19', 'bun', 'src/cli/normalize.ts']
+    network_mode: none
+    volumes:
+      - '/home/gib/Public/TV/Dragnet (1951):/media/source:ro'
+      - ./data/gibtv/channel:/media/channel

server/home/Gbrown/podman-gbrown.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Gbrown
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Gbrown
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Home/compose.yml

@@ -0,0 +1,24 @@
+services:
+  homeassistant:
+    image: homeassistant/home-assistant:latest
+    container_name: home
+    hostname: home
+    domainname: home.gbrown.org
+    network_mode: host
+    cap_add: ['NET_ADMIN','NET_RAW']
+    #networks:
+      #- nginx-bridge
+    #ports:
+      #- "8123:8123"
+    volumes:
+      - ./volume:/config
+      - /run/dbus:/run/dbus:ro
+      - /etc/localtime:/etc/localtime:ro
+    labels:
+      com.centurylinklabs.watchtower.enable: 'true'
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+#networks:
+  #nginx-bridge:
+    #external: true

server/home/Home/podman-home.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Home
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Home
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Immich/.env.example

@@ -0,0 +1,7 @@
+UPLOAD_LOCATION=CHANGE_ME
+DB_DATA_LOCATION=CHANGE_ME
+IMMICH_VERSION=CHANGE_ME
+DB_PASSWORD=CHANGE_ME
+DB_USERNAME=CHANGE_ME
+DB_DATABASE_NAME=CHANGE_ME
+TZ=CHANGE_ME

server/home/Immich/compose.yml

@@ -0,0 +1,86 @@
+name: immich
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  immich-server:
+    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
+    extends:
+      file: hwaccel.transcoding.yml
+      service: nvenc
+    container_name: immich-server
+    hostname: immich-server
+    domainname: immich.gbrown.org
+    networks: ['nginx-bridge']
+    #ports: ['2283:2283']
+    env_file: [.env]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ${UPLOAD_LOCATION}:/data
+      - /etc/localtime:/etc/localtime:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    depends_on:
+      - redis
+      - database
+    healthcheck:
+      disable: false
+    privileged: true
+
+  immich-machine-learning:
+    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
+    extends:
+      file: hwaccel.transcoding.yml
+      service: nvenc
+    container_name: immich-machine-learning
+    hostname: immich-machine-learning
+    networks: ['nginx-bridge']
+    #ports: ['3003:3003']
+    env_file: [.env]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes: ['./volumes/model-cache:/cache:z']
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    healthcheck:
+      disable: false
+    privileged: true
+
+  redis:
+    image: docker.io/valkey/valkey:9@sha256:fb8d272e529ea567b9bf1302245796f21a2672b8368ca3fcb938ac334e613c8f
+    container_name: immich-redis
+    hostname: immich-redis
+    networks: ['nginx-bridge']
+    env_file: [.env]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    healthcheck:
+      test: redis-cli ping || exit 1
+
+  database:
+    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
+    container_name: immich-postgres
+    hostname: immich-postgres
+    networks: ['nginx-bridge']
+    #ports: ['5432:5432']
+    env_file: [.env]
+    environment:
+      POSTGRES_PASSWORD: ${DB_PASSWORD}
+      POSTGRES_USER: ${DB_USERNAME}
+      POSTGRES_DB: ${DB_DATABASE_NAME}
+      POSTGRES_INITDB_ARGS: '--data-checksums'
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    shm_size: 128mb

server/home/Immich/podman-immich.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Immich
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Immich
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Media/.env.example

@@ -0,0 +1,9 @@
+PLEX_CLAIM=CHANGE_ME
+PLEX_TOKEN=CHANGE_ME
+JELLYFIN_TOKEN=CHANGE_ME
+OPENVPN_PASSWORD=CHANGE_ME
+QBIT_PASS=CHANGE_ME
+SONARR_API_KEY=CHANGE_ME
+RADARR_API_KEY=CHANGE_ME
+UN_SONARR_0_API_KEY=CHANGE_ME
+UN_RADARR_0_API_KEY=CHANGE_ME

server/home/Media/compact-tv/README.md

@@ -0,0 +1,109 @@
+# compact-tv
+
+Sonarr and Radarr each have exactly four quality profiles. Two of them (**Standard** and
+**Mini**) also have their files shrunk by `compact-tv.py`, which runs every night.
+
+| Profile | Used for | What gets kept |
+|---|---|---|
+| **Mini** | shows I demote hard | TV: 720p at 800 kbps (~0.42 GB per hour). Movies: 720p at 1500 kbps |
+| **Standard** | **every request from other users (Seerr default)**, and anything I demote | TV: 1080p at 1200 kbps (~0.6 GB per hour). Movies: 1080p at 2500 kbps (~1.2 GB per hour) |
+| **HD** | my own library, unchanged | whatever downloads, up to 1080p |
+| **UHD** | only me, through Seerr's 4K servers | 4K |
+
+SD sources keep their resolution, at 600 kbps for TV and 900 kbps for movies. Movies get
+higher rates because dark scenes, grain and action break down first at low bitrates. Nothing is ever
+upscaled. Audio becomes Opus: 96 kbps for stereo, 160 for 5.1, 192 for 7.1. Subtitles,
+chapters and attachments are copied as they are. I picked the rates from sample encodes of
+Shameless S05E06 (VMAF ~91 for Standard, ~86 for Mini).
+
+## How the profiles work
+
+- **Standard and Mini** are copies of HD that upgrade only toward small releases (see below),
+  so Sonarr and Radarr never replace a file with a bigger one.
+- **Small downloads first.** Both profiles prefer small releases when they grab:
+  - The "Small Encode" format (MeGusta, PSA, ELiTE, GalaxyTV/RG, YTS/YIFY) scores +3000.
+  - LQ scores 0. In HD and UHD it scores -10000, which keeps those groups out of HD.
+  - Standard merges HDTV/WEB/Bluray at each resolution, so the format score decides between
+    sources rather than the source type.
+  - Mini merges 720p and 1080p into one group and adds +1000 for 720p.
+  - Many files therefore arrive already small, and the job skips them.
+- **Minimum sizes are shared by every profile.** For 720p I lowered them to 5 MB per minute
+  and for 1080p to 6 MB per minute (8 for Bluray-1080p), so small encodes are not rejected.
+  HD still never takes the LQ groups.
+- **Seerr:**
+  - The "Sonarr - HD" and "Radarr" servers default to Standard. Only my admin account sees
+    the profile picker, so other users always get Standard. I pick HD or UHD per request.
+  - Other users no longer have "Request 4K".
+- **Configarr is opt-in** (`docker compose --profile configarr up configarr`). Its templates
+  would recreate the old TRaSH profiles and size limits.
+- Backups from before all this: `backups/pre-four-profiles-20261028/`. That covers both
+  apps' profiles, custom formats and size definitions, which title was on which profile,
+  and Seerr's settings. `setup-small-profiles.py` reapplies the Standard and Mini settings.
+
+## How the job works
+
+The nightly run (00:30 to 08:00) has two phases.
+
+**1. Search first.** It asks Sonarr and Radarr to search titles on Standard or Mini that it
+hasn't searched before, biggest first: 5 shows and 20 movies a night, so downloads and
+indexer use stay spread out. Standard and Mini upgrade only toward small releases:
+- Every allowed quality sits in one group, so no resolution or source upgrade can happen.
+- An upgrade needs a custom-format score jump of at least 2500, and upgrades stop at 3000.
+  Only gaining "Small Encode" clears that bar.
+
+So a search replaces a big file only with a small encode, never with a big HEVC release.
+After that, the apps' regular RSS checks keep picking up small releases on their own.
+
+**2. Encode what's left.** A title becomes eligible for encoding 7 days after its search,
+and only its files that are still big get encoded. Files that arrive after the title's
+search work differently:
+- This covers new episodes of ongoing shows, and fresh requests.
+- The first night they're seen, they get their own search (one episode or movie at a time).
+- They're encoded only if they're still big 3 days later.
+
+Downloads never wait for any of this, so new episodes and requests land in the library as
+soon as they download. Only encoding waits. For each file, the job encodes to
+`~/.cache/compact-tv/work.mkv` on the NVMe and then checks the result:
+- the duration matches the original
+- the audio and subtitle track counts match
+- the file is smaller
+
+It then renames the new file over the original, keeping the same name with `.mkv`, and asks
+Sonarr or Radarr to rescan. Plex picks the change up through its folder watcher, and the
+episode keeps its Plex ID, so watch history stays. If a small release shows up later, the
+app may still replace an encoded file with it, which costs bandwidth, not GPU.
+
+Other details:
+- Each output carries a `COMPACT_TIER` tag. A file is skipped if re-encoding would save less
+  than 30%. Ruled-out files are cached by size in `state.json`, so they aren't probed every
+  night. Search dates and failures are kept there too.
+- A downloaded HEVC file no bigger than twice the target size counts as done. That is the
+  small release a search brought in (a MeGusta 1080p episode runs about 2.3 Mbps against
+  Standard's 1.2), and it is neither re-encoded nor searched again. The job's own tagged
+  outputs are exempt, so Standard to Mini still works.
+- **Skipped:** HDR and Dolby Vision sources (they need tone mapping), subtitle tracks ffprobe
+  can't identify, and files that have failed twice.
+- Speed is about 0.19 hours of GPU per hour of 1080p video (NVENC is the bottleneck). The job
+  stops starting new encodes at 08:00.
+- `cost-report.py` estimates, per title, the GPU hours the job would need against the size of
+  a fresh small download.
+- A `PAUSE` file in this folder makes the nightly run do nothing.
+
+## Going back up
+
+Moving down works: Standard to Mini re-encodes the Standard copy. Moving up does not
+restore quality. To get quality back, I set the title to HD and search, which redownloads
+it at full size.
+
+## Running it
+
+    ./compact-tv.py --dry-run                      # what tonight would do
+    ./compact-tv.py --search-all --limit 0         # search every unsearched title now, encode nothing
+    touch PAUSE                                    # skip nightly runs until PAUSE is deleted
+    ./compact-tv.py --series "Shameless (US)" --limit 1
+    systemctl status compact-tv.timer compact-tv.service
+    tail -f compact-tv.log
+
+The installed units are copies of `compact-tv.service` and `compact-tv.timer` from this
+folder. `ExecStart` goes through `/usr/bin/python3` because SELinux won't let systemd
+execute a file labeled as home content.

server/home/Media/compact-tv/compact-tv.py

@@ -0,0 +1,348 @@
+#!/usr/bin/env python3
+"""Shrink shows and movies that Sonarr/Radarr have on the "Standard" or "Mini" profile.
+
+Demoting something is just picking one of those profiles. Each night this job first asks
+Sonarr/Radarr to search a few titles it hasn't searched before. Both profiles upgrade only
+toward small releases, so the search swaps whatever big files have a small release for
+bandwidth instead of GPU. A week after a title's search, the job re-encodes whatever is
+still big to HEVC on the GPU (NVENC) at the tier's bitrate, verifies the result, swaps it in
+place of the original, and has Sonarr/Radarr rescan. Plex and Jellyfin pick up changes
+through their own folder watchers.
+
+Moving something back up means switching it to HD and searching, which redownloads it at
+full size. See README.md.
+
+Usage:
+  compact-tv.py [--until HH:MM] [--series TITLE] [--limit N] [--dry-run]
+
+--series matches a show or movie title. Runs as gib (files are gib-owned) from
+compact-tv.service, nightly.
+"""
+import argparse, datetime, fcntl, json, os, pathlib, re, shutil, subprocess, sys, time, urllib.request
+
+HERE = pathlib.Path(__file__).resolve().parent
+STATE = HERE / "state.json"
+LOG = HERE / "compact-tv.log"
+WORK = pathlib.Path.home() / ".cache" / "compact-tv"   # NVMe, so half-done encodes never land in a ZFS snapshot
+VOLUMES = pathlib.Path.home() / "Server/Media/volumes"
+
+# Per app: API port, container path -> host path, and how to list one title's files.
+APPS = {
+    "sonarr": {"port": 8989, "paths": ("/tv/", "/home/gib/Public/TV/"), "titles": "/series",
+               "files": lambda call, t: call(f"/episodefile?seriesId={t['id']}"),
+               "rescan": lambda t: {"name": "RescanSeries", "seriesId": t["id"]}},
+    "radarr": {"port": 7878, "paths": ("/movies/", "/home/gib/Public/Movies/"), "titles": "/movie",
+               "files": lambda call, t: [t["movieFile"]] if t.get("movieFile") else [],
+               "rescan": lambda t: {"name": "RescanMovie", "movieId": t["id"]}},
+}
+
+# Video bitrate (kbps) per output class, and the widest output each tier allows.
+# TV floors come from watching samples: 1200 kbps at 1080p, 800 at 720p, 600 at SD.
+# Movies get roughly double: dark scenes, grain and action break down first at low rates,
+# and a 2-hour movie at 2500 kbps is still ~2.3 GB (a quarter of a typical 1080p file).
+# That also leaves already-small releases (YTS-style, ~1.5-2 GB) under the 30% bar.
+# Nothing is ever upscaled: a 720p or SD source keeps its size and uses that class's rate.
+TIERS = {
+    "Standard": {"max_width": 1920, "kbps": {"sonarr": {"1080": 1200, "720": 1200, "sd": 600},
+                                             "radarr": {"1080": 2500, "720": 1800, "sd": 900}}},
+    "Mini":     {"max_width": 1280, "kbps": {"sonarr": {"1080": 800,  "720": 800,  "sd": 600},
+                                             "radarr": {"1080": 1500, "720": 1500, "sd": 900}}},
+}
+MIN_SAVING = 0.30            # leave a file alone unless re-encoding saves at least this much
+MAX_FAILURES = 2             # after this many failed attempts a file is skipped for good
+
+# Search first, encode later. Standard/Mini upgrade only toward small releases (see
+# setup-small-profiles.py), so asking the *arr to search a title swaps whatever big files
+# have a small release, for bandwidth instead of GPU. Each night searches a few titles not
+# searched before, biggest first, so downloads and indexer use stay spread out. A title's
+# files are only encoded once its search is SEARCH_GRACE_DAYS old.
+#
+# Files that arrive after their title's search (new episodes, fresh requests) were never
+# searched for a small copy, so the job searches that episode or movie on its own first,
+# and encodes it only if it is still big FILE_SEARCH_GRACE_DAYS later. Downloads are never
+# delayed by any of this; only encoding waits.
+SEARCHES_PER_NIGHT = {"sonarr": 5, "radarr": 20}
+SEARCH_GRACE_DAYS = 7
+FILE_SEARCH_GRACE_DAYS = 3        # small encodes usually appear within hours to a day; 3 leaves room
+HDR_TRANSFERS = {"smpte2084", "arib-std-b67"}
+
+
+def log(msg):
+    line = f"{datetime.datetime.now():%Y-%m-%d %H:%M:%S}  {msg}"
+    print(line, flush=True)
+    with open(LOG, "a") as f:
+        f.write(line + "\n")
+
+
+# ── Sonarr / Radarr ─────────────────────────────────────────────────────────
+def arr(app):
+    """Return a call(path, body=None) for the app's v3 API."""
+    key = re.search(r"<ApiKey>([^<]+)", (VOLUMES / app / "config.xml").read_text()).group(1)
+    ip = subprocess.run(["docker", "inspect", app, "--format",
+                         "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"],
+                        capture_output=True, text=True, check=True).stdout.split()[0]
+    base = f"http://{ip}:{APPS[app]['port']}/api/v3"
+
+    def call(path, body=None):
+        req = urllib.request.Request(base + path, headers={"X-Api-Key": key, "Content-Type": "application/json"},
+                                     data=json.dumps(body).encode() if body is not None else None)
+        with urllib.request.urlopen(req, timeout=60) as r:
+            return json.load(r)
+    return call
+
+
+# ── probing and planning ────────────────────────────────────────────────────
+def probe(path):
+    out = subprocess.run(["ffprobe", "-v", "error", "-print_format", "json", "-show_format", "-show_streams", str(path)],
+                         capture_output=True, text=True)
+    return json.loads(out.stdout) if out.returncode == 0 else None
+
+
+def subtitles(info):
+    """Subtitle tracks ffmpeg can carry over. Tracks ffprobe can't identify are dropped."""
+    return [s for s in info["streams"] if s["codec_type"] == "subtitle" and s.get("codec_name")]
+
+
+def audio_kbps(channels):
+    return 96 if channels <= 2 else 160 if channels <= 6 else 192
+
+
+def plan(info, tier, app):
+    """Return (ffmpeg args, estimated output bytes) for one file, or (None, reason) to skip it."""
+    t = TIERS[tier]
+    video = next((s for s in info["streams"] if s["codec_type"] == "video"
+                  and not s.get("disposition", {}).get("attached_pic")), None)
+    if not video:
+        return None, "no video stream"
+    if video.get("color_transfer") in HDR_TRANSFERS or any("Dolby Vision" in str(sd) for sd in video.get("side_data_list", [])):
+        return None, "HDR/Dolby Vision source (not handled yet)"
+    if (info["format"].get("tags") or {}).get("COMPACT_TIER") == tier:
+        return None, f"already {tier}"
+
+    src_w = int(video["width"])
+    out_w = min(src_w, t["max_width"])
+    cls = "1080" if out_w >= 1700 else "720" if out_w >= 1200 else "sd"
+    vkbps = t["kbps"][app][cls]
+    duration = float(info["format"]["duration"])
+    audios = [s for s in info["streams"] if s["codec_type"] == "audio"]
+    est = int((vkbps + sum(audio_kbps(int(a.get("channels") or 2)) for a in audios)) * 1000 / 8 * duration * 1.03)
+    if int(info["format"]["size"]) * (1 - MIN_SAVING) < est:
+        return None, f"already small ({int(info['format']['size']) / 1e9 * 3600 / duration:.2f} GB/h)"
+    # A downloaded HEVC file within 2x the target is a small release that already replaced a
+    # big one. Re-encoding it would cost GPU and quality for little space. Our own outputs
+    # (tagged) still go through, so moving a title from Standard to Mini works.
+    tags = info["format"].get("tags") or {}
+    if video.get("codec_name") == "hevc" and "COMPACT_TIER" not in tags and int(info["format"]["size"]) <= 2 * est:
+        return None, f"small HEVC release ({int(info['format']['size']) / 1e9 * 3600 / duration:.2f} GB/h)"
+
+    vf = []
+    if video.get("field_order") not in (None, "progressive", "unknown"):
+        vf.append("bwdif")
+    if out_w < src_w:
+        vf.append(f"scale={out_w}:-2:flags=lanczos")
+    args = ["-map", f"0:{video['index']}"]
+    if vf:
+        args += ["-filter:v", ",".join(vf)]
+    args += ["-c:v", "hevc_nvenc", "-preset", "p7", "-tune", "hq", "-profile:v", "main10", "-pix_fmt", "p010le",
+             "-rc", "vbr", "-multipass", "fullres", "-spatial-aq", "1", "-temporal-aq", "1", "-rc-lookahead", "32",
+             "-bf", "4", "-b_ref_mode", "middle",
+             "-b:v", f"{vkbps}k", "-maxrate", f"{vkbps * 2}k", "-bufsize", f"{vkbps * 4}k"]
+    for i, a in enumerate(audios):
+        ch = int(a.get("channels") or 2)
+        layout = {1: "mono", 2: "stereo", 6: "5.1", 8: "7.1"}.get(ch)
+        args += ["-map", f"0:{a['index']}", f"-c:a:{i}", "libopus", f"-b:a:{i}", f"{audio_kbps(ch)}k",
+                 f"-filter:a:{i}", f"aformat=channel_layouts={layout or 'stereo'}"]
+    args += ["-mapping_family", "1"]
+    for i, s in enumerate(subtitles(info)):
+        args += ["-map", f"0:{s['index']}", f"-c:s:{i}", "srt" if s["codec_name"] == "mov_text" else "copy"]
+    args += ["-map", "0:t?", "-c:t", "copy", "-metadata", f"COMPACT_TIER={tier}"]
+    return args, est
+
+
+# ── encode, verify, swap ────────────────────────────────────────────────────
+def encode(src, args, tier):
+    WORK.mkdir(parents=True, exist_ok=True)
+    tmp = WORK / "work.mkv"
+    tmp.unlink(missing_ok=True)
+    cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-hwaccel", "cuda", "-i", str(src),
+           *args, "-metadata", f"COMPACT_SOURCE={src.name} ({src.stat().st_size} bytes)", str(tmp)]
+    r = subprocess.run(cmd, capture_output=True, text=True)
+    if r.returncode != 0:
+        tmp.unlink(missing_ok=True)
+        raise RuntimeError("ffmpeg failed: " + r.stderr.strip()[-300:])
+    return tmp
+
+
+def verify(src_info, tmp):
+    new = probe(tmp)
+    if not new:
+        raise RuntimeError("output does not probe")
+    a, b = float(src_info["format"]["duration"]), float(new["format"]["duration"])
+    if abs(a - b) > max(2.0, a * 0.005):
+        raise RuntimeError(f"duration mismatch {a:.1f}s vs {b:.1f}s")
+    counts = lambda i: (sum(1 for s in i["streams"] if s["codec_type"] == "audio"), len(subtitles(i)))
+    if counts(src_info) != counts(new):
+        raise RuntimeError(f"audio/subtitle track counts {counts(src_info)} -> {counts(new)}")
+    if int(new["format"]["size"]) >= int(src_info["format"]["size"]):
+        raise RuntimeError("output is not smaller")
+    return int(new["format"]["size"])
+
+
+def swap(src, tmp):
+    """Copy the encode next to the original, then rename it over the original in one step."""
+    final = src.with_suffix(".mkv")
+    staged = src.with_name(f".{final.name}.compact-tmp")
+    shutil.copyfile(tmp, staged)
+    os.replace(staged, final)
+    if final != src:
+        src.unlink()
+    tmp.unlink(missing_ok=True)
+    return final
+
+
+# ── main ────────────────────────────────────────────────────────────────────
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--until", help="stop starting new encodes after this local time (HH:MM)")
+    ap.add_argument("--series", help="only this series title")
+    ap.add_argument("--limit", type=int, help="encode at most N files")
+    ap.add_argument("--dry-run", action="store_true", help="plan only, change nothing")
+    ap.add_argument("--search-all", action="store_true",
+                    help="search every not-yet-searched title now instead of a few per night")
+    opt = ap.parse_args()
+
+    if (HERE / "PAUSE").exists() and not opt.dry_run:
+        log("PAUSE file present, not encoding anything (delete it to resume)")
+        return
+
+    lock = open(HERE / ".lock", "w")
+    try:
+        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
+    except BlockingIOError:
+        sys.exit("another compact-tv run is active")
+
+    deadline = None
+    if opt.until:
+        h, m = map(int, opt.until.split(":"))
+        now = datetime.datetime.now()
+        deadline = now.replace(hour=h, minute=m, second=0, microsecond=0)
+        if deadline <= now:
+            deadline += datetime.timedelta(days=1)
+
+    state = json.loads(STATE.read_text()) if STATE.exists() else {}
+    for k in ("failures", "searched", "skip"):
+        state.setdefault(k, {})
+    save = lambda: STATE.write_text(json.dumps(state, indent=1))
+    today = datetime.date.today()
+    work = []                                  # (app, call, title, tier); TV first, then movies
+    for app, a in APPS.items():
+        call = arr(app)
+        profiles = {p["id"]: p["name"] for p in call("/qualityprofile")}
+        work += [(app, call, t, profiles[t["qualityProfileId"]]) for t in call(a["titles"])
+                 if profiles.get(t["qualityProfileId"]) in TIERS and (not opt.series or t["title"] == opt.series)]
+    log(f"=== run start{' [dry run]' if opt.dry_run else ''}: {len(work)} title(s) on "
+        f"{'/'.join(TIERS)}{f', until {deadline:%H:%M}' if deadline else ''} ===")
+
+    # ── 1. search: ask the *arr for small replacements before spending GPU ──
+    size_of = lambda t: (t.get("statistics") or {}).get("sizeOnDisk") or t.get("sizeOnDisk") or 0
+    for app in APPS:
+        fresh = sorted((w for w in work if w[0] == app and f"{app}:{w[2]['id']}" not in state["searched"]
+                        and size_of(w[2]) > 0), key=lambda w: -size_of(w[2]))
+        fresh = fresh if opt.search_all else fresh[:SEARCHES_PER_NIGHT[app]]
+        for _, call, t, tier in fresh:
+            log(f"  {'would search' if opt.dry_run else 'search'} {t['title']} ({size_of(t) / 1e9:.0f} GB, {tier})")
+        if fresh and not opt.dry_run:
+            call = fresh[0][1]
+            if app == "sonarr":
+                for _, _, t, _ in fresh:
+                    call("/command", {"name": "SeriesSearch", "seriesId": t["id"]})
+            else:
+                call("/command", {"name": "MoviesSearch", "movieIds": [t["id"] for _, _, t, _ in fresh]})
+            for _, _, t, _ in fresh:
+                state["searched"][f"{app}:{t['id']}"] = today.isoformat()
+            save()
+
+    # ── 2. encode what is still big after its search ────────────────────────
+    age = lambda iso: (today - datetime.date.fromisoformat(iso)).days
+
+    def ready(app, ef, title_searched, to_search):
+        """True if this big file has had its chance to be replaced by a small download."""
+        if (ef.get("dateAdded") or "")[:10] <= title_searched:   # was on disk when the title was searched
+            return age(title_searched) >= SEARCH_GRACE_DAYS
+        key = f"{app}:file:{ef['id']}"                            # arrived later: search it on its own first
+        if key not in state["searched"]:
+            to_search.append(ef)
+            return False
+        return age(state["searched"][key]) >= FILE_SEARCH_GRACE_DAYS
+
+    def search_files(app, call, show, files):
+        if opt.dry_run:
+            log(f"  would search {len(files)} newer file(s) of {show['title']}"); return
+        if app == "sonarr":
+            ids = {f["id"] for f in files}
+            eps = [e["id"] for e in call(f"/episode?seriesId={show['id']}") if e.get("episodeFileId") in ids]
+            call("/command", {"name": "EpisodeSearch", "episodeIds": eps})
+        else:
+            call("/command", {"name": "MoviesSearch", "movieIds": [show["id"]]})
+        for f in files:
+            state["searched"][f"{app}:file:{f['id']}"] = today.isoformat()
+        log(f"  search {len(files)} newer file(s) of {show['title']} before encoding"); save()
+
+    done = saved = 0
+    for app, call, show, tier in work:
+        title_searched = state["searched"].get(f"{app}:{show['id']}")
+        if not title_searched:
+            continue                          # its title search comes first, on a later night
+        container, host = APPS[app]["paths"]
+        files = sorted(APPS[app]["files"](call, show), key=lambda f: f["path"])
+        changed, to_search = False, []
+        for ef in files:
+            if deadline and datetime.datetime.now() >= deadline:
+                log("deadline reached, stopping"); break
+            if opt.limit is not None and done >= opt.limit:
+                break
+            src = pathlib.Path(ef["path"].replace(container, host, 1))
+            if state["failures"].get(str(src), 0) >= MAX_FAILURES or not src.exists():
+                continue
+            # Files already ruled out at this tier and size need no re-probe (most of them, most nights).
+            seen = f"{tier}:{src.stat().st_size}"
+            if state["skip"].get(str(src)) == seen:
+                continue
+            info = probe(src)
+            if not info:
+                log(f"  SKIP {src.name}: does not probe"); continue
+            args, est = plan(info, tier, app)
+            if args is None:
+                state["skip"][str(src)] = seen  # reason in `est`; quiet, since most nights most files are done
+                continue
+            if not ready(app, ef, title_searched, to_search):
+                continue
+            old = int(info["format"]["size"])
+            if opt.dry_run:
+                log(f"  would encode {src.name}: {old / 1e6:.0f} MB -> ~{est / 1e6:.0f} MB ({tier})")
+                done += 1; continue
+            t0 = time.monotonic()
+            try:
+                tmp = encode(src, args, tier)
+                new = verify(info, tmp)
+                final = swap(src, tmp)
+            except Exception as e:
+                state["failures"][str(src)] = state["failures"].get(str(src), 0) + 1
+                save()
+                log(f"  FAIL {src.name}: {e}"); continue
+            done += 1; saved += old - new; changed = True
+            log(f"  {final.name}: {old / 1e6:.0f} MB -> {new / 1e6:.0f} MB "
+                f"(-{(1 - new / old) * 100:.0f}%, {time.monotonic() - t0:.0f}s, {tier})")
+        if to_search:
+            search_files(app, call, show, to_search)
+        if changed:
+            call("/command", APPS[app]["rescan"](show))
+        if (deadline and datetime.datetime.now() >= deadline) or (opt.limit is not None and done >= opt.limit):
+            break
+    if not opt.dry_run:
+        save()
+    log(f"=== run end: {done} file(s){' planned' if opt.dry_run else ''}, saved {saved / 1e9:.1f} GB ===")
+
+
+if __name__ == "__main__":
+    main()

server/home/Media/compact-tv/compact-tv.service

@@ -0,0 +1,17 @@
+# Installed to /etc/systemd/system/. Runs as gib because the TV files are gib-owned,
+# so the job needs no root. A system unit (rather than a user unit) because gib has
+# no lingering session on this headless box. ExecStart goes through python3 because
+# SELinux will not let systemd execute a file labeled as home content (user_home_t).
+[Unit]
+Description=Re-encode TV shows on Sonarr's Compact/Supercompact profiles
+Documentation=file:///home/gib/Server/Media/compact-tv/README.md
+After=docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+User=gib
+Group=gib
+ExecStart=/usr/bin/python3 /home/gib/Server/Media/compact-tv/compact-tv.py --until 08:00
+Nice=15
+IOSchedulingClass=idle

server/home/Media/compact-tv/compact-tv.timer

@@ -0,0 +1,11 @@
+# Starts after the 00:00 hour and stops starting new encodes at 08:00 (see --until in
+# the service), so Plex and Frigate keep the GPU's encoder during the day.
+[Unit]
+Description=Nightly TV re-encode for Compact/Supercompact shows
+
+[Timer]
+OnCalendar=*-*-* 00:30:00
+Persistent=false
+
+[Install]
+WantedBy=timers.target

server/home/Media/compact-tv/cost-report.py

@@ -0,0 +1,65 @@
+#!/usr/bin/env python3
+"""Per-title cost of shrinking everything on Standard/Mini: re-encode vs redownload.
+
+For each show or movie on those profiles, probes every file with compact-tv's own planner
+and reports what the nightly job would do (encode, skip as already small, skip as HDR),
+the space it would save, and the GPU time it would take. Alongside that it estimates what a
+fresh small download would cost instead, using typical small-encode sizes.
+
+GPU time is measured, not guessed: the Shameless pilot took 669 s for 57.5 minutes of
+1080p (about 0.19 h of GPU per hour of video). 4K sources decode slower; the 1.5x for them
+is an estimate until the first 4K night is logged.
+
+Usage: cost-report.py > report.tsv     (also prints a summary to stderr)
+"""
+import importlib.util, json, pathlib, sys
+
+HERE = pathlib.Path(__file__).resolve().parent
+spec = importlib.util.spec_from_file_location("ct", HERE / "compact-tv.py")
+ct = importlib.util.module_from_spec(spec); spec.loader.exec_module(ct)
+
+GPU_H_PER_H = 669 / 3450                 # pilot: hours of GPU per hour of 1080p video
+UHD_FACTOR = 1.5                         # estimate for 2160p sources
+SMALL_GB_PER_H = {"Standard": 0.85, "Mini": 0.57}   # MeGusta-class 1080p / 720p HEVC, measured on S.W.A.T.
+
+rows = []
+for app, a in ct.APPS.items():
+    call = ct.arr(app)
+    profiles = {p["id"]: p["name"] for p in call("/qualityprofile")}
+    container, host = a["paths"]
+    for t in call(a["titles"]):
+        tier = profiles.get(t["qualityProfileId"])
+        if tier not in ct.TIERS:
+            continue
+        r = {"app": app, "title": t["title"], "tier": tier, "files": 0, "hours": 0.0, "size": 0,
+             "enc_files": 0, "enc_size": 0, "enc_est": 0, "gpu_h": 0.0, "hdr_size": 0, "small_size": 0}
+        for f in a["files"](call, t):
+            p = pathlib.Path(f["path"].replace(container, host, 1))
+            info = ct.probe(p) if p.exists() else None
+            if not info:
+                continue
+            size, dur = int(info["format"]["size"]), float(info["format"]["duration"]) / 3600
+            r["files"] += 1; r["hours"] += dur; r["size"] += size
+            args, why = ct.plan(info, tier, app)
+            if args:
+                w = max((s.get("width") or 0) for s in info["streams"] if s["codec_type"] == "video")
+                r["enc_files"] += 1; r["enc_size"] += size; r["enc_est"] += why
+                r["gpu_h"] += dur * GPU_H_PER_H * (UHD_FACTOR if w > 1920 else 1)
+            elif "HDR" in str(why):
+                r["hdr_size"] += size
+            else:
+                r["small_size"] += size
+        if r["files"]:
+            r["redownload_gb"] = round(r["hours"] * SMALL_GB_PER_H[tier], 1)
+            rows.append(r)
+
+rows.sort(key=lambda r: -(r["enc_size"] - r["enc_est"]))
+cols = ["app", "title", "tier", "files", "hours", "size", "enc_files", "enc_size", "enc_est", "gpu_h", "hdr_size",
+        "small_size", "redownload_gb"]
+print("\t".join(cols))
+for r in rows:
+    print("\t".join(str(round(r[c], 2) if isinstance(r[c], float) else r[c]) for c in cols))
+tot = lambda k: sum(r[k] for r in rows)
+print(f"{len(rows)} titles; encode {tot('enc_files')} files {tot('enc_size')/1e12:.2f} TB -> ~{tot('enc_est')/1e12:.2f} TB, "
+      f"{tot('gpu_h'):.0f} GPU hours; HDR skipped {tot('hdr_size')/1e12:.2f} TB; already small {tot('small_size')/1e12:.2f} TB",
+      file=sys.stderr)

server/home/Media/compact-tv/setup-small-profiles.py

@@ -0,0 +1,99 @@
+#!/usr/bin/env python3
+"""Make Standard and Mini prefer small releases, and upgrade only toward small ones.
+
+Run once per app; it is idempotent, so re-running updates everything in place.
+
+- Adds "Small Encode" (release groups known for small HEVC/x264 encodes) and "720p" /
+  "1080p" resolution formats. They score 0 in every profile except Standard and Mini.
+- HD and UHD are otherwise untouched: they keep LQ at -10000, so they never pick these.
+- Standard and Mini:
+  - All allowed qualities share ONE group, so no "better resolution" or "better source"
+    upgrade can ever download a big file. Only the custom-format score decides.
+  - Scores: Small Encode +3000, x265 +2000 (so first grabs still prefer HEVC), LQ 0,
+    release-tier formats 0 (one big release must never "beat" another big one).
+    Standard adds +500 for 1080p and +250 for 720p; Mini adds +1000 for 720p and +500 for
+    1080p. Those preferences stay below the upgrade threshold, so they never cause churn.
+  - Upgrades are on, but an upgrade needs a score jump of at least 2500, and upgrading
+    stops at 3000. In practice only gaining "Small Encode" clears that bar: a big H.264
+    file gets replaced by a small encode, but never by a big HEVC one.
+
+Usage: setup-small-profiles.py <sonarr|radarr>
+"""
+import json, re, subprocess, sys, urllib.request, pathlib
+
+APP = sys.argv[1]
+cfg = pathlib.Path.home() / f"Server/Media/volumes/{APP}/config.xml"
+key = re.search(r"<ApiKey>([^<]+)", cfg.read_text()).group(1)
+ip = subprocess.run(["docker", "inspect", APP, "--format", "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"],
+                    capture_output=True, text=True, check=True).stdout.split()[0]
+base = f"http://{ip}:{8989 if APP == 'sonarr' else 7878}/api/v3"
+
+def call(path, body=None, method=None):
+    req = urllib.request.Request(base + path, method=method, headers={"X-Api-Key": key, "Content-Type": "application/json"},
+                                 data=json.dumps(body).encode() if body is not None else None)
+    with urllib.request.urlopen(req, timeout=60) as r:
+        return json.load(r)
+
+GROUPS = r"^(MeGusta|PSA|ELiTE|GalaxyTV|GalaxyRG|GalaxyRG265|YIFY|YTS(\.(MX|LT|AG))?)$"
+resolution = lambda r: [{"name": f"{r}p", "implementation": "ResolutionSpecification",
+                         "negate": False, "required": True, "fields": [{"name": "value", "value": r}]}]
+WANTED = {
+    "Small Encode": [{"name": "Small encode groups", "implementation": "ReleaseGroupSpecification",
+                      "negate": False, "required": False, "fields": [{"name": "value", "value": GROUPS}]}],
+    "720p": resolution(720),
+    "1080p": resolution(1080),
+}
+TIERS = {
+    "Standard": {"Small Encode": 3000, "x265 (HD)": 2000, "LQ": 0, "LQ (Release Title)": 0, "1080p": 500, "720p": 250},
+    "Mini":     {"Small Encode": 3000, "x265 (HD)": 2000, "LQ": 0, "LQ (Release Title)": 0, "720p": 1000, "1080p": 500},
+}
+RELEASE_TIER = re.compile(r"Tier \d+|WEB Scene")
+MIN_UPGRADE_JUMP, UPGRADE_UNTIL = 2500, 3000
+
+cfs = {c["name"]: c for c in call("/customformat")}
+for name, specs in WANTED.items():
+    body = {"name": name, "includeCustomFormatWhenRenaming": False, "specifications": specs}
+    if name in cfs:
+        body["id"] = cfs[name]["id"]; call(f"/customformat/{body['id']}", body, "PUT")
+    else:
+        call("/customformat", body)
+cfs = {c["name"]: c["id"] for c in call("/customformat")}
+
+
+def flatten(items):
+    """Every quality in the profile with whether it is allowed, groups expanded, lowest rank first."""
+    for it in items:
+        for sub in (it["items"] or [it]):
+            yield sub["quality"], it["allowed"]
+
+
+def one_group(profile):
+    """Put every allowed quality into a single top-ranked group; leave disallowed ones as they are."""
+    flat = list(flatten(profile["items"]))
+    group = {"id": 1100, "name": "Any allowed", "allowed": True,
+             "items": [{"quality": q, "items": [], "allowed": True} for q, ok in flat if ok]}
+    profile["items"] = [{"quality": q, "items": [], "allowed": False} for q, ok in flat if not ok] + [group]
+    profile["cutoff"] = group["id"]
+    return profile
+
+
+for prof in call("/qualityprofile"):
+    have = {f["format"] for f in prof["formatItems"]}
+    for name in WANTED:
+        if cfs[name] not in have:
+            prof["formatItems"].append({"format": cfs[name], "name": name, "score": 0})
+    scores = TIERS.get(prof["name"])
+    if scores:
+        prof = one_group(prof)
+        prof.update(upgradeAllowed=True, minUpgradeFormatScore=MIN_UPGRADE_JUMP, cutoffFormatScore=UPGRADE_UNTIL)
+        for f in prof["formatItems"]:
+            if f["name"] in scores:
+                f["score"] = scores[f["name"]]
+            elif RELEASE_TIER.search(f["name"]):
+                f["score"] = 0
+    call(f"/qualityprofile/{prof['id']}", prof, "PUT")
+    if scores:
+        allowed = [q["quality"]["name"] for it in prof["items"] if it["allowed"] for q in it["items"]]
+        print(f"{APP} {prof['name']}: one group of {len(allowed)} qualities ({allowed[0]} .. {allowed[-1]}); "
+              f"upgrades on, jump >= {MIN_UPGRADE_JUMP}, until {UPGRADE_UNTIL}")
+        print("   nonzero scores: " + ", ".join(f"{f['name']}={f['score']}" for f in prof["formatItems"] if f["score"]))

server/home/Media/compose.yml

@@ -0,0 +1,434 @@
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  plex:
+    image: plexinc/pms-docker:latest
+    container_name: plex
+    hostname: plex
+    domainname: plex.gbrown.org
+    networks: [nginx-bridge]
+    group_add: [105, 39]
+    ports:
+      - 32400:32400/tcp # REQUIRED
+      #- 5353:5353/udp # For older Bonjour/Avahi network discovery
+      #- 8324:8324/tcp # For Plex Roku App
+      - 32410:32410/udp # For network discovery
+      - 32412:32412/udp # For network discovery
+      - 32413:32413/udp # For network discovery
+      - 32414:32414/udp # For network discovery
+      #- 1900:1900/udp # For DLNA
+      - 32469:32469/tcp # For DLNA
+    environment:
+      - TZ=America/New_York
+      - PLEX_UID=1000
+      - PLEX_GID=1000
+      - PLEX_CLAIM=${PLEX_CLAIM}
+      - ADVERTISE_IP=http://192.168.0.43
+      - NVIDIA_VISIBLE_DEVICES=all
+      - NVIDIA_DRIVER_CAPABILITIES=compute,video,utility
+    labels: ["com.centurylinklabs.watchtower.enable=true"]
+    volumes:
+      - /run:/run
+      - ./volumes/plex/config:/config
+      - ./volumes/plex/transcode:/transcode
+      - /home/gib/Public/Movies:/data/movies
+      - /home/gib/Public/TV:/data/tvshows
+      - /home/gib/Public/Music:/data/music
+      - /home/gib/Public/Audiobooks:/data/audiobooks
+      - /home/gib/Public/Podcasts:/data/podcasts
+    devices: [/dev/dri:/dev/dri]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    security_opt: ["seccomp:unconfined"]
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  jellyfin:
+    image: jellyfin/jellyfin
+    container_name: jellyfin
+    hostname: jellyfin
+    domainname: jellyfin.gbrown.org
+    networks: [nginx-bridge]
+    group_add: [105, 39]
+    ports: [7359:7359/udp]
+    user: 1000:1000
+    environment:
+      - JELLYFIN_PublishedServerUrl=https://jellyfin.gbrown.org
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    volumes:
+      - ./volumes/jellyfin/config:/config
+      - ./volumes/jellyfin/cache:/cache
+      - /home/gib/Public/Movies:/data/movies
+      - /home/gib/Public/TV:/data/tvshows
+      - /home/gib/Public/Music:/data/music
+      - /home/gib/Public/Audiobooks:/data/audiobooks
+      - /home/gib/Public/Podcasts:/data/podcasts
+      - /home/gib/.local/share/fonts:/usr/local/share/fonts/custom:ro
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    devices: [/dev/dri:/dev/dri]
+    security_opt: [seccomp:unconfined]
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  audiobookshelf:
+    image: ghcr.io/advplyr/audiobookshelf:latest
+    container_name: audiobookshelf
+    hostname: audiobookshelf
+    domainname: audio.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [80:80]
+    environment: [TZ=America/New_York, PUID=1000, PGID=1000]
+    labels: ["com.centurylinklabs.watchtower.enable=true"]
+    volumes:
+      - ./volumes/audiobookshelf/config:/config
+      - ./volumes/audiobookshelf/metadata:/metadata
+      - /home/gib/Public/Audiobooks:/audiobooks
+      - /home/gib/Public/Podcasts:/podcasts
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  jellyplex-watched:
+    image: luigi311/jellyplex-watched:latest
+    container_name: jellyplex-watched
+    hostname: jellyplex-watched
+    networks: [nginx-bridge]
+    environment:
+      DRYRUN: False
+      DEBUG_LEVEL: INFO # Or debug for verbose logging
+      RUN_ONLY_ONCE: False
+      SLEEP_DURATION: 3600
+      REQUEST_TIMEOUT: 300
+      MAX_THREADS: 2
+      GENERATE_GUIDS: False
+      GENERATE_LOCATIONS: True
+      USER_MAPPING: '{ "gib": "gibby_b", "adrian": "Adrian"}'
+      LIBRARY_MAPPING: '{ "TV Shows": "Shows"}'
+      PLEX_BASEURL: http://plex:32400
+      PLEX_TOKEN: ${PLEX_TOKEN}
+      SSL_BYPASS: False
+      JELLYFIN_BASEURL: http://jellyfin:8096
+      JELLYFIN_TOKEN: ${JELLYFIN_TOKEN}
+      SYNC_FROM_PLEX_TO_JELLYFIN: True
+      SYNC_FROM_JELLYFIN_TO_PLEX: True
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  gluetun:
+    image: qmcgaw/gluetun:latest
+    container_name: gluetun
+    hostname: gluetun
+    networks: [nginx-bridge]
+    cap_add: [NET_ADMIN]
+    devices: [/dev/net/tun:/dev/net/tun]
+    #ports:
+    #- 8000:8000 # HTTP control server
+    #- 8080:8080 # For SABnzbd
+    #- 6969:6969 # For qBittorrent
+    #- 6881:6881 # For torrent connectivity
+    #- 6881:6881/udp # For torrent connectivity
+    environment:
+      - VPN_SERVICE_PROVIDER=private internet access
+      - VPN_TYPE=openvpn
+      - SERVER_REGIONS=US East
+      - OPENVPN_USER=p7671057
+      - OPENVPN_PASSWORD=${OPENVPN_PASSWORD}
+      - TZ=America/New_York
+      - PUID=1000
+      - PGID=100
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    restart: unless-stopped
+    tty: true
+    stdin_open: true
+
+  sabnzbd:
+    image: lscr.io/linuxserver/sabnzbd:latest
+    container_name: sabnzbd
+    domainname: sabnzbd.gbrown.org
+    network_mode: service:gluetun
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    volumes:
+      - ./volumes/sabnzbd:/config
+      - /home/gib/Public/Downloads:/downloads
+      - /home/gib/Public/Downloads/incomplete:/incomplete-downloads
+    depends_on: [gluetun]
+    restart: unless-stopped
+    tty: true
+    stdin_open: true
+
+  qbittorrent:
+    image: linuxserver/qbittorrent:latest
+    container_name: qbittorrent
+    network_mode: service:gluetun
+    environment:
+      - PUID=1000
+      - PGID=1000
+      - TZ=America/New_York
+      - WEBUI_PORT=6969
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    volumes:
+      - ./volumes/qbittorrent:/config
+      - /home/gib/Public/Downloads:/downloads
+    depends_on: [gluetun]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  qbit-guard:
+    image: ghcr.io/gengines/qbit-guard:latest
+    container_name: qbitguard
+    hostname: qbitguard
+    networks: [nginx-bridge]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    environment:
+      - QBIT_HOST=http://gluetun:6969
+      - QBIT_USER=gib
+      - QBIT_PASS=${QBIT_PASS}
+      - QBIT_ALLOWED_CATEGORIES=tv-sonarr,radarr
+      - ENABLE_PREAIR_CHECK=1
+      - SONARR_URL=http://sonarr:8989
+      - SONARR_API_KEY=${SONARR_API_KEY}
+      - SONARR_CATEGORIES=tv-sonarr
+      - RADARR_URL=http://radarr:7878
+      - RADARR_API_KEY=${RADARR_API_KEY}
+      - RADARR_CATEGORIES=radarr
+      - ENABLE_ISO_CHECK=1
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  seerr:
+    image: ghcr.io/seerr-team/seerr:latest
+    container_name: seerr
+    init: true
+    hostname: seerr
+    domainname: seerr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [5055:5055]
+    environment: [LOG_LEVEL=debug, TZ=America/New_York]
+    volumes: [./volumes/seerr:/app/config]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    depends_on: [radarr, sonarr]
+    healthcheck:
+      test: wget --no-verbose --tries=1 --spider http://localhost:5055/api/v1/status || exit 1
+      start_period: 20s
+      timeout: 3s
+      interval: 15s
+      retries: 3
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  prowlarr:
+    image: lscr.io/linuxserver/prowlarr:latest
+    container_name: prowlarr
+    hostname: prowlarr
+    domainname: prowlarr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [9696:9696]
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    volumes: [./volumes/prowlarr:/config]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    depends_on: [qbittorrent, sabnzbd]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  lidarr:
+    image: lscr.io/linuxserver/lidarr:latest
+    container_name: lidarr
+    hostname: lidarr
+    domainname: lidarr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [8686:8686]
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    volumes:
+      - ./volumes/lidarr:/config
+      - /home/gib/Public/Music:/music
+      - /home/gib/Public/Downloads:/downloads
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    depends_on: [prowlarr, qbittorrent, sabnzbd]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  radarr:
+    image: ghcr.io/linuxserver/radarr:latest
+    container_name: radarr
+    hostname: radarr
+    domainname: radarr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [7878:7878]
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    volumes:
+      - ./volumes/radarr:/config
+      - /home/gib/Public/Movies:/movies
+      - /home/gib/Public/Downloads:/downloads
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    depends_on: [prowlarr, qbittorrent, sabnzbd]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  sonarr:
+    image: ghcr.io/linuxserver/sonarr:latest
+    container_name: sonarr
+    hostname: sonarr
+    domainname: sonarr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [8989:8989]
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    volumes:
+      - ./volumes/sonarr:/config
+      - /home/gib/Public/TV:/tv
+      - /home/gib/Public/Downloads:/downloads
+    depends_on: [prowlarr, qbittorrent, sabnzbd]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  bazarr:
+    image: lscr.io/linuxserver/bazarr:latest
+    container_name: bazarr
+    hostname: bazarr
+    domainname: bazarr.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [6767:6767]
+    environment: [PUID=1000, PGID=1000, TZ=America/New_York]
+    volumes:
+      - ./volumes/bazarr:/config
+      - /home/gib/Public/Movies:/movies
+      - /home/gib/Public/TV:/tv
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  whisperasr:
+    image: onerahmet/openai-whisper-asr-webservice:latest-gpu
+    container_name: whisperasr
+    hostname: whisperasr
+    networks: [nginx-bridge]
+    #ports: [9000:9000]
+    environment: [TZ=America/New_York, PUID=1000, PGID=1000, ASR_MODEL=small]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    devices: [/dev/dri:/dev/dri]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+    security_opt: ["seccomp:unconfined"]
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  unpackerr:
+    image: golift/unpackerr:latest
+    container_name: unpackerr
+    hostname: unpackerr
+    networks: [nginx-bridge]
+    volumes:
+      - /home/gib/Public/Downloads:/downloads
+      - /home/gib/Server/Media/volumes/unpackerr:/config
+    environment:
+      - TZ=America/New_York
+      # log lives in its own volume, NOT in /downloads where the *arr apps scan
+      - UN_LOG_FILE=/config/unpackerr.log
+      - UN_LOG_FILES=4
+      - UN_LOG_FILE_MB=5
+      - UN_SONARR_0_URL=http://sonarr:8989
+      - UN_SONARR_0_API_KEY=${UN_SONARR_0_API_KEY}
+      - UN_RADARR_0_URL=http://radarr:7878
+      - UN_RADARR_0_API_KEY=${UN_RADARR_0_API_KEY}
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    tty: true
+    stdin_open: true
+    restart: always
+
+  metube:
+    image: ghcr.io/alexta69/metube:latest
+    container_name: metube
+    hostname: metube
+    domainname: metube.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [8081:8081]
+    environment:
+      - TZ=America/New_York
+      - PUID=1000
+      - PGID=1000
+      - YTDL_OPTIONS={"cookiefile":"/cookies/cookies.txt"}
+    volumes:
+      - ./volumes/metube:/cookies:rw
+      - /home/gib/Public/Downloads:/downloads:rw
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  pinchflat:
+    image: ghcr.io/kieraneglin/pinchflat:latest
+    container_name: pinchflat
+    hostname: pinchflat
+    domainname: pinchflat.gbrown.org
+    networks: [nginx-bridge]
+    #ports: [8945:8945]
+    environment: [TZ=America/New_York, PUID=1000, PGID=1000]
+    volumes:
+      - ./volumes/pinchflat:/config:rw
+      - /home/gib/Public/Downloads:/downloads:rw
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    tty: true
+    stdin_open: true
+    restart: unless-stopped
+
+  configarr:
+    image: ghcr.io/raydak-labs/configarr:latest
+    container_name: configarr
+    hostname: configarr
+    domainname: configarr.gbrown.org
+    networks: [nginx-bridge]
+    environment: [TZ=America/New_York, PUID=1000, PGID=1000]
+    labels: [com.centurylinklabs.watchtower.enable=true]
+    volumes:
+      - ./volumes/configarr/config:/app/config
+      - ./volumes/configarr/repos:/app/repos
+      - ./volumes/configarr/cfs:/app/cfs
+      - ./volumes/configarr/templates:/app/templates
+    depends_on: [radarr, sonarr]
+    tty: true
+    stdin_open: true
+    # Run-to-completion: configarr syncs TRaSH guides into Sonarr/Radarr,
+    # prints an execution summary, and exits 0. It has no built-in scheduler
+    # (upstream points at Ofelia or cron for that), so `unless-stopped` here
+    # meant a completed sync was immediately restarted -- a hot loop of ~9
+    # full syncs every 3 minutes. `no` lets it run once per `up` and stop.
+    restart: "no"
+    # Opt-in only (`docker compose --profile configarr up configarr`). Its templates
+    # create the TRaSH profiles and size limits that were removed on 2026-10-28 in
+    # favour of four hand-managed profiles (Mini/Standard/HD/UHD), so a plain `up`
+    # must never run it. See Media/compact-tv/README.md.
+    profiles: [configarr]

server/home/Media/podman-media.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Media
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Media
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Minecraft/Bedrock/Adrian/compose.yml

@@ -0,0 +1,28 @@
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  bedrock-minecraft:
+    image: itzg/minecraft-bedrock-server
+    container_name: mc-bedrock-adrian
+    network_mode: host
+    #hostname: mc-bedrock-adrian
+    #domainname: mc-bedrock-adrian.gbrown.org
+    #networks: ['nginx-bridge']
+    #ports:
+      #- 19132:19132
+      #- 19132:19132/udp
+    environment:
+     - EULA=TRUE
+     #- UID=1000
+     #- GID=1000
+     - TZ=America/New_York
+    #user: 1000:1000
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - './data:/data'
+    tty: true
+    stdin_open: true
+    restart: unless-stopped

server/home/Minecraft/Bedrock/Adrian/podman-minecraft-bedrock-adrian.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Minecraft/Bedrock/Adrian
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Minecraft/Bedrock/Adrian
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Minecraft/Java/compose.yml

@@ -0,0 +1,104 @@
+# Java Edition Minecraft behind mc-router. Live since 2026-10-06 at mc.gbrown.org.
+#
+# mc-router owns the one public port (25565) and routes each player by the hostname
+# they typed: Java clients send it in their first packet, so several servers can
+# share one port and one IP. Bedrock clients do not send it, which is why the
+# Bedrock server in ../Bedrock stays separate on its own UDP port.
+#
+# Start:    docker compose up -d        (from this directory)
+#
+# Each hostname needs a DNS-only (grey cloud) A record in Cloudflare pointing at the
+# home IP. The proxied *.gbrown.org wildcard can't carry Minecraft traffic, and a
+# specific record overrides it. UniFi forwards 25565/tcp to this host. Home
+# Assistant's Cloudflare integration keeps these records on the current home IP;
+# select each new record there too.
+#
+# Add a server:
+#   1. mkdir servers/<name>, then copy the "gib" service below and point it there
+#   2. add "<hostname>=<service>:25565" to MAPPING
+#   3. add the DNS-only record for <hostname>
+# Servers publish no ports of their own; only mc-router is reachable from outside.
+
+services:
+  mc-router:
+    image: itzg/mc-router:latest
+    container_name: mc-router
+    environment:
+      # <hostname players type>=<service name>:<port inside the network>
+      MAPPING: |
+        mc.gbrown.org=gib:25565
+    ports: ["25565:25565"]
+    networks: [minecraft]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    restart: unless-stopped
+
+  gib:
+    image: itzg/minecraft-server:latest
+    container_name: mc-gib
+    environment:
+      # 26.1.2 matches the newest OptiFine (SEUS shaders) on the desktop. The 1.20.4
+      # world is in backups/; upgrades rewrite the world and cannot be undone.
+      - VERSION=26.1.2
+      - DIFFICULTY=normal
+      - MAX_PLAYERS=30
+      # Paper since 2026-10-13. The last vanilla state is backups/gib-26.1.2-vanilla-*.
+      - TYPE=PAPER
+      # DonutSMP-style plugin set, pulled from Modrinth on each start. Every entry
+      # needs a 26.1.2 Paper build; check Modrinth before adding or bumping VERSION.
+      # Config lives in servers/gib/plugins/<Plugin>/. GrimAC only publishes alpha builds.
+      - MODRINTH_DOWNLOAD_DEPENDENCIES=required
+      - |
+        MODRINTH_PROJECTS=
+        luckperms
+        vaultunlocked
+        essentialsx
+        essentialsx-spawn
+        placeholderapi
+        playerpoints
+        zmenu
+        zauctionhouse
+        ultimateshop
+        smartspawner
+        wildrtp
+        ultimate-teams
+        asbounty
+        axafkzone
+        tab-was-taken
+        deluxemenus
+        coreprotect
+        worldedit
+        worldguard
+        grimac:alpha
+        chunky
+      - MEMORY=12G
+      - SIMULATION_DISTANCE=16
+      - VIEW_DISTANCE=16
+      - ALLOW_FLIGHT=true
+      - ENABLE_COMMAND_BLOCK=true
+      - ENABLE_WHITELIST=true
+      - UID=1000
+      - GID=1000
+      - EULA=true
+      - TZ=America/New_York
+      - ENABLE_ROLLING_LOGS=true
+      - LANG=en_US.UTF-8
+      - LANGUAGE=en_US:en
+      - LC_ALL=en_US.UTF-8
+      - SERVER_NAME=Gib's Minecraft Server
+      - SNOOPER_ENABLED=false
+      - WHITELIST_FILE=/data/whitelist.json
+      #- ENABLE_AUTOPAUSE=true
+      # Don't change MOTD it was really hard to make it look good.
+      - MOTD=           \u00A7aGib's Minecraft Server\u00A7r                                            \u00A74Welcome to HECK
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./servers/gib:/data
+    networks: [minecraft]
+    stdin_open: true
+    tty: true
+    restart: unless-stopped
+
+networks:
+  minecraft: {}

server/home/Minecraft/Java/podman-minecraft-java.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Minecraft/Java
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Minecraft/Java
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Nginx/.env.example

@@ -0,0 +1,2 @@
+DB_POSTGRES_PASSWORD=CHANGE_ME
+POSTGRES_PASSWORD=CHANGE_ME

server/home/Nginx/compose.yml

@@ -0,0 +1,55 @@
+services:
+  app:
+    image: 'jc21/nginx-proxy-manager:latest'
+    container_name: nginx-proxy-manager
+    hostname: nginx-proxy-manager
+    domainname: nginx.gbrown.org
+    networks:
+      - nginx-bridge
+    restart: unless-stopped
+    ports:
+      - '80:80'
+      - '443:443'
+      - '81:81' # Admin Web Port
+      #- '21:21' # FTP
+      #- '22:22' # SSH
+      #- '25565:25565' # Minecraft
+    environment:
+      PUID: 1000
+      PGID: 1000
+      DB_POSTGRES_HOST: 'db'
+      DB_POSTGRES_PORT: '5432'
+      DB_POSTGRES_USER: 'npm'
+      DB_POSTGRES_PASSWORD: ${DB_POSTGRES_PASSWORD}
+      DB_POSTGRES_NAME: 'npm'
+    labels:
+      com.centurylinklabs.watchtower.enable: 'true'
+    volumes:
+      - ./volumes/data:/data
+      - ./volumes/letsencrypt:/etc/letsencrypt
+    depends_on:
+      - db
+
+  db:
+    image: postgres:17
+    container_name: nginx-proxy-manager-db
+    hostname: nginx-proxy-manager-db
+    networks:
+      - nginx-bridge
+    environment:
+      POSTGRES_USER: 'npm'
+      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
+      POSTGRES_DB: 'npm'
+    labels:
+      com.centurylinklabs.watchtower.enable: 'true'
+    volumes:
+      - ./volumes/postgres:/var/lib/postgresql/data
+    restart: unless-stopped
+
+networks:
+  nginx-bridge:
+    driver: bridge
+    ipam:
+      config:
+        - subnet: 172.25.0.0/24
+    name: nginx-bridge

server/home/Nginx/podman-nginx.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Nginx
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Nginx
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/OpenWebUI/.env.example

@@ -0,0 +1,4 @@
+BW_SESSION=CHANGE_ME
+OPENAI_API_KEY=CHANGE_ME
+OAUTH_CLIENT_ID=CHANGE_ME
+OAUTH_CLIENT_SECRET=CHANGE_ME

server/home/OpenWebUI/compose.yml

@@ -0,0 +1,147 @@
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  openwebui:
+    image: ghcr.io/open-webui/open-webui:main
+    #image: ghcr.io/open-webui/open-webui:ollama
+    container_name: openwebui
+    hostname: openwebui
+    domainname: chat.gbrown.org
+    networks:
+      - nginx-bridge
+    #ports:
+    #- 8080:8080
+    environment:
+      - WEBUI_NAME=Gib's Chat
+      - WEBUI_URL=https://chat.gbrown.org
+      - OLLAMA_BASE_URL=http://ollama:11434
+      - [email protected]
+      - DEFAULT_MODELS=llama3.1
+      - TASK_MODEL=llama3.1
+      - ENABLE_AUTOCOMPLETE_GENERATION=false
+      - OPENAI_API_KEY=${OPENAI_API_KEY}
+      - AUTOMATIC1111_BASE_URL=http://192.168.0.43:7860
+      - COMFYUI_BASE_URL=http://comfyui:8188
+      - WHISPER_MODEL_AUTO_UPDATE=True
+      - PDF_EXTRACT_IMAGES=True
+      - ENABLE_RAG_WEB_SEARCH=True
+      - RAG_WEB_SEARCH_ENGINE=searxng
+      - SEARXNG_QUERY_URL=https://search.gbrown.org/search?q=<query>
+      - ENABLE_IMAGE_GENERATION=True
+      - IMAGE_GENERATION_ENGINE=ComfyUI
+      - IMAGE_SIZE=1216x832
+      #- USE_OLLAMA_DOCKER=True
+      #- USE_CUDA_DOCKER=True
+      - DO_NOT_TRACK=True
+      - ENABLE_OAUTH_SIGNUP=true
+      - OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
+      #- ENABLE_OAUTH_ROLE_MANAGEMENT=true
+      #- OAUTH_ROLES_CLAIM=roles
+      #- OAUTH_ADMIN_ROLES=admin,internal,superuser
+      - OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID}
+      - OAUTH_CLIENT_SECRET=${OAUTH_CLIENT_SECRET}
+      - OPENID_PROVIDER_URL=https://auth.gbrown.org/application/o/openwebui/.well-known/openid-configuration
+      - OPENID_REDIRECT_URI=https://chat.gbrown.org/oauth/oidc/callback
+      - OAUTH_PROVIDER_NAME=Gib's Auth
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - ./volumes/openwebui:/app/backend/data
+      #- /home/gib/Public/Server_Backup/Ollama_Models:/root/.ollama
+    #devices:
+    #- /dev/dri:/dev/dri
+    tty: true
+    restart: unless-stopped
+    #security_opt: ['seccomp:unconfined','label:disable']
+    #deploy:
+    #resources:
+    #reservations:
+    #devices:
+    #- driver: nvidia
+    #count: 1
+    #capabilities: [gpu]
+
+  ollama:
+    #image: ollama/ollama:rocm
+    image: ollama/ollama:latest
+    container_name: ollama
+    hostname: ollama
+    domainname: ollama.gbrown.org
+    networks:
+      - nginx-bridge
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    ports:
+      - 11434:11434
+    volumes:
+      - /home/gib/Public/Server_Backup/Ollama_Models:/root/.ollama
+    devices:
+      - /dev/dri:/dev/dri
+    tty: true
+    restart: unless-stopped
+    security_opt: ["seccomp:unconfined", "label:disable"]
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  comfyui:
+    #image: yanwk/comfyui-boot:rocm
+    image: yanwk/comfyui-boot:cu128-megapak
+    init: true
+    container_name: comfyui
+    hostname: comfyui
+    domainname: comfyui.gbrown.org
+    networks:
+      - nginx-bridge
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    #ports: [8188:8188]
+    volumes:
+      - /home/gib/Public/Server_Backup/ComfyUI/:/root
+    #environment:
+    #- HSA_OVERRIDE_GFX_VERSION=11.0.0
+    #- CLI_ARGS=''
+    devices:
+      - /dev/dri:/dev/dri
+      #- /dev/kfd:/dev/kfd
+    tty: true
+    security_opt: ["seccomp:unconfined", "label:disable"]
+    restart: unless-stopped
+    group_add: [video]
+    #cap_add: [SYS_PTRACE]
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  mcpo:
+    image: ghcr.io/open-webui/mcpo:main
+    container_name: mcpo
+    hostname: mcpo.gbrown.org
+    domainname: mcpo.gbrown.org
+    command: 'mcpo --port 8010 --config /app/config.json --api-key "Tragedy8-Unfazed-Craftily-Hedging-Gem-Velcro"'
+    restart: unless-stopped
+    tty: true
+    networks:
+      - nginx-bridge
+    ports:
+      - 8010:8010
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    env_file:
+      - .env
+    volumes:
+      - ./volumes/mcpo/openapi.json:/app/config.json
+      - ./volumes/mcpo/bin/gitea-mcp:/usr/local/bin/gitea-mcp
+      - ./volumes/mcpo/bin/bw:/usr/local/bin/bw

server/home/OpenWebUI/podman-openwebui.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: OpenWebUI
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/OpenWebUI
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/Plausible/.env.example

@@ -0,0 +1,8 @@
+BASE_URL=CHANGE_ME
+SECRET_KEY_BASE=CHANGE_ME
+HTTP_PORT=CHANGE_ME
+HTTPS_PORT=CHANGE_ME
+SMTP_USER_NAME=CHANGE_ME
+SMTP_USER_PWD=CHANGE_ME
+POSTGRES_PASSWORD=CHANGE_ME
+TOTP_VAULT_KEY=CHANGE_ME

server/home/Plausible/UPSTREAM

@@ -0,0 +1,6 @@
+checkout: ~/Server/Plausible/
+repo:     https://github.com/plausible/community-edition
+ref:      tag v3.0.1
+commit:   20385b05aa8e5645c88fc1a526360608e0fba324
+
+compose.yml here is the local, modified copy of the upstream file.

server/home/Plausible/compose.yml

@@ -0,0 +1,131 @@
+networks:
+  nginx-bridge:
+    external: true
+
+services:
+  plausible_db:
+    image: postgres:16-alpine
+    container_name: plausible_db
+    hostname: plausible_db
+    networks: [nginx-bridge]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - db-data:/var/lib/postgresql/data
+    environment:
+      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U postgres"]
+      start_period: 1m
+    restart: always
+    tty: true
+
+  plausible_events_db:
+    image: clickhouse/clickhouse-server:24.12-alpine
+    container_name: plausible_events_db
+    hostname: plausible_events_db
+    networks: [nginx-bridge]
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - event-data:/var/lib/clickhouse
+      - event-logs:/var/log/clickhouse-server
+      - ./clickhouse/logs.xml:/etc/clickhouse-server/config.d/logs.xml:ro
+      # This makes ClickHouse bind to IPv4 only, since Docker doesn't enable IPv6 in bridge networks by default.
+      # Fixes "Listen [::]:9000 failed: Address family for hostname not supported" warnings.
+      - ./clickhouse/ipv4-only.xml:/etc/clickhouse-server/config.d/ipv4-only.xml:ro
+      # This makes ClickHouse consume less resources, which is useful for small setups.
+      # https://clickhouse.com/docs/en/operations/tips#using-less-than-16gb-of-ram
+      #- ./clickhouse/low-resources.xml:/etc/clickhouse-server/config.d/low-resources.xml:ro
+    ulimits:
+      nofile:
+        soft: 262144
+        hard: 262144
+    environment:
+      - CLICKHOUSE_SKIP_USER_SETUP=1
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          "wget --no-verbose --tries=1 -O - http://127.0.0.1:8123/ping || exit 1",
+        ]
+      start_period: 1m
+    restart: always
+    tty: true
+
+  plausible:
+    image: ghcr.io/plausible/community-edition:v3.2.1
+    container_name: plausible
+    hostname: plausible
+    networks: [nginx-bridge]
+    #ports:
+    #- ${HTTP_PORT}:8000
+    #- ${HTTPS_PORT}:443
+    command: sh -c "/entrypoint.sh db createdb && /entrypoint.sh db migrate && /entrypoint.sh run"
+    depends_on:
+      plausible_db:
+        condition: service_healthy
+      plausible_events_db:
+        condition: service_healthy
+    labels:
+      com.centurylinklabs.watchtower.enable: "true"
+    volumes:
+      - plausible-data:/var/lib/plausible
+    ulimits:
+      nofile:
+        soft: 65535
+        hard: 65535
+    extra_hosts:
+      # Pin the relay to the WireGuard address. Docker containers do NOT read the host
+      # /etc/hosts, and the public *.gbrown.org wildcard still returns Cloudflare AAAA
+      # records alongside the UniFi A record -- so without this, delivery depends on
+      # resolver address-ordering. The name (not the IP) is used so the wildcard cert
+      # still validates by hostname.
+      - "smtp.usesend.gbrown.org:192.168.2.2"
+    environment:
+      - TMPDIR=/var/lib/plausible/tmp
+      # required: https://github.com/plausible/community-edition/wiki/configuration#required
+      - BASE_URL=${BASE_URL}
+      - SECRET_KEY_BASE=${SECRET_KEY_BASE}
+      # optional: https://github.com/plausible/community-edition/wiki/configuration#optional
+      # registration: https://github.com/plausible/community-edition/wiki/configuration#registration
+      - TOTP_VAULT_KEY=${TOTP_VAULT_KEY}
+      - DISABLE_REGISTRATION=invite_only
+      - ENABLE_EMAIL_VERIFICATION=true
+      # web: https://github.com/plausible/community-edition/wiki/configuration#web
+      - HTTP_PORT=${HTTP_PORT}
+      - HTTPS_PORT=${HTTPS_PORT}
+      # databases: https://github.com/plausible/community-edition/wiki/configuration#database
+      - DATABASE_URL
+      - CLICKHOUSE_DATABASE_URL
+      # Google: https://github.com/plausible/community-edition/wiki/configuration#google
+      - GOOGLE_CLIENT_ID
+      - GOOGLE_CLIENT_SECRET
+      # geolocation: https://github.com/plausible/community-edition/wiki/configuration#ip-geolocation
+      - IP_GEOLOCATION_DB
+      - GEONAMES_SOURCE_FILE
+      - MAXMIND_LICENSE_KEY
+      - MAXMIND_EDITION
+      # email: https://github.com/plausible/community-edition/wiki/configuration#email
+      - MAILER_ADAPTER
+      - [email protected]
+      - MAILER_NAME=Gib's Plausible Analytics
+      - SMTP_HOST_ADDR=smtp.usesend.gbrown.org
+      - SMTP_HOST_PORT=587
+      - SMTP_USER_NAME=${SMTP_USER_NAME}
+      - SMTP_USER_PWD=${SMTP_USER_PWD}
+      - SMTP_HOST_SSL_ENABLED=false
+      - POSTMARK_API_KEY
+      - MAILGUN_API_KEY
+      - MAILGUN_DOMAIN
+      - MAILGUN_BASE_URI
+      - MANDRILL_API_KEY
+      - SENDGRID_API_KEY
+    restart: always
+    tty: true
+
+volumes:
+  db-data:
+  event-data:
+  event-logs:
+  plausible-data:

server/home/Plausible/podman-plausible.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: Plausible
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/Plausible
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/PostHog/.env.example

@@ -0,0 +1,29 @@
+POSTHOG_SECRET=CHANGE_ME
+ENCRYPTION_SALT_KEYS=CHANGE_ME
+BROWSERLESS_SECRET=CHANGE_ME
+DOMAIN=CHANGE_ME
+REGISTRY_URL=CHANGE_ME
+POSTHOG_APP_TAG=CHANGE_ME
+TLS_BLOCK=CHANGE_ME
+CADDY_TLS_BLOCK=CHANGE_ME
+CADDY_HOST=CHANGE_ME
+POSTHOG_HTTP_PORT=CHANGE_ME
+ANTHROPIC_API_KEY=CHANGE_ME
+OPENAI_API_KEY=CHANGE_ME
+USESEND_API_KEY=CHANGE_ME
+POSTHOG_GITLAB_LOGIN_KEY=CHANGE_ME
+POSTHOG_GITLAB_LOGIN_SECRET=CHANGE_ME
+SECRET_KEY=CHANGE_ME
+OBJECT_STORAGE_ACCESS_KEY_ID=CHANGE_ME
+OBJECT_STORAGE_SECRET_ACCESS_KEY=CHANGE_ME
+SESSION_RECORDING_V2_S3_ACCESS_KEY_ID=CHANGE_ME
+SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY=CHANGE_ME
+INTERNAL_API_SECRET=CHANGE_ME
+CSP_REPORT_ENDPOINT=CHANGE_ME
+DOCKER_REGISTRY_PREFIX=CHANGE_ME
+ELAPSED=CHANGE_ME
+HEATMAP_BROWSERLESS_SCREENSHOT_COOKIES_ENABLED=CHANGE_ME
+OPT_OUT_CAPTURE=CHANGE_ME
+POSTHOG_NODE_TAG=CHANGE_ME
+SEAWEEDFS_DOCKER_NAME=CHANGE_ME
+TIMEOUT=CHANGE_ME

server/home/PostHog/README.md

@@ -0,0 +1,391 @@
+# PostHog (self-hosted) on Fedora
+
+PostHog's official installer (`bin/deploy-hobby`) assumes Ubuntu/Debian and a
+bare host. This directory is that install done by hand, adapted to this server.
+
+## Why the official installer doesn't work here
+
+| What `deploy-hobby` does | Why it fails / what we did instead |
+|---|---|
+| `apt update`, `apt install git brotli curl` | Not Debian. All three are already installed via dnf. |
+| Installs Docker from `download.docker.com/linux/ubuntu` | Docker 29.8 is already installed. |
+| Downloads the v1 `docker-compose` binary to `/usr/local/bin` | We use the Compose v5 plugin (`docker compose`). |
+| Runs Caddy on host ports `80:80` / `443:443` and gets its own Let's Encrypt certs | nginx-proxy-manager owns 80/443. Caddy now serves plain HTTP on `127.0.0.1:8002` and NPM terminates TLS in front of it. |
+| Publishes Temporal on `0.0.0.0:7233` and its UI on `0.0.0.0:8081` | Rebound to `127.0.0.1`. |
+| Phones home to PostHog Cloud with your domain | Skipped. |
+| Generates `compose/start` calling `./compose/wait` | Upstream bug, fixed here. WORKDIR is `/code` but the mount is at `/compose`, so the relative path never resolves. Harmless (`bin/migrate` retries 10x) but it meant the readiness check never ran. Ours calls `/compose/wait`. |
+
+Note: Docker here runs *without* SELinux labelling (`docker info` lists only
+seccomp and cgroupns), so the bind mounts need no `:z` / `:Z` relabelling even
+though the host is SELinux-enforcing.
+
+Caddy is kept rather than replaced by NPM because it does path-based routing
+that NPM can't express in one proxy host: `/e`, `/i/v0`, `/batch`, `/capture`
+go to `capture`; `/s` to `replay-capture`; `/i/v1/logs` to `capture-logs`;
+`/livestream` to `livestream`; `/posthog/*` to object storage; everything else
+to `web`. NPM just forwards the whole hostname to Caddy.
+
+## Layout
+
+    compose.yml               docker-compose.hobby.yml + local patches
+    compose.override.yml      additive local settings, auto-merged by compose
+    docker-compose.base.yml   verbatim from the repo (compose.yml `extends` it)
+    .env                      secrets + domain  (chmod 600, never regenerate)
+    .env.services             verbatim from the repo
+    posthog/                  upstream checkout, only used for config bind mounts
+    share/                    GeoLite2-City.mmdb
+    compose/                  entrypoint scripts the web/worker containers run,
+                              plus local-patches.py (see "Multiple projects" below)
+    bin/upgrade               Fedora-native replacement for bin/upgrade-hobby
+    bin/apply-local-patches.py  replays our compose.yml deltas after an upgrade
+
+## nginx-proxy-manager
+
+Add a proxy host:
+
+- Domain: `posthog.gbrown.org`
+- Scheme: `http`, Forward host: `posthog`, Forward port: `80`
+- **Websockets: ON** (required — the `/livestream` endpoint)
+- Request an SSL cert, force SSL
+
+`posthog` is a network alias on `nginx-bridge` for the Caddy container.
+
+Also worth raising NPM's `client_max_body_size` for this host (Advanced tab:
+`client_max_body_size 50m;`) — session-recording batches and source-map
+uploads exceed the 1 MB default.
+
+## Two config files, two reasons
+
+`compose.yml` holds changes that *replace* upstream values -- the proxy's ports
+and networks. These have to be patched into the file directly, because Compose
+merges list-valued keys like `ports` by appending: an override file saying
+`8002:80` would leave you with 80, 443 *and* 8002. `bin/apply-local-patches.py`
+replays them after an upgrade overwrites the file.
+
+`compose.override.yml` holds purely *additive* settings (currently
+`TRUST_ALL_PROXIES` for the three Django services). Compose merges it
+automatically and no upgrade ever touches it, so nothing needs replaying.
+
+Rule of thumb: if you're adding an env var, put it in the override file. If
+you're changing a port or a network, patch compose.yml and add it to
+`bin/apply-local-patches.py`. `command` is the exception: Compose replaces it
+on merge rather than appending, so the worker's command lives in the override.
+
+## Operating it
+
+    docker compose up -d           # start
+    docker compose stop            # stop (keeps volumes)
+    docker compose logs -f web     # watch migrations / app
+    docker compose ps              # status
+    bin/upgrade                    # upgrade (runs async migration checks first)
+
+First boot runs Django + ClickHouse migrations and takes 5-15 minutes before
+`/_health` returns 200. The first person to visit the site creates the admin
+account, so log in promptly.
+
+## Multiple projects (source patches)
+
+Without a license, upstream PostHog allows one project per org and hides the
+"New project" button. PostHog no longer sells self-hosted licenses, so
+`compose/local-patches.py` patches two files in `/code` when `web`, `worker`
+and `temporal-django-worker` start:
+
+- `posthog/utils.py`: `can_create_org` follows `MULTI_ORG_ENABLED` alone, which
+  shows the button.
+- `posthog/models/organization.py`: the org keeps the `organizations_projects`
+  feature, which the project API checks. Without this, the hourly
+  `sync_all_organization_available_product_features` task resets the org's
+  features to `[]` and the API refuses a second project.
+
+Both are gated on `MULTI_ORG_ENABLED` in `compose.override.yml`. Because they run
+at container start, recreating a container no longer loses them. An earlier
+version patched running containers by hand and was wiped when the containers
+were recreated on 2026-09-22.
+
+If upstream changes the code, the script logs `PATTERN NOT FOUND` and the app
+starts unpatched. Check after upgrades (`bin/upgrade` prints it):
+
+    docker compose logs web worker | grep LOCAL-PATCH
+
+## Rebooting the host
+
+A host boot is not `docker compose up`. Dockerd starts containers from each one's
+restart policy alone, in no order, and never reads `depends_on`, so the dependency
+ordering the stack is built around does not exist at boot time.
+
+Upstream ships `restart: no` or `on-failure` on nearly every service, and
+`on-failure` does not retry a clean exit. Several services exit *zero* on purpose:
+`hypercache-server` when it loses the race to `redis7` on a cold boot, and
+`capture` when it loses Kafka (on 2026-09-27 a host memory spike stalled Kafka,
+capture shut down gracefully, and it stayed down for three days). So
+`compose.override.yml` sets `restart: unless-stopped` on every long-running
+service. A manual `docker compose stop` is still respected.
+
+Expect `hypercache-server` to flap a few times early in boot until `redis7`
+answers. That is the retry working, not a fault.
+
+`kafka-init` sitting at `Exited (0)` is correct. It is a one-shot init job and
+must keep `restart: no`.
+
+The web UI serves fine while these four are missing, so check
+`docker ps -a --filter name=posthog` for anything not `Up` before assuming a
+reboot went cleanly. To bring back anything stopped, in dependency order, without
+recreating running containers:
+
+    docker compose up -d --no-recreate
+
+## Watchtower is deliberately not enabled
+
+Every other stack here carries `com.centurylinklabs.watchtower.enable=true`.
+PostHog does not, on purpose: its upgrades run schema migrations that must be
+checked with `asyncmigrationscheck` before the app restarts, and a silent image
+swap underneath a running ClickHouse/Kafka pipeline can corrupt data. Use
+`bin/upgrade` instead.
+
+## Resource notes
+
+This stack is 37 containers (ClickHouse, Kafka, Zookeeper, Postgres, Redis,
+Valkey, Temporal, Elasticsearch, SeaweedFS, Browserless, and ~12 Rust/Node
+consumers). PostHog asks for 8 GB minimum; with Sentry already running here,
+keep an eye on `free -h`.
+
+If memory gets tight, these can be commented out of `compose.yml` with little
+loss on a personal instance:
+
+- `elasticsearch` — Temporal is configured with `ENABLE_ES=false`, so it is
+  only there for Temporal's advanced visibility. Saves ~1 GB.
+- `temporal-ui` — debugging UI for Temporal workflows only.
+- `cymbal` / `cymbal-resolution` — error-tracking stack-trace symbolication.
+  Skip if you're keeping errors in Sentry rather than PostHog.
+
+## Ad blockers and the hostname
+
+The host is `analytics.gbrown.org`. It was originally `posthog.gbrown.org`, which
+uBlock Origin blocked, leaving the dashboard a blank white page.
+
+The culprit is EasyPrivacy line 2964 (EasyPrivacy is enabled by default in uBO):
+
+    ://posthog.$script,domain=~posthog.com
+
+That is an *unanchored substring* pattern, not a domain rule, so any URL
+containing `://posthog.` matches -- including `https://posthog.gbrown.org/...`.
+The `$script` option is why the symptom was a white page rather than a failed
+connection: the HTML document is a `document` request and loaded fine, but every
+`/static/*.js` was blocked, so React never mounted.
+
+Renaming fixes it. Checked against all five lists uBO enables by default
+(EasyPrivacy, EasyList, uBO filters/privacy/badware, ~160k rules), with controls
+validating in both directions:
+
+    posthog.gbrown.org      BLOCKED (rule above)
+    analytics.gbrown.org    clean
+    stats/insights/metrics  clean
+    bills.gbrown.org        clean (control)
+
+Also clean on the new host: `/static/array.js`, `/static/recorder.js`,
+`/static/recorder-v2.js`, `/e/`, `/i/v0/e/`, `/batch/`, `/capture/`, `/decide/`.
+
+Generic words like "analytics" survive because the lists anchor to specific
+vendor domains (`||google-analytics.com`) rather than bare words -- blocking the
+word outright would break the web. The `://posthog.` rule is the unusual one.
+
+### To change the hostname again
+
+1. Edit `DOMAIN` in `.env`.
+2. Add the proxy host in nginx-proxy-manager: http -> `posthog` -> 80,
+   websockets on, `client_max_body_size 50m;`.
+3. `docker compose up -d`.
+
+Caddy listens on `:80` regardless of hostname, so nothing inside the stack needs
+touching -- that is the payoff of terminating TLS at NPM instead of in Caddy.
+Note that `SITE_URL` feeds `CSRF_TRUSTED_ORIGINS`, so existing browser sessions
+may need a fresh login after a rename. Capture/API traffic is unaffected.
+
+Still worth proxying capture through the app's own origin (a Next.js `/ingest`
+rewrite): it is same-origin, so no CORS and no preflight, and it survives future
+filter-list changes regardless of what this host is called.
+
+## Capture settings for the bill-tracker app (decided 2026-09-17)
+
+The bill tracker (Next + Expo + Convex) sends to this instance. Gib asked for
+everything on, so both clients ship with session replay, autocapture, heatmaps,
+console log capture and exception capture ENABLED, on posthog-js defaults:
+
+    maskAllInputs: true        (default — form inputs are masked)
+    rendered text: NOT masked  (default)
+
+The tradeoff, recorded deliberately rather than discovered later: this is a
+finance app, so bill amounts, balances, payee names and due dates are rendered
+*text*, not inputs, and therefore land in session replays verbatim. Autocapture
+also stores the text of clicked elements, so a button reading "Pay $1,247.33 to
+Chase" becomes an event property. Console log capture ships whatever the app
+logs, which in dev is often a whole transaction object.
+
+This is considered acceptable because the instance is self-hosted on Gib's own
+hardware and serves his own household — the data never leaves the house.
+
+To tighten later:
+
+    maskTextSelector: '*'                  // or scope to the money elements
+    class="ph-no-capture"                  // on balance / amount / payee nodes
+
+One asymmetry worth remembering: tightening does NOT retroactively clean stored
+replays. Anything recorded before the change stays recorded, so the cheap moment
+to decide is before real household data flows, not after.
+
+Client config lives in the bill-tracker repo, not here. The host is read from
+NEXT_PUBLIC_POSTHOG_HOST / EXPO_PUBLIC_POSTHOG_HOST (Infisical), and web routes
+capture through a same-origin /ingest rewrite rather than hitting this host
+directly.
+
+## Blocked asset filenames — why the client requests ext-a.js / ext-b.js
+
+Renaming the host to `analytics.gbrown.org` fixed the dashboard, but two of
+PostHog's own script filenames are blocked by rules that match the FILENAME, not
+the host, so a reverse proxy cannot dodge them:
+
+    easyprivacy.txt line 1906:   /posthog-recorder.js
+    easyprivacy.txt line  809:   /dead-clicks-autocapture.js
+
+Both are unanchored substrings with no type or domain options, so they match on
+any host, any request type, first-party included. Confirmed blocked both direct
+and through the app's /ingest proxy. Symptom: session replay and dead-click
+capture silently never start for anyone running uBlock.
+
+The bill-tracker app works around this with posthog-js's
+`prepare_external_dependency_script` hook, serving them under inert names:
+
+    posthog-recorder.js        -> ext-a.js
+    dead-clicks-autocapture.js -> ext-b.js
+
+(`strict_script_versioning` off, so only the plain `/static/<name>.js?v=` form is
+requested.) The Next rewrite maps the disguised names back. Verified: replay
+blobs land in `posthog.session_replay_events`, which was stuck at 0 before.
+
+Everything else PostHog serves is clean on both paths — `array.js`,
+`array.full.js`, `recorder.js`, `recorder-v2.js`, `surveys.js`,
+`exception-autocapture.js`, `web-vitals-with-attribution.js`,
+`tracing-headers.js`, `toolbar.js`, `external-scripts-loader.js`, `/e/`,
+`/flags/`. Note the legacy `recorder.js` and `recorder-v2.js` names are NOT
+listed; only the newer `posthog-recorder.js` is.
+
+One more, shape-dependent: `/s/array/<token>/config` is blocked by
+ubo-privacy.txt line 3159 (`$script,xhr,1p` — first-party too), but
+`/array/<token>/config.js` is clean. posthog-js 1.433.10 requests the latter, so
+it does not bite today. Worth re-checking after a major posthog-js upgrade.
+
+### Expected absences (not bugs)
+
+- `$pageleave` does NOT fire on client-side SPA navigation. posthog-js hangs it
+  off the real unload/visibilitychange lifecycle, so it tracks actual page exits
+  only. Verified against every pageleave in the instance.
+- `$web_vitals` flushes on page hide; `$dead_click` needs a click that resolves
+  to nothing. Both stay absent until those conditions occur.
+
+## nginx-proxy-manager buffer sizes (required, or replay breaks)
+
+Symptom: the Session Replay list page shows "Error while trying to load
+recordings". `GET /api/environments/<id>/session_recordings` returns 502 from
+openresty in ~130ms, while a SINGLE recording
+(`.../session_recordings/<session_id>`) returns 200 and `/events` returns 200.
+
+Cause is nginx, not PostHog. From `/data/logs/proxy-host-121_error.log`:
+
+    upstream sent too big header while reading response header from upstream
+
+nginx's default `proxy_buffer_size` is 4-8KB. PostHog's session_recordings LIST
+endpoint sends response headers larger than that, so nginx aborts before reading
+the body and returns 502. The request never reaches Django, which is why nothing
+appears in the web container logs and Caddy logs no 5xx at all. The single
+recording endpoint sends normal-sized headers, hence the asymmetry.
+
+Required in the NPM Advanced tab for this proxy host:
+
+    client_max_body_size 50m;
+    proxy_buffering on;
+    proxy_buffer_size 128k;
+    proxy_buffers 4 256k;
+
+The Sentry proxy host (1.conf) already carried the same three buffer directives —
+same nginx default, same fix, just never applied to PostHog's host.
+
+Verified in `/data/logs/proxy-host-121_access.log`: 502 at 22:46:06 before the
+change, 200 at 22:46:36 after, no further "too big header" entries.
+
+Do NOT edit the generated `.conf` files directly — NPM regenerates them from its
+own database, so a direct edit reverts on the next save.
+
+### Verifying replay from the server side
+
+ClickHouse `session_replay_events` proves events ARRIVED; it does not prove they
+can be PLAYED BACK. Those are different paths (ingestion -> Kafka -> object
+storage vs. API -> proxy -> browser). To check the storage side:
+
+    docker compose exec seaweedfs sh -c \
+      "echo 'fs.ls -l /buckets/posthog/session_recordings/5y' | weed shell -master=localhost:9333"
+
+Blobs live under `session_recordings/<retention>/`, named `<epoch_ms>-<hash>`.
+
+## "No recordings found" with replay working fine
+
+The Session Replay list ships a DEFAULT filter of `> 5 active seconds`. Any
+session with 5 or fewer *active* seconds is hidden, even though it recorded
+perfectly and plays back fine.
+
+Active time is not wall-clock time. The first verified session here ran 458s
+(7.6 min) but logged only 643ms of activity — one click and a little mouse
+movement in an otherwise idle tab. PostHog counts interaction, not duration,
+deliberately, to suppress bounces and accidental loads.
+
+That default makes quick dev checks vanish and looks identical to "replay is
+broken". To see them: open the duration chip on /replay/home and set the
+condition to `> 0 active seconds`, or save a filter preset. Date range and the
+"filter out internal and test users" toggle are usually NOT the culprit.
+
+Check the real numbers server-side before assuming anything is missing:
+
+    docker compose exec clickhouse clickhouse-client -q \
+      "select session_id, sum(active_milliseconds) active_ms,
+              dateDiff('second', min(min_first_timestamp), max(max_last_timestamp)) duration_s,
+              sum(click_count) clicks, sum(event_count) evts
+       from posthog.session_replay_events group by session_id"
+
+Verified 2026-09-17: session 01a0b177-d35c-7292-ac59-021af608d29a recorded,
+listed once the threshold was lowered, and played back in the browser.
+
+## Real client IPs through NPM -> Caddy
+
+By default Django logged `x_forwarded_for=172.25.0.8` — nginx-proxy-manager's own
+container IP — so every event's `$ip` resolved to the proxy and GeoIP failed on a
+private address.
+
+NPM was not at fault; it sends `X-Forwarded-For $proxy_add_x_forwarded_for` and
+`X-Real-IP $remote_addr` from its global proxy include. The loss was in Caddy:
+its global options block was empty, so `trusted_proxies` was unset, and Caddy's
+default is to distrust an incoming X-Forwarded-For and OVERWRITE it with the
+direct peer. Proven by sending `X-Forwarded-For: 9.9.9.9` to Caddy and watching
+Django log `172.25.0.1`.
+
+Fixed in `.env` — no compose patch, because `CADDY_TLS_BLOCK` is substituted
+directly into Caddy's global options block:
+
+    CADDY_TLS_BLOCK="servers {
+    	trusted_proxies static 172.25.0.0/24
+    }"
+
+Multi-line values work in `.env`; `docker compose config` resolves it correctly
+and `caddy validate` accepts it. Apply with `docker compose up -d proxy`.
+
+Verified after: a spoofed `X-Forwarded-For: 9.9.9.9` arrives as `ip=9.9.9.9`, and
+a real request through NPM arrives as the true client address.
+
+TRADEOFF: `/24` trusts every container on nginx-bridge (~42 of them), any of
+which could forge an X-Forwarded-For that Caddy now believes — compounded by
+`TRUST_ALL_PROXIES=true` on Django (see compose.override.yml). Acceptable for a
+LAN-only household instance; tighten to NPM's address if this is ever exposed to
+the internet. Pinning `172.25.0.8/32` is narrower but breaks silently whenever
+Docker reassigns NPM's IP.
+
+Note this does NOT produce geolocation: LAN clients are 192.168.x.x, which is not
+in the MaxMind database, so GeoIP still errors. What it buys is the ability to
+tell devices apart.

server/home/PostHog/UPSTREAM

@@ -0,0 +1,6 @@
+checkout: ~/Server/PostHog/posthog/
+repo:     https://github.com/PostHog/posthog
+ref:      master at this commit
+commit:   ddfaf8843fca8e796bf1f795976842d2d1adb380
+
+docker-compose.base.yml is the upstream file at that commit, unchanged.

server/home/PostHog/bin/apply-local-patches.py

@@ -0,0 +1,57 @@
+#!/usr/bin/env python3
+"""Re-apply this server's deltas to a freshly-copied docker-compose.hobby.yml.
+
+Upstream's bin/upgrade-hobby overwrites compose.yml from the repo on every
+upgrade, so these three changes have to be replayed afterwards. Idempotent:
+running it on an already-patched file is a no-op.
+
+Usage: bin/apply-local-patches.py [compose.yml]
+"""
+import sys
+
+path = sys.argv[1] if len(sys.argv) > 1 else 'compose.yml'
+s = open(path).read()
+changed = []
+
+# 1) nginx-proxy-manager owns host 80/443. Publish Caddy on loopback instead and
+#    put it on nginx-bridge so NPM can reach it by name.
+old = """        ports:
+            - '80:80'
+            - '443:443'
+"""
+new = """        ports:
+            # nginx-proxy-manager owns host 80/443. Loopback only; NPM reaches
+            # this container over nginx-bridge as "posthog".
+            - '127.0.0.1:${POSTHOG_HTTP_PORT:-8002}:80'
+        networks:
+            default:
+            nginx-bridge:
+                aliases:
+                    - posthog
+"""
+if old in s:
+    s = s.replace(old, new)
+    changed.append('proxy ports + nginx-bridge')
+
+# 2) TLS terminates at NPM, so Caddy must not try to get its own certs. Take the
+#    site address from .env (CADDY_HOST=:80) instead of the hardcoded domain.
+old = "            CADDY_HOST: '$DOMAIN, http://, https://'\n"
+if old in s:
+    s = s.replace(old, "            CADDY_HOST: $CADDY_HOST\n")
+    changed.append('CADDY_HOST from .env')
+
+# 3) Temporal gRPC + UI on loopback, not every interface.
+for a, b in (("            - 7233:7233\n", "            - 127.0.0.1:7233:7233\n"),
+             ("            - 8081:8080\n", "            - 127.0.0.1:8081:8080\n")):
+    if a in s:
+        s = s.replace(a, b)
+        changed.append(a.strip())
+
+# 4) nginx-bridge is created and owned by ~/Server/Nginx.
+old = "networks:\n    otel_network:\n"
+if old in s and 'nginx-bridge:\n        external: true' not in s:
+    s = s.replace(old, "networks:\n    otel_network:\n    nginx-bridge:\n        external: true\n")
+    changed.append('nginx-bridge external')
+
+open(path, 'w').write(s)
+print(f"{path}: " + (", ".join(changed) if changed else "already patched, no changes"))

server/home/PostHog/bin/upgrade

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+#
+# Fedora-native replacement for posthog/bin/upgrade-hobby.
+#
+# Upstream's script is unusable here: it calls apt-get, re-downloads the v1
+# `docker-compose` binary, and overwrites compose.yml from the repo (throwing
+# away this server's nginx-proxy-manager and port changes). This does the same
+# work using dnf-era tooling and replays the local patches afterwards.
+#
+set -euo pipefail
+cd "$(dirname "$0")/.."
+
+echo "==> Refreshing the PostHog checkout"
+git -C posthog fetch origin
+git -C posthog reset --hard origin/master
+echo "    now on $(git -C posthog rev-parse --short HEAD)"
+
+echo "==> Refreshing compose files and re-applying local patches"
+cp posthog/docker-compose.base.yml docker-compose.base.yml
+cp posthog/docker-compose.hobby.yml compose.yml
+cp posthog/.env.services .env.services
+bin/apply-local-patches.py compose.yml
+docker compose config >/dev/null && echo "    compose config OK"
+
+echo "==> Refreshing the GeoIP database"
+curl -sL 'https://mmdbcdn.posthog.net/' --http1.1 | brotli --decompress > share/GeoLite2-City.mmdb
+echo "{\"date\": \"$(date +%F)\"}" > share/GeoLite2-City.json
+chmod 644 share/GeoLite2-City.mmdb share/GeoLite2-City.json
+
+echo "==> Pulling images"
+docker compose pull
+
+echo "==> Checking async migrations against the new version"
+docker compose up -d db redis7 clickhouse
+docker compose run --rm asyncmigrationscheck
+
+echo "==> Restarting the stack"
+docker compose up -d --remove-orphans
+
+# web, worker and temporal-django-worker apply compose/local-patches.py on start.
+# Surface the result so an upstream change that breaks a patch is not missed.
+echo "==> Local source patches"
+sleep 15
+docker compose logs --since 5m web worker temporal-django-worker 2>&1 | grep LOCAL-PATCH || \
+    echo "    no LOCAL-PATCH lines yet, check: docker compose logs web worker | grep LOCAL-PATCH"
+
+echo
+echo "Done. Watch the migrations with:  docker compose logs -f web"

server/home/PostHog/compose.override.yml

@@ -0,0 +1,99 @@
+services:
+  web: &id001
+    restart: unless-stopped
+    environment:
+      TRUST_ALL_PROXIES: 'true'
+      MULTI_ORG_ENABLED: 'true'
+      # SSO: the "GitLab" login button is really Authentik. NPM on the VPS maps GitLab's
+      # /oauth/* and /api/v4/user paths on auth.gbrown.org to Authentik, the same shim
+      # Infisical uses. GitLab login needs no license here; only Google is gated.
+      SOCIAL_AUTH_GITLAB_API_URL: https://auth.gbrown.org
+      SOCIAL_AUTH_GITLAB_KEY: ${POSTHOG_GITLAB_LOGIN_KEY:?set POSTHOG_GITLAB_LOGIN_KEY in .env}
+      SOCIAL_AUTH_GITLAB_SECRET: ${POSTHOG_GITLAB_LOGIN_SECRET:?set POSTHOG_GITLAB_LOGIN_SECRET in .env}
+      PERSONHOG_ADDR: personhog-router:50052
+      PERSISTED_FEATURE_FLAGS: sql-editor-vim-mode,cdp-hog-sources,customer-analytics-roadmap,marketing-analytics,web-experiments,dashboard-widgets,twig,tracing,metrics
+      EMAIL_ENABLED: 'true'
+      EMAIL_HOST: smtp.usesend.gbrown.org
+      EMAIL_PORT: '587'
+      EMAIL_HOST_USER: usesend
+      EMAIL_HOST_PASSWORD: ${USESEND_API_KEY:?set USESEND_API_KEY in .env}
+      EMAIL_USE_TLS: 'true'
+      EMAIL_USE_SSL: 'false'
+      EMAIL_DEFAULT_FROM: PostHog <[email protected]>
+      EMAIL_REPLY_TO: [email protected]
+  worker:
+    <<: *id001
+    # compose/worker applies compose/local-patches.py, then runs upstream's
+    # command. `command` is replaced on merge (not appended), so it is safe here.
+    command: /compose/worker
+    volumes:
+      - ./compose:/compose
+  temporal-django-worker: *id001
+  # Every long-running service restarts regardless of exit code. Upstream ships
+  # `on-failure` (or `no`), which never retries a clean exit, and these services
+  # exit 0 on purpose: capture shuts down gracefully when it loses Kafka
+  # (2026-09-27, down 3 days), hypercache-server exits 0 when it beats redis7 at
+  # boot. web/worker/temporal-django-worker get this via the anchor above.
+  # kafka-init and asyncmigrationscheck are one-shots and must stay `no`.
+  browserless:
+    restart: unless-stopped
+  capture:
+    restart: unless-stopped
+  capture-logs:
+    restart: unless-stopped
+  clickhouse:
+    restart: unless-stopped
+  cymbal:
+    restart: unless-stopped
+  cymbal-resolution:
+    restart: unless-stopped
+  db:
+    restart: unless-stopped
+  elasticsearch:
+    restart: unless-stopped
+  feature-flags:
+    restart: unless-stopped
+  hypercache-server:
+    restart: unless-stopped
+  ingestion-error-tracking:
+    restart: unless-stopped
+  ingestion-general:
+    restart: unless-stopped
+  ingestion-logs:
+    restart: unless-stopped
+  ingestion-sessionreplay:
+    restart: unless-stopped
+  ingestion-traces:
+    restart: unless-stopped
+  kafka:
+    restart: unless-stopped
+  livestream:
+    restart: unless-stopped
+  objectstorage:
+    restart: unless-stopped
+  personhog-replica:
+    restart: unless-stopped
+  personhog-router:
+    restart: unless-stopped
+  plugins:
+    restart: unless-stopped
+  property-defs-rs:
+    restart: unless-stopped
+  recording-api:
+    restart: unless-stopped
+  redis7:
+    restart: unless-stopped
+  replay-capture:
+    restart: unless-stopped
+  seaweedfs:
+    restart: unless-stopped
+  temporal:
+    restart: unless-stopped
+  temporal-admin-tools:
+    restart: unless-stopped
+  temporal-ui:
+    restart: unless-stopped
+  valkey:
+    restart: unless-stopped
+  zookeeper:
+    restart: unless-stopped

server/home/PostHog/compose.yml

@@ -0,0 +1,767 @@
+#
+# `docker-compose` file used ONLY for hobby deployments.
+#
+# Please take a look at https://posthog.com/docs/self-host/deploy/hobby
+# for more info.
+#
+# PostHog has sunset support for self-hosted K8s deployments.
+# See: https://posthog.com/blog/sunsetting-helm-support-posthog
+#
+
+x-logging: &default-logging
+    driver: json-file
+    options:
+        max-size: '50m'
+        max-file: '3'
+
+services:
+    db:
+        extends:
+            file: docker-compose.base.yml
+            service: db
+        image: ${DOCKER_REGISTRY_PREFIX:-}postgres:15.12-alpine
+        logging: *default-logging
+        volumes:
+            - postgres-data:/var/lib/postgresql/data
+
+    redis7:
+        extends:
+            file: docker-compose.base.yml
+            service: redis7
+        logging: *default-logging
+        volumes:
+            - redis7-data:/data
+
+    valkey:
+        extends:
+            file: docker-compose.base.yml
+            service: valkey
+        logging: *default-logging
+
+    clickhouse:
+        extends:
+            file: docker-compose.base.yml
+            service: clickhouse
+        restart: on-failure
+        logging: *default-logging
+        depends_on:
+            - kafka
+            - zookeeper
+        volumes:
+            - ./posthog/posthog/idl:/idl
+            - ./posthog/docker/clickhouse/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
+            - ./posthog/docker/clickhouse/config.xml:/etc/clickhouse-server/config.xml
+            - ./posthog/docker/clickhouse/config.d/default.xml:/etc/clickhouse-server/config.d/default.xml
+            - ./posthog/docker/clickhouse/users.xml:/etc/clickhouse-server/users.xml
+            - ./posthog/docker/clickhouse/user_defined_function.xml:/etc/clickhouse-server/user_defined_function.xml
+            - ./posthog/posthog/user_scripts:/var/lib/clickhouse/user_scripts
+            - clickhouse-data:/var/lib/clickhouse
+
+    zookeeper:
+        extends:
+            file: docker-compose.base.yml
+            service: zookeeper
+        logging: *default-logging
+        environment:
+            ZOO_AUTOPURGE_PURGEINTERVAL: 1
+            ZOO_AUTOPURGE_SNAPRETAINCOUNT: 3
+        volumes:
+            - zookeeper-datalog:/datalog
+            - zookeeper-data:/data
+            - zookeeper-logs:/logs
+    kafka:
+        extends:
+            file: docker-compose.base.yml
+            service: kafka
+        logging: *default-logging
+        depends_on:
+            - zookeeper
+        environment:
+            KAFKA_LOG_RETENTION_MS: 3600000
+            KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 300000
+            KAFKA_LOG_RETENTION_HOURS: 1
+        volumes:
+            - kafka-data:/bitnami/kafka
+
+    worker:
+        extends:
+            file: docker-compose.base.yml
+            service: worker
+        logging: *default-logging
+        environment:
+            SITE_URL: https://$DOMAIN
+            SECRET_KEY: ${SECRET_KEY}
+            OBJECT_STORAGE_ACCESS_KEY_ID: ${OBJECT_STORAGE_ACCESS_KEY_ID}
+            OBJECT_STORAGE_SECRET_ACCESS_KEY: ${OBJECT_STORAGE_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENDPOINT: http://objectstorage:19000
+            OBJECT_STORAGE_PUBLIC_ENDPOINT: https://$DOMAIN
+            SESSION_RECORDING_V2_S3_ENDPOINT: http://seaweedfs:8333
+            SESSION_RECORDING_V2_S3_ACCESS_KEY_ID: ${SESSION_RECORDING_V2_S3_ACCESS_KEY_ID}
+            SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY: ${SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENABLED: true
+            RECORDING_API_URL: http://recording-api:6738
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET:-$POSTHOG_SECRET}
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            POSTHOG_SKIP_MIGRATION_CHECKS: '1'
+            PERSONHOG_ADDR: 'personhog-router:50052'
+            PERSONHOG_ENABLED: 'true'
+            BROWSERLESS_CDP_URL: 'ws://browserless:3000'
+            BROWSERLESS_TOKEN: ${BROWSERLESS_SECRET:-$POSTHOG_SECRET}
+            HEATMAP_BROWSERLESS_URL: 'http://browserless:3000'
+            HEATMAP_BROWSERLESS_TOKEN: ${BROWSERLESS_SECRET:-$POSTHOG_SECRET}
+            HEATMAP_BROWSERLESS_SCREENSHOT_COOKIES_ENABLED: ${HEATMAP_BROWSERLESS_SCREENSHOT_COOKIES_ENABLED:-false}
+            # Point Django at the Rust feature-flags container; the setting
+            # otherwise defaults to localhost:3001.
+            FEATURE_FLAGS_SERVICE_URL: 'http://feature-flags:3001'
+            # The logs product uses a dedicated ClickHouse cluster config (separate
+            # from CLICKHOUSE_HOST) whose host defaults to localhost and whose SECURE
+            # defaults to true outside DEBUG. Point it at the stack's ClickHouse over
+            # plaintext; the other logs-cluster settings already default correctly
+            # (native port 9000, user "default", empty password).
+            CLICKHOUSE_LOGS_CLUSTER_HOST: 'clickhouse'
+            CLICKHOUSE_LOGS_CLUSTER_SECURE: 'false'
+            # AI provider keys for PostHog's AI features. Blank unless set in .env.
+            ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
+            OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+        image: $REGISTRY_URL:$POSTHOG_APP_TAG
+        depends_on:
+            db:
+                condition: service_healthy
+            redis7:
+                condition: service_healthy
+            clickhouse:
+                condition: service_started
+            kafka:
+                condition: service_healthy
+            web:
+                condition: service_started
+            personhog-router:
+                condition: service_started
+            browserless:
+                condition: service_started
+    web:
+        extends:
+            file: docker-compose.base.yml
+            service: web
+        command: /compose/start
+        logging: *default-logging
+        volumes:
+            - ./compose:/compose
+        image: $REGISTRY_URL:$POSTHOG_APP_TAG
+        environment:
+            SITE_URL: https://$DOMAIN
+            LIVESTREAM_HOST: 'https://${DOMAIN}/livestream'
+            SECRET_KEY: ${SECRET_KEY}
+            OBJECT_STORAGE_ACCESS_KEY_ID: ${OBJECT_STORAGE_ACCESS_KEY_ID}
+            OBJECT_STORAGE_SECRET_ACCESS_KEY: ${OBJECT_STORAGE_SECRET_ACCESS_KEY}
+            SESSION_RECORDING_V2_S3_ACCESS_KEY_ID: ${SESSION_RECORDING_V2_S3_ACCESS_KEY_ID}
+            SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY: ${SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENDPOINT: http://objectstorage:19000
+            OBJECT_STORAGE_PUBLIC_ENDPOINT: https://$DOMAIN
+            SESSION_RECORDING_V2_S3_ENDPOINT: http://seaweedfs:8333
+            OBJECT_STORAGE_ENABLED: true
+            RECORDING_API_URL: http://recording-api:6738
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET:-$POSTHOG_SECRET}
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            OTEL_SERVICE_NAME: 'posthog'
+            OTEL_EXPORTER_OTLP_ENDPOINT: ''
+            OTEL_SDK_DISABLED: 'true'
+            OPT_OUT_CAPTURE: ${OPT_OUT_CAPTURE:-false}
+            # Blank sends no CSP reports. Point it at this instance's own /report/ endpoint to
+            # collect them yourself.
+            CSP_REPORT_ENDPOINT: ${CSP_REPORT_ENDPOINT:-}
+            DEPLOYMENT: 'hobby'
+            PERSONHOG_ADDR: 'personhog-router:50052'
+            PERSONHOG_ENABLED: 'true'
+            # Point Django at the Rust feature-flags container; the setting
+            # otherwise defaults to localhost:3001.
+            FEATURE_FLAGS_SERVICE_URL: 'http://feature-flags:3001'
+            # The logs product uses a dedicated ClickHouse cluster config (separate
+            # from CLICKHOUSE_HOST) whose host defaults to localhost and whose SECURE
+            # defaults to true outside DEBUG. Point it at the stack's ClickHouse over
+            # plaintext; the other logs-cluster settings already default correctly
+            # (native port 9000, user "default", empty password).
+            CLICKHOUSE_LOGS_CLUSTER_HOST: 'clickhouse'
+            CLICKHOUSE_LOGS_CLUSTER_SECURE: 'false'
+            # AI provider keys for PostHog's AI features. Blank unless set in .env.
+            ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
+            OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+        depends_on:
+            - db
+            - redis7
+            - clickhouse
+            - kafka
+            - objectstorage
+            - seaweedfs
+            - personhog-router
+
+    plugins:
+        extends:
+            file: docker-compose.base.yml
+            service: plugins
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            SITE_URL: https://$DOMAIN
+            SECRET_KEY: ${SECRET_KEY}
+            OBJECT_STORAGE_ACCESS_KEY_ID: ${OBJECT_STORAGE_ACCESS_KEY_ID}
+            OBJECT_STORAGE_SECRET_ACCESS_KEY: ${OBJECT_STORAGE_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENDPOINT: http://objectstorage:19000
+            OBJECT_STORAGE_PUBLIC_ENDPOINT: https://$DOMAIN
+            OBJECT_STORAGE_ENABLED: true
+            CDP_REDIS_HOST: redis7
+            CDP_REDIS_PORT: 6379
+            LOGS_REDIS_HOST: redis7
+            LOGS_REDIS_PORT: 6379
+            LOGS_REDIS_TLS: 'false'
+            TRACES_REDIS_HOST: redis7
+            TRACES_REDIS_PORT: 6379
+            TRACES_REDIS_TLS: 'false'
+            # Every CDP process dual-writes to Valkey and refuses to start without a host.
+            CDP_VALKEY_HOST: valkey
+            CDP_VALKEY_PORT: 6379
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            PERSONS_DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            BEHAVIORAL_COHORTS_DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            OTEL_EXPORTER_OTLP_ENDPOINT: ''
+        volumes:
+            - ./share:/share
+        depends_on:
+            - db
+            - redis7
+            - valkey
+            - clickhouse
+            - kafka
+            - objectstorage
+            - seaweedfs
+
+    ingestion-general:
+        extends:
+            file: docker-compose.base.yml
+            service: ingestion-general
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            PERSONS_DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            BEHAVIORAL_COHORTS_DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            # The combined-mode AI consumer reads persons through PersonHog
+            PERSONHOG_ADDR: 'personhog-router:50052'
+            PERSONHOG_ENABLED: 'true'
+        depends_on:
+            - db
+            - redis7
+            - clickhouse
+            - kafka
+            # The combined AI consumer's blob store healthchecks the ai-blobs bucket at startup
+            - objectstorage
+            - personhog-router
+
+    ingestion-sessionreplay:
+        extends:
+            file: docker-compose.base.yml
+            service: ingestion-sessionreplay
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            SESSION_RECORDING_V2_S3_ACCESS_KEY_ID: ${SESSION_RECORDING_V2_S3_ACCESS_KEY_ID}
+            SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY: ${SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY}
+            SESSION_RECORDING_V2_S3_TIMEOUT_MS: 120000
+            SESSION_RECORDING_V2_S3_ENDPOINT: http://seaweedfs:8333
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+        depends_on:
+            - db
+            - redis7
+            - kafka
+            - seaweedfs
+
+    recording-api:
+        extends:
+            file: docker-compose.base.yml
+            service: recording-api
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            # Read side of the session recording V2 blob store. Must point at the same
+            # SeaweedFS the ingestion-sessionreplay writer uses — without these the
+            # service falls back to its localhost:8333 default, so every playback
+            # request 500s with ECONNREFUSED ("We're unable to play this recording").
+            SESSION_RECORDING_V2_S3_ACCESS_KEY_ID: ${SESSION_RECORDING_V2_S3_ACCESS_KEY_ID}
+            SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY: ${SESSION_RECORDING_V2_S3_SECRET_ACCESS_KEY}
+            SESSION_RECORDING_V2_S3_ENDPOINT: http://seaweedfs:8333
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            # Must resolve to the same value as the web and worker services, which send it as
+            # X-Internal-Api-Secret. The JWT signing secret is deliberately left unset here, so hobby
+            # stays on the legacy shared secret until that scheme is rolled out.
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET:-$POSTHOG_SECRET}
+        depends_on:
+            - db
+            - redis7
+            - clickhouse
+            - seaweedfs
+
+    ingestion-error-tracking:
+        extends:
+            file: docker-compose.base.yml
+            service: ingestion-error-tracking
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            ENCRYPTION_SALT_KEYS: ${ENCRYPTION_SALT_KEYS}
+            ERROR_TRACKING_CYMBAL_BASE_URL: 'http://cymbal:3302'
+        depends_on:
+            - db
+            - redis7
+            - kafka
+
+    ingestion-logs:
+        extends:
+            file: docker-compose.base.yml
+            service: ingestion-logs
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            LOGS_REDIS_TLS: 'false'
+        depends_on:
+            - db
+            - redis7
+            - kafka
+
+    ingestion-traces:
+        extends:
+            file: docker-compose.base.yml
+            service: ingestion-traces
+        image: ${REGISTRY_URL}-node:${POSTHOG_NODE_TAG:-latest}
+        logging: *default-logging
+        environment:
+            TRACES_REDIS_TLS: 'false'
+        depends_on:
+            - db
+            - redis7
+            - kafka
+
+    proxy:
+        extends:
+            file: docker-compose.base.yml
+            service: proxy
+        logging: *default-logging
+        ports:
+            # nginx-proxy-manager owns host 80/443. Loopback only; NPM reaches
+            # this container over nginx-bridge as "posthog".
+            - '127.0.0.1:${POSTHOG_HTTP_PORT:-8002}:80'
+        networks:
+            default:
+            nginx-bridge:
+                aliases:
+                    - posthog
+        volumes:
+            - caddy-data:/data
+            - caddy-config:/config
+        environment:
+            CADDY_TLS_BLOCK: $TLS_BLOCK
+            CADDY_HOST: $CADDY_HOST
+        depends_on:
+            - web
+            - livestream
+    objectstorage:
+        # SeaweedFS on the same objectstorage:19000 endpoint and credentials this service
+        # served when it ran MinIO, so no other service in this file changes.
+        extends:
+            file: docker-compose.base.yml
+            service: objectstorage
+        restart: on-failure
+        logging: *default-logging
+        # The dev/CI bootstrap this replaces also registers an `anonymous` identity with
+        # Admin rights. Hobby must not, because Caddy proxies /posthog/* straight to this
+        # service: an anonymous identity opens every object to the internet. With only the
+        # posthog identity, SeaweedFS rejects unsigned requests and the presigned URLs
+        # Django hands out keep working. There is no duckgres here, so there is no
+        # ducklake-dev bucket either. The loop holds no shell variables on purpose:
+        # bin/upgrade-hobby pipes this file through envsubst, which collapses a
+        # doubled-dollar variable reference to a bare dollar sign.
+        entrypoint:
+            - /bin/sh
+            - -c
+            - |
+                (
+                    while true; do
+                        printf 's3.configure -access_key=object_storage_root_user -secret_key=object_storage_root_password -user=posthog -actions=Admin -apply\ns3.bucket.create -name posthog\ns3.bucket.create -name ai-blobs\n' \
+                                | /usr/bin/weed shell -master=localhost:19001 >/dev/null 2>&1 || true
+                        if echo 's3.bucket.list' | /usr/bin/weed shell -master=localhost:19001 2>/dev/null | grep -q ai-blobs \
+                                && echo 's3.configure' | /usr/bin/weed shell -master=localhost:19001 2>/dev/null | grep -q object_storage_root_user; then
+                            touch /tmp/objectstorage-ready
+                            echo "Objectstorage SeaweedFS ready with required buckets and identities"
+                            break
+                        fi
+                        sleep 1
+                    done
+                ) &
+                exec /usr/bin/weed "$$@"
+            - --
+        # A new volume. SeaweedFS cannot read the MinIO layout, so the old objects stay on
+        # the `objectstorage` volume until the operator copies them across with
+        # bin/migrate-storage-hobby or deletes the volume.
+        volumes:
+            - objectstorage-data:/data
+        ports:
+            # Localhost only. MinIO published this port on every interface.
+            - '127.0.0.1:19000:19000'
+
+    seaweedfs:
+        extends:
+            file: docker-compose.base.yml
+            service: seaweedfs
+        container_name: '${SEAWEEDFS_DOCKER_NAME:-seaweedfs-main}'
+        restart: on-failure
+        logging: *default-logging
+        ports:
+            - '127.0.0.1:8333:8333' # S3 API
+            - '127.0.0.1:9333:9333' # Master server
+        volumes:
+            - seaweedfs:/data
+        healthcheck:
+            test:
+                [
+                    'CMD',
+                    'sh',
+                    '-c',
+                    "echo 's3.bucket.list' | /usr/bin/weed shell -master=localhost:9333 2>&1 | grep -q posthog",
+                ]
+            interval: 5s
+            timeout: 10s
+            retries: 30
+            start_period: 10s
+
+    browserless:
+        # Renders image exports and heatmap screenshots (no browser ships in the PostHog image).
+        image: ghcr.io/browserless/chromium:v2.51.2
+        restart: on-failure
+        logging: *default-logging
+        environment:
+            TOKEN: ${BROWSERLESS_SECRET:-$POSTHOG_SECRET}
+            LOG_LEVEL: warn
+            DEBUG: '-*'
+
+    asyncmigrationscheck:
+        extends:
+            file: docker-compose.base.yml
+            service: asyncmigrationscheck
+        image: $REGISTRY_URL:$POSTHOG_APP_TAG
+        logging: *default-logging
+        environment:
+            SITE_URL: https://$DOMAIN
+            SECRET_KEY: ${SECRET_KEY}
+            SKIP_ASYNC_MIGRATIONS_SETUP: 0
+
+    # Temporal containers
+    temporal:
+        extends:
+            file: docker-compose.base.yml
+            service: temporal
+        logging: *default-logging
+        environment:
+            - ENABLE_ES=false
+        ports:
+            - 127.0.0.1:7233:7233
+        volumes:
+            - ./posthog/docker/temporal/dynamicconfig:/etc/temporal/config/dynamicconfig
+    elasticsearch:
+        extends:
+            file: docker-compose.base.yml
+            service: elasticsearch
+        logging: *default-logging
+    temporal-admin-tools:
+        extends:
+            file: docker-compose.base.yml
+            service: temporal-admin-tools
+        logging: *default-logging
+        depends_on:
+            - temporal
+    temporal-ui:
+        extends:
+            file: docker-compose.base.yml
+            service: temporal-ui
+        logging: *default-logging
+        ports:
+            - 127.0.0.1:8081:8080
+        depends_on:
+            temporal:
+                condition: service_started
+            db:
+                condition: service_healthy
+    temporal-django-worker:
+        command: /compose/temporal-django-worker
+        extends:
+            file: docker-compose.base.yml
+            service: temporal-django-worker
+        logging: *default-logging
+        volumes:
+            - ./compose:/compose
+        image: $REGISTRY_URL:$POSTHOG_APP_TAG
+        environment:
+            SITE_URL: https://$DOMAIN
+            SECRET_KEY: ${SECRET_KEY}
+            BROWSERLESS_CDP_URL: 'ws://browserless:3000'
+            BROWSERLESS_TOKEN: ${BROWSERLESS_SECRET:-$POSTHOG_SECRET}
+            HEATMAP_BROWSERLESS_URL: 'http://browserless:3000'
+            HEATMAP_BROWSERLESS_TOKEN: ${BROWSERLESS_SECRET:-$POSTHOG_SECRET}
+            HEATMAP_BROWSERLESS_SCREENSHOT_COOKIES_ENABLED: ${HEATMAP_BROWSERLESS_SCREENSHOT_COOKIES_ENABLED:-false}
+            # This worker runs the recording deletion Temporal activities, which call
+            # recording-api. Without these two the calls go out unauthenticated and
+            # recording-api rejects them with a 401.
+            RECORDING_API_URL: http://recording-api:6738
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET:-$POSTHOG_SECRET}
+            # Point Django at the Rust feature-flags container; the setting
+            # otherwise defaults to localhost:3001.
+            FEATURE_FLAGS_SERVICE_URL: 'http://feature-flags:3001'
+            # The logs product uses a dedicated ClickHouse cluster config (separate
+            # from CLICKHOUSE_HOST) whose host defaults to localhost and whose SECURE
+            # defaults to true outside DEBUG. Point it at the stack's ClickHouse over
+            # plaintext; the other logs-cluster settings already default correctly
+            # (native port 9000, user "default", empty password). This worker runs
+            # the logs alert-check Temporal activities, which query ClickHouse under
+            # Workload.LOGS.
+            CLICKHOUSE_LOGS_CLUSTER_HOST: 'clickhouse'
+            CLICKHOUSE_LOGS_CLUSTER_SECURE: 'false'
+            # AI provider keys for PostHog's AI features. Blank unless set in .env.
+            ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
+            OPENAI_API_KEY: ${OPENAI_API_KEY:-}
+        depends_on:
+            - db
+            - redis7
+            - clickhouse
+            - kafka
+            - objectstorage
+            - seaweedfs
+            - temporal
+            - browserless
+
+    capture:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: capture
+        logging: *default-logging
+
+    capture-logs:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: capture-logs
+        logging: *default-logging
+
+    replay-capture:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: replay-capture
+        logging: *default-logging
+
+    property-defs-rs:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: property-defs-rs
+        logging: *default-logging
+        depends_on:
+            kafka-init:
+                condition: service_completed_successfully
+            db:
+                condition: service_healthy
+
+    livestream:
+        extends:
+            file: docker-compose.base.yml
+            service: livestream
+        logging: *default-logging
+        environment:
+            - LIVESTREAM_JWT_SECRET=${POSTHOG_SECRET}
+        volumes:
+            - ./posthog/docker/livestream/configs-hobby.yml:/configs/configs.yml
+
+    personhog-replica:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: personhog-replica
+        logging: *default-logging
+        environment:
+            PRIMARY_DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+        depends_on:
+            db:
+                condition: service_healthy
+
+    personhog-router:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: personhog-router
+        logging: *default-logging
+        depends_on:
+            - personhog-replica
+
+    feature-flags:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: feature-flags
+        logging: *default-logging
+        depends_on:
+            - db
+            - redis7
+
+    hypercache-server:
+        build:
+            context: ./posthog/rust
+        extends:
+            file: docker-compose.base.yml
+            service: hypercache-server
+        logging: *default-logging
+        depends_on:
+            - redis7
+
+    kafka-init:
+        image: docker.io/redpandadata/redpanda:v25.1.9
+        entrypoint: /bin/sh
+        logging: *default-logging
+        command:
+            - -c
+            - |
+                set -x
+                echo "Waiting for Kafka broker to accept connections..."
+                TIMEOUT=60
+                ELAPSED=0
+                # Use rpk topic list as the readiness check - it's a simple operation
+                # that confirms the broker is accepting Kafka protocol requests
+                until rpk topic list --brokers kafka:9092 2>/dev/null; do
+                    echo "Kafka broker not ready yet (elapsed: $${ELAPSED}s)..."
+                    sleep 2
+                    ELAPSED=$$((ELAPSED + 2))
+                    if [ $$ELAPSED -ge $$TIMEOUT ]; then
+                        echo "Timeout waiting for Kafka broker after $${TIMEOUT}s"
+                        echo "Final attempt to list topics:"
+                        rpk topic list --brokers kafka:9092 || true
+                        exit 1
+                    fi
+                done
+                echo "Kafka broker is accepting requests, creating topics..."
+                # The topic list is docker/kafka/topics.txt, mounted below, shared with
+                # docker-compose.dev.yml so the two stacks cannot drift apart. That header
+                # explains what needs pre-creating.
+                # Note: $$ escapes $ for docker-compose variable substitution
+                TOPICS=$$(grep -vE '^[[:space:]]*#|^[[:space:]]*$$' /topics.txt)
+                for topic in $$TOPICS; do
+                    if rpk topic create "$$topic" --brokers kafka:9092 -p 1 -r 1 2>&1; then
+                        echo "Topic $$topic created successfully"
+                    else
+                        if rpk topic list --brokers kafka:9092 | awk '{print $$1}' | grep -qx "$$topic"; then
+                            echo "Topic $$topic already exists, continuing"
+                        else
+                            echo "Failed to create topic $$topic"
+                            exit 1
+                        fi
+                    fi
+                done
+                echo "Final topic list:"
+                rpk topic list --brokers kafka:9092
+                echo "Topics ready"
+        volumes:
+            - ./posthog/docker/kafka/topics.txt:/topics.txt:ro
+        depends_on:
+            kafka:
+                condition: service_healthy
+
+    cymbal:
+        image: ghcr.io/posthog/posthog/cymbal:master
+        build:
+            context: ./posthog/rust
+            args:
+                BIN: cymbal
+        restart: on-failure
+        logging: *default-logging
+        volumes:
+            - ./share:/share
+        environment:
+            KAFKA_HOSTS: 'kafka:9092'
+            OBJECT_STORAGE_BUCKET: posthog
+            OBJECT_STORAGE_ACCESS_KEY_ID: ${OBJECT_STORAGE_ACCESS_KEY_ID}
+            OBJECT_STORAGE_SECRET_ACCESS_KEY: ${OBJECT_STORAGE_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENDPOINT: 'http://seaweedfs:8333'
+            OBJECT_STORAGE_FORCE_PATH_STYLE: 'true'
+            BIND_HOST: '0.0.0.0'
+            BIND_PORT: '3302'
+            DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            PERSONS_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            MAXMIND_DB_PATH: '/share/GeoLite2-City.mmdb'
+            REDIS_URL: 'redis://redis7:6379/'
+            ISSUE_BUCKETS_REDIS_URL: 'redis://redis7:6379/'
+            RUST_LOG: 'info'
+            # Processing offloads all symbol resolution to cymbal-resolution over gRPC. Both are
+            # mandatory: startup aborts when either is empty, so omitting them crash-loops cymbal.
+            CYMBAL_REMOTE_RESOLUTION_HOST: 'cymbal-resolution'
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET}
+        depends_on:
+            kafka-init:
+                condition: service_completed_successfully
+            seaweedfs:
+                condition: service_healthy
+            db:
+                condition: service_healthy
+            redis7:
+                condition: service_healthy
+            cymbal-resolution:
+                condition: service_started
+
+    # The cymbal binary in resolution mode: the gRPC symbol-resolution service that cymbal's
+    # processing mode calls. Needs only Postgres and object storage — no Kafka or Redis — hence
+    # the narrower env than cymbal above.
+    cymbal-resolution:
+        image: ghcr.io/posthog/posthog/cymbal:master
+        build:
+            context: ./posthog/rust
+            args:
+                BIN: cymbal
+        restart: on-failure
+        logging: *default-logging
+        environment:
+            CYMBAL_MODE: resolution
+            GRPC_ADDRESS: '0.0.0.0:50061'
+            OBJECT_STORAGE_BUCKET: posthog
+            OBJECT_STORAGE_ACCESS_KEY_ID: ${OBJECT_STORAGE_ACCESS_KEY_ID}
+            OBJECT_STORAGE_SECRET_ACCESS_KEY: ${OBJECT_STORAGE_SECRET_ACCESS_KEY}
+            OBJECT_STORAGE_ENDPOINT: 'http://seaweedfs:8333'
+            OBJECT_STORAGE_FORCE_PATH_STYLE: 'true'
+            DATABASE_URL: 'postgres://posthog:posthog@db:5432/posthog'
+            INTERNAL_API_SECRET: ${INTERNAL_API_SECRET}
+            RUST_LOG: 'info'
+        depends_on:
+            seaweedfs:
+                condition: service_healthy
+            db:
+                condition: service_healthy
+
+networks:
+    otel_network:
+    nginx-bridge:
+        external: true
+
+volumes:
+    zookeeper-data:
+    zookeeper-datalog:
+    zookeeper-logs:
+    objectstorage-data:
+    seaweedfs:
+    postgres-data:
+    clickhouse-data:
+    caddy-data:
+    caddy-config:
+    redis7-data:
+    kafka-data:
+    redpanda-data:

server/home/PostHog/compose/local-patches.py

@@ -0,0 +1,72 @@
+"""Local source patches, applied to /code at container start.
+
+Called from compose/start, compose/worker and compose/temporal-django-worker
+before the app boots, so every recreated container comes back patched. (The old
+approach patched running containers by hand and silently vanished on the next
+`docker compose up`.)
+
+Both patches exist because PostHog no longer sells self-hosted licenses, so the
+license branches below are unreachable here. Both are gated on
+MULTI_ORG_ENABLED (set in compose.override.yml), so unsetting it restores
+upstream behaviour without un-patching. Only posthog/ files are touched, which
+is MIT-licensed; ee/ is not involved.
+
+Idempotent. If upstream changes shape, this warns loudly and lets the app start
+unpatched rather than taking PostHog down. Check with:
+    docker compose logs web worker | grep LOCAL-PATCH
+"""
+
+import sys
+
+PATCHES = [
+    (
+        # Shows the "New project" / "New organization" buttons. Upstream gates
+        # them on preflight's can_create_org, which demands a license.
+        "can_create_org",
+        "/code/posthog/utils.py",
+        """    if settings.MULTI_ORG_ENABLED:
+        license = get_cached_instance_license()
+        if license is not None and AvailableFeature.ZAPIER in license.available_features:
+            return True
+        else:
+            logger.warning("You have configured MULTI_ORG_ENABLED, but not the required premium PostHog plan!")
+""",
+        """    if settings.MULTI_ORG_ENABLED:
+        # Local patch, see compose/local-patches.py
+        return True
+""",
+    ),
+    (
+        # Lifts the 1-project-per-org limit in the project API. Without a
+        # license, every org save and the periodic feature sync reset the org's
+        # features to [], and an org without `organizations_projects` may only
+        # have one project. With it and no `limit`, the cap is upstream's
+        # MAX_ALLOWED_PROJECTS_PER_ORG (2000).
+        "organizations_projects",
+        "/code/posthog/models/organization.py",
+        """        return self.available_product_features
+
+    def get_available_feature(""",
+        """        # Local patch, see compose/local-patches.py
+        if settings.MULTI_ORG_ENABLED and not self.get_available_feature("organizations_projects"):
+            self.available_product_features = [
+                *self.available_product_features,
+                {"key": "organizations_projects", "name": "Organizations projects"},
+            ]
+
+        return self.available_product_features
+
+    def get_available_feature(""",
+    ),
+]
+
+for name, path, old, new in PATCHES:
+    src = open(path).read()
+    if new in src:
+        status = "already applied"
+    elif src.count(old) == 1:
+        open(path, "w").write(src.replace(old, new))
+        status = "applied"
+    else:
+        status = "PATTERN NOT FOUND, upstream changed, running UNPATCHED"
+    print(f"LOCAL-PATCH {name}: {status}", file=sys.stderr, flush=True)

server/home/PostHog/compose/start

@@ -0,0 +1,5 @@
+#!/bin/bash
+python /compose/local-patches.py
+/compose/wait
+./bin/migrate
+./bin/docker-server

server/home/PostHog/compose/temporal-django-worker

@@ -0,0 +1,3 @@
+#!/bin/bash
+python /compose/local-patches.py
+./bin/temporal-django-worker

server/home/PostHog/compose/wait

@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+
+import socket
+import time
+
+def loop():
+    print("Waiting for ClickHouse and Postgres to be ready")
+    try:
+        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+            s.connect(('clickhouse', 9000))
+        print("Clickhouse is ready")
+        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+            s.connect(('db', 5432))
+        print("Postgres is ready")
+    except ConnectionRefusedError as e:
+        time.sleep(5)
+        loop()
+
+loop()

server/home/PostHog/compose/worker

@@ -0,0 +1,3 @@
+#!/bin/bash
+python /compose/local-patches.py
+exec ./bin/docker-worker-celery --with-scheduler

server/home/PostHog/podman-posthog.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=Podman Compose: PostHog
+After=network-online.target podman.socket
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+WorkingDirectory=%h/Server/PostHog
+ExecStart=/usr/bin/podman compose up -d
+ExecStop=/usr/bin/podman compose down
+RemainAfterExit=yes
+TimeoutStartSec=900
+TimeoutStopSec=90
+Restart=on-failure
+RestartSec=30
+
+[Install]
+WantedBy=default.target

server/home/RESTORE.md

@@ -0,0 +1,140 @@
+# Restoring ~/Server on server.gib from the ZFS-pool backup
+
+Written 2026-09-17. Read this *before* you need it.
+
+## What this protects against
+
+The NVMe/LUKS btrfs volume (`/`, `/home`, and `/var/lib/docker`) dying. The backup lives on
+the ZFS pool in the same machine, so it does **not** protect against fire, theft, or the
+whole box failing. It is a disk-failure backup, not an offsite one.
+
+## Layout
+
+    /home/gib/Public/Server_Backup/Containers/server.gib/current/
+
+**One directory, overwritten daily at 02:00** by `server-backup.timer`. History lives in ZFS:
+`Media/Server_Backup` has `zfs-auto-snapshot` keeping **14 dailies**, taken by `cron.daily`
+between roughly 03:22 and 03:52. The 02:00 schedule is deliberate — it must finish before
+that window, or a snapshot captures a half-written backup as that day's history. It also
+staggers against the VPS backup, which pushes into this same pool at 01:00.
+
+Older copies are under the hidden snapshot dir (`snapdir=hidden`, so `ls` will not show it,
+but you can `cd` in):
+
+    ls /home/gib/Public/Server_Backup/.zfs/snapshot/
+    ls /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
+
+The `03-16-2026` and `03-31-2026` directories are leftovers from the old manual script.
+
+## Why this runs as root
+
+This box runs Docker, so most container data under `~/Server` is root-owned. The old script
+used `sudo rsync`, which worked but meant it could never be automated without a sudoers
+rule. The replacement runs as root via `server-backup.service`, a systemd **system** unit —
+already root, so ownership is preserved natively and **no sudoers entry is needed at
+runtime**. If you see `/etc/sudoers.d/99-claude-setup` on this machine, it was temporary
+setup scaffolding and should be deleted.
+
+(This differs from the VPS, which is rootless podman and needs `rsync --fake-super` to carry
+subuid ownership through. Nothing like that is needed here.)
+
+The script lives at `/usr/local/sbin/server-backup`, deliberately **not** in `~/Server` — a
+root-executed script inside a user-writable directory is a footgun. `~/Server/backup` is a
+thin wrapper that triggers the systemd unit so manual runs behave identically to nightly ones.
+
+## What is in a backup
+
+| Path | What it is |
+|---|---|
+| `Media/`, `Frigate/`, `Home/`, `OpenWebUI/`, ... | Raw bind-mount data per service |
+| `_backup_state/sqlite/<original path>` | Online `.backup` of every SQLite DB — **authoritative** |
+| `_backup_state/convex/<container>.zip` | `npx convex export` snapshot of each SQLite-backed Convex deployment (gbrown) — **authoritative** |
+
+> **Use `_backup_state/` for databases, not the raw copies.** Containers keep running during
+> the backup, so the raw files are crash-consistent at best. The dumps use SQLite's online
+> backup API and are what will actually restore cleanly. That covers the ones that matter:
+> Plex's `com.plexapp.plugins.library.db` (watch history, ratings), Jellyfin, Frigate, Home
+> Assistant, OpenWebUI, Audiobookshelf, and every *arr config.
+>
+> **Convex is the exception.** It writes to its SQLite file constantly, so a raw `.backup`
+> never completes. Its `_backup_state/convex/` export is the restore source, and the raw
+> `db.sqlite3` is not dumped at all.
+
+## What is deliberately NOT backed up
+
+- **`Agents/Gitea/volumes/runner-{1,2,3}/dind` — ~203 GB.** Docker-in-Docker containerd
+  image layers for the CI runners. Pure cache that churns constantly; a rebuilt runner just
+  re-pulls it. The parts that matter — `runner-N/data/.runner` (the 648-byte registration
+  token) and `certs/` — **are** backed up.
+  > Measured as root. A plain `du` as `gib` reports these as ~13 MB because gib cannot
+  > descend into them. Do not trust non-root sizes here.
+- Docker **named volumes** (`/var/lib/docker/volumes`, 47 in use, ~61 GB): Sentry (13) and
+  Plausible (4) were consciously excluded — Sentry's event data is not worth restoring and
+  Plausible had nothing in it. Romm looked like a gap but is not: its anonymous `/romm`
+  volume is only a parent directory, with `assets/`, `config/` and `resources/` bind-mounted
+  from `~/Server` and `library/` from the pool.
+- **Nextcloud AIO** — has its own borg backup at `~/Public/Nextcloud/Backups`, run by
+  hand from the AIO admin UI. Last run 2026-09-17 (finished 09-18). Runs are slow and
+  AIO stops the Nextcloud containers while one is going.
+- `node_modules`, build output, `*.log`, `cache`/`tmp` dirs, `*.mp4`/`*.mkv`.
+
+## Restoring
+
+Stop the stack first so nothing overwrites what you restore:
+
+    cd ~/Server/<service> && docker compose down     # per service, or stop docker entirely
+
+Pull it back — as root, so ownership is restored:
+
+    sudo rsync -a --numeric-ids --delete \
+      /home/gib/Public/Server_Backup/Containers/server.gib/current/ \
+      /home/gib/Server/
+
+For an older day, use a snapshot path instead:
+
+    /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
+
+Then put the databases back from the dumps. For each one, stop the owning container, copy the
+dump over the live file, and match the original ownership. Plex, for example:
+
+    docker compose -f ~/Server/Media/compose.yml stop plex
+    D=~/Server/_backup_state/sqlite/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
+    L=~/Server/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
+    sudo cp "$D/com.plexapp.plugins.library.db" "$L/"
+    sudo chown --reference="$L/com.plexapp.dlna.db" "$L/com.plexapp.plugins.library.db"
+    docker compose -f ~/Server/Media/compose.yml start plex
+
+`chown --reference` copies ownership from a neighbouring file, which saves looking up what
+uid each container expects.
+
+For Convex, start the backend (it can be empty) and import the export through it. This
+replaces every table with the snapshot:
+
+    Z=~/Server/_backup_state/convex/gbrown-backend.zip
+    KEY=$(docker exec gbrown-backend ./generate_admin_key.sh | tail -1)
+    docker run --rm -it --network nginx-bridge -v "$Z:/backup.zip:ro" \
+      -e CONVEX_SELF_HOSTED_URL=http://gbrown-backend:3210 -e CONVEX_SELF_HOSTED_ADMIN_KEY="$KEY" \
+      docker.io/library/node:22-alpine sh -ec 'mkdir /w && cd /w && echo "{}" > package.json &&
+        npm install --silent convex && npx convex import --replace-all /backup.zip'
+
+Functions and schema are not in the export. Redeploy them from the app's repo as usual.
+
+After a full restore the CI runners will have no `dind` storage. That is fine — they re-pull
+images on the next job. The registration in `runner-N/data/.runner` is intact, so they
+reconnect to Gitea without re-registering.
+
+## Checking it is working
+
+    systemctl list-timers server-backup.timer
+    systemctl status server-backup.service
+    tail -40 ~/Server/logs/backup.log
+
+A healthy run ends with `=== backup complete ===`. Watch for:
+
+- `ZFS history OK` — confirms snapshots are still being taken. If this ever says **WARNING**,
+  versioning is broken and `current/` is just a mirror: a mistake would propagate with
+  nothing to roll back to. Fix `zfs-auto-snapshot` before anything else.
+- `rsync exit 24` is a NOTE, not a failure — a file changed mid-transfer, normal on a live box.
+
+Optional: put an uptime-kuma push URL in `~/Server/.backup-heartbeat-url` and successful runs
+will ping it, turning a silent failure into a down monitor.