Panama/server

The server role

What a Panama machine runs when the interview answered server: the same shell environment as every other machine, minus everything that needs a screen, plus rootless podman compose services managed as systemd user units.

server/
  vps/            The catalog panama-server installs from, one directory
                  per service: compose.yml, the podman-<name>.service user
                  unit, and an .env.example naming what the service needs told.
                  Loose files beside the directories are the VPS's own
                  host tooling (see "Host tooling" below)
  home/           The home server's services in the same shape (see below)
  scripts/        update-containers, the nightly image updater
  systemd/        The units behind it, linked by setup-server

The home server

server/home/ mirrors ~/Server on the home server, where the services still run under Docker. Each directory has the same three files as the VPS catalog, and the unit is drafted for the eventual move to rootless podman. It has never run, and the composes still use Docker-only features (host networking, the GPU, the Docker socket) that need converting first. Nothing here is installable: panama-server reads server/vps/ only, so a service moves into the catalog when it actually moves to podman.

Paths keep their nesting (server/home/Agents/Gitea is ~/Server/Agents/Gitea). Media/compact-tv/ is the nightly TV re-encode job that runs beside the Media stack, with its own README. Three services are checkouts of upstream repositories. For those, only the local files are kept, and an UPSTREAM file names the repository and the exact commit the checkout sits at: PostHog, Sentry and Plausible.

Both trees are snapshots of the live machines with every secret replaced by a ${VAR} named in .env.example, or by CHANGE_ME in files that are not compose files. The live servers still carry some secrets inline, so a live compose and its copy here can differ on those lines alone.

The contract with ~/Server

The repository carries the definitions; the machine carries the state. panama server enable <Name> creates ~/Server/<Name>/ as a real directory and symlinks the service's visible top-level files into it: compose.yml, plus any helper script or README. The units symlink into ~/.config/systemd/user/. The .env (seeded once from .env.example, then yours) and the bind-mounted data/ live in ~/Server/<Name>/ and are never inside the checkout — so nothing git can do, clean -fdx included, can reach a database.

Because of that split, compose files must reference secrets as ${VAR} interpolations resolved from the .env beside them, never inline. tests/server/compose-secrets-contract fails the suite when a tracked file under server/ carries anything that looks like a secret: this repository is public, and the gitignore is a seatbelt, not the brakes.

Placeholders in .env.example are spelled CHANGE_ME, exactly — that token is what panama server enable refuses to start on.

Conventions

• Service directories are TitleCase, matching ~/Server as it has always looked: server/vps/Gitea becomes ~/Server/Gitea.
• Bind mounts live under ./data/ in new services. Imported services keep the volume path they were running with until their cutover, because the repo copy must render identically to the live one — renames happen with the service stopped, not in git.
• Only 80, 443 and 81 are open, and setup-server opens each one only where nothing already grants it. A port reachable through an enabled service or an accepting rich rule — a CDN-scoped grant, say — is left exactly as it is, because turning a narrow grant into a public one during an unattended panama update is not a decision an installer gets to make. A machine that manages its own firewall opts out for good with echo manual > ~/.local/state/panama/firewall. Everything else is reverse-proxied over the nginx-bridge network by container name, with no published ports of its own. A service that truly needs another host port says so in a README in its directory, and the port is opened by hand.
• Every compose joins nginx-bridge (external; created by setup-server).

Enabling, updating, cutting over

panama server list             # the catalog, and what this machine runs
panama server enable Gitea     # link, seed .env, enable --now
panama server status           # is what is enabled actually up
panama server disable Gitea    # stop and unlink; data and .env stay

panama update relinks and reloads but never restarts a running service — it names the ones whose definitions changed and leaves the restart to you.

Moving a service that already runs from hand-managed files onto the repo's (the cutover): stop the unit, standardize the volume directory to data/ (updating the repo compose to match), then panama server enable <Name> — it backs up the hand-written files as *.pre-panama and links the tracked ones — and verify it came back up. One service at a time, quiet hours.

Adding a service

Add server/vps/<Name>/ with the three files, modelled on any existing service. The unit is the standard oneshot podman compose up -d shape (WorkingDirectory=%h/Server/<Name>); the shape contract pins the details. Secrets go in .env.example as KEY=CHANGE_ME lines.

A service that needs a scheduled job of its own carries it beside the compose unit: a helper <job>.service and a <job>.timer with the same name, plus the script it runs. Enable links all three and enables the timer. Disable stops the timer and unlinks the units. UseSend is the example: its usesend-cert-refresh timer keeps the SMTP proxy's certificate in step with Nginx Proxy Manager's.

Host tooling

Each server's own scripts and runbooks sit as loose files at the top of its tree. They are reference copies for rebuilding a machine. Nothing installs them.

• vps/backup, vps/server-backup.service, vps/server-backup.timer: the VPS's nightly backup, run as a user unit. It dumps every database into ~/Server/_backup_state/, then rsyncs ~/Server to the home server's ZFS pool with --fake-super, so subuid ownership survives the trip.
• vps/verify-rootless: the post-reboot health check for the rootless stack. It covers linger, the unprivileged port floor, unit state and boot enablement, postgres, endpoints, and admin ports bound to the WireGuard address.
• vps/AGENTS.md: the VPS runbook, with both public IPs redacted.
• home/RESTORE.md: how to restore the home server's ~/Server from the ZFS backup.
• home/backup and home/systemd/: the home server's backup trigger and its root system units (server-backup nightly, media-backup weekly).
• home/server-backup and home/media-backup: the scripts those units run. On that machine they live in /usr/local/sbin, root-owned, because a root-run script must not sit in a directory the user can write to. Restore them there, not into ~/Server.

The VPS units carry RequiresMountsFor=/home/gib/Media because that host keeps its podman graphroot on an attached volume mounted there. The setting is harmless on a machine without the mount.

Nightly updates

server/scripts/update-containers pulls images and restarts changed services through their units, nightly at midnight via podman-update.timer. It replaced watchtower after watchtower recreated a container outside its compose pod and took Gitea down for three days — the script's header carries the full story, plus the SKIP list for services that must only ever be updated by hand (postgresql, authentik).