Gabriel Brown
f4fca1797f
update-containers kept LOCAL_BUILD, a hand-written list of services whose images are built on the machine, so it would not try to pull them. The compose files already say this. A service with a build: stanza names a local image, and there is nothing in a registry to pull. images_for now drops the image of any compose service that has a build: stanza. Those services resolve no images, and the run skips them without a log line. The list, and the app names it carried, are gone. On the VPS, --list resolves the same images as before for every other service.
The server role
What a Panama machine runs when the interview answered server: the same
shell environment as every other machine, minus everything that needs a
screen, plus rootless podman compose services managed as systemd user units.
server/
vps/ The catalog panama-server installs from, one directory
per service: compose.yml, the podman-<name>.service user
unit, and an .env.example naming what the service needs told
home/ The home server's services in the same shape (see below)
scripts/ update-containers, the nightly image updater
systemd/ The units behind it, linked by setup-server
The home server
server/home/ mirrors ~/Server on the home server, where the services still
run under Docker. Each directory has the same three files as the VPS catalog,
and the unit is drafted for the eventual move to rootless podman. It has never
run, and the composes still use Docker-only features (host networking, the
GPU, the Docker socket) that need converting first. Nothing here is
installable: panama-server reads server/vps/ only, so a service moves into
the catalog when it actually moves to podman.
Paths keep their nesting (server/home/Agents/Gitea is ~/Server/Agents/Gitea).
Three services are checkouts of upstream repositories. For those, only the
local files are kept, and an UPSTREAM file names the repository and the exact
commit the checkout sits at: PostHog, Sentry and Plausible.
Both trees are snapshots of the live machines with every secret replaced by a
${VAR} named in .env.example, or by CHANGE_ME in files that are not
compose files. The live servers still carry some secrets inline, so a live
compose and its copy here can differ on those lines alone.
The contract with ~/Server
The repository carries the definitions; the machine carries the state.
panama server enable <Name> creates ~/Server/<Name>/ as a real
directory and symlinks only compose.yml into it; the unit symlinks into
~/.config/systemd/user/. The .env (seeded once from .env.example, then
yours) and the bind-mounted data/ live in ~/Server/<Name>/ and are never
inside the checkout — so nothing git can do, clean -fdx included, can reach
a database.
Because of that split, compose files must reference secrets as ${VAR}
interpolations resolved from the .env beside them, never inline.
tests/server/compose-secrets-contract fails the suite when a tracked file
under server/ carries anything that looks like a secret: this repository is
public, and the gitignore is a seatbelt, not the brakes.
Placeholders in .env.example are spelled CHANGE_ME, exactly — that token
is what panama server enable refuses to start on.
Conventions
- Service directories are TitleCase, matching
~/Serveras it has always looked:server/vps/Giteabecomes~/Server/Gitea. - Bind mounts live under
./data/in new services. Imported services keep the volume path they were running with until their cutover, because the repo copy must render identically to the live one — renames happen with the service stopped, not in git. - Only 80, 443 and 81 are open, and
setup-serveropens each one only where nothing already grants it. A port reachable through an enabled service or an accepting rich rule — a CDN-scoped grant, say — is left exactly as it is, because turning a narrow grant into a public one during an unattendedpanama updateis not a decision an installer gets to make. A machine that manages its own firewall opts out for good withecho manual > ~/.local/state/panama/firewall. Everything else is reverse-proxied over thenginx-bridgenetwork by container name, with no published ports of its own. A service that truly needs another host port says so in a README in its directory, and the port is opened by hand. - Every compose joins
nginx-bridge(external; created by setup-server).
Enabling, updating, cutting over
panama server list # the catalog, and what this machine runs
panama server enable Gitea # link, seed .env, enable --now
panama server status # is what is enabled actually up
panama server disable Gitea # stop and unlink; data and .env stay
panama update relinks and reloads but never restarts a running
service — it names the ones whose definitions changed and leaves the
restart to you.
Moving a service that already runs from hand-managed files onto the repo's
(the cutover): stop the unit, standardize the volume directory to data/
(updating the repo compose to match), then panama server enable <Name> —
it backs up the hand-written files as *.pre-panama and links the tracked
ones — and verify it came back up. One service at a time, quiet hours.
Adding a service
Add server/vps/<Name>/ with the three files, modelled on any
existing service. The unit is the standard oneshot podman compose up -d
shape (WorkingDirectory=%h/Server/<Name>); the shape contract pins the
details. Secrets go in .env.example as KEY=CHANGE_ME lines.
Nightly updates
server/scripts/update-containers pulls images and restarts changed
services through their units, nightly at midnight via podman-update.timer.
It replaced watchtower after watchtower recreated a container outside its
compose pod and took Gitea down for three days — the script's header carries
the full story, plus the SKIP list for services that must only ever be
updated by hand (postgresql, authentik).