setup-server queried --query-port, saw "not open", and blanket-opened 80, 443 and 81. On a machine reaching those ports through rich rules -- one CDN's address sets, say -- neither appears in --list-ports, so the step read a deliberate, narrow grant as "closed" and replaced it with a public one. It did this during `panama update`, which asks nothing and runs unattended, so there was no prompt and no diff for anyone to read. A production server lost its firewall posture that way, port 81 and the proxy's admin portal included. Each port is now checked for existing coverage first -- an open port, an enabled service, or an accepting rich rule, ranges included -- and left alone when it has some. The bias stays where it was: only a rule that positively reads as a grant counts, and anything unrecognised still falls through to opening the port, because a fresh server that is silently unreachable is the worse failure. A rule that DROPS a port is not coverage either; reading a block as a grant would leave the proxy unreachable on a machine nobody thought to re-check. A machine that manages its own firewall opts out once, with `echo manual > ~/.local/state/panama/firewall`, and Panama never touches firewalld on it again. A file rather than an environment variable, because the run that needs to respect it is the unattended one. tests/setup/server-firewall-contract pins all of it, and fails against the previous code on exactly the cases that cost the posture. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
4.2 KiB
The server role
What a Panama machine runs when the interview answered server: the same
shell environment as every other machine, minus everything that needs a
screen, plus rootless podman compose services managed as systemd user units.
server/
containers/ One directory per service: compose.yml, the
podman-<name>.service user unit, and an .env.example
naming what the service needs told
scripts/ update-containers, the nightly image updater
systemd/ The units behind it, linked by setup-server
The contract with ~/Server
The repository carries the definitions; the machine carries the state.
panama server enable <Name> creates ~/Server/<Name>/ as a real
directory and symlinks only compose.yml into it; the unit symlinks into
~/.config/systemd/user/. The .env (seeded once from .env.example, then
yours) and the bind-mounted data/ live in ~/Server/<Name>/ and are never
inside the checkout — so nothing git can do, clean -fdx included, can reach
a database.
Because of that split, compose files must reference secrets as ${VAR}
interpolations resolved from the .env beside them, never inline.
tests/server/compose-secrets-contract fails the suite when a tracked file
under server/ carries anything that looks like a secret: this repository is
public, and the gitignore is a seatbelt, not the brakes.
Placeholders in .env.example are spelled CHANGE_ME, exactly — that token
is what panama server enable refuses to start on.
Conventions
- Service directories are TitleCase, matching
~/Serveras it has always looked:server/containers/Giteabecomes~/Server/Gitea. - Bind mounts live under
./data/in new services. Imported services keep the volume path they were running with until their cutover, because the repo copy must render identically to the live one — renames happen with the service stopped, not in git. - Only 80, 443 and 81 are open, and
setup-serveropens each one only where nothing already grants it. A port reachable through an enabled service or an accepting rich rule — a CDN-scoped grant, say — is left exactly as it is, because turning a narrow grant into a public one during an unattendedpanama updateis not a decision an installer gets to make. A machine that manages its own firewall opts out for good withecho manual > ~/.local/state/panama/firewall. Everything else is reverse-proxied over thenginx-bridgenetwork by container name, with no published ports of its own. A service that truly needs another host port says so in a README in its directory, and the port is opened by hand. - Every compose joins
nginx-bridge(external; created by setup-server).
Enabling, updating, cutting over
panama server list # the catalog, and what this machine runs
panama server enable Gitea # link, seed .env, enable --now
panama server status # is what is enabled actually up
panama server disable Gitea # stop and unlink; data and .env stay
panama update relinks and reloads but never restarts a running
service — it names the ones whose definitions changed and leaves the
restart to you.
Moving a service that already runs from hand-managed files onto the repo's
(the cutover): stop the unit, standardize the volume directory to data/
(updating the repo compose to match), then panama server enable <Name> —
it backs up the hand-written files as *.pre-panama and links the tracked
ones — and verify it came back up. One service at a time, quiet hours.
Adding a service
Add server/containers/<Name>/ with the three files, modelled on any
existing service. The unit is the standard oneshot podman compose up -d
shape (WorkingDirectory=%h/Server/<Name>); the shape contract pins the
details. Secrets go in .env.example as KEY=CHANGE_ME lines.
Nightly updates
server/scripts/update-containers pulls images and restarts changed
services through their units, nightly at midnight via podman-update.timer.
It replaced watchtower after watchtower recreated a container outside its
compose pod and took Gitea down for three days — the script's header carries
the full story, plus the SKIP list for services that must only ever be
updated by hand (postgresql, authentik).