148 lines
7.4 KiB
Markdown
148 lines
7.4 KiB
Markdown
# The server role
|
|
|
|
What a Panama machine runs when the interview answered `server`: the same
|
|
shell environment as every other machine, minus everything that needs a
|
|
screen, plus rootless podman compose services managed as systemd user units.
|
|
|
|
```
|
|
server/
|
|
vps/ The catalog panama-server installs from, one directory
|
|
per service: compose.yml, the podman-<name>.service user
|
|
unit, and an .env.example naming what the service needs told.
|
|
Loose files beside the directories are the VPS's own
|
|
host tooling (see "Host tooling" below)
|
|
home/ The home server's services in the same shape (see below)
|
|
scripts/ update-containers, the nightly image updater
|
|
systemd/ The units behind it, linked by setup-server
|
|
```
|
|
|
|
## The home server
|
|
|
|
`server/home/` mirrors `~/Server` on the home server, where the services still
|
|
run under Docker. Each directory has the same three files as the VPS catalog,
|
|
and the unit is drafted for the eventual move to rootless podman. It has never
|
|
run, and the composes still use Docker-only features (host networking, the
|
|
GPU, the Docker socket) that need converting first. Nothing here is
|
|
installable: panama-server reads `server/vps/` only, so a service moves into
|
|
the catalog when it actually moves to podman.
|
|
|
|
Paths keep their nesting (`server/home/Agents/Gitea` is `~/Server/Agents/Gitea`).
|
|
`Media/compact-tv/` is the nightly TV re-encode job that runs beside the Media
|
|
stack, with its own README.
|
|
Three services are checkouts of upstream repositories. For those, only the
|
|
local files are kept, and an `UPSTREAM` file names the repository and the exact
|
|
commit the checkout sits at: `PostHog`, `Sentry` and `Plausible`.
|
|
|
|
Both trees are snapshots of the live machines with every secret replaced by a
|
|
`${VAR}` named in `.env.example`, or by `CHANGE_ME` in files that are not
|
|
compose files. The live servers still carry some secrets inline, so a live
|
|
compose and its copy here can differ on those lines alone.
|
|
|
|
## The contract with ~/Server
|
|
|
|
The repository carries the definitions; the machine carries the state.
|
|
`panama server enable <Name>` creates `~/Server/<Name>/` as a **real
|
|
directory** and symlinks the service's visible top-level files into it:
|
|
`compose.yml`, plus any helper script or README. The units symlink into
|
|
`~/.config/systemd/user/`. The `.env` (seeded once from `.env.example`, then
|
|
yours) and the bind-mounted `data/` live in `~/Server/<Name>/` and are never
|
|
inside the checkout — so nothing git can do, `clean -fdx` included, can reach
|
|
a database.
|
|
|
|
Because of that split, compose files must reference secrets as `${VAR}`
|
|
interpolations resolved from the `.env` beside them, never inline.
|
|
`tests/server/compose-secrets-contract` fails the suite when a tracked file
|
|
under `server/` carries anything that looks like a secret: this repository is
|
|
public, and the gitignore is a seatbelt, not the brakes.
|
|
|
|
Placeholders in `.env.example` are spelled `CHANGE_ME`, exactly — that token
|
|
is what `panama server enable` refuses to start on.
|
|
|
|
## Conventions
|
|
|
|
- Service directories are TitleCase, matching `~/Server` as it has always
|
|
looked: `server/vps/Gitea` becomes `~/Server/Gitea`.
|
|
- Bind mounts live under `./data/` in new services. Imported services keep
|
|
the volume path they were running with until their cutover, because the
|
|
repo copy must render identically to the live one — renames happen with
|
|
the service stopped, not in git.
|
|
- Only 80, 443 and 81 are open, and `setup-server` opens each one only where
|
|
nothing already grants it. A port reachable through an enabled service or an
|
|
accepting rich rule — a CDN-scoped grant, say — is left exactly as it is,
|
|
because turning a narrow grant into a public one during an unattended
|
|
`panama update` is not a decision an installer gets to make. A machine that
|
|
manages its own firewall opts out for good with
|
|
`echo manual > ~/.local/state/panama/firewall`. Everything else is
|
|
reverse-proxied over the `nginx-bridge` network by container name, with no
|
|
published ports of its own. A service that truly needs another host port
|
|
says so in a README in its directory, and the port is opened by hand.
|
|
- Every compose joins `nginx-bridge` (external; created by setup-server).
|
|
|
|
## Enabling, updating, cutting over
|
|
|
|
```sh
|
|
panama server list # the catalog, and what this machine runs
|
|
panama server enable Gitea # link, seed .env, enable --now
|
|
panama server status # is what is enabled actually up
|
|
panama server disable Gitea # stop and unlink; data and .env stay
|
|
```
|
|
|
|
`panama update` relinks and reloads but **never restarts a running
|
|
service** — it names the ones whose definitions changed and leaves the
|
|
restart to you.
|
|
|
|
Moving a service that already runs from hand-managed files onto the repo's
|
|
(the cutover): stop the unit, standardize the volume directory to `data/`
|
|
(updating the repo compose to match), then `panama server enable <Name>` —
|
|
it backs up the hand-written files as `*.pre-panama` and links the tracked
|
|
ones — and verify it came back up. One service at a time, quiet hours.
|
|
|
|
## Adding a service
|
|
|
|
Add `server/vps/<Name>/` with the three files, modelled on any
|
|
existing service. The unit is the standard oneshot `podman compose up -d`
|
|
shape (`WorkingDirectory=%h/Server/<Name>`); the shape contract pins the
|
|
details. Secrets go in `.env.example` as `KEY=CHANGE_ME` lines.
|
|
|
|
A service that needs a scheduled job of its own carries it beside the
|
|
compose unit: a helper `<job>.service` and a `<job>.timer` with the same
|
|
name, plus the script it runs. Enable links all three and enables the timer.
|
|
Disable stops the timer and unlinks the units. `UseSend` is the example: its
|
|
`usesend-cert-refresh` timer keeps the SMTP proxy's certificate in step with
|
|
Nginx Proxy Manager's.
|
|
|
|
## Host tooling
|
|
|
|
Each server's own scripts and runbooks sit as loose files at the top of its
|
|
tree. They are reference copies for rebuilding a machine. Nothing installs
|
|
them.
|
|
|
|
- `vps/backup`, `vps/server-backup.service`, `vps/server-backup.timer`: the
|
|
VPS's nightly backup, run as a user unit. It dumps every database into
|
|
`~/Server/_backup_state/`, then rsyncs `~/Server` to the home server's ZFS
|
|
pool with `--fake-super`, so subuid ownership survives the trip.
|
|
- `vps/verify-rootless`: the post-reboot health check for the rootless
|
|
stack. It covers linger, the unprivileged port floor, unit state and boot
|
|
enablement, postgres, endpoints, and admin ports bound to the WireGuard
|
|
address.
|
|
- `vps/AGENTS.md`: the VPS runbook, with both public IPs redacted.
|
|
- `home/RESTORE.md`: how to restore the home server's `~/Server` from the
|
|
ZFS backup.
|
|
- `home/backup` and `home/systemd/`: the home server's backup trigger and its
|
|
root system units (`server-backup` nightly, `media-backup` weekly). The
|
|
scripts those units run live in `/usr/local/sbin` on that machine. They are
|
|
readable by root alone, so this tree has no copy of them.
|
|
|
|
The VPS units carry `RequiresMountsFor=/home/gib/Media` because that host
|
|
keeps its podman graphroot on an attached volume mounted there. The setting
|
|
is harmless on a machine without the mount.
|
|
|
|
## Nightly updates
|
|
|
|
`server/scripts/update-containers` pulls images and restarts changed
|
|
services through their units, nightly at midnight via `podman-update.timer`.
|
|
It replaced watchtower after watchtower recreated a container outside its
|
|
compose pod and took Gitea down for three days — the script's header carries
|
|
the full story, plus the SKIP list for services that must only ever be
|
|
updated by hand (postgresql, authentik).
|