Gabriel Brown
f33da41cc6
A machine's role is now the interview's first question and the one answer Panama records. Servers get the same shell minus the screen: core packages, nvm, Bun, Claude Code and Codex (desktops get Codex too), linger, rootless ports from 80, firewalld, the nginx-bridge network, and a nightly image updater that replaced watchtower for cause. server/containers/ carries junior's 23 compose services -- secrets moved to per-machine .env files that never enter this public repo, every transformed compose proven to render byte-identical to what is live. 'panama server' enables, disables and relinks them; nothing here restarts a running service. 'boot --server' walks a fresh VPS from its root login to a normal install. Five new contracts pin the secrets rule, the catalog's shape, panama-server's behavior, the role plumbing, and the dotfile classification. Claude-Session: https://claude.ai/code/session_01NU5JGiN3JfzqrLQB6wmJ1E
231 lines
13 KiB
Markdown
231 lines
13 KiB
Markdown
# Panama server role: headless installs and versioned containers
|
||
|
||
2026-08-25. Designed with Gib across two interview rounds; every decision below
|
||
was made explicitly, none inferred.
|
||
|
||
## Goal
|
||
|
||
Panama becomes the one repo for every Fedora machine Gib runs. A desktop gets
|
||
what it gets today. A server — a Hetzner VPS, a future home server — gets the
|
||
same shell environment (bash profile, prompt, tmux-on-ssh, agents) minus
|
||
everything that needs a screen, plus rootless podman, systemd user units, and a
|
||
versioned catalog of compose services that `panama server enable` brings up.
|
||
The repo stays public; secrets never enter it.
|
||
|
||
## Decisions of record
|
||
|
||
| Decision | Choice |
|
||
|---|---|
|
||
| Repo | One public monorepo. No companion repo. `.gitignore` + a secrets contract keep it safe |
|
||
| Role | Asked first in the interview (`desktop` / `server`), persisted to `~/.local/state/panama/role`; `--server` on `boot`/`install` presets it |
|
||
| Layout | Top-level `server/`, services under `server/containers/<Name>/` (TitleCase, matching junior) |
|
||
| Tracked per service | `compose.yml`, `podman-<name>.service`, `.env.example`. Nothing else |
|
||
| `~/Server` | Real directories. Only `compose.yml` is symlinked into them; units symlink into `~/.config/systemd/user/`. Live `.env` and data stay outside the repo tree entirely |
|
||
| Data dirs | Standardized to `./data/`, applied per service at cutover (a stop/rename/start), never at import |
|
||
| Enablement | `panama server` subcommand family, not the interview |
|
||
| Units | Checked in per service (they legitimately differ), installed as symlinks so edits flow with `panama update` |
|
||
| Updates | `panama update` relinks and `daemon-reload`s but never restarts a running service; it names services whose definitions changed |
|
||
| Nightly image updates | junior's `update-containers.sh` + `podman-update.timer` adopted into the repo (it replaced watchtower for cause) |
|
||
| Codex | Installed everywhere, same pattern as Claude Code |
|
||
| Ports | 80/443/81 only. Anything else is opened by hand and documented in the service's folder |
|
||
| VPS bootstrap | `boot` run as root walks user-creation/keys/sshd-hardening, each step skipped when already true, then re-runs as the user |
|
||
| Out of scope | server.gib (docker; translate later), backup and verify-rootless (own design later), cutover of junior's live services (later, per service) |
|
||
|
||
## 1. Machine role
|
||
|
||
- New interview question, asked first: "What is this machine?" — `desktop` or
|
||
`server` via `gum choose`. `./install --server` (and `boot --server`) records
|
||
the answer without asking.
|
||
- Persisted to `~/.local/state/panama/role`, plain text, one word. This is the
|
||
first durable answer Panama keeps, and deliberately so: the transient-answers
|
||
principle exists to keep *personal* data out of durable paths, and a machine
|
||
role is a fact about the machine. `install`, `panama update`, `panama doctor`
|
||
and future migrations read it. A machine with no role file is a desktop —
|
||
every machine that predates this work is one.
|
||
- On role=server the interview skips: hardware probes (NVIDIA, Secure Boot,
|
||
firmware — and their bootstrap tools; only `gum` is bootstrapped), the
|
||
debloat question, and the extras checklist. It keeps: hostname, git
|
||
identity, gh sign-in, SSH key, user-content.
|
||
|
||
## 2. Stages by role
|
||
|
||
Role=desktop is exactly today's behavior. Role=server runs:
|
||
|
||
| Stage | Server behavior |
|
||
|---|---|
|
||
| `install-packages` | Core + server lists (§3). No flatpaks, no COPRs, no desktop repos. Claude Code and Codex install for both roles |
|
||
| `link-dotfiles` | Universal dot dirs only: `btop`, `nvim`, `ohmyposh`, `tmux`, `vim`, `panama`. Desktop-only (everything else: `hypr`, `quickshell`, `kitty`, `ghostty`, `espanso`, `gtk-*`, `uwsm`, `vicinae`, `wofi`, `xdg-desktop-portal`) is skipped via an explicit classification list in the stage — a new dot dir must be classified or the contract fails. Bash profile links as today; Firefox/wallpaper/cursor seeding skipped |
|
||
| `link-user` | Unchanged (SSH aliases and agent rules are exactly what a server wants) |
|
||
| `setup-identity` | Unchanged |
|
||
| `setup-server` (new) | §4 |
|
||
| `link-server` (new) | Creates `~/Server`, then delegates to `panama-server relink` so link repair has one owner |
|
||
| Skipped | `link-skills` (all three shipped skills are desktop-specific today; revisit when a server skill exists), `change-settings`, `link-vicinae-scripts`, `install-hardware` |
|
||
|
||
Migrations run on servers too — they are self-guarding no-ops where they do
|
||
not apply. `panama doctor` learns the role: on a server it reports service
|
||
health via `panama-server status` instead of probing desktop units.
|
||
|
||
## 3. Package restructure
|
||
|
||
`setup/packages/` gains `core-packages` (both roles) and `server-packages`;
|
||
`initial-packages` keeps what is desktop-only. Nothing is added or dropped for
|
||
desktops — the union of core + desktop lists equals today's set.
|
||
|
||
- **core-packages**: awk, bat, btop, curl, dnf5-plugins, eza, fontconfig, fzf,
|
||
gh, git-all, gum, jq, neovim, oh-my-posh, openssh, openssl, podman,
|
||
podman-compose, python3-dnf, ripgrep, tmux, unzip, wget, zoxide.
|
||
(`podman` moves here from development-packages; `podman-compose` is what
|
||
provides `podman compose` on junior — verified, 1.6.0.)
|
||
- **server-packages**: nvm (Node via nvm, as on desktop — agents and MCP
|
||
servers want it, and Codex installs through npm).
|
||
- **initial-packages** (desktop-only remainder): cargo, rustup, flatpak, fwupd,
|
||
kitty, ksshaskpass, libselinux-utils, pciutils, python3-neovim,
|
||
wireguard-tools, vim-X11 — plus everything else currently there that the
|
||
core list did not claim.
|
||
- Codex CLI: installed in `install-packages` beside the Claude Code step —
|
||
skip when `codex` is on PATH, `npm install -g @openai/codex` (after nvm/node
|
||
is available), soft-fail like Claude Code does.
|
||
- The package-hash mechanic in `install` is untouched; new files under
|
||
`-maxdepth 1` join the hash automatically.
|
||
|
||
## 4. `setup-server` stage
|
||
|
||
Idempotent, like every stage; each item checks before acting:
|
||
|
||
1. `loginctl enable-linger $USER`.
|
||
2. `/etc/sysctl.d/99-rootless-ports.conf` → `net.ipv4.ip_unprivileged_port_start=80`,
|
||
applied with `sysctl --system` (junior already has exactly this file).
|
||
3. firewalld: open 80, 443, 81 (NPM admin portal), permanent + runtime.
|
||
Nothing else — reverse-proxy philosophy, ports open only when unavoidable.
|
||
4. `podman network create nginx-bridge` if absent (every compose expects it,
|
||
`external: true`).
|
||
5. Nightly image updates: `update-containers.sh` is adopted into the repo as
|
||
`server/scripts/update-containers` (no `.sh`, per repo rule — junior's copy
|
||
is imported and its `ExecStart`/log paths updated), with
|
||
`server/systemd/podman-update.service` and `.timer` symlinked into
|
||
`~/.config/systemd/user/` and the timer enabled. Logs go to
|
||
`~/Server/logs/` as today.
|
||
|
||
## 5. The `panama server` command
|
||
|
||
`bin/panama-server`, surfaced as `panama server <verb>` (README's subcommand
|
||
table and its contract update with it):
|
||
|
||
- `list` — every service in `server/containers/`, with enabled/active state.
|
||
- `enable <Name>` — create `~/Server/<Name>/`; symlink `compose.yml`; seed
|
||
`~/Server/<Name>/.env` from `.env.example` when the example exists and the
|
||
live file does not; if the seeded `.env` still contains placeholder values,
|
||
stop there and say what to fill in — do **not** start; otherwise symlink the
|
||
unit, `daemon-reload`, `enable --now`.
|
||
- `disable <Name>` — `disable --now`, remove the unit symlink,
|
||
`daemon-reload`. `~/Server/<Name>` and its data are never touched.
|
||
- `status [<Name>]` — unit state plus `podman compose ps` per service.
|
||
- `relink` — for every service whose unit symlink exists: refresh both
|
||
symlinks, `daemon-reload`, and report services whose definitions changed
|
||
since the last run (so the operator restarts deliberately). Called by the
|
||
`link-server` stage and by `panama update` on role=server.
|
||
|
||
Live `.env` files live in `~/Server/<Name>/`, not in the repo tree — the repo
|
||
carries only `.env.example`. `.gitignore` still covers `.env` everywhere under
|
||
`server/` as belt and braces.
|
||
|
||
## 6. VPS bootstrap (`boot` as root)
|
||
|
||
`boot` gains a root path. When `EUID == 0`, before anything else, walk these
|
||
steps, each checked and skipped when already true:
|
||
|
||
1. Ask for a username (default `gib`). Create it with `wheel` if absent; if it
|
||
exists, use it.
|
||
2. Ensure the user has a password (needed for sudo) — `passwd` interactively
|
||
if none is set.
|
||
3. Copy root's `authorized_keys` to the user when the user has none.
|
||
4. Offer sshd hardening (yes/no, default yes): drop-in under
|
||
`/etc/ssh/sshd_config.d/` with `PermitRootLogin no`,
|
||
`PasswordAuthentication no`, then reload sshd. Skipped when already set.
|
||
5. Move/clone the checkout under the user's home, chown it, and re-exec
|
||
`install --server` as that user.
|
||
|
||
Run as a normal user, `boot` behaves exactly as today (plus passing `--server`
|
||
through).
|
||
|
||
## 7. Importing junior's services (the first population)
|
||
|
||
A one-time operation, performed against junior over SSH, that **never stops,
|
||
starts, or restarts anything**. For each dedicated compose dir:
|
||
|
||
1. Copy `compose.yml` into `server/containers/<Name>/`.
|
||
2. Take the **installed** unit (`~/.config/systemd/user/podman-<name>.service`)
|
||
as truth; diff against the copy in the service dir and flag any divergence.
|
||
3. Move inline secrets out: secret-bearing values in the repo copy of
|
||
`compose.yml` become `${VAR}` interpolations; extracted values are written
|
||
to `~/Server/<Name>/.env` **on junior** (inert until cutover — the live
|
||
compose still carries them inline); `.env.example` with placeholder values
|
||
is committed. Services already using `.env`/`env_file` get an
|
||
`.env.example` generated from the live `.env` with values blanked.
|
||
4. Verify without touching the stack: `podman compose config` rendered from
|
||
the repo copy (with the live `.env`) must match the render of the live
|
||
compose. Volume paths are kept **verbatim** at import (`volumes/`,
|
||
`volume/`, `pg_data/` and all) so this diff can be exact; the `data/`
|
||
standardization happens per service at cutover.
|
||
5. Commit. The secrets contract (§8) gates every one of these commits.
|
||
|
||
**Import list (23):** Adminer, Agentchat, Authentik, Beszel, CalCom,
|
||
ConvexMonorepo, Documenso, Gitea, Infisical, ISponsorBlockTV, LashAddict, N8n,
|
||
Nginx_Proxy_Manager, Penpot, Portainer, PostgreSQL, Rustdesk, Spoon, StPeteIT,
|
||
Tools, Uptime, UseSend, Vaultwarden. CalCom, Documenso, Penpot and N8n are
|
||
currently parked (dir present, unit not active) — imported anyway, noted as
|
||
parked in the catalog.
|
||
|
||
**Excluded:** Watchtower (retired — replaced by the update timer after it took
|
||
gitea down; its dir stays on junior until cleanup), Completeuphoria,
|
||
sierraandtyler, Bang (source checkouts deployed by CI from their own repos —
|
||
Panama owning a second copy of their compose would drift), `ci/`, `logs/`,
|
||
and junior's loose scripts other than `update-containers.sh` (§4).
|
||
|
||
**Cutover** (per service, later, not this work): stop → rename the volume dir
|
||
to `data/` and update the repo compose to match → replace live compose with
|
||
the symlink, installed unit with the symlink → `daemon-reload` → start →
|
||
verify. Ten minutes a service, done when Gib chooses, favoring quiet hours.
|
||
|
||
## 8. Contracts and docs
|
||
|
||
New `tests/server/`, same discipline as the rest:
|
||
|
||
- `compose-secrets-contract` — fails when any tracked file under `server/`
|
||
contains an inline secret-looking value (the `user-content-contract`
|
||
pattern), when `.gitignore` stops covering `.env` under `server/`, or when
|
||
`git ls-files` shows a tracked `.env`.
|
||
- `containers-shape-contract` — every `server/containers/<Name>/` carries
|
||
`compose.yml` + `podman-<name>.service`; every unit's `WorkingDirectory` is
|
||
`%h/Server/<Name>`; every `${VAR}` used in a compose file appears in that
|
||
service's `.env.example`.
|
||
- `panama-server-contract` — throwaway `HOME`, stubbed `systemctl`/`podman` on
|
||
PATH: `enable` creates the right symlinks and seeds `.env`; placeholder
|
||
values block start; `disable` leaves data alone; `relink` reports changed
|
||
definitions.
|
||
- `role-contract` — the interview asks role first, `--server` presets it, the
|
||
role file is written, stages filter correctly per role (via the stage
|
||
scripts' static/stubbed seams).
|
||
- `dotfile-classification-contract` — every dir in `config/dot/` appears in
|
||
exactly one of link-dotfiles' universal/desktop lists.
|
||
- README: layout section gains `server/`, the subcommand table gains
|
||
`panama server`, the pinned contract count moves — `readme-contract` and the
|
||
interview/link-dotfiles contracts that assert today's behavior update in the
|
||
same change.
|
||
|
||
## Phasing
|
||
|
||
1. **Role plumbing** — interview question, role file, `--server`, package
|
||
split, stage filters, `setup-server`, boot's root path.
|
||
2. **Server skeleton** — `server/` tree, `panama-server`, `link-server`,
|
||
update-timer adoption, contracts, README/docs.
|
||
3. **Import** — junior's 23 services into `server/containers/` per §7.
|
||
4. **Later, separately** — per-service cutover on junior; backup and
|
||
verify-rootless design; server.gib docker→podman translation; a real
|
||
end-to-end test on a fresh VPS.
|
||
|
||
Testing note: contracts for phases 1–2 run with stubs and throwaway HOMEs, but
|
||
per standing practice all suite runs batch at the end of the session with
|
||
Gib's go-ahead. Phase 3 touches junior only with read commands plus the inert
|
||
`.env` writes described in §7.
|