# Panama server role: headless installs and versioned containers

2026-08-25. Designed with Gib across two interview rounds; every decision below
was made explicitly, none inferred.

## Goal

Panama becomes the one repo for every Fedora machine Gib runs. A desktop gets
what it gets today. A server — a Hetzner VPS, a future home server — gets the
same shell environment (bash profile, prompt, tmux-on-ssh, agents) minus
everything that needs a screen, plus rootless podman, systemd user units, and a
versioned catalog of compose services that `panama server enable` brings up.
The repo stays public; secrets never enter it.

## Decisions of record

| Decision | Choice |
|---|---|
| Repo | One public monorepo. No companion repo. `.gitignore` + a secrets contract keep it safe |
| Role | Asked first in the interview (`desktop` / `server`), persisted to `~/.local/state/panama/role`; `--server` on `boot`/`install` presets it |
| Layout | Top-level `server/`, services under `server/containers/<Name>/` (TitleCase, matching junior) |
| Tracked per service | `compose.yml`, `podman-<name>.service`, `.env.example`. Nothing else |
| `~/Server` | Real directories. Only `compose.yml` is symlinked into them; units symlink into `~/.config/systemd/user/`. Live `.env` and data stay outside the repo tree entirely |
| Data dirs | Standardized to `./data/`, applied per service at cutover (a stop/rename/start), never at import |
| Enablement | `panama server` subcommand family, not the interview |
| Units | Checked in per service (they legitimately differ), installed as symlinks so edits flow with `panama update` |
| Updates | `panama update` relinks and `daemon-reload`s but never restarts a running service; it names services whose definitions changed |
| Nightly image updates | junior's `update-containers.sh` + `podman-update.timer` adopted into the repo (it replaced watchtower for cause) |
| Codex | Installed everywhere, same pattern as Claude Code |
| Ports | 80/443/81 only. Anything else is opened by hand and documented in the service's folder |
| VPS bootstrap | `boot` run as root walks user-creation/keys/sshd-hardening, each step skipped when already true, then re-runs as the user |
| Out of scope | server.gib (docker; translate later), backup and verify-rootless (own design later), cutover of junior's live services (later, per service) |

## 1. Machine role

- New interview question, asked first: "What is this machine?" — `desktop` or
  `server` via `gum choose`. `./install --server` (and `boot --server`) records
  the answer without asking.
- Persisted to `~/.local/state/panama/role`, plain text, one word. This is the
  first durable answer Panama keeps, and deliberately so: the transient-answers
  principle exists to keep *personal* data out of durable paths, and a machine
  role is a fact about the machine. `install`, `panama update`, `panama doctor`
  and future migrations read it. A machine with no role file is a desktop —
  every machine that predates this work is one.
- On role=server the interview skips: hardware probes (NVIDIA, Secure Boot,
  firmware — and their bootstrap tools; only `gum` is bootstrapped), the
  debloat question, and the extras checklist. It keeps: hostname, git
  identity, gh sign-in, SSH key, user-content.

## 2. Stages by role

Role=desktop is exactly today's behavior. Role=server runs:

| Stage | Server behavior |
|---|---|
| `install-packages` | Core + server lists (§3). No flatpaks, no COPRs, no desktop repos. Claude Code and Codex install for both roles |
| `link-dotfiles` | Universal dot dirs only: `btop`, `nvim`, `ohmyposh`, `tmux`, `vim`, `panama`. Desktop-only (everything else: `hypr`, `quickshell`, `kitty`, `ghostty`, `espanso`, `gtk-*`, `uwsm`, `vicinae`, `wofi`, `xdg-desktop-portal`) is skipped via an explicit classification list in the stage — a new dot dir must be classified or the contract fails. Bash profile links as today; Firefox/wallpaper/cursor seeding skipped |
| `link-user` | Unchanged (SSH aliases and agent rules are exactly what a server wants) |
| `setup-identity` | Unchanged |
| `setup-server` (new) | §4 |
| `link-server` (new) | Creates `~/Server`, then delegates to `panama-server relink` so link repair has one owner |
| Skipped | `link-skills` (all three shipped skills are desktop-specific today; revisit when a server skill exists), `change-settings`, `link-vicinae-scripts`, `install-hardware` |

Migrations run on servers too — they are self-guarding no-ops where they do
not apply. `panama doctor` learns the role: on a server it reports service
health via `panama-server status` instead of probing desktop units.

## 3. Package restructure

`setup/packages/` gains `core-packages` (both roles) and `server-packages`;
`initial-packages` keeps what is desktop-only. Nothing is added or dropped for
desktops — the union of core + desktop lists equals today's set.

- **core-packages**: awk, bat, btop, curl, dnf5-plugins, eza, fontconfig, fzf,
  gh, git-all, gum, jq, neovim, oh-my-posh, openssh, openssl, podman,
  podman-compose, python3-dnf, ripgrep, tmux, unzip, wget, zoxide.
  (`podman` moves here from development-packages; `podman-compose` is what
  provides `podman compose` on junior — verified, 1.6.0.)
- **server-packages**: nvm (Node via nvm, as on desktop — agents and MCP
  servers want it, and Codex installs through npm).
- **initial-packages** (desktop-only remainder): cargo, rustup, flatpak, fwupd,
  kitty, ksshaskpass, libselinux-utils, pciutils, python3-neovim,
  wireguard-tools, vim-X11 — plus everything else currently there that the
  core list did not claim.
- Codex CLI: installed in `install-packages` beside the Claude Code step —
  skip when `codex` is on PATH, `npm install -g @openai/codex` (after nvm/node
  is available), soft-fail like Claude Code does.
- The package-hash mechanic in `install` is untouched; new files under
  `-maxdepth 1` join the hash automatically.

## 4. `setup-server` stage

Idempotent, like every stage; each item checks before acting:

1. `loginctl enable-linger $USER`.
2. `/etc/sysctl.d/99-rootless-ports.conf` → `net.ipv4.ip_unprivileged_port_start=80`,
   applied with `sysctl --system` (junior already has exactly this file).
3. firewalld: open 80, 443, 81 (NPM admin portal), permanent + runtime.
   Nothing else — reverse-proxy philosophy, ports open only when unavoidable.
4. `podman network create nginx-bridge` if absent (every compose expects it,
   `external: true`).
5. Nightly image updates: `update-containers.sh` is adopted into the repo as
   `server/scripts/update-containers` (no `.sh`, per repo rule — junior's copy
   is imported and its `ExecStart`/log paths updated), with
   `server/systemd/podman-update.service` and `.timer` symlinked into
   `~/.config/systemd/user/` and the timer enabled. Logs go to
   `~/Server/logs/` as today.

## 5. The `panama server` command

`bin/panama-server`, surfaced as `panama server <verb>` (README's subcommand
table and its contract update with it):

- `list` — every service in `server/containers/`, with enabled/active state.
- `enable <Name>` — create `~/Server/<Name>/`; symlink `compose.yml`; seed
  `~/Server/<Name>/.env` from `.env.example` when the example exists and the
  live file does not; if the seeded `.env` still contains placeholder values,
  stop there and say what to fill in — do **not** start; otherwise symlink the
  unit, `daemon-reload`, `enable --now`.
- `disable <Name>` — `disable --now`, remove the unit symlink,
  `daemon-reload`. `~/Server/<Name>` and its data are never touched.
- `status [<Name>]` — unit state plus `podman compose ps` per service.
- `relink` — for every service whose unit symlink exists: refresh both
  symlinks, `daemon-reload`, and report services whose definitions changed
  since the last run (so the operator restarts deliberately). Called by the
  `link-server` stage and by `panama update` on role=server.

Live `.env` files live in `~/Server/<Name>/`, not in the repo tree — the repo
carries only `.env.example`. `.gitignore` still covers `.env` everywhere under
`server/` as belt and braces.

## 6. VPS bootstrap (`boot` as root)

`boot` gains a root path. When `EUID == 0`, before anything else, walk these
steps, each checked and skipped when already true:

1. Ask for a username (default `gib`). Create it with `wheel` if absent; if it
   exists, use it.
2. Ensure the user has a password (needed for sudo) — `passwd` interactively
   if none is set.
3. Copy root's `authorized_keys` only after OpenSSH parses every key line. The
   destination directory and file are created as the target UID at `0700/0600`
   and revalidated before hardening is available.
4. Offer sshd hardening (yes/no, default yes): atomically install
   `/etc/ssh/sshd_config.d/00-panama.conf` with `PermitRootLogin no`,
   `PasswordAuthentication no`, and `KbdInteractiveAuthentication no`. Run
   `sshd -t` plus effective root/target `sshd -T -C` checks before reloading the
   detected unit. Restore a prior regular file with metadata on failure. A
   missing unit or unsupported existing drop-in leaves SSH unchanged and
   bootstrap continues. The later binding transaction design owns the details.
5. Move/clone the checkout under the user's home, chown it, and re-exec
   `install --server` as that user.

Run as a normal user, `boot` behaves exactly as today (plus passing `--server`
through).

## 7. Importing junior's services (the first population)

A one-time operation, performed against junior over SSH, that **never stops,
starts, or restarts anything**. For each dedicated compose dir:

1. Copy `compose.yml` into `server/containers/<Name>/`.
2. Take the **installed** unit (`~/.config/systemd/user/podman-<name>.service`)
   as truth; diff against the copy in the service dir and flag any divergence.
3. Move inline secrets out: secret-bearing values in the repo copy of
   `compose.yml` become `${VAR}` interpolations; extracted values are written
   to `~/Server/<Name>/.env` **on junior** (inert until cutover — the live
   compose still carries them inline); `.env.example` with placeholder values
   is committed. Services already using `.env`/`env_file` get an
   `.env.example` generated from the live `.env` with values blanked.
4. Verify without touching the stack: `podman compose config` rendered from
   the repo copy (with the live `.env`) must match the render of the live
   compose. Volume paths are kept **verbatim** at import (`volumes/`,
   `volume/`, `pg_data/` and all) so this diff can be exact; the `data/`
   standardization happens per service at cutover.
5. Commit. The secrets contract (§8) gates every one of these commits.

**Import list (23):** Adminer, Agentchat, Authentik, Beszel, CalCom,
ConvexMonorepo, Documenso, Gitea, Infisical, ISponsorBlockTV, LashAddict, N8n,
Nginx_Proxy_Manager, Penpot, Portainer, PostgreSQL, Rustdesk, Spoon, StPeteIT,
Tools, Uptime, UseSend, Vaultwarden. CalCom, Documenso, Penpot and N8n are
currently parked (dir present, unit not active) — imported anyway, noted as
parked in the catalog.

**Excluded:** Watchtower (retired — replaced by the update timer after it took
gitea down; its dir stays on junior until cleanup), Completeuphoria,
sierraandtyler, Bang (source checkouts deployed by CI from their own repos —
Panama owning a second copy of their compose would drift), `ci/`, `logs/`,
and junior's loose scripts other than `update-containers.sh` (§4).

**Cutover** (per service, later, not this work): stop → rename the volume dir
to `data/` and update the repo compose to match → replace live compose with
the symlink, installed unit with the symlink → `daemon-reload` → start →
verify. Ten minutes a service, done when Gib chooses, favoring quiet hours.

## 8. Contracts and docs

New `tests/server/`, same discipline as the rest:

- `compose-secrets-contract` — fails when any tracked file under `server/`
  contains an inline secret-looking value (the `user-content-contract`
  pattern), when `.gitignore` stops covering `.env` under `server/`, or when
  `git ls-files` shows a tracked `.env`.
- `containers-shape-contract` — every `server/containers/<Name>/` carries
  `compose.yml` + `podman-<name>.service`; every unit's `WorkingDirectory` is
  `%h/Server/<Name>`; every `${VAR}` used in a compose file appears in that
  service's `.env.example`.
- `panama-server-contract` — throwaway `HOME`, stubbed `systemctl`/`podman` on
  PATH: `enable` creates the right symlinks and seeds `.env`; placeholder
  values block start; `disable` leaves data alone; `relink` reports changed
  definitions.
- `role-contract` — the interview asks role first, `--server` presets it, the
  role file is written, stages filter correctly per role (via the stage
  scripts' static/stubbed seams).
- `dotfile-classification-contract` — every dir in `config/dot/` appears in
  exactly one of link-dotfiles' universal/desktop lists.
- README: layout section gains `server/`, the subcommand table gains
  `panama server`, the pinned contract count moves — `readme-contract` and the
  interview/link-dotfiles contracts that assert today's behavior update in the
  same change.

## Phasing

1. **Role plumbing** — interview question, role file, `--server`, package
   split, stage filters, `setup-server`, boot's root path.
2. **Server skeleton** — `server/` tree, `panama-server`, `link-server`,
   update-timer adoption, contracts, README/docs.
3. **Import** — junior's 23 services into `server/containers/` per §7.
4. **Later, separately** — per-service cutover on junior; backup and
   verify-rootless design; server.gib docker→podman translation; a real
   end-to-end test on a fresh VPS.

Testing note: contracts for phases 1–2 run with stubs and throwaway HOMEs, but
per standing practice all suite runs batch at the end of the session with
Gib's go-ahead. Phase 3 touches junior only with read commands plus the inert
`.env` writes described in §7.