Panama/.claude/docs/specs/2026-08-25-server-role.md

Panama server role: headless installs and versioned containers

2026-08-25. Designed with Gib across two interview rounds; every decision below was made explicitly, none inferred.

Goal

Panama becomes the one repo for every Fedora machine Gib runs. A desktop gets what it gets today. A server — a Hetzner VPS, a future home server — gets the same shell environment (bash profile, prompt, tmux-on-ssh, agents) minus everything that needs a screen, plus rootless podman, systemd user units, and a versioned catalog of compose services that panama server enable brings up. The repo stays public; secrets never enter it.

Decisions of record

Decision	Choice
Repo	One public monorepo. No companion repo. .gitignore + a secrets contract keep it safe
Role	Asked first in the interview (desktop / server), persisted to ~/.local/state/panama/role; --server on boot/install presets it
Layout	Top-level server/, services under server/containers/<Name>/ (TitleCase, matching junior)
Tracked per service	compose.yml, podman-<name>.service, .env.example. Nothing else
~/Server	Real directories. Only compose.yml is symlinked into them; units symlink into ~/.config/systemd/user/. Live .env and data stay outside the repo tree entirely
Data dirs	Standardized to ./data/, applied per service at cutover (a stop/rename/start), never at import
Enablement	panama server subcommand family, not the interview
Units	Checked in per service (they legitimately differ), installed as symlinks so edits flow with panama update
Updates	panama update relinks and daemon-reloads but never restarts a running service; it names services whose definitions changed
Nightly image updates	junior's update-containers.sh + podman-update.timer adopted into the repo (it replaced watchtower for cause)
Codex	Installed everywhere, same pattern as Claude Code
Ports	80/443/81 only. Anything else is opened by hand and documented in the service's folder
VPS bootstrap	boot run as root walks user-creation/keys/sshd-hardening, each step skipped when already true, then re-runs as the user
Out of scope	server.gib (docker; translate later), backup and verify-rootless (own design later), cutover of junior's live services (later, per service)

1. Machine role

• New interview question, asked first: "What is this machine?" — desktop or server via gum choose. ./install --server (and boot --server) records the answer without asking.
• Persisted to ~/.local/state/panama/role, plain text, one word. This is the first durable answer Panama keeps, and deliberately so: the transient-answers principle exists to keep personal data out of durable paths, and a machine role is a fact about the machine. install, panama update, panama doctor and future migrations read it. A machine with no role file is a desktop — every machine that predates this work is one.
• On role=server the interview skips: hardware probes (NVIDIA, Secure Boot, firmware — and their bootstrap tools; only gum is bootstrapped), the debloat question, and the extras checklist. It keeps: hostname, git identity, gh sign-in, SSH key, user-content.

2. Stages by role

Role=desktop is exactly today's behavior. Role=server runs:

Stage	Server behavior
install-packages	Core + server lists (§3). No flatpaks, no COPRs, no desktop repos. Claude Code and Codex install for both roles
link-dotfiles	Universal dot dirs only: btop, nvim, ohmyposh, tmux, vim, panama. Desktop-only (everything else: hypr, quickshell, kitty, ghostty, espanso, gtk-*, uwsm, vicinae, wofi, xdg-desktop-portal) is skipped via an explicit classification list in the stage — a new dot dir must be classified or the contract fails. Bash profile links as today; Firefox/wallpaper/cursor seeding skipped
link-user	Unchanged (SSH aliases and agent rules are exactly what a server wants)
setup-identity	Unchanged
setup-server (new)	§4
link-server (new)	Creates ~/Server, then delegates to panama-server relink so link repair has one owner
Skipped	link-skills (all three shipped skills are desktop-specific today; revisit when a server skill exists), change-settings, link-vicinae-scripts, install-hardware

Migrations run on servers too — they are self-guarding no-ops where they do not apply. panama doctor learns the role: on a server it reports service health via panama-server status instead of probing desktop units.

3. Package restructure

setup/packages/ gains core-packages (both roles) and server-packages; initial-packages keeps what is desktop-only. Nothing is added or dropped for desktops — the union of core + desktop lists equals today's set.

• core-packages: awk, bat, btop, curl, dnf5-plugins, eza, fontconfig, fzf, gh, git-all, gum, jq, neovim, oh-my-posh, openssh, openssl, podman, podman-compose, python3-dnf, ripgrep, tmux, unzip, wget, zoxide. (podman moves here from development-packages; podman-compose is what provides podman compose on junior — verified, 1.6.0.)
• server-packages: nvm (Node via nvm, as on desktop — agents and MCP servers want it, and Codex installs through npm).
• initial-packages (desktop-only remainder): cargo, rustup, flatpak, fwupd, kitty, ksshaskpass, libselinux-utils, pciutils, python3-neovim, wireguard-tools, vim-X11 — plus everything else currently there that the core list did not claim.
• Codex CLI: installed in install-packages beside the Claude Code step — skip when codex is on PATH, npm install -g @openai/codex (after nvm/node is available), soft-fail like Claude Code does.
• The package-hash mechanic in install is untouched; new files under -maxdepth 1 join the hash automatically.

4. setup-server stage

Idempotent, like every stage; each item checks before acting:

1. loginctl enable-linger $USER.
2. /etc/sysctl.d/99-rootless-ports.conf → net.ipv4.ip_unprivileged_port_start=80, applied with sysctl --system (junior already has exactly this file).
3. firewalld: open 80, 443, 81 (NPM admin portal), permanent + runtime. Nothing else — reverse-proxy philosophy, ports open only when unavoidable.
4. podman network create nginx-bridge if absent (every compose expects it, external: true).
5. Nightly image updates: update-containers.sh is adopted into the repo as server/scripts/update-containers (no .sh, per repo rule — junior's copy is imported and its ExecStart/log paths updated), with server/systemd/podman-update.service and .timer symlinked into ~/.config/systemd/user/ and the timer enabled. Logs go to ~/Server/logs/ as today.

5. The panama server command

bin/panama-server, surfaced as panama server <verb> (README's subcommand table and its contract update with it):

• list — every service in server/containers/, with enabled/active state.
• enable <Name> — create ~/Server/<Name>/; symlink compose.yml; seed ~/Server/<Name>/.env from .env.example when the example exists and the live file does not; if the seeded .env still contains placeholder values, stop there and say what to fill in — do not start; otherwise symlink the unit, daemon-reload, enable --now.
• disable <Name> — disable --now, remove the unit symlink, daemon-reload. ~/Server/<Name> and its data are never touched.
• status [<Name>] — unit state plus podman compose ps per service.
• relink — for every service whose unit symlink exists: refresh both symlinks, daemon-reload, and report services whose definitions changed since the last run (so the operator restarts deliberately). Called by the link-server stage and by panama update on role=server.

Live .env files live in ~/Server/<Name>/, not in the repo tree — the repo carries only .env.example. .gitignore still covers .env everywhere under server/ as belt and braces.

6. VPS bootstrap (boot as root)

boot gains a root path. When EUID == 0, before anything else, walk these steps, each checked and skipped when already true:

1. Ask for a username (default gib). Create it with wheel if absent; if it exists, use it.
2. Ensure the user has a password (needed for sudo) — passwd interactively if none is set.
3. Copy root's authorized_keys only after OpenSSH parses every key line. The destination directory and file are created as the target UID at 0700/0600 and revalidated before hardening is available.
4. Offer sshd hardening (yes/no, default yes): atomically install /etc/ssh/sshd_config.d/00-panama.conf with PermitRootLogin no, PasswordAuthentication no, and KbdInteractiveAuthentication no. Run sshd -t plus effective root/target sshd -T -C checks before reloading the detected unit. Restore a prior regular file with metadata on failure. A missing unit or unsupported existing drop-in leaves SSH unchanged and bootstrap continues. The later binding transaction design owns the details.
5. Move/clone the checkout under the user's home, chown it, and re-exec install --server as that user.

Run as a normal user, boot behaves exactly as today (plus passing --server through).

7. Importing junior's services (the first population)

A one-time operation, performed against junior over SSH, that never stops, starts, or restarts anything. For each dedicated compose dir:

1. Copy compose.yml into server/containers/<Name>/.
2. Take the installed unit (~/.config/systemd/user/podman-<name>.service) as truth; diff against the copy in the service dir and flag any divergence.
3. Move inline secrets out: secret-bearing values in the repo copy of compose.yml become ${VAR} interpolations; extracted values are written to ~/Server/<Name>/.env on junior (inert until cutover — the live compose still carries them inline); .env.example with placeholder values is committed. Services already using .env/env_file get an .env.example generated from the live .env with values blanked.
4. Verify without touching the stack: podman compose config rendered from the repo copy (with the live .env) must match the render of the live compose. Volume paths are kept verbatim at import (volumes/, volume/, pg_data/ and all) so this diff can be exact; the data/ standardization happens per service at cutover.
5. Commit. The secrets contract (§8) gates every one of these commits.

Import list (23): Adminer, Agentchat, Authentik, Beszel, CalCom, ConvexMonorepo, Documenso, Gitea, Infisical, ISponsorBlockTV, LashAddict, N8n, Nginx_Proxy_Manager, Penpot, Portainer, PostgreSQL, Rustdesk, Spoon, StPeteIT, Tools, Uptime, UseSend, Vaultwarden. CalCom, Documenso, Penpot and N8n are currently parked (dir present, unit not active) — imported anyway, noted as parked in the catalog.

Excluded: Watchtower (retired — replaced by the update timer after it took gitea down; its dir stays on junior until cleanup), Completeuphoria, sierraandtyler, Bang (source checkouts deployed by CI from their own repos — Panama owning a second copy of their compose would drift), ci/, logs/, and junior's loose scripts other than update-containers.sh (§4).

Cutover (per service, later, not this work): stop → rename the volume dir to data/ and update the repo compose to match → replace live compose with the symlink, installed unit with the symlink → daemon-reload → start → verify. Ten minutes a service, done when Gib chooses, favoring quiet hours.

8. Contracts and docs

New tests/server/, same discipline as the rest:

• compose-secrets-contract — fails when any tracked file under server/ contains an inline secret-looking value (the user-content-contract pattern), when .gitignore stops covering .env under server/, or when git ls-files shows a tracked .env.
• containers-shape-contract — every server/containers/<Name>/ carries compose.yml + podman-<name>.service; every unit's WorkingDirectory is %h/Server/<Name>; every ${VAR} used in a compose file appears in that service's .env.example.
• panama-server-contract — throwaway HOME, stubbed systemctl/podman on PATH: enable creates the right symlinks and seeds .env; placeholder values block start; disable leaves data alone; relink reports changed definitions.
• role-contract — the interview asks role first, --server presets it, the role file is written, stages filter correctly per role (via the stage scripts' static/stubbed seams).
• dotfile-classification-contract — every dir in config/dot/ appears in exactly one of link-dotfiles' universal/desktop lists.
• README: layout section gains server/, the subcommand table gains panama server, the pinned contract count moves — readme-contract and the interview/link-dotfiles contracts that assert today's behavior update in the same change.

Phasing

1. Role plumbing — interview question, role file, --server, package split, stage filters, setup-server, boot's root path.
2. Server skeleton — server/ tree, panama-server, link-server, update-timer adoption, contracts, README/docs.
3. Import — junior's 23 services into server/containers/ per §7.
4. Later, separately — per-service cutover on junior; backup and verify-rootless design; server.gib docker→podman translation; a real end-to-end test on a fresh VPS.

Testing note: contracts for phases 1–2 run with stubs and throwaway HOMEs, but per standing practice all suite runs batch at the end of the session with Gib's go-ahead. Phase 3 touches junior only with read commands plus the inert .env writes described in §7.