Panama/docs/superpowers/specs/2026-08-25-escalation-ladder-and-usage.md

The escalation ladder, and usage worth trusting

Approved from mock agents.html (2026-08-25). Two builds in one cycle: no failure surface is a dead end (crash → one click → your agent mid-investigation), and the bar's usage number becomes a multi-agent panel backed by Omarchy's battle-tested collectors. Borrowed code is MIT — carry Copyright (c) David Heinemeier Hansson + the MIT permission notice in a header comment on every derived file, and add an Omarchy entry to docs/UPSTREAM-INSPIRATION.md.

Decisions log

Decision	Choice
Scope	Ladder + usage panel, one cycle
Ladder surfaces v1	App crashes, shell reload failures, Health red checks, update/migration failures, plus panama diagnose by hand
Agent	Configurable via preferredAgent; Claude and Codex both fully supported on the ladder (per-agent argv table); default none — quiet until chosen, like Omarchy
Permission posture	Auto-approve (user's explicit choice), behind a toggle; the diagnose skill still teaches reads-not-fixes; root still via panama-sudo
Usage	Collectors ported near-verbatim (Claude + Codex), Panama-native popover panel, no cross-machine sync
Settings home	New agents leaf under System
Notification click mechanism	Omarchy's command-as-data hint (panama-exec), executed by the shell — survives restarts, never blocks the watcher

Schema (orchestrator adds first — do not re-add)

key	type	def	group	notes
preferredAgent	enum none/claude/codex	none	agents	none = crash toasts carry no action
crashDiagnoseOffer	bool	true	agents	gates the crash rung (still requires an agent chosen)
reloadFailureOffer	bool	true	agents	shell-reload rung
healthAgentHandoff	bool	true	agents	Health-page rung
agentAutoApprove	bool	true	agents	off → agents launch in their default prompting mode
agentUsageClaude / agentUsageCodex	bool	true	agents	per-collector enables
agentUsageRefreshMinutes	int 5–60	15	agents	collector fan-out interval

showAgentUsage stays where it is (vitals) and keeps its meaning; the Bar page row remains, the Agents page gets the full set (declared mirror in settings-ownership-contract if both surface the same key). New group agents needs a groupPages route to the new leaf (search-routing-contract).

The launcher: bin/panama-agent

Ported from omarchy-agent, trimmed to Panama's needs, attribution header included.

• Reads preferredAgent and agentAutoApprove from ~/.config/panama/settings.json with jq at press time (the power-button precedent) — none exits 0 silently.
• cd "$PANAMA_PATH" before spawning: the checkout is where the skills, the repo trust, and the pre-approved diagnostic permissions live.
• Per-agent argv table — verify every flag against the installed binaries' --help, do not trust the Omarchy source or this spec: Claude ≈ claude [auto-approve flag] -- "$prompt", Codex ≈ codex [auto-approve flag] -- "$prompt" (Omarchy used --permission-mode auto and --approve-for-me; use whatever the real flags are, and the prompting-mode variants when agentAutoApprove is off).
• Spawn: setsid kitty --directory "$PANAMA_PATH" -e <argv…> (kitty is the house terminal, its own health check exists; --app-id-style fixed class panama-agent so window rules can target it later).
• --prompt "text" input; no --pick menu in v1 (the Agents page is the picker).
• Repo .claude/settings.json gains a pre-approved allowlist for the read-only diagnostic commands (coredumpctl, journalctl, rpm -q, panama doctor) so even non-auto-approve launches investigate smoothly.

The rungs

1. Crashes. bin/panama-crash-watch gains COREDUMP_PID + COREDUMP_SIGNAL_NAME extraction. When preferredAgent != none and crashDiagnoseOffer, the notification carries the command as data: notify-send … --hint=string:panama-exec:"$(printf 'panama-agent-crash %q %q %q %q' pid comm exe signal)" with body "Click to diagnose with ". Otherwise today's actionless notification, verbatim. Existing session-dedupe and portal-crash rationale untouched. New bin/panama-agent-crash ports omarchy-agent-crash: coredump facts + coredumpctl list timestamp into a heredoc prompt naming the diagnose-crash skill with its absolute path as fallback (the harness-agnostic trick — this is what makes the ladder work for Codex too), then exec panama-agent --prompt "$prompt".

2. The panama-exec hint. services/Notifs.qml: read hint panama-exec into the notification model (one field, carried through popup persistence like any other). modules/notifications/NotificationCard.qml: on body click, if the entry carries an exec, run it via Quickshell.execDetached(["sh", "-c", command]) and dismiss — the branch sits beside the existing defaultAction handling and defers to it when both exist. Security note in-file: the session bus is local; any sender could carry the hint, which grants nothing an local process lacks.

3. Shell reload failure. shell.qml's onReloadFailed (old shell keeps running — it can still notify): when reloadFailureOffer and an agent is chosen, send a notification through the shell's own Notifs with the exec hint → new bin/panama-agent-reload builds the prompt from the failure string plus the last ~40 matching journal lines.

4. Health page. Red check with no repair, or whose repair just failed → an "Ask the agent" SettingsButton beside it (visible only when healthAgentHandoff and an agent chosen), running panama-agent --prompt with the check's JSON snapshot (panama doctor check <id>) embedded.

5. Update/migration failures. The failure summary in install gains one line: Hand it to an agent: panama diagnose. bin/panama-migrate-notify's failure path likewise.

6. panama diagnose (cmd_diagnose). By-hand rung: bundles panama doctor --summary, the last journal errors, and "what feels wrong" free text (panama diagnose [words…]) into a prompt and hands it to panama-agent. House three-touchpoint rule + README list (readme-contract).

The skill: skills/diagnose-crash/

Written by the orchestrator, shipped un-gated like the other two, structure adapted from Omarchy's: triggers name the notification and panama-agent-crash; coredumpctl info <pid> first and read the command line; rule out OOM before blaming the program; correlate the crash time against recent package updates and journal context; Fedora debuginfod for symbols; the security rule (a core holds secrets — scratch files via mktemp, deleted after); "never invent function names"; "diagnosis reads; it does not fix"; report structure. Claims auto-pinned by skills-contract (it already scans every skill in skills/).

Usage collectors and panel

• config/dot/quickshell/scripts/panama-agent-usage-claude — port of Omarchy's Python collector (attribution header): OAuth usage endpoint + model-scoped limits[], transcript scan with message-id dedupe and scan cache, cached-limits-expire-on-window-rollover, waiting-vs-expired auth states, retryAdvised transport/HTTP distinction, percent-scale voting. Keep Panama's secrets discipline: the token stays out of argv and out of the record (their in-process Python read already satisfies it; the contract proves it).
• …/panama-agent-usage-codex — port: app-server JSON-RPC limits, session-file token stats with the last_token_usage rule.
• …/panama-agent-usage-update — fan-out: runs enabled collectors in parallel, jq -e validity gate, atomic writes to $XDG_STATE_HOME/panama/agents/usage/<agent>.json.
• services/AgentUsage.qml → directory model: one FileView per record file, discovered by listing the dir; aggregate headline = fullest window across enabled agents; timer from agentUsageRefreshMinutes, running only while showAgentUsage.
• modules/bar/AgentUsageWidget.qml: keeps the two-tier amber-75/red-90 coloring and the hide-when-nothing rule; click now opens the panel popover (right-click keeps Settings).
• New modules/bar/AgentUsagePanel.qml (or popover home matching how other bar popovers are built — follow the house pattern found in the code): per the mock — agent tabs, tier chip, updated-ago line, limit meters with reset countdowns (model-scoped rows included), tokens by day (today bolded), by-model bars, honest auth states. No keyboard-nav requirement in v1; Escape closes like every popover.
• Old scripts/panama-agent-usage retires (delete; its consumers migrate). The five pins of tests/quickshell/agent-usage-contract survive against the new collectors: token never in argv, never in any record, credentials never written, expired token → no request + honest state, widget hidden unless asked-for AND real numbers. Extend for: per-agent enables honored, record validity gate, atomic write.

New settings leaf

agents under the System category: 4-file ceremony (SettingsRoutes categories entry, SettingsShell case + Component, qmldir, AgentsPage.qml), search entries, regen docs+commands (orchestrator). Page per the mock: preferred-agent picker (install-state aware: show "not installed" detail for an absent binary), the four escalation toggles (auto-approve copy states the trade in plain words), usage card (master switch mirroring showAgentUsage, per-agent enables, refresh interval).

Contracts

• crash-watch-contract: extend for PID/signal fields and hint gating (none-agent → no hint; chosen → hint present; never both mechanisms).
• agent-usage-contract: as above.
• New tests/quickshell/panama-agent-contract: hermetic — stubbed settings.json + stub kitty/claude/codex on PATH; none → silent exit; claude/codex → correct argv shape including the auto-approve/prompting variants; prompt passed as one argv element; cwd is the repo. panama-agent-crash: prompt contains pid/comm/exe/signal + the skill path; never runs a real agent (stub PATH). NOT in the desktop-hijacking ledger — it must run stubbed.
• skills-contract covers diagnose-crash automatically once the skill lands.
• settings-nav/search-routing/settings-docs/readme-contract obligations as usual.

Ownership (disjoint)

• Orchestrator: this spec; schema keys first; the diagnose-crash skill; UPSTREAM-INSPIRATION.md + license attribution file check; docs/commands regen; README count; seam audit.
• Agent A — usage: the three collector scripts, AgentUsage.qml, AgentUsageWidget.qml, AgentUsagePanel.qml (+ its qmldir if bar modules use one), agent-usage-contract, deletion of the old collector.
• Agent B — ladder scripts: bin/panama-agent, bin/panama-agent-crash, bin/panama-agent-reload, bin/panama-crash-watch, bin/panama (cmd_diagnose), install failure line, bin/panama-migrate-notify failure line, .claude/settings.json allowlist, crash-watch-contract, panama-agent-contract, README subcommand row.
• Agent C — shell & settings UI: services/Notifs.qml (hint field), NotificationCard.qml (exec branch), shell.qml (onReloadFailed rung), modules/settings/HealthPage.qml ("Ask the agent"), AgentsPage.qml + routes/shell-case/qmldir, search entries in SettingsSearch.qml.

Standing rules apply: live desktop (valid QML every save, journal checks), no real agent launches or notifications left uncleaned during builds, contracts as you go, never the desktop-hijacking set unannounced.