# The escalation ladder, and usage worth trusting

Approved from mock `agents.html` (2026-08-25). Two builds in one cycle: no failure surface is a
dead end (crash → one click → your agent mid-investigation), and the bar's usage number becomes a
multi-agent panel backed by Omarchy's battle-tested collectors. Borrowed code is MIT — carry
`Copyright (c) David Heinemeier Hansson` + the MIT permission notice in a header comment on every
derived file, and add an Omarchy entry to `docs/UPSTREAM-INSPIRATION.md`.

## Decisions log

| Decision | Choice |
|---|---|
| Scope | Ladder + usage panel, one cycle |
| Ladder surfaces v1 | App crashes, shell reload failures, Health red checks, update/migration failures, plus `panama diagnose` by hand |
| Agent | Configurable via `preferredAgent`; **Claude and Codex both fully supported on the ladder** (per-agent argv table); default `none` — quiet until chosen, like Omarchy |
| Permission posture | Auto-approve (user's explicit choice), behind a toggle; the diagnose skill still teaches reads-not-fixes; root still via `panama-sudo` |
| Usage | Collectors ported near-verbatim (Claude + Codex), Panama-native popover panel, no cross-machine sync |
| Settings home | New `agents` leaf under System |
| Notification click mechanism | Omarchy's command-as-data hint (`panama-exec`), executed by the shell — survives restarts, never blocks the watcher |

## Schema (orchestrator adds first — do not re-add)

| key | type | def | group | notes |
|---|---|---|---|---|
| `preferredAgent` | enum `none/claude/codex` | `none` | agents | `none` = crash toasts carry no action |
| `crashDiagnoseOffer` | bool | true | agents | gates the crash rung (still requires an agent chosen) |
| `reloadFailureOffer` | bool | true | agents | shell-reload rung |
| `healthAgentHandoff` | bool | true | agents | Health-page rung |
| `agentAutoApprove` | bool | true | agents | off → agents launch in their default prompting mode |
| `agentUsageClaude` / `agentUsageCodex` | bool | true | agents | per-collector enables |
| `agentUsageRefreshMinutes` | int 5–60 | 15 | agents | collector fan-out interval |

`showAgentUsage` stays where it is (vitals) and keeps its meaning; the Bar page row remains, the
Agents page gets the full set (declared mirror in `settings-ownership-contract` if both surface
the same key). New group `agents` needs a `groupPages` route to the new leaf
(`search-routing-contract`).

## The launcher: `bin/panama-agent`

Ported from `omarchy-agent`, trimmed to Panama's needs, attribution header included.

- Reads `preferredAgent` and `agentAutoApprove` from `~/.config/panama/settings.json` with jq at
  press time (the power-button precedent) — `none` exits 0 silently.
- `cd "$PANAMA_PATH"` before spawning: the checkout is where the skills, the repo trust, and the
  pre-approved diagnostic permissions live.
- Per-agent argv table — **verify every flag against the installed binaries' `--help`, do not
  trust the Omarchy source or this spec**: Claude ≈ `claude [auto-approve flag] -- "$prompt"`,
  Codex ≈ `codex [auto-approve flag] -- "$prompt"` (Omarchy used `--permission-mode auto` and
  `--approve-for-me`; use whatever the real flags are, and the prompting-mode variants when
  `agentAutoApprove` is off).
- Spawn: `setsid kitty --directory "$PANAMA_PATH" -e <argv…>` (kitty is the house terminal, its
  own health check exists; `--app-id`-style fixed class `panama-agent` so window rules can target
  it later).
- `--prompt "text"` input; no `--pick` menu in v1 (the Agents page is the picker).
- Repo `.claude/settings.json` gains a pre-approved allowlist for the read-only diagnostic
  commands (`coredumpctl`, `journalctl`, `rpm -q`, `panama doctor`) so even non-auto-approve
  launches investigate smoothly.

## The rungs

**1. Crashes.** `bin/panama-crash-watch` gains `COREDUMP_PID` + `COREDUMP_SIGNAL_NAME`
extraction. When `preferredAgent != none` and `crashDiagnoseOffer`, the notification carries the
command as data: `notify-send … --hint=string:panama-exec:"$(printf 'panama-agent-crash %q %q %q %q' pid comm exe signal)"`
with body "Click to diagnose with <Agent>". Otherwise today's actionless notification, verbatim.
Existing session-dedupe and portal-crash rationale untouched. New `bin/panama-agent-crash` ports
`omarchy-agent-crash`: coredump facts + `coredumpctl list` timestamp into a heredoc prompt naming
the `diagnose-crash` skill **with its absolute path as fallback** (the harness-agnostic trick —
this is what makes the ladder work for Codex too), then `exec panama-agent --prompt "$prompt"`.

**2. The `panama-exec` hint.** `services/Notifs.qml`: read hint `panama-exec` into the
notification model (one field, carried through popup persistence like any other).
`modules/notifications/NotificationCard.qml`: on body click, if the entry carries an exec, run it
via `Quickshell.execDetached(["sh", "-c", command])` and dismiss — the branch sits beside the
existing `defaultAction` handling and defers to it when both exist. Security note in-file: the
session bus is local; any sender could carry the hint, which grants nothing an local process
lacks.

**3. Shell reload failure.** `shell.qml`'s `onReloadFailed` (old shell keeps running — it can
still notify): when `reloadFailureOffer` and an agent is chosen, send a notification through the
shell's own Notifs with the exec hint → new `bin/panama-agent-reload` builds the prompt from the
failure string plus the last ~40 matching journal lines.

**4. Health page.** Red check with no repair, or whose repair just failed → an "Ask the agent"
SettingsButton beside it (visible only when `healthAgentHandoff` and an agent chosen), running
`panama-agent --prompt` with the check's JSON snapshot (`panama doctor check <id>`) embedded.

**5. Update/migration failures.** The failure summary in `install` gains one line:
`Hand it to an agent: panama diagnose`. `bin/panama-migrate-notify`'s failure path likewise.

**6. `panama diagnose` (cmd_diagnose).** By-hand rung: bundles `panama doctor --summary`, the
last journal errors, and "what feels wrong" free text (`panama diagnose [words…]`) into a prompt
and hands it to `panama-agent`. House three-touchpoint rule + README list (readme-contract).

## The skill: `skills/diagnose-crash/`

Written by the orchestrator, shipped un-gated like the other two, structure adapted from
Omarchy's: triggers name the notification and `panama-agent-crash`; `coredumpctl info <pid>`
first and read the command line; rule out OOM before blaming the program; correlate the crash
time against recent package updates and journal context; Fedora debuginfod for symbols; the
security rule (a core holds secrets — scratch files via mktemp, deleted after); "never invent
function names"; "diagnosis reads; it does not fix"; report structure. Claims auto-pinned by
`skills-contract` (it already scans every skill in `skills/`).

## Usage collectors and panel

- `config/dot/quickshell/scripts/panama-agent-usage-claude` — port of Omarchy's Python collector
  (attribution header): OAuth usage endpoint + model-scoped `limits[]`, transcript scan with
  message-id dedupe and scan cache, cached-limits-expire-on-window-rollover, waiting-vs-expired
  auth states, `retryAdvised` transport/HTTP distinction, percent-scale voting. Keep Panama's
  secrets discipline: the token stays out of argv and out of the record (their in-process Python
  read already satisfies it; the contract proves it).
- `…/panama-agent-usage-codex` — port: app-server JSON-RPC limits, session-file token stats with
  the last_token_usage rule.
- `…/panama-agent-usage-update` — fan-out: runs enabled collectors in parallel, `jq -e` validity
  gate, atomic writes to `$XDG_STATE_HOME/panama/agents/usage/<agent>.json`.
- `services/AgentUsage.qml` → directory model: one FileView per record file, discovered by
  listing the dir; aggregate `headline` = fullest window across enabled agents; timer from
  `agentUsageRefreshMinutes`, running only while `showAgentUsage`.
- `modules/bar/AgentUsageWidget.qml`: keeps the two-tier amber-75/red-90 coloring and the
  hide-when-nothing rule; click now opens the panel popover (right-click keeps Settings).
- New `modules/bar/AgentUsagePanel.qml` (or popover home matching how other bar popovers are
  built — follow the house pattern found in the code): per the mock — agent tabs, tier chip,
  updated-ago line, limit meters with reset countdowns (model-scoped rows included), tokens by
  day (today bolded), by-model bars, honest auth states. No keyboard-nav requirement in v1;
  Escape closes like every popover.
- Old `scripts/panama-agent-usage` retires (delete; its consumers migrate). The five pins of
  `tests/quickshell/agent-usage-contract` survive against the new collectors: token never in
  argv, never in any record, credentials never written, expired token → no request + honest
  state, widget hidden unless asked-for AND real numbers. Extend for: per-agent enables honored,
  record validity gate, atomic write.

## New settings leaf

`agents` under the System category: 4-file ceremony (SettingsRoutes categories entry,
SettingsShell case + Component, qmldir, `AgentsPage.qml`), search entries, regen docs+commands
(orchestrator). Page per the mock: preferred-agent picker (install-state aware: show "not
installed" detail for an absent binary), the four escalation toggles (auto-approve copy states
the trade in plain words), usage card (master switch mirroring `showAgentUsage`, per-agent
enables, refresh interval).

## Contracts

- `crash-watch-contract`: extend for PID/signal fields and hint gating (none-agent → no hint;
  chosen → hint present; never both mechanisms).
- `agent-usage-contract`: as above.
- New `tests/quickshell/panama-agent-contract`: hermetic — stubbed `settings.json` +
  stub `kitty`/`claude`/`codex` on PATH; `none` → silent exit; claude/codex → correct argv shape
  including the auto-approve/prompting variants; prompt passed as one argv element; cwd is the
  repo. panama-agent-crash: prompt contains pid/comm/exe/signal + the skill path; never runs a
  real agent (stub PATH). NOT in the desktop-hijacking ledger — it must run stubbed.
- `skills-contract` covers `diagnose-crash` automatically once the skill lands.
- `settings-nav`/`search-routing`/`settings-docs`/`readme-contract` obligations as usual.

## Ownership (disjoint)

- **Orchestrator**: this spec; schema keys first; the `diagnose-crash` skill;
  `UPSTREAM-INSPIRATION.md` + license attribution file check; docs/commands regen; README count;
  seam audit.
- **Agent A — usage**: the three collector scripts, `AgentUsage.qml`, `AgentUsageWidget.qml`,
  `AgentUsagePanel.qml` (+ its qmldir if bar modules use one), `agent-usage-contract`, deletion
  of the old collector.
- **Agent B — ladder scripts**: `bin/panama-agent`, `bin/panama-agent-crash`,
  `bin/panama-agent-reload`, `bin/panama-crash-watch`, `bin/panama` (`cmd_diagnose`), `install`
  failure line, `bin/panama-migrate-notify` failure line, `.claude/settings.json` allowlist,
  `crash-watch-contract`, `panama-agent-contract`, README subcommand row.
- **Agent C — shell & settings UI**: `services/Notifs.qml` (hint field), `NotificationCard.qml`
  (exec branch), `shell.qml` (`onReloadFailed` rung), `modules/settings/HealthPage.qml` ("Ask
  the agent"), `AgentsPage.qml` + routes/shell-case/qmldir, search entries in
  `SettingsSearch.qml`.

Standing rules apply: live desktop (valid QML every save, journal checks), no real agent
launches or notifications left uncleaned during builds, contracts as you go, never the
desktop-hijacking set unannounced.