docs: multi-phase implementation plan (63 tasks across 5 phases)
Phase 1 Stabilize (13), Phase 2 Runtime unification (12), Phase 3 Dev-box surface (11), Phase 4 Sync correctness (12), Phase 5 Notifications & polish (15), plus index.
This commit is contained in:
Gabriel Brown
@@ -0,0 +1,409 @@
|
||||
# Spoon Phase 3 — Dev Box as a First-Class Surface: Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Give every user a standalone "My Machine" page at `/machine` that exposes their persistent per-user Fedora box directly: live status + start/stop/restart controls, a full-page terminal rooted at `~`, and a home-scoped file browser (reuse `FileTree`/`CodeEditor`). Make the workspace terminal survive tab switches, and add terminal QoL (web-links, copy-on-select). The box is USER-scoped (there is no `jobId`): the browser reaches it only through Next routes that mint short-lived HMAC tokens **after** Convex auth resolves the caller's own box username server-side — the browser never sees a worker secret and can only ever reach its OWN box.
|
||||
|
||||
**Architecture:** Same ownership-proxy pattern already used for `/jobs/:id/*`, replicated for a user-scoped `/box/*` surface:
|
||||
- **Worker (Bun/dockerode/ws):** new user-scoped HMAC terminal token (`terminal-token.ts`), a box helper module (`box.ts`) for home-path derivation + home-scoped file access + status, HTTP routes `GET /box/status`, `POST /box/lifecycle`, `GET /box/tree`, `GET|PUT /box/file`, and a WS route `GET /box/terminal`. All authed by the internal bearer token (HTTP) or the signed user token (WS), which embeds the username.
|
||||
- **Next (App Router):** `agent-worker-proxy.ts` gains `mintBoxTerminalToken(username)`, `resolveBoxUsername()`, and `proxyBox(...)`; `/api/box/*` route handlers derive the caller's username from Convex auth and proxy to the worker; a client `/machine` page reuses an extracted `XtermSession` component plus `FileTree` + `CodeEditor`.
|
||||
|
||||
**Tech Stack:** Next.js 16 (React 19), Convex, agent-worker (Bun, dockerode, ws, xterm).
|
||||
|
||||
**Depends on:** Phase 1 (terminal sizing fix + box handle/mutex lifecycle). Phase 3 assumes the Phase-1 `acquireUserBox` handle API (see "Phase-1 assumptions" below) and the Phase-1 realpath-based path containment. It benefits from Phase 2 (one-box-three-agents) but does **not** require it: the `/box` surface only needs the persistent box to exist, not any particular agent runtime.
|
||||
|
||||
## Phase-1 assumptions (read before starting)
|
||||
|
||||
Phase 1 reworks `apps/agent-worker/src/user-container.ts` to a per-username mutex + handle API. This plan is written against:
|
||||
|
||||
```ts
|
||||
// apps/agent-worker/src/user-container.ts (Phase-1 shape)
|
||||
export type BoxHandle = { boxName: string; release: () => void }; // release is idempotent
|
||||
export const acquireUserBox: (args: {
|
||||
username: string; workdir: string; containerHome: string;
|
||||
}) => Promise<BoxHandle>;
|
||||
```
|
||||
|
||||
**If Phase 1 kept the pre-Phase-1 signature** (`acquireUserBox(...) => Promise<string>` + `releaseUserBox(username)`, as in commit `b092955`), adapt every `handle.release()` in this plan to `releaseUserBox(username)` and treat the returned string as `boxName`. Check the real signature in `user-container.ts` before writing Task 5, and follow it.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **Next tests:** jsdom + @testing-library/react. Component tests live in `apps/next/tests/component/*.test.tsx`, unit tests in `apps/next/tests/unit/*.test.ts`. Run `cd apps/next && bun run test:component` and `bun run test:unit`. Mock `convex/react`, `next/navigation`, and `sonner` exactly as `apps/next/tests/component/render.test.tsx:10-38` does. Monaco (`CodeEditor`) and xterm do not render under jsdom — mock those child components in page tests (see Task 9).
|
||||
- **Worker tests:** `apps/agent-worker/tests/unit/*.test.ts` (vitest). Run `cd apps/agent-worker && bun run test:unit`. Match `apps/agent-worker/tests/unit/terminal-token.test.ts` for token tests. Docker-dependent code (status/lifecycle/exec) is **not** unit-testable without a daemon — document manual verification instead; unit-test only the pure logic (path containment, token parsing, tree building against a real tmpdir).
|
||||
- **Conventional commits, one per task** (e.g. `feat(worker): user-scoped box terminal token`). Commit only after the task's tests pass.
|
||||
- **Browser NEVER receives a worker token.** Tokens are minted server-side in Next route handlers after Convex auth. The username is derived server-side from the authed user (`api.userEnvironment.getMine`), never taken from the request body/query — so a user can only ever reach their own box.
|
||||
- **Reuse, don't duplicate.** Extract the xterm wiring from `workspace-terminal.tsx` into a shared `XtermSession` (Task 8) before building the box terminal; reuse `FileTree` and `CodeEditor` verbatim on `/machine`. Reuse `safeHomeJoin`-style realpath containment for `/box/file`.
|
||||
|
||||
## Reference facts (verified against the codebase)
|
||||
|
||||
- **Job terminal token** (`apps/agent-worker/src/terminal-token.ts`): format `${expiresAtMs}.${jobId}.${hmacHex}`, HMAC over `${expiresAtMs}.${jobId}`. Minted in `apps/next/src/lib/agent-worker-proxy.ts:20` (`mintTerminalToken`). Secret precedence (both sides): `SPOON_AGENT_TERMINAL_SECRET` → `SPOON_AGENT_WORKER_INTERNAL_TOKEN` → `SPOON_WORKER_TOKEN`. Worker reads it as `env.terminalSecret` (`apps/agent-worker/src/env.ts:43-47`); Next builds it in `terminalSecret()` (`agent-worker-proxy.ts:12-15`).
|
||||
- **Worker HTTP auth**: `requireAuth` (`apps/agent-worker/src/server.ts:45-51`) compares the `Authorization: Bearer` header to `env.internalToken`. Routes are dispatched in `startWorkerServer` (`server.ts:59-186`); only `/health`, `/cleanup`, and `/jobs/:id/*` exist today. The `jobRoute` regex (`server.ts:53-57`) is the pattern to mirror for `/box/*`.
|
||||
- **Worker WS**: `attachTerminalServer` (`terminal.ts:157-181`) handles `server.on('upgrade')`, matches `/^\/jobs\/([^/]+)\/terminal$/`, verifies the token, then calls `bridge(ws, jobId)`. Only enabled when `env.runtime === 'docker'`.
|
||||
- **`bridge()`** (`terminal.ts:21-150`): resolves the workspace via `getTerminalWorkspace(jobId)`, acquires the box via `acquireUserBox`, then spawns `<runtime> exec -i -e TERM=... -e HOME=... -w <containerRepo> <box> /bin/bash -lc 'exec script -qfc <launcher> /dev/null'`. The launcher does `stty rows.. cols..` then `exec tmux new-session -A -s spoon` (or `bash -il`). Input/resize buffered until the exec is ready; `cleanup()` kills the proc and releases the box.
|
||||
- **Box home layout** (`apps/agent-worker/src/worker.ts:1315-1334`): `username = userEnv?.username ?? 'user'`; `homeDir = path.resolve(env.workdir, 'homes', username)`; `containerHome = path.posix.join('/home', username)`. The box is acquired with `{ username, workdir: homeDir, containerHome }`.
|
||||
- **Box container** (`apps/agent-worker/src/runtime/docker.ts`): `userContainerName(username)` = `spoon-box-<sanitized>` (`docker.ts:338-339`); `ensureUserContainer({username,workdir,containerHome})` starts `spoon-box-*` with `--memory 4g --cpus 2` mounting the home (`docker.ts:341-383`); `stopWorkspaceContainer(name)` = `rm -f` (`docker.ts:438-442`); `inspectWorkspaceContainer(name)` runs `inspect` (`docker.ts:444-453`). Container runtime binary = `containerRuntime()`.
|
||||
- **Username resolution (server)**: `api.userEnvironment.getMine` (`packages/backend/convex/userEnvironment.ts:15-34`) returns `{ enabled, username, firstName, dotfilesRepoUrl, dotfilesRepoRef, setupCommand }` where `username = settings?.homeUsername ?? deriveHomeUsername(user?.name)` — the exact value the worker uses. This is the authoritative source for the caller's box username on the Next side.
|
||||
- **Existing file/tree worker fns** (`worker.ts`): `listWorkspaceTree` (1440), `readWorkspaceFile` (1477), `writeWorkspaceFile` (1483) — model the box variants on these but root at `homeDir` and use realpath containment.
|
||||
- **Reusable UI**: `FileTree` (`apps/next/src/components/agent-workspace/file-tree.tsx`) props `{ tree: FileTreeNode|null; selectedPath?; expandedPaths: string[]; onSelect; onToggleDirectory }`. `CodeEditor` (`code-editor.tsx:33-51`) props `{ path?; content; savedContent; readOnly; vimEnabled; onSave; onChange; onVimEnabledChange }`. `FileTreeNode`, `FileResponse`, `DiffResponse` live in `agent-workspace/types.ts`. Nav items are declared in `apps/next/src/components/layout/header/index.tsx:23-45` (`NavItem` = `{ href; icon; label; external? }`).
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Worker — user-scoped box terminal token (mint helper is Next-side; verify is worker-side)
|
||||
|
||||
Add a user-scoped token alongside the existing job-scoped one. It must be unambiguous from the 3-part job token, so it embeds a literal `box` segment.
|
||||
|
||||
**Files:**
|
||||
- `apps/agent-worker/src/terminal-token.ts` — add `verifyBoxTerminalToken`; keep `verifyTerminalToken` unchanged.
|
||||
- `apps/agent-worker/tests/unit/box-terminal-token.test.ts` — new test.
|
||||
|
||||
**Interfaces:**
|
||||
- **Token format (Produces/Consumes):** `${expiresAtMs}.box.${username}.${hmacSha256Hex}`, HMAC over the string `${expiresAtMs}.box.${username}`. Four dot-separated parts (job token has three), so the two can never collide. Username is `[a-z0-9_-]+` (already sanitized by `deriveHomeUsername`) and therefore never contains `.`.
|
||||
- **Produces:**
|
||||
```ts
|
||||
export const verifyBoxTerminalToken = (
|
||||
token: string,
|
||||
username: string,
|
||||
secret: string,
|
||||
): boolean;
|
||||
```
|
||||
Returns `true` only when: token has exactly 4 parts, part[1] === `'box'`, part[2] === `username`, expiry not passed, and the HMAC (over `${parts[0]}.box.${parts[2]}`) matches via `timingSafeEqual` on equal-length buffers. Empty token or empty secret → `false`.
|
||||
|
||||
**Steps:**
|
||||
- [ ] Write `apps/agent-worker/tests/unit/box-terminal-token.test.ts` mirroring `terminal-token.test.ts`: a local `mintBox(username, expiresAt, secret)` helper producing the 4-part token; cases: accepts valid unexpired username-matched token; rejects expired; rejects token minted for another username; rejects wrong secret; rejects malformed/empty; **rejects a 3-part job token passed to `verifyBoxTerminalToken`** (cross-scheme confusion guard).
|
||||
- [ ] Run `cd apps/agent-worker && bun run test:unit` → FAIL (function missing).
|
||||
- [ ] Implement `verifyBoxTerminalToken` in `terminal-token.ts` reusing the existing `signature()` helper. Split on `.`, require `parts.length === 4 && parts[1] === 'box'`, check `parts[2] === username`, parse+check expiry, `timingSafeEqual`.
|
||||
- [ ] Run tests → PASS.
|
||||
- [ ] Commit: `feat(worker): user-scoped box terminal token verification`.
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Worker — box helper module (home paths, status, lifecycle by username)
|
||||
|
||||
Centralize per-user box path derivation, status inspection, and lifecycle so both HTTP routes and the WS bridge share one source of truth.
|
||||
|
||||
**Files:**
|
||||
- `apps/agent-worker/src/box.ts` — new module.
|
||||
- `apps/agent-worker/src/runtime/docker.ts` — add `inspectUserBoxStatus`.
|
||||
- `apps/agent-worker/tests/unit/box-paths.test.ts` — new test (pure path logic only).
|
||||
|
||||
**Interfaces:**
|
||||
- **Produces (`box.ts`):**
|
||||
```ts
|
||||
export const boxHomePaths = (username: string) => ({
|
||||
homeDir: string, // path.resolve(env.workdir, 'homes', username)
|
||||
containerHome: string, // path.posix.join('/home', username)
|
||||
});
|
||||
|
||||
export type BoxStatus = {
|
||||
running: boolean;
|
||||
image: string | null;
|
||||
startedAt: string | null; // ISO from container State.StartedAt, null if stopped/absent
|
||||
memoryLimitBytes: number | null;
|
||||
containerName: string; // userContainerName(username)
|
||||
};
|
||||
export const getBoxStatus = (username: string) => Promise<BoxStatus>;
|
||||
|
||||
export const startBox = (username: string) => Promise<void>; // ensureUserContainer(boxHomePaths)
|
||||
export const stopBox = (username: string) => Promise<void>; // stop container + drop registry entry
|
||||
export const restartBox = (username: string) => Promise<void>; // stopBox then startBox
|
||||
```
|
||||
- **Produces (`docker.ts`):**
|
||||
```ts
|
||||
export const inspectUserBoxStatus = (username: string) => Promise<{
|
||||
running: boolean; image: string | null; startedAt: string | null; memoryLimitBytes: number | null;
|
||||
}>;
|
||||
```
|
||||
Implementation: `execa(containerRuntime(), ['inspect','--format','{{.State.Running}}|{{.Config.Image}}|{{.State.StartedAt}}|{{.HostConfig.Memory}}', userContainerName(username)], { reject:false, stdin:'ignore' })`. On non-zero exit (no such container) return `{ running:false, image:null, startedAt:null, memoryLimitBytes:null }`. Parse the pipe-split line; `running = field==='true'`; `memoryLimitBytes = Number(field) || null` (0 means unlimited → null); `startedAt` null when not running.
|
||||
- **Consumes:** `ensureUserContainer`, `stopWorkspaceContainer`, `userContainerName` (docker.ts); `env.workdir` (env.ts). `stopBox` must also clear the in-memory registry so a stopped box isn't considered "held" — add `export const resetBox = (username: string) => void` to `user-container.ts` (clears the idle timer and deletes the map entry) and call it from `stopBox`.
|
||||
|
||||
**Steps:**
|
||||
- [ ] Add `resetBox(username)` to `user-container.ts` (clear `box.idleTimer`, `boxes.delete(username)`).
|
||||
- [ ] Write `apps/agent-worker/tests/unit/box-paths.test.ts`: assert `boxHomePaths('gib')` returns `homeDir` ending in `homes/gib` and `containerHome === '/home/gib'`; assert two usernames produce distinct homeDirs. (Do NOT test docker-touching fns here.)
|
||||
- [ ] Run `cd apps/agent-worker && bun run test:unit` → FAIL.
|
||||
- [ ] Implement `inspectUserBoxStatus` in `docker.ts`, then `box.ts` (`boxHomePaths`, `getBoxStatus` wrapping `inspectUserBoxStatus` + `userContainerName`, `startBox`/`stopBox`/`restartBox`).
|
||||
- [ ] Run tests → PASS.
|
||||
- [ ] **Manual verification (docker required), record in the commit body:** with the worker running and a box up, `curl -H "Authorization: Bearer $TOKEN" localhost:3921/box/status?user=<you>` (after Task 4) shows `running:true` + image + startedAt; after `POST /box/lifecycle {"action":"stop"}` status flips to `running:false`; `start` brings it back.
|
||||
- [ ] Commit: `feat(worker): per-user box status and lifecycle helpers`.
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Worker — home-scoped file access (tree/read/write with realpath containment)
|
||||
|
||||
Home-scoped equivalents of `listWorkspaceTree`/`readWorkspaceFile`/`writeWorkspaceFile`, rooted at the user's `homeDir`, hardened against symlink escape (reuse the Phase-1 containment approach: resolve `fs.realpath` of the parent and re-check containment; reject symlinked write targets).
|
||||
|
||||
**Files:**
|
||||
- `apps/agent-worker/src/box.ts` — add tree/read/write.
|
||||
- `apps/agent-worker/tests/unit/box-files.test.ts` — new test against a real tmpdir.
|
||||
|
||||
**Interfaces:**
|
||||
- **Produces:**
|
||||
```ts
|
||||
export const listBoxTree = (username: string) => Promise<FileTreeNode>;
|
||||
export const readBoxFile = (username: string, relPath: string) => Promise<string>;
|
||||
export const writeBoxFile = (username: string, relPath: string, content: string) => Promise<{ success: true }>;
|
||||
// internal: safeHomePath(homeDir, relPath) => Promise<string> (realpath-checked absolute path)
|
||||
```
|
||||
- `listBoxTree`: walk `homeDir` (root node `name: '~'`, `path: ''`), skipping a small ignore set: `.git`, `node_modules`, `.cache`, `.local/share/Trash`, `.npm`, `.bun`, `.cargo`, `dist`, `build`, `.next`. Cap traversal (e.g. skip directories once total node count exceeds ~5000) so a huge home can't hang the request. Node shape matches `FileTreeNode` from `agent-workspace/types.ts` (`{ name; path; type: 'file'|'directory'; children? }`).
|
||||
- `readBoxFile`/`writeBoxFile`: resolve via `safeHomePath`. `writeBoxFile` `mkdir -p` the parent, then write. **No job/diff/event recording** (there is no job) — this is the key difference from `writeWorkspaceFile`.
|
||||
- `safeHomePath`: lexical resolve, then `realpath` the existing ancestor and re-check `startsWith(homeDir + sep)`; throw `Refusing to access path outside home: <relPath>` on escape. For writes, if the target itself exists and is a symlink, reject.
|
||||
- **Consumes:** `boxHomePaths`, `node:fs/promises` (`stat`, `readdir`, `readFile`, `writeFile`, `mkdir`, `realpath`), `FileTreeNode` type (import from where the worker declares it — check `worker.ts` for the local `FileTreeNode` type; if not exported, define a matching local type in `box.ts`).
|
||||
|
||||
**Steps:**
|
||||
- [ ] Write `apps/agent-worker/tests/unit/box-files.test.ts`: create a tmp home dir (`os.tmpdir()` + `mkdtemp`), stub `env.workdir` so `boxHomePaths('t').homeDir` points at it (either set `SPOON_AGENT_WORKDIR` before importing, or test `safeHomePath` directly by exporting it). Cases: `writeBoxFile` then `readBoxFile` round-trips; `readBoxFile('../../etc/passwd')` rejects; a symlink inside home pointing outside is rejected on read AND write; `listBoxTree` includes a created file and excludes `node_modules`.
|
||||
- [ ] Run `cd apps/agent-worker && bun run test:unit` → FAIL.
|
||||
- [ ] Implement `safeHomePath`, `listBoxTree`, `readBoxFile`, `writeBoxFile` in `box.ts`.
|
||||
- [ ] Run tests → PASS.
|
||||
- [ ] Commit: `feat(worker): home-scoped box file tree/read/write with realpath containment`.
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Worker — `/box/*` HTTP routes
|
||||
|
||||
Wire the box helpers into the HTTP server, authed by the internal bearer token like every other route.
|
||||
|
||||
**Files:**
|
||||
- `apps/agent-worker/src/server.ts` — add box route dispatch (import from `./box`).
|
||||
|
||||
**Interfaces (route → response shape):**
|
||||
- `GET /box/status?user=<username>` → `200 { status: BoxStatus }`.
|
||||
- `POST /box/lifecycle?user=<username>` body `{ action: 'start' | 'stop' | 'restart' }` → `200 { status: BoxStatus }` (return the post-action status; unknown action → `400 { error }`).
|
||||
- `GET /box/tree?user=<username>` → `200 { tree: FileTreeNode }`.
|
||||
- `GET /box/file?user=<username>&path=<rel>` → `200 { path, content }`.
|
||||
- `PUT /box/file?user=<username>` body `{ path, content }` → `200 { success: true }`.
|
||||
- Missing/empty `user` → `400 { error: 'Missing user' }`.
|
||||
|
||||
**Notes:** the `user` query param is trusted here because this route is only reachable with the internal bearer token, and the only caller (Next, Task 7) derives it from Convex auth. Add a `boxRoute(pathname)` matcher (mirror `jobRoute`, `server.ts:53-57`) matching `/^\/box\/(status|lifecycle|tree|file)$/`, and dispatch inside the existing try/catch in `startWorkerServer` **before** the `jobRoute` block. Read `user` via `url.searchParams.get('user')`.
|
||||
|
||||
**Steps:**
|
||||
- [ ] (No new unit test — this is HTTP glue over already-tested helpers; verify manually.) Add the `boxRoute` matcher and the five handlers in `server.ts`, importing `getBoxStatus, startBox, stopBox, restartBox, listBoxTree, readBoxFile, writeBoxFile` from `./box`. Reuse `sendJson`/`parseJson`.
|
||||
- [ ] `cd apps/agent-worker && bun run build` (or `bun run typecheck`) → passes.
|
||||
- [ ] **Manual verification (record in commit body):** with a box for your username, `curl -H "Authorization: Bearer $SPOON_WORKER_TOKEN" 'localhost:3921/box/status?user=<you>'` returns status JSON; `curl -X PUT ... 'localhost:3921/box/file?user=<you>' -d '{"path":"spoon-test.txt","content":"hi"}'` then `GET .../box/file?...&path=spoon-test.txt` round-trips; `GET .../box/tree?user=<you>` lists the home.
|
||||
- [ ] Commit: `feat(worker): /box status, lifecycle, tree, and file HTTP routes`.
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Worker — `/box/terminal` WebSocket bridge
|
||||
|
||||
Add a user-scoped terminal WS that acquires the box and opens a login shell rooted at `~`. Extract the shared PTY-bridging logic so the job and box bridges don't duplicate the exec/buffer/cleanup dance.
|
||||
|
||||
**Files:**
|
||||
- `apps/agent-worker/src/terminal.ts` — add `bridgeBox`, refactor shared exec bridging into a helper, add the `/box/terminal` upgrade match.
|
||||
|
||||
**Interfaces:**
|
||||
- **WS route:** `GET /box/terminal?token=<boxToken>`. Verify with `verifyBoxTerminalToken(token, username, env.terminalSecret)` — but the **username comes from the token itself** (parse part[2] after a successful format check), since the token is what authorizes the connection. Concretely: parse the token, extract the candidate username, then `if (!verifyBoxTerminalToken(token, candidateUsername, env.terminalSecret)) reject`. On success call `bridgeBox(ws, candidateUsername)`.
|
||||
- **`bridgeBox(ws, username)`:** derive `{ homeDir, containerHome } = boxHomePaths(username)`; `const handle = await acquireUserBox({ username, workdir: homeDir, containerHome })`; ensure a login shell works even for a brand-new home by writing a minimal `.bash_profile` if absent (reuse the snippet from `user-environment.ts:62-68`, or extract that into a shared `ensureBashProfile(homeDir)` and call it here); spawn the same `exec -i` shell as `bridge()` but with `-w <containerHome>` (root at `~`, not a repo) and env `TERM` + `HOME=<containerHome>` only (**no per-job secrets** — the box terminal is not job-scoped). Reuse the buffered-input + resize + cleanup logic; `cleanup()` calls `handle.release()` (or `releaseUserBox(username)` per the Phase-1 assumption).
|
||||
|
||||
**Refactor detail:** extract from the current `bridge()` (`terminal.ts:21-150`) a helper like:
|
||||
```ts
|
||||
const runShellBridge = (ws, boxName, opts: { cwd: string; envFlags: string[]; getSize: () => {cols:number;rows:number} }) => { /* buffered input, spawn, forward, exit */ };
|
||||
```
|
||||
so both `bridge(ws, jobId)` and `bridgeBox(ws, username)` call it. Keep behavior identical for the job path (regression-guard by leaving its manual smoke test intact).
|
||||
|
||||
**Steps:**
|
||||
- [ ] In `attachTerminalServer` (`terminal.ts:157-181`), add a second upgrade match `/^\/box\/terminal$/`: read `token`, parse username, verify with `verifyBoxTerminalToken`, on failure `401` + destroy, on success `wss.handleUpgrade(... => bridgeBox(ws, username))`. Keep the existing `/jobs/:id/terminal` branch working.
|
||||
- [ ] Implement `bridgeBox` and the shared `runShellBridge` (refactor `bridge` to use it). Add `ensureBashProfile` (shared with `user-environment.ts` or duplicated minimal).
|
||||
- [ ] `cd apps/agent-worker && bun run test:unit` → existing tests still PASS; `bun run build`/typecheck passes.
|
||||
- [ ] **Manual verification (record in commit body):** after Task 8/9 the browser flow covers this; for now, mint a box token by hand and connect a `websocat`/wscat client to `ws://localhost:3921/box/terminal?token=...`, confirm you land in `~` (`pwd` shows `/home/<you>`), typing works, resize works (`stty size` reflects the client), and disconnect releases the box (status idle-reaps after `boxIdleMs`).
|
||||
- [ ] Commit: `feat(worker): user-scoped /box/terminal websocket bridge`.
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Next — box proxy helpers (mint token, resolve username, proxy)
|
||||
|
||||
Server-only helpers that mint the box token and proxy `/box/*` to the worker, deriving the username from Convex auth so the caller can only reach their own box.
|
||||
|
||||
**Files:**
|
||||
- `apps/next/src/lib/agent-worker-proxy.ts` — add `mintBoxTerminalToken`, `resolveBoxUsername`, `proxyBox`, `withBox`.
|
||||
- `apps/next/tests/unit/box-terminal-token.test.ts` — new unit test for the mint format.
|
||||
|
||||
**Interfaces:**
|
||||
- **Produces:**
|
||||
```ts
|
||||
// Mirrors mintTerminalToken (agent-worker-proxy.ts:20). Returns null if secret/WS base unset.
|
||||
export const mintBoxTerminalToken = (username: string): { url: string; expiresAt: number } | null;
|
||||
// payload = `${expiresAt}.box.${username}`; token = `${payload}.${hmacHex}`;
|
||||
// url = `${wsBase}/box/terminal?token=${encodeURIComponent(token)}` (NOTE: /box/terminal, no jobId)
|
||||
|
||||
// Resolves the authed caller's own box username, or an unauthorized/again response.
|
||||
export const resolveBoxUsername = (): Promise<
|
||||
| { ok: true; username: string }
|
||||
| { ok: false; response: NextResponse }
|
||||
>;
|
||||
// uses convexAuthNextjsToken() then fetchQuery(api.userEnvironment.getMine, {}, { token }) -> .username
|
||||
|
||||
// Proxies to the worker with the internal token, always appending user=<derived username>.
|
||||
export const proxyBox = (
|
||||
username: string, action: 'status'|'lifecycle'|'tree'|'file',
|
||||
init?: RequestInit, search?: URLSearchParams,
|
||||
): Promise<NextResponse>;
|
||||
|
||||
// Convenience wrapper: resolve username -> run handler, 401/500 on failure (mirror withOwnedJob).
|
||||
export const withBox = (
|
||||
handler: (username: string) => Promise<Response>,
|
||||
): Promise<Response>;
|
||||
```
|
||||
- **Consumes:** `terminalSecret()` and `workerToken()` (existing, `agent-worker-proxy.ts:12-15,41-42`), `env.NEXT_PUBLIC_SPOON_AGENT_WORKER_WS_URL`, `env.SPOON_AGENT_WORKER_URL`, `convexAuthNextjsToken`, `fetchQuery`, `api.userEnvironment.getMine`.
|
||||
|
||||
**Steps:**
|
||||
- [ ] Write `apps/next/tests/unit/box-terminal-token.test.ts`: import `createHmac`, replicate the expected token string for a fixed secret+username+expiry, and assert `mintBoxTerminalToken`'s `url` contains `/box/terminal?token=` and the token has 4 dot-parts with `parts[1]==='box'`. To make the secret deterministic, set `process.env.SPOON_AGENT_TERMINAL_SECRET` and `NEXT_PUBLIC_SPOON_AGENT_WORKER_WS_URL` before import (match how `apps/next/tests/unit/environment.test.ts` handles env). If importing `agent-worker-proxy.ts` pulls in `server-only`/Convex and fails under vitest, extract the pure token-building into a tiny `buildBoxToken(username, secret, wsBase, now)` and unit-test that instead (keep `mintBoxTerminalToken` a thin wrapper). Prefer the extraction — it keeps the test hermetic.
|
||||
- [ ] Run `cd apps/next && bun run test:unit` → FAIL.
|
||||
- [ ] Implement the four helpers (+ `buildBoxToken` if extracted). `resolveBoxUsername` returns `{ ok:false, response: 401 }` when no auth token; `proxyBox` mirrors `proxyWorker` (`agent-worker-proxy.ts:95-131`) but targets `/box/${action}` and force-sets `user` in the search params from the derived username.
|
||||
- [ ] Run tests → PASS.
|
||||
- [ ] Commit: `feat(web): box terminal token mint + ownership proxy helpers`.
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Next — `/api/box/*` route handlers
|
||||
|
||||
App-Router handlers that derive the caller's username and proxy to the worker.
|
||||
|
||||
**Files (all new):**
|
||||
- `apps/next/src/app/api/box/status/route.ts` — `GET`.
|
||||
- `apps/next/src/app/api/box/lifecycle/route.ts` — `POST`.
|
||||
- `apps/next/src/app/api/box/tree/route.ts` — `GET`.
|
||||
- `apps/next/src/app/api/box/file/route.ts` — `GET` + `PUT`.
|
||||
- `apps/next/src/app/api/box/terminal-token/route.ts` — `GET`.
|
||||
|
||||
**Interfaces (client contract):**
|
||||
- `GET /api/box/status` → `{ status: BoxStatus }`.
|
||||
- `POST /api/box/lifecycle` body `{ action:'start'|'stop'|'restart' }` → `{ status: BoxStatus }`.
|
||||
- `GET /api/box/tree` → `{ tree: FileTreeNode }`.
|
||||
- `GET /api/box/file?path=<rel>` → `{ path, content }`; `PUT /api/box/file` body `{ path, content }` → `{ success:true }`.
|
||||
- `GET /api/box/terminal-token` → `{ url, expiresAt }` or `503 { error }` when unconfigured (mirror `agent-jobs/[jobId]/terminal-token/route.ts`).
|
||||
- All return `401` when unauthenticated (via `resolveBoxUsername`/`withBox`).
|
||||
|
||||
**Implementation:** each handler uses `withBox`/`resolveBoxUsername`. Example (`tree`):
|
||||
```ts
|
||||
export const GET = async () => await withBox(async (username) => await proxyBox(username, 'tree', { method: 'GET' }));
|
||||
```
|
||||
`terminal-token` uses `resolveBoxUsername` then `mintBoxTerminalToken(username)` (503 if null). `file` forwards `path` and request body like `agent-jobs/[jobId]/file/route.ts:1-28`. `lifecycle` forwards the JSON body.
|
||||
|
||||
**Steps:**
|
||||
- [ ] Create the five route files following the `agent-jobs/[jobId]/*` handlers as templates but with `withBox`/`resolveBoxUsername` instead of `withOwnedJob`.
|
||||
- [ ] `cd apps/next && bun run typecheck` (or the repo's lint/build) → passes.
|
||||
- [ ] **Manual verification (record in commit body):** signed in, hit `/api/box/status` in the browser/devtools → `200` with your box status; signed out → `401`. `PUT`+`GET /api/box/file` round-trips a file into your home.
|
||||
- [ ] Commit: `feat(web): /api/box status, lifecycle, tree, file, terminal-token routes`.
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Next — extract a reusable `XtermSession` component
|
||||
|
||||
Factor the xterm wiring out of `workspace-terminal.tsx` so `/machine` and the workspace both use one implementation (DRY the WS/fit/fonts/theme logic). This also sets up Task 10's persistence work.
|
||||
|
||||
**Files:**
|
||||
- `apps/next/src/components/agent-workspace/xterm-session.tsx` — new shared component (move the themes + effect from `workspace-terminal.tsx:16-209`).
|
||||
- `apps/next/src/components/agent-workspace/workspace-terminal.tsx` — becomes a thin wrapper over `XtermSession`.
|
||||
- `apps/next/tests/component/xterm-session.test.tsx` — new (behavioral, jsdom-limited).
|
||||
|
||||
**Interfaces:**
|
||||
- **Produces:**
|
||||
```ts
|
||||
type XtermStatus = 'idle' | 'connecting' | 'connected' | 'closed' | 'error' | 'unconfigured';
|
||||
export const XtermSession = (props: {
|
||||
active: boolean; // mount/connect gate
|
||||
tokenUrl: string; // e.g. `/api/box/terminal-token` or `/api/agent-jobs/${jobId}/terminal-token`
|
||||
sessionKey: string; // identity for the connection effect deps (jobId or username)
|
||||
waitingLabel?: string; // shown when active is false-but-expected (see Task 10)
|
||||
copyOnSelect?: boolean; // Task 11
|
||||
}) => JSX.Element;
|
||||
```
|
||||
Preserve current behavior: lazy-import `@xterm/xterm`, `@xterm/addon-fit`, `@xterm/addon-web-links`; fetch `tokenUrl` for `{ url }`; open WS; `fit()` after `term.open()`; refit on `document.fonts.ready` / Nerd-Font load (Phase-1 sizing fix — keep it); theme swap without teardown; reconnect button. Move the two `ITheme` objects into this file.
|
||||
- **`WorkspaceTerminal`** keeps its `{ jobId, active }` signature and renders `<XtermSession active={active} tokenUrl={`/api/agent-jobs/${jobId}/terminal-token`} sessionKey={jobId} />`.
|
||||
|
||||
**Steps:**
|
||||
- [ ] Write `apps/next/tests/component/xterm-session.test.tsx`. Since xterm can't render in jsdom, mock the three `@xterm/*` dynamic imports (`vi.mock('@xterm/xterm', ...)` returning a fake `Terminal` class with `open`/`loadAddon`/`onData`/`onResize`/`dispose`/`refresh` spies, etc.) and stub `global.WebSocket`. Assert: with `active:false` it renders the waiting label and does NOT construct a WebSocket; with `active:true` it fetches `tokenUrl` and opens a WebSocket to the returned `url`. (Mock `fetch` to return `{ url: 'ws://x' }`.) Keep assertions coarse — this is a smoke/behavior guard, not pixel-level.
|
||||
- [ ] Run `cd apps/next && bun run test:component` → FAIL.
|
||||
- [ ] Extract `XtermSession`, rewrite `WorkspaceTerminal` as the wrapper. Ensure the existing terminal tab still works (`agent-workspace-shell.tsx:664-667` passes `active`).
|
||||
- [ ] Run tests → PASS.
|
||||
- [ ] Commit: `refactor(web): extract reusable XtermSession from WorkspaceTerminal`.
|
||||
|
||||
---
|
||||
|
||||
## Task 9: Next — `/machine` page + nav entry ("My Machine")
|
||||
|
||||
The standalone dev-box surface: status card + lifecycle controls, home file browser (`FileTree` + `CodeEditor`), and a full-page `~`-rooted terminal.
|
||||
|
||||
**Files (new unless noted):**
|
||||
- `apps/next/src/app/(app)/machine/page.tsx` — the page.
|
||||
- `apps/next/src/components/machine/machine-shell.tsx` — client shell (status card, actions, split file-browser/editor + terminal). Mirrors the load/save/tree logic from `agent-workspace-shell.tsx` but against `/api/box/*` and with no job concepts.
|
||||
- `apps/next/src/components/machine/box-status-card.tsx` — status + start/stop/restart buttons.
|
||||
- `apps/next/src/components/layout/header/index.tsx` — add the `/machine` nav item (edit, not new).
|
||||
- `apps/next/tests/component/machine-shell.test.tsx` — new component test.
|
||||
|
||||
**Interfaces / behavior:**
|
||||
- **Nav:** add `{ href: '/machine', icon: <lucide icon, e.g. `Server` or `HardDrive`>, label: 'Machine' }` to the authenticated `navItems` array (`index.tsx:23-45`), placed after Threads.
|
||||
- **Status card:** fetch `GET /api/box/status` on mount + after each lifecycle action; poll every ~10s. Show running/stopped badge, image, uptime (derive from `startedAt`), memory cap (format `memoryLimitBytes` → GiB, or "unlimited" when null). Buttons call `POST /api/box/lifecycle` with `start`/`stop`/`restart`; disable while a request is in flight; `toast.error` on failure and only `toast.success` after the awaited response resolves (per the repo's error-handling principle). Refresh status from the response.
|
||||
- **File browser:** reuse `FileTree` (props exactly as `agent-workspace-shell.tsx:574-580`) fed by `GET /api/box/tree`; open files via `GET /api/box/file?path=`; edit in `CodeEditor`; save via `PUT /api/box/file`. Reuse the `OpenFileState`/`openFile`/`loadFile`/`writeFileContent` shapes from `agent-workspace-shell.tsx` (copy the minimal subset — no diff/UI-state persistence needed). `readOnly={false}`, `vimEnabled` local state.
|
||||
- **Terminal:** `<XtermSession active tokenUrl="/api/box/terminal-token" sessionKey={username} waitingLabel="Starting your machine…" />`, laid out full-page (own tab or a resizable pane — a Radix `Tabs` with "Files" and "Terminal", or a two-pane layout). The page is `~`-rooted by construction (worker `bridgeBox` uses `containerHome`).
|
||||
- **Username:** the page reads `api.userEnvironment.getMine` via `useQuery` for display (firstName/username), but all box access is server-derived — the client never sends the username.
|
||||
|
||||
**Steps:**
|
||||
- [ ] Write `apps/next/tests/component/machine-shell.test.tsx`: mock `convex/react` (`useQuery` → `{ username:'gib', firstName:'Gib', enabled:true }`), `next/navigation`, `sonner`, and `@/components/agent-workspace/xterm-session` (render `<div>terminal</div>`) and `@/components/agent-workspace/code-editor` (Monaco won't run in jsdom). Mock `global.fetch` so `/api/box/status` returns a stopped box and `/api/box/tree` returns a small tree. Assert: the status card renders "Stopped" and a "Start" button; clicking "Start" POSTs `/api/box/lifecycle` with `{action:'start'}` (assert on the `fetch` mock); the file tree renders a known file; the terminal placeholder renders.
|
||||
- [ ] Run `cd apps/next && bun run test:component` → FAIL.
|
||||
- [ ] Build `box-status-card.tsx`, `machine-shell.tsx`, `machine/page.tsx`, and add the nav item. The page is a server component that renders the client `MachineShell` (like other `(app)` pages); auth-gating follows the existing `(app)` layout — no special guard needed beyond what the layout provides.
|
||||
- [ ] Run tests → PASS.
|
||||
- [ ] **Manual verification (record in commit body):** sign in, open `/machine`. (1) Status card shows correct running/stopped + image + uptime + mem cap; Start/Stop/Restart each change the status within a few seconds. (2) Terminal drops you into `~` (`pwd` = `/home/<you>`), typing + resize work, output survives idle. (3) File browser lists your home; open+edit+save a file, confirm it persists (re-open shows the change; `cat` in the terminal confirms).
|
||||
- [ ] Commit: `feat(web): /machine dev-box page with status, terminal, and home file browser`.
|
||||
|
||||
---
|
||||
|
||||
## Task 10: Next — persistent workspace terminal (survive tab switches)
|
||||
|
||||
Make the workspace Terminal tab keep its PTY/scrollback when switching Radix tabs, refit + resize-send when it becomes visible, and distinguish "waiting for workspace" from "connecting".
|
||||
|
||||
**Files:**
|
||||
- `apps/next/src/components/agent-workspace/agent-workspace-shell.tsx` — the Terminal `TabsContent` (currently `660-668`).
|
||||
- `apps/next/src/components/agent-workspace/xterm-session.tsx` — add refit-on-visible + `waiting` handling.
|
||||
- `apps/next/tests/component/xterm-session.test.tsx` — extend from Task 8.
|
||||
|
||||
**Behavior / interfaces:**
|
||||
- **Persistence:** give the Terminal `TabsContent` `forceMount` and hide it with CSS when inactive instead of unmounting (Radix `TabsContent` unmounts by default — that destroys the PTY, audit item #15). Pattern: `<TabsContent value='terminal' forceMount className={cn('m-0 min-h-0 flex-1 overflow-hidden', activeWorkspaceTab !== 'terminal' && 'hidden')}>`. The other tabs stay as-is. Keep `<XtermSession active={workspaceReady} ... />` **mounted regardless of tab** so the WS/PTY persists; use a separate `visible` prop for fit behavior.
|
||||
- **Refit on visible:** add a `visible: boolean` prop to `XtermSession`. When it transitions to `true`, call `fitAddon.fit()` inside `requestAnimationFrame` and send a resize frame (the Phase-1 pattern). Keep `active` as the connect gate (`workspaceReady`), `visible` as the CSS-visibility gate (`activeWorkspaceTab === 'terminal'`). In the shell, pass `active={workspaceReady}` and `visible={activeWorkspaceTab === 'terminal'}`.
|
||||
- **Waiting vs connecting:** `XtermSession` status starts `'idle'`; when `active` is false but the workspace is expected (job pending), show `waitingLabel` ("Waiting for workspace…") — distinct from `'connecting'` (token fetched, WS opening). The shell already computes `workspacePending`/`workspaceReady` (`agent-workspace-shell.tsx:109-122`); pass an appropriate `waitingLabel`.
|
||||
|
||||
**Steps:**
|
||||
- [ ] Extend `xterm-session.test.tsx`: assert that toggling `visible` false→true triggers a `fit()` call on the mocked fit addon and sends a resize frame over the mocked WebSocket; assert that with `active:false` + a `waitingLabel`, the waiting label renders and no WebSocket is constructed; assert the component stays mounted (WS not closed) when `visible` goes true→false while `active` stays true.
|
||||
- [ ] Run `cd apps/next && bun run test:component` → FAIL.
|
||||
- [ ] Add the `visible` prop + refit effect to `XtermSession`; update `WorkspaceTerminal` to forward `visible`; change the Terminal `TabsContent` in `agent-workspace-shell.tsx` to `forceMount` + CSS-hide, and mount `<WorkspaceTerminal active={workspaceReady} visible={activeWorkspaceTab==='terminal'} .../>` (update `WorkspaceTerminal` props to accept `visible`).
|
||||
- [ ] Run tests → PASS.
|
||||
- [ ] **Manual verification (record in commit body):** open a running thread workspace, go to Terminal, run `top` (or type a long-running output), switch to Editor then back — the session and scrollback are intact (not a fresh shell), and the terminal is correctly sized (not quarter-size) on return. Before a worker claims the job, the Terminal tab shows "Waiting for workspace…", not "Connecting…".
|
||||
- [ ] Commit: `feat(web): persistent workspace terminal across tab switches`.
|
||||
|
||||
---
|
||||
|
||||
## Task 11: Next — terminal QoL (copy-on-select; confirm web-links)
|
||||
|
||||
Web-links is already loaded (`workspace-terminal.tsx:144` / carried into `XtermSession`). Add a copy-on-select toggle available in both terminals.
|
||||
|
||||
**Files:**
|
||||
- `apps/next/src/components/agent-workspace/xterm-session.tsx` — copy-on-select via `term.onSelectionChange`.
|
||||
- `apps/next/tests/component/xterm-session.test.tsx` — extend.
|
||||
|
||||
**Interfaces / behavior:**
|
||||
- Add a `copyOnSelect?: boolean` prop (already declared in Task 8) with a small toggle control in the terminal header (a button/checkbox). When enabled, on `term.onSelectionChange` copy `term.getSelection()` to the clipboard (`navigator.clipboard.writeText`, guarded — clipboard may be unavailable/denied; swallow errors). Persist the toggle in `localStorage` (`spoon.terminal.copyOnSelect`) so it sticks across sessions. Confirm `WebLinksAddon` is loaded in `XtermSession` (it is — keep it; it makes URLs clickable).
|
||||
|
||||
**Steps:**
|
||||
- [ ] Extend `xterm-session.test.tsx`: mock `navigator.clipboard.writeText`; with `copyOnSelect` on, simulate the mocked terminal's selection-change callback with a non-empty selection and assert `writeText` was called with it; with it off, assert not called. Assert the header toggle renders and flipping it updates behavior.
|
||||
- [ ] Run `cd apps/next && bun run test:component` → FAIL.
|
||||
- [ ] Implement the toggle + `onSelectionChange` handler + `localStorage` persistence in `XtermSession`; ensure `WorkspaceTerminal` and the `/machine` terminal both expose it.
|
||||
- [ ] Run tests → PASS.
|
||||
- [ ] **Manual verification (record in commit body):** in `/machine` and a workspace terminal, enable copy-on-select, select text → it's on the clipboard; a printed URL is clickable (web-links) and opens in a new tab.
|
||||
- [ ] Commit: `feat(web): terminal copy-on-select toggle and web-links polish`.
|
||||
|
||||
---
|
||||
|
||||
## Final verification (run before declaring the phase done)
|
||||
|
||||
- [ ] `cd apps/agent-worker && bun run test:unit` — all green (token, box-paths, box-files).
|
||||
- [ ] `cd apps/next && bun run test:unit && bun run test:component` — all green.
|
||||
- [ ] Typecheck/lint per repo scripts for both apps.
|
||||
- [ ] End-to-end manual smoke on a docker-enabled deployment: `/machine` status+lifecycle+terminal+file-edit all work; workspace terminal persists across tab switches and is correctly sized; a user signed in as A cannot reach user B's box (all `/api/box/*` derive the username server-side — confirm by inspecting that no request carries a username).
|
||||
|
||||
## Self-review against the Phase 3 spec
|
||||
|
||||
- ✅ `/machine` page: status (running/stopped, image, uptime, mem cap), start/stop/restart, full-page `~`-rooted terminal, home file browser reusing FileTree/CodeEditor — Tasks 9 (+2,3,4 backing).
|
||||
- ✅ Worker user-scoped HMAC tokens + `/box/tree|file|terminal` + status/lifecycle endpoints — Tasks 1–5.
|
||||
- ✅ Same ownership-proxy pattern (Next mints after Convex auth, browser never sees worker secret) — Tasks 6–7; username derived server-side (a user can only reach their own box).
|
||||
- ✅ Persistent workspace terminal (forceMount + CSS hide, refit-on-visible, waiting-vs-connecting) — Task 10.
|
||||
- ✅ Terminal QoL (web-links, copy-on-select) — Task 11.
|
||||
- ✅ Secret precedence + payload around username+expiry — Task 1/6.
|
||||
Reference in New Issue
Block a user