diff --git a/README.md b/README.md index d2e0df3..1f10691 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,8 @@

Spoon is a self-hostable fork maintenance cockpit built around managed forks, - durable maintenance threads, and OpenCode-powered workspaces. + durable maintenance threads, and a persistent per-user dev box that agents, + terminals, and project commands all run inside.

@@ -61,14 +62,38 @@ in the data model, but GitHub is the active automation surface today. Spoon keeps raw GitHub drift visible while also tracking ignored upstream changes so irrelevant commits do not keep a fork permanently actionable. -- **OpenCode workspaces** - Agent work happens in an isolated workspace with a file tree, browser editor, - diff viewer, command panel, logs, artifacts, and draft PR actions. +- **One persistent per-user box** + Every user owns a single long-running Fedora container `spoon-box-{username}` + with a persistent home. Every thread's agent turn, terminal session, and + project command `docker exec`s into that same box — there is no per-job + throwaway container. + +- **Three agent runtimes** + Codex, OpenCode, and Claude Code all run inside the box behind one adapter + interface, selected per thread and validated at queue time against the AI + provider profile. + +- **Workspaces** + Agent work happens in a workspace with a file tree, browser editor, diff + viewer, command panel, logs, artifacts, an interactive terminal, and draft PR + actions. + +- **My Machine** + A `/machine` surface to start/stop/restart the box, open a `~`-rooted + terminal, and browse the persistent home. + +- **GitHub webhooks** + A signature-verified webhook keeps drift fresh on `push` and flips connection + status on `installation` changes; the hourly cron is a fallback. + +- **Notifications** + An in-app bell plus preference-gated transactional email for maintenance + threads, agent turns, needed input, sync failures, and connection re-auth. - **User-owned providers and secrets** - AI provider profiles, Codex/OpenCode auth, and per-Spoon project secrets are - encrypted. Secrets are redacted from logs and refused from commits when - materialized into env files. + AI provider profiles, Codex/OpenCode/Anthropic auth, and per-Spoon project + secrets are encrypted. Secrets are written `0600`, redacted from logs, and + refused from commits when materialized into env files. - **Draft PR handoff** Code changes become branches and draft pull requests. Spoon does not @@ -138,12 +163,23 @@ custom work exists.

-OpenCode workspaces +The box and workspaces Spoon's optional agent worker is designed to run outside Convex actions. The -worker claims queued jobs, clones the current GitHub fork, creates a branch, -starts an isolated workspace, and exposes workspace operations to the Next app -through server-only API proxies. +worker claims queued jobs, clones the current GitHub fork into the owner's +persistent home, creates a branch, and exposes workspace operations to the Next +app through server-only API proxies. + +Everything runs inside **one long-running container per user**, +`spoon-box-{username}`. The box is a Fedora image (`docker/agent-job.Dockerfile`) +started with `sleep infinity` and a `--memory 4g` / `--cpus 2` cap. Each +thread's agent turn, terminal, and project commands `docker exec` into that same +box; the box's `/home/{username}` is a bind-mounted persistent home so dotfiles, +installed tools, shell history, and thread checkouts under `~/Code/{spoon}/{branch}` +survive across sessions. The box is reference-counted by the worker and reaped +after `SPOON_AGENT_BOX_IDLE_MS` idle (default 30m). There is no per-job +`docker run --rm` container and no separate `opencode serve` container — that +path was removed. Workspace capabilities: @@ -154,16 +190,18 @@ Workspace capabilities: - inspect diffs - send thread messages to the agent - run configured commands +- use an interactive terminal (xterm.js → box PTY; see + [docs/agent-terminal.md](docs/agent-terminal.md)) - store logs and artifacts - push a branch - open a draft PR The browser never receives worker tokens and never talks directly to the worker -or job container. +or the box. -Worker cleanup is available in `Settings -> Worker`. It can delete old terminal -workspace records and ask the active worker to remove orphaned job containers -and inactive work directories. +Worker cleanup is available in `Settings -> Worker`. It can delete stale +workspace records and ask the active worker to remove orphaned containers and +inactive work directories (persistent per-user homes are preserved). Local worker development: @@ -174,20 +212,53 @@ bun dev:next:worker bun dev:next:worker:staging ``` -Local host-run worker commands still load env through Infisical, then -`scripts/dev-agent-worker` selects Podman when available, falls back to Docker, -and publishes the OpenCode server on a localhost port so the host worker can -reach the job container. Override with: +Local host-run worker commands load env through Infisical, then +`scripts/dev-agent-worker` selects Podman when available and falls back to +Docker. Override the container CLI with: ```env SPOON_AGENT_CONTAINER_RUNTIME=podman -SPOON_AGENT_CONTAINER_ACCESS=host_port ```
-Production agent runtime images +Agent runtimes + +Three agent CLIs run inside the box behind one `AgentRuntime` adapter interface +(`apps/agent-worker/src/runtime/*-adapter.ts`), selected by `job.runtime`: + +| Runtime | CLI | Selected for | +| ---------- | ----------------------------------- | -------------------------------------------------- | +| `codex` | `@openai/codex` | OpenAI providers and Codex ChatGPT-login snapshots | +| `opencode` | `opencode-ai` | OpenAI-compatible API-key providers (default) | +| `claude` | `@anthropic-ai/claude-code@2.1.207` | Anthropic providers | + +The runtime is validated at queue time (`runtimeSupport.ts` + +`agentJobs.insertJob`): a job is rejected unless the resolved runtime is one the +AI provider profile supports. `runtimesForProfile` maps profiles to runtimes — +ChatGPT-login snapshots → `codex` only; `openai` API keys → `opencode`/`codex`; +Anthropic API keys → `claude`/`opencode`; Anthropic `anthropic_oauth_json` +credential snapshots → `claude` only; every other OpenAI-compatible API key → +`opencode`. + +Auth is materialized into the box before a turn: + +- **Codex ChatGPT-login** profiles get the encrypted `auth.json` written to + `CODEX_HOME/.codex/auth.json`. +- **Anthropic API-key** profiles authenticate via `ANTHROPIC_API_KEY` in the + environment (no file needed). +- **Anthropic `anthropic_oauth_json`** credential-snapshot profiles get the + encrypted OAuth blob written to `~/.claude/.credentials.json`. +- **API-key** profiles run through OpenCode. + +All materialized auth files are written `0600`. Treat those saved auth files +like a password and only use them on trusted workers. + +
+ +
+Production runtime images Gitea CI builds and pushes three production images: @@ -197,32 +268,36 @@ git.gbrown.org/gib/spoon-agent-worker:latest git.gbrown.org/gib/spoon-agent-job:latest ``` -The worker image is the long-running service that polls Convex. The job image is -the isolated workbench that the worker launches for each agent job. For the MVP, -production should use the repo-provided JS/TS workbench image: +The worker image is the long-running service that polls Convex. The +`spoon-agent-job` image is the **box** image — the per-user Fedora dev box the +worker `exec`s into (it is not launched fresh per job). Point the worker at it +with: ```env SPOON_AGENT_JOB_IMAGE="git.gbrown.org/gib/spoon-agent-job:latest" ``` -The job image includes Node 22, Bun, pnpm and yarn through Corepack, npm, git, -ripgrep, Python, build tools, OpenCode, the Codex CLI, and the Claude Code CLI. -It is not the forked project's production runtime; it is the agent execution -environment. +The box image is Fedora 41 with Node, Bun, pnpm and yarn, npm, git, ripgrep, +Python, build tools, plus interactive tooling (neovim, tmux, fzf, fd, bat, eza, +zoxide, gh, gum, oh-my-posh) and the pinned agent CLIs OpenCode, Codex, and +Claude Code. It is not the forked project's production runtime; it is the agent +execution environment. Production worker runtime requirements: - `spoon-agent-worker` must run as a separate service. -- The worker needs `/var/run/docker.sock` mounted so it can launch job - containers. -- Production should keep `SPOON_AGENT_CONTAINER_RUNTIME=docker` and - `SPOON_AGENT_CONTAINER_ACCESS=network`. -- The production Docker host must be logged into `git.gbrown.org` so worker jobs - can pull the private `spoon-agent-job` image. +- The worker needs `/var/run/docker.sock` mounted so it can create and `exec` + into per-user boxes on the host daemon. +- Production should keep `SPOON_AGENT_CONTAINER_RUNTIME=docker`. +- `SPOON_AGENT_HOST_WORKDIR` must equal the absolute host path backing + `SPOON_AGENT_WORKDIR` (identical inside and outside the worker container) so + the host daemon can bind-mount per-user homes under `${WORKDIR}/homes/{username}`. +- The production Docker host must be logged into `git.gbrown.org` so the worker + can pull the private `spoon-agent-job` box image. - `SPOON_WORKER_TOKEN` must match the value stored in Convex production env. - `spoon-next` needs `SPOON_AGENT_WORKER_URL=http://spoon-agent-worker:3921` and - `SPOON_AGENT_WORKER_INTERNAL_TOKEN` so Next API routes can proxy workspace - file, diff, message, command, and draft PR actions. + `SPOON_AGENT_WORKER_INTERNAL_TOKEN` so Next API routes can proxy workspace and + box file, diff, message, command, terminal, and draft PR actions. - `spoon-agent-worker` also needs `GITHUB_APP_ID` and `GITHUB_APP_PRIVATE_KEY`. If the private key is stored in a single-line dotenv value, encode newlines as literal `\n` characters so the worker can restore the PEM before using it. @@ -241,26 +316,57 @@ curl -H "Authorization: Bearer $SPOON_AGENT_WORKER_INTERNAL_TOKEN" \ Deployment readiness checklist: 1. Production Convex env has `SPOON_WORKER_TOKEN`, `SPOON_ENCRYPTION_KEY`, - GitHub App env, and Convex Auth signing keys. + GitHub App env (including `GITHUB_APP_WEBHOOK_SECRET`), and Convex Auth + signing keys. 2. Compose env has `SPOON_AGENT_WORKER_URL`, `SPOON_AGENT_WORKER_INTERNAL_TOKEN`, `SPOON_AGENT_JOB_IMAGE`, and the GitHub - App private key. + App private key; the `spoon-next` image is built with + `NEXT_PUBLIC_SPOON_AGENT_WORKER_WS_URL`. 3. The production Docker host can pull private images from `git.gbrown.org`. -4. `Settings -> Worker` reports the expected job image, runtime, network, and - active workspace count. +4. `Settings -> Worker` reports the expected box image, runtime, network, and + active box count. 5. The first test thread uses a configured API-key provider or a trusted Codex login profile. -6. If a worker restart leaves stale workspace state, use the workspace recovery - panel or `Settings -> Worker` cleanup. +6. If a worker restart leaves stale state, use the workspace recovery panel or + `Settings -> Worker` cleanup. -API-key based AI provider profiles run through OpenCode. Codex ChatGPT login -profiles run through the Codex CLI: Spoon writes the encrypted `auth.json` into -the isolated job workspace as `CODEX_HOME/.codex/auth.json` before execution. -Anthropic profiles run through the Claude Code CLI: API-key profiles authenticate -via `ANTHROPIC_API_KEY`, while `anthropic_oauth_json` credential-snapshot profiles -have their encrypted OAuth blob written into the isolated job workspace as -`~/.claude/.credentials.json` before execution. Treat those saved auth files like -a password and only use them on trusted workers. +
+ +
+GitHub webhooks + +Spoon exposes a signature-verified webhook as a Convex `httpAction` at +`POST /webhooks/github` (`packages/backend/convex/githubWebhooks.ts`, +`http.ts`). Configure the GitHub App webhook URL as +`/webhooks/github` and set `GITHUB_APP_WEBHOOK_SECRET`; the +endpoint fails closed (HTTP 503) when no secret is configured and rejects any +payload whose `x-hub-signature-256` does not verify. + +- `push` → a targeted drift refresh for every Spoon tracking that repository. +- `installation` / `installation_repositories` → the matching connection's + status flips to `revoked` (deleted/suspended), `active` (created/unsuspended), + or otherwise `needs_reauth`, surfaced in `Settings -> Integrations`. + +The hourly `refreshDueSpoons` cron remains a fallback, so drift still refreshes +even if a webhook is missed. + +
+ +
+Notifications + +Spoon delivers an in-app bell plus preference-gated transactional email +(`packages/backend/convex/notifications.ts`). Five kinds are emitted: +`maintenance_thread`, `agent_turn_finished`, `agent_needs_input`, `sync_failed`, +and `connection_needs_reauth`. + +- Every event inserts an in-app notification (bell badge via `unreadCount`, + list via `listMine`, `markRead` / `markAllRead`). +- Email is sent through UseSend (`USESEND_*`) only when the user has an email + and the matching per-kind preference is not disabled. Unset preferences + default to enabled. +- Per-kind email toggles live in `Settings -> Notifications`. Web push is out of + scope.
@@ -289,45 +395,54 @@ a password and only use them on trusted workers.
Core tables -| Table | Purpose | -| ------------------------ | --------------------------------------------------------- | -| `spoons` | Managed fork records | -| `threads` | Durable maintenance and work conversations | -| `threadMessages` | Messages inside threads | -| `syncRuns` | Upstream checks, sync attempts, and maintenance decisions | -| `ignoredUpstreamChanges` | Intentional ignore records that affect effective drift | -| `gitConnections` | Git provider connection metadata | -| `spoonRepositoryStates` | Latest cached upstream/fork state | -| `spoonCommits` | Cached upstream and fork-only commits | -| `spoonPullRequests` | Cached fork/upstream pull requests | -| `spoonSecrets` | Encrypted per-Spoon environment variables | -| `spoonAgentSettings` | Per-Spoon runtime, branch, command, and env-file settings | -| `aiProviderProfiles` | Encrypted provider/auth profiles used by OpenCode | -| `agentJobs` | Worker-executed workspace jobs and PR lifecycle | -| `agentJobEvents` | Append-only worker event log | -| `agentJobArtifacts` | Diffs, summaries, command output, PR body drafts | -| `agentWorkspaceChanges` | Recorded user, agent, and command file changes | +| Table | Purpose | +| ------------------------- | ----------------------------------------------------------- | +| `spoons` | Managed fork records | +| `threads` | Durable maintenance and work conversations | +| `threadMessages` | Messages inside threads | +| `syncRuns` | Upstream checks, sync attempts, and maintenance decisions | +| `ignoredUpstreamChanges` | Intentional ignore records that affect effective drift | +| `gitConnections` | Git provider connection metadata | +| `spoonRepositoryStates` | Latest cached upstream/fork state | +| `spoonCommits` | Cached upstream and fork-only commits | +| `spoonPullRequests` | Cached fork/upstream pull requests | +| `spoonSecrets` | Encrypted per-Spoon environment variables | +| `spoonAgentSettings` | Per-Spoon runtime, branch, command, and env-file settings | +| `aiProviderProfiles` | Encrypted provider/auth profiles (Codex/OpenCode/Anthropic) | +| `agentJobs` | Worker-executed workspace jobs and PR lifecycle | +| `agentJobEvents` | Append-only worker event log | +| `agentJobArtifacts` | Diffs, summaries, command output, PR body drafts | +| `agentWorkspaceChanges` | Recorded user, agent, and command file changes | +| `userDotfiles` | Encrypted per-user dotfiles overlay + repo/setup config | +| `userEnvironment` | Per-user box/home environment config | +| `notifications` | In-app notification rows (bell) | +| `notificationPreferences` | Per-user email notification toggles |
Important routes -| Route | Purpose | -| --------------------------------- | --------------------------------------- | -| `/` | Public product landing page | -| `/dashboard` | Maintenance overview | -| `/spoons` | Managed fork list | -| `/spoons/new` | Manual/GitHub Spoon creation | -| `/spoons/[spoonId]` | Spoon detail dashboard | -| `/spoons/[spoonId]/agent/[jobId]` | Interactive workspace | -| `/threads` | Global thread queue | -| `/threads/[threadId]` | Thread detail | -| `/settings/profile` | User profile settings | -| `/settings/integrations` | GitHub and service integration settings | -| `/settings/ai-providers` | AI/OpenCode provider profiles | +| Route | Purpose | +| --------------------------------- | ----------------------------------------------- | +| `/` | Public product landing page | +| `/dashboard` | Maintenance overview | +| `/spoons` | Managed fork list | +| `/spoons/new` | Manual/GitHub Spoon creation | +| `/spoons/[spoonId]` | Spoon detail dashboard | +| `/spoons/[spoonId]/agent/[jobId]` | Interactive workspace | +| `/threads` | Global thread queue | +| `/threads/[threadId]` | Thread detail | +| `/machine` | My Machine: box status, terminal, home | +| `/settings/profile` | User profile settings | +| `/settings/integrations` | GitHub and service integration settings | +| `/settings/ai-providers` | AI provider profiles (Codex/OpenCode/Anthropic) | +| `/settings/dotfiles` | Per-user dotfiles overlay + repo | +| `/settings/notifications` | Email notification preferences | -Legacy `/updates` and `/agents` routes redirect into `/threads`. +The Convex `httpAction` `POST /webhooks/github` (mounted at +`/webhooks/github`) handles GitHub App webhooks. Legacy +`/updates` and `/agents` routes redirect into `/threads`.
@@ -428,16 +543,17 @@ not call Infisical.
Public Next variables -| Variable | Used for | -| --------------------------------- | ------------------------------------------- | -| `NEXT_PUBLIC_SITE_URL` | Canonical Spoon web URL | -| `NEXT_PUBLIC_CONVEX_URL` | Convex client URL | -| `NEXT_PUBLIC_DEPLOYMENT_URL` | Convex dashboard/deployment URL when needed | -| `NEXT_PUBLIC_PLAUSIBLE_URL` | Plausible analytics endpoint | -| `NEXT_PUBLIC_SENTRY_DSN` | Browser Sentry DSN | -| `NEXT_PUBLIC_SENTRY_URL` | Sentry instance URL | -| `NEXT_PUBLIC_SENTRY_ORG` | Sentry organization | -| `NEXT_PUBLIC_SENTRY_PROJECT_NAME` | Sentry project name | +| Variable | Used for | +| --------------------------------------- | ------------------------------------------------------------------- | +| `NEXT_PUBLIC_SITE_URL` | Canonical Spoon web URL | +| `NEXT_PUBLIC_CONVEX_URL` | Convex client URL | +| `NEXT_PUBLIC_SPOON_AGENT_WORKER_WS_URL` | Browser-facing worker WS base for the terminal (**build-time** var) | +| `NEXT_PUBLIC_DEPLOYMENT_URL` | Convex dashboard/deployment URL when needed | +| `NEXT_PUBLIC_PLAUSIBLE_URL` | Plausible analytics endpoint | +| `NEXT_PUBLIC_SENTRY_DSN` | Browser Sentry DSN | +| `NEXT_PUBLIC_SENTRY_URL` | Sentry instance URL | +| `NEXT_PUBLIC_SENTRY_ORG` | Sentry organization | +| `NEXT_PUBLIC_SENTRY_PROJECT_NAME` | Sentry project name |
@@ -492,15 +608,16 @@ not call Infisical. | `SPOON_AGENT_WORKER_URL` | Internal worker HTTP URL used by Next | | `SPOON_AGENT_WORKER_HTTP_PORT` | Worker HTTP port | | `SPOON_AGENT_WORKER_INTERNAL_TOKEN` | Server-only token for Next-to-worker proxy | -| `SPOON_AGENT_JOB_IMAGE` | Agent job container image | +| `SPOON_AGENT_TERMINAL_SECRET` | HMAC secret for terminal/box tokens (falls back to the internal/worker token) | +| `SPOON_AGENT_JOB_IMAGE` | Per-user box (Fedora) image the worker `exec`s into | | `SPOON_AGENT_RUNTIME` | Runtime mode, currently Docker/Podman-oriented | | `SPOON_AGENT_CONTAINER_RUNTIME` | Container CLI used by worker, `docker`/`podman` | -| `SPOON_AGENT_CONTAINER_ACCESS` | `network` in prod, `host_port` for host dev | +| `SPOON_AGENT_BOX_IDLE_MS` | Idle time before a per-user box is reaped (default `1800000`) | | `SPOON_AGENT_MAX_CONCURRENT_JOBS` | Worker concurrency limit | -| `SPOON_AGENT_JOB_TIMEOUT_MS` | Job timeout | -| `SPOON_AGENT_WORKDIR` | Worker work directory | +| `SPOON_AGENT_JOB_TIMEOUT_MS` | Agent turn timeout | +| `SPOON_AGENT_WORKDIR` | Worker work directory; per-user homes live under `homes/{username}` | | `SPOON_AGENT_HOST_WORKDIR` | Host path matching `SPOON_AGENT_WORKDIR` when the worker runs in Docker and controls the host Docker socket | -| `SPOON_AGENT_NETWORK` | Optional job container network | +| `SPOON_AGENT_NETWORK` | Optional box container network | @@ -528,12 +645,20 @@ not call Infisical. - GitHub drift refresh, commit cache, PR cache, and sync-run history - Effective drift and ignored upstream change records - Global Threads page and Spoon-scoped Threads tab -- OpenCode/Codex-oriented agent worker and browser workspace foundation +- Persistent per-user box the worker `exec`s into for agent turns, terminal, and + commands +- Three agent runtimes (Codex, OpenCode, Claude Code) behind one adapter, with + queue-time runtime validation +- My Machine surface (box status/start/stop/restart, terminal, home browser) +- Interactive workspace terminal (xterm.js → box PTY) +- Per-user dotfiles overlay + optional dotfiles repo +- Signature-verified GitHub webhook with hourly cron fallback +- In-app + email notifications with per-user preferences - Monaco editor with optional Vim mode - Diff viewer, command panel, worker logs, and artifacts - Encrypted Spoon secrets and bulk `.env` import -- Encrypted AI provider profiles, including Codex auth JSON and API-key - provider support +- Encrypted AI provider profiles, including Codex/Anthropic auth JSON and + API-key provider support - Authentik, GitHub, and password auth through Convex Auth - Self-hosted Convex/Postgres deployment model diff --git a/docs/agent-terminal.md b/docs/agent-terminal.md index e474969..f96b8ba 100644 --- a/docs/agent-terminal.md +++ b/docs/agent-terminal.md @@ -1,34 +1,44 @@ -# Workspace interactive terminal +# Interactive terminal -A real shell inside the agent workspace, shown as the **Terminal** tab in the -workspace UI. It's an xterm.js front end bridged to a bash/tmux PTY running in a -persistent per-job container (the agent job image), mounting the same workspace -the editor and agent use. +A real shell inside the user's persistent box (`spoon-box-{username}`). There are +two entry points, both bridging to the **same** box: + +- The workspace **Terminal** tab — a shell opened at the thread's checkout + (`~/Code/{spoon}/{branch}`). +- The **My Machine** page (`/machine`) — a `~`-rooted shell into the same box. + +Both are an xterm.js front end bridged to a bash/tmux PTY via `docker exec` into +the box. There is no per-job `spoon-agent-term-*` container anymore — the box is +long-running and shared by the agent, the editor, and both terminals. ## Architecture ``` browser (xterm.js) - │ 1. GET /api/agent-jobs/:id/terminal-token (Convex-auth'd, owner only) - │ → { url: "wss://worker…/jobs/:id/terminal?token=…", expiresAt } - │ - │ 2. WebSocket wss://worker./jobs/:id/terminal?token=… + │ 1a. GET /api/agent-jobs/:id/terminal-token (Convex-auth'd, owner only) + │ → { url: "wss://worker…/jobs/:id/terminal?token=…", expiresAt } + │ 1b. GET /api/box/terminal-token (Convex-auth'd; username + │ → { url: "wss://worker…/box/terminal?token=…", expiresAt } derived + │ server-side) + │ 2. WebSocket wss://worker./{jobs/:id|box}/terminal?token=… ▼ nginx ── upgrade ──► spoon-agent-worker :3921 - │ verifyTerminalToken(token, jobId, secret) - │ dockerode exec -t → bash/tmux PTY + │ verify HMAC token (job-scoped or username-scoped) + │ docker exec -it → bash/tmux PTY ▼ - spoon-agent-term- (job image, mounts the workspace) + spoon-box-{username} (Fedora box; persistent home mounted) ``` -- The browser **never** holds the worker secret. The Next app (which has already - verified job ownership) mints a short-lived HMAC token; the worker verifies it. +- The browser **never** holds the worker secret. The Next app mints a short-lived + HMAC token; the worker verifies it. The job terminal token is minted only after + a Convex ownership check; the box terminal token carries the username derived + **server-side** from the authed user (never from the request). - Frames: **binary** = stdin/stdout bytes; **text JSON** `{type:"resize",cols,rows}` = resize. The token's 2-minute expiry is a _connect_ window; an established session persists. -- The shell runs `tmux new-session -A -s spoon` (falls back to `bash -l`), so - reconnecting reattaches the same session. Idle containers are removed after - `SPOON_AGENT_TERMINAL_IDLE_MS` (default 30m). +- The shell prefers `tmux new-session -A -s spoon` (falls back to `bash -l`), so + reconnecting reattaches the same session. The box itself is idle-reaped after + `SPOON_AGENT_BOX_IDLE_MS` (default 30m) once no job or terminal holds it. ## Configuration @@ -37,8 +47,7 @@ nginx ── upgrade ──► spoon-agent-worker :3921 | Next app | `NEXT_PUBLIC_SPOON_AGENT_WORKER_WS_URL` | **Yes** | Browser-facing worker WS base, e.g. `wss://worker.spoon.gbrown.org` (prod) or `ws://localhost:3921` (dev). **Build-time** (`NEXT_PUBLIC`): for the Docker image it must be passed as a build arg (wired in `docker/Dockerfile` + `docker/compose.yml`, sourced from the build env file), not a runtime env. Unset → the Terminal tab shows "not configured". | | Next app | `SPOON_AGENT_TERMINAL_SECRET` | No | HMAC secret for signing tokens. Falls back to `SPOON_AGENT_WORKER_INTERNAL_TOKEN`. | | Worker | `SPOON_AGENT_TERMINAL_SECRET` | No | Must match the Next app's. Falls back to `SPOON_AGENT_WORKER_INTERNAL_TOKEN` (already shared), so by default **no new secret is needed**. | -| Worker | `SPOON_AGENT_TERMINAL_IMAGE` | No | Shell container image. Defaults to `SPOON_AGENT_JOB_IMAGE`. | -| Worker | `SPOON_AGENT_TERMINAL_IDLE_MS` | No | Idle-container reap delay (default `1800000`). | +| Worker | `SPOON_AGENT_BOX_IDLE_MS` | No | Idle-box reap delay, shared with agent jobs (default `1800000`). The terminal holds the box open while connected. | Because the secret defaults to the already-shared worker token, the **only** required step is setting `NEXT_PUBLIC_SPOON_AGENT_WORKER_WS_URL` and exposing the @@ -70,9 +79,10 @@ server { Then set on the Next app: `NEXT_PUBLIC_SPOON_AGENT_WORKER_WS_URL=wss://worker.spoon.gbrown.org`. -> The worker's HTTP routes (`/jobs/:id/tree` etc.) require the internal bearer -> token, so exposing the worker host only usefully exposes the token-gated -> `/jobs/:id/terminal` upgrade. Still, restrict the server block to TLS. +> The worker's HTTP routes (`/jobs/:id/tree`, `/box/*`, etc.) require the +> internal bearer token, so exposing the worker host only usefully exposes the +> token-gated `/jobs/:id/terminal` and `/box/terminal` upgrades. Still, restrict +> the server block to TLS. ## Dev testing (no nginx) @@ -90,15 +100,19 @@ Docker mode. Prod (Docker socket mounted) works as-is. ## Security -- Owner-only: the token route uses Convex auth + `assertOwned`. -- Tokens are short-lived (2m connect window), job-scoped, HMAC-signed. -- A shell in the workspace can reach the network and the repo's git credentials. - This is intended for the single-user self-hosted deployment; do not expose the +- Owner-only: the job token route uses Convex auth + ownership assertion; the + box token derives the username server-side from the authed user. +- Tokens are short-lived (2m connect window), scoped (job-scoped or + username-scoped), and HMAC-signed. +- A shell in the box can reach the network and the repo's git credentials. This + is intended for the single-user self-hosted deployment; do not expose the worker domain without TLS, and keep the deployment single-tenant. ## Tools in the shell -The job image ships `bash`, `tmux`, `neovim`, `git`, `ripgrep`, `jq`, `python3`, -`node`, `bun`, `pnpm`, `yarn`, `curl`/`wget`, `unzip`. Bring your own dotfiles by -cloning them in-session (e.g. `git clone ~/.config/...`); persistent -auto-cloning of a dotfiles repo is a planned follow-up. +The box image ships `bash`, `tmux`, `neovim`, `git`, `ripgrep`, `jq`, `python3`, +`node`, `bun`, `pnpm`, `yarn`, `curl`/`wget`, `unzip`, plus QoL tooling (`fzf`, +`fd`, `bat`, `eza`, `zoxide`, `gh`, `gum`, `oh-my-posh`) and the agent CLIs +(`codex`, `opencode`, `claude`). Personalize the shell in **Settings → Dotfiles** +(overlay files + an optional dotfiles repo), applied to the persistent home; see +[dotfiles.md](dotfiles.md). diff --git a/docs/compose.prod.yml b/docs/compose.prod.yml index 26b0072..8412646 100644 --- a/docs/compose.prod.yml +++ b/docs/compose.prod.yml @@ -7,18 +7,18 @@ # If you change them, read the comments first: # # 1. AGENT WORKDIR (spoon-agent-worker): the worker is containerized but -# launches the Codex job container by talking to the HOST Docker daemon. -# The host can only bind-mount real HOST paths, so the work directory MUST -# be a bind mount whose path is IDENTICAL inside and outside the container, -# and SPOON_AGENT_HOST_WORKDIR must match it. A named volume does NOT work -# here because its real host path is hidden from the worker. All three -# references to /var/lib/spoon-agent/work below must stay in sync; change -# them together if you want the data somewhere else. +# creates and `exec`s into the per-user box (spoon-box-{username}) by talking +# to the HOST Docker daemon. The host can only bind-mount real HOST paths, so +# the work directory MUST be a bind mount whose path is IDENTICAL inside and +# outside the container, and SPOON_AGENT_HOST_WORKDIR must match it. A named +# volume does NOT work here because its real host path is hidden from the +# worker. All three references to /var/lib/spoon-agent/work below must stay +# in sync; change them together if you want the data somewhere else. # # 2. IMAGE FRESHNESS: services use `pull_policy: always` + Watchtower labels so -# a redeploy / new push always lands. The Codex *job* image is pulled by the -# worker itself on startup (see SPOON_AGENT_JOB_IMAGE); restarting the worker -# (which Watchtower does on a new image) re-pulls a fresh job image. +# a redeploy / new push always lands. The *box* image is pulled by the worker +# itself on startup (see SPOON_AGENT_JOB_IMAGE); restarting the worker (which +# Watchtower does on a new image) re-pulls a fresh box image. networks: nginx-bridge: # Change to network you plan to use @@ -83,9 +83,9 @@ services: - SPOON_WORKER_TOKEN=${SPOON_WORKER_TOKEN} volumes: - /var/run/docker.sock:/var/run/docker.sock - # Identical host:container path so the sibling Codex job containers can - # bind-mount the workspace via the host daemon. Do NOT switch this to a - # named volume. See header note (1). + # Identical host:container path so the sibling per-user box can bind-mount + # the persistent home (homes/{username}) via the host daemon. Do NOT switch + # this to a named volume. See header note (1). - /var/lib/spoon-agent/work:/var/lib/spoon-agent/work labels: ['com.centurylinklabs.watchtower.enable=true'] tty: true diff --git a/docs/dotfiles.md b/docs/dotfiles.md index 26716e5..ebf006d 100644 --- a/docs/dotfiles.md +++ b/docs/dotfiles.md @@ -6,10 +6,11 @@ preloaded with QoL CLI tooling, a persistent per-user home, and user dotfiles. ## The model - **Persistent per-user home.** Each user gets a home directory on the worker - host at `${SPOON_AGENT_WORKDIR}/homes/{username}`, bind-mounted into every - job/terminal container at `/home/{username}` (`HOME`). It survives across - sessions, so dotfiles, installed tools, nvim plugins, and shell history persist. - `username` is derived from the user's profile first name (sanitized). + host at `${SPOON_AGENT_WORKDIR}/homes/{username}`, bind-mounted into the + per-user box (`spoon-box-{username}`) at `/home/{username}` (`HOME`). It + survives across sessions, so dotfiles, installed tools, nvim plugins, and shell + history persist. `username` is derived from the user's profile first name + (sanitized). - **Threads as folders.** Each thread's checkout lives at `~/Code/{spoon}/{branch}` inside that home, so every thread shows up as a folder in one home. The agent (`codex --cd …`) and the terminal both open there. @@ -61,18 +62,21 @@ Secrets: dotfiles are encrypted, but real API keys/tokens belong in a Spoon's No new required env. The home is a host directory under the existing workdir, so the prod bind-mount + `SPOON_AGENT_HOST_WORKDIR` translation already covers it. -## Notes / limits (Phase 1) +## Notes / limits - **Repo auth:** public repos only. Private/self-hosted (e.g. Gitea) dotfiles repos are a follow-up (store a token/deploy key). - **Binary files:** the overlay is text-first. - **Cleanup:** `~/Code/{spoon}/{branch}` checkouts persist (threads as folders); a per-thread "delete checkout" action is a follow-up. -- **Concurrency:** jobs share one home; fine at the default +- **Concurrency:** jobs and terminals share one box/home; fine at the default `SPOON_AGENT_MAX_CONCURRENT_JOBS=1`. -## Phase 2 north star +## The box -A single long-running per-user container that every thread `exec`s into (agent -via `docker exec`, not `docker run --rm`). The per-user home + `~/Code/{spoon}/ -{branch}` layout built here is its foundation. +The per-user home + `~/Code/{spoon}/{branch}` layout is the foundation of the +box model: a single long-running per-user container (`spoon-box-{username}`) that +every thread's agent turn, terminal, and command `exec`s into (`docker exec`, not +`docker run --rm`). Home materialization runs once per box start (and again when +the dotfiles config hash changes), so the overlay and repo setup apply to every +session in that box. diff --git a/docs/server-deploy-changes.md b/docs/server-deploy-changes.md index 947eaf6..b082d87 100644 --- a/docs/server-deploy-changes.md +++ b/docs/server-deploy-changes.md @@ -1,7 +1,7 @@ -# Server deploy changes (terminal + dotfiles + Fedora + Phase 2) +# Server deploy changes (box + terminal + dotfiles + webhooks + notifications) -Everything the production host / compose / `.env` needs for the workspace -terminal, personalized dev environment, Nerd Font, and the per-user container. +Everything the production host / compose / `.env` needs for the per-user box, the +terminal, the personalized dev environment, GitHub webhooks, and notifications. Most items have safe defaults; the **Required** ones are the only must-dos. ## Required @@ -44,9 +44,10 @@ Most items have safe defaults; the **Required** ones are the only must-dos. restart picks up the new Fedora job image. Make sure the prod registry has the new `spoon-agent-job:latest`. -4. **Deploy Convex functions** (new tables `userDotfiles`, `userEnvironment`). - `SPOON_ENCRYPTION_KEY` (or `INSTANCE_SECRET`) is already required and is what - encrypts dotfiles at rest — no change, just confirm it's set. +4. **Deploy Convex functions** (new tables `userDotfiles`, `userEnvironment`, + `notifications`, `notificationPreferences`). `SPOON_ENCRYPTION_KEY` (or + `INSTANCE_SECRET`) is already required and is what encrypts dotfiles at rest — + no change, just confirm it's set. 5. **Confirm `SPOON_AGENT_HOST_WORKDIR`** on the `spoon-agent-worker` service is the absolute host path backing `SPOON_AGENT_WORKDIR` (the fix from the terminal @@ -54,17 +55,36 @@ Most items have safe defaults; the **Required** ones are the only must-dos. and are bind-mounted into the box via the host daemon — this only resolves if the host-workdir translation is correct. (No new var; just verify.) +6. **Configure the GitHub App webhook.** Set the App's webhook URL to + `/webhooks/github` and set `GITHUB_APP_WEBHOOK_SECRET` in the + **Convex** production env (the signature-verified `httpAction` runs in Convex, + not the worker). Without the secret the endpoint fails closed (HTTP 503) and + drift only refreshes via the hourly cron. `push` refreshes affected Spoons; + `installation` / `installation_repositories` update connection status. + +7. **Notifications email (optional).** In-app notifications work with no config. + To also send email, set `USESEND_API_KEY`, `USESEND_URL`, and + `USESEND_FROM_EMAIL` in the Convex env. Per-user, per-kind email toggles live + in `Settings → Notifications`; unset preferences default to on. + +8. **Claude Code runtime.** The box image pins `@anthropic-ai/claude-code`, so + Anthropic provider profiles (API key or `anthropic_oauth_json` credential + snapshot) can drive the `claude` runtime with no extra host env. Anthropic + API-key auth flows through `ANTHROPIC_API_KEY`; OAuth snapshots are written to + `~/.claude/.credentials.json` in the box. + ## Optional (safe defaults — only set to override) On the `spoon-agent-worker` service: -| Var | Default | Purpose | -| ------------------------------ | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -| `SPOON_AGENT_TERMINAL_SECRET` | falls back to `SPOON_AGENT_WORKER_INTERNAL_TOKEN` | HMAC secret for terminal tokens (must match the Next app's, which also falls back). Leave unset to use the shared token. | -| `SPOON_AGENT_BOX_IDLE_MS` | `1800000` (30m) | How long a per-user box survives idle before being reaped. | -| `SPOON_AGENT_TERMINAL_IDLE_MS` | `1800000` | (Legacy; box idle now governs cleanup.) | +| Var | Default | Purpose | +| ----------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| `SPOON_AGENT_TERMINAL_SECRET` | falls back to `SPOON_AGENT_WORKER_INTERNAL_TOKEN` | HMAC secret for terminal/box tokens (must match the Next app's, which also falls back). Leave unset to use the shared token. | +| `SPOON_AGENT_BOX_IDLE_MS` | `1800000` (30m) | How long a per-user box survives idle before being reaped. | No new env is needed for dotfiles, the per-user home, or the Nerd Font. +`SPOON_AGENT_TERMINAL_IMAGE` and `SPOON_AGENT_TERMINAL_IDLE_MS` were removed — +the box uses `SPOON_AGENT_JOB_IMAGE` and `SPOON_AGENT_BOX_IDLE_MS`. ## Notes / one-time cleanup @@ -84,7 +104,10 @@ No new env is needed for dotfiles, the per-user home, or the Nerd Font. ```bash docker exec spoon-agent-worker docker --version # CLI present (29.x) -docker run --rm git.gbrown.org/gib/spoon-agent-job:latest codex --version # 0.142 +docker run --rm git.gbrown.org/gib/spoon-agent-job:latest codex --version # 0.142 +docker run --rm git.gbrown.org/gib/spoon-agent-job:latest claude --version # 2.1.207 +docker run --rm git.gbrown.org/gib/spoon-agent-job:latest opencode --version docker run --rm git.gbrown.org/gib/spoon-agent-job:latest bash -lc 'eza --version; zoxide --version; oh-my-posh --version' -# then: open a thread → Terminal tab; Settings → Dotfiles add a .bashrc alias. +# then: /machine → box status + terminal; open a thread → Terminal tab; +# Settings → Dotfiles add a .bashrc alias; Settings → Notifications toggles. ```