ci: read the production env from Infisical and deploy on push

Each job logs in as the read-only spoon-ci machine identity and exports the Infisical prod environment, so production values no longer live in Gitea. After pushing the images, CI asks the VPS to redeploy podman-spoon.service over the forced-command deploy key, with the key passed through env so the printed script never contains it. Installs survive a half-written runner cache, and one production deploy runs at a time.

Infisical prod carries a valid JWKS; the copy in DOTENV_PROD had escaped quotes, so the Convex env sync had been serving invalid JSON at /.well-known/jwks.json. The first sync from Infisical replaces it.

.gitea/workflows/build-next.yml

@@ -1,4 +1,4 @@
-name: Build and Push Spoon Images
+name: Build and Deploy Spoon
 
 on:
   push:
@@ -15,6 +15,14 @@ on:
       - 'bun.lock'
       - 'turbo.json'
 
+# One production deploy at a time; a newer push waits rather than racing.
+concurrency:
+  group: build-spoon-production
+  cancel-in-progress: false
+
+# Production env comes from the Infisical `prod` environment. Each job logs in
+# as the read-only `spoon-ci` machine identity and writes the env to a temp
+# file (scripts/ci-export-env.mjs), so no prod values live in Gitea.
 jobs:
   quality:
     runs-on: ubuntu-latest
@@ -23,14 +31,15 @@ jobs:
       - uses: oven-sh/setup-bun@v2
         with:
           bun-version: 1.3.10
-      - run: bun install --frozen-lockfile
+      - run: ./scripts/ci-install
       - name: Lint, typecheck, and test
         env:
-          DOTENV_PROD: ${{ secrets.DOTENV_PROD }}
+          INFISICAL_CLIENT_ID: ${{ secrets.INFISICAL_CLIENT_ID }}
+          INFISICAL_CLIENT_SECRET: ${{ secrets.INFISICAL_CLIENT_SECRET }}
         run: |
           env_file="$(mktemp)"
           trap 'rm -f "$env_file"' EXIT
-          printf '%s\n' "$DOTENV_PROD" > "$env_file"
+          node scripts/ci-export-env.mjs prod > "$env_file"
           bunx dotenv -e "$env_file" -- env NODE_ENV=test SKIP_E2E=1 bun run ci:check
 
   build-images:
@@ -43,14 +52,15 @@ jobs:
       - uses: oven-sh/setup-bun@v2
         with:
           bun-version: 1.3.10
-      - run: bun install --frozen-lockfile
+      - run: ./scripts/ci-install
       - name: Build Next image
         env:
-          DOTENV_PROD: ${{ secrets.DOTENV_PROD }}
+          INFISICAL_CLIENT_ID: ${{ secrets.INFISICAL_CLIENT_ID }}
+          INFISICAL_CLIENT_SECRET: ${{ secrets.INFISICAL_CLIENT_SECRET }}
         run: |
           env_file="$(mktemp)"
           trap 'rm -f "$env_file"' EXIT
-          printf '%s\n' "$DOTENV_PROD" > "$env_file"
+          node scripts/ci-export-env.mjs prod > "$env_file"
           CI_ENV_FILE="$env_file" ./scripts/build-next-app production
       - name: Build agent images
         env:
@@ -72,3 +82,23 @@ jobs:
           docker tag spoon-agent-job:latest git.gbrown.org/gib/spoon-agent-job:latest
           docker push git.gbrown.org/gib/spoon-agent-job:${{ gitea.sha }}
           docker push git.gbrown.org/gib/spoon-agent-job:latest
+
+      # Asks the VPS to pull the new images and restart podman-spoon.service
+      # through the forced-command deploy key (~/Server/AGENTS.md §15). A
+      # failure only warns: the images are pushed and the midnight
+      # podman-update.timer deploys them anyway. The key comes in through env,
+      # not ${{ secrets.* }} in the script: the runner prints the script, and
+      # Gitea does not mask a multi-line secret split across script lines.
+      - name: Trigger immediate deploy
+        env:
+          DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }}
+        run: |
+          install -m600 /dev/null "$HOME/.deploy_key"
+          printf '%s\n' "$DEPLOY_SSH_KEY" > "$HOME/.deploy_key"
+          ssh -i "$HOME/.deploy_key" \
+              -o BatchMode=yes \
+              -o StrictHostKeyChecking=accept-new \
+              -o ConnectTimeout=15 \
+              [email protected] spoon \
+            || echo "::warning::deploy trigger failed; nightly sweep will deploy instead"
+          rm -f "$HOME/.deploy_key"

.infisical.json

@@ -1,5 +1,6 @@
 {
-  "workspaceId": "57f4b778-582f-49e2-89e1-5c7e6077facb",
+    "workspaceId": "57f4b778-582f-49e2-89e1-5c7e6077facb",
-  "defaultEnvironment": "",
+    "defaultEnvironment": "",
-  "gitBranchToEnvironmentMapping": null
+    "gitBranchToEnvironmentMapping": null,
+    "domain": "https://infisical.gbrown.org/api"
 }

AGENTS.md

@@ -70,7 +70,13 @@
 - Old terminal workspaces can be deleted from `Settings -> Worker`; orphaned
   containers/workdirs are cleaned through the worker HTTP API, not from the
   browser directly.
-- CI uses Gitea-injected secrets or `CI_ENV_FILE` and must not call Infisical.
+- CI reads Infisical `prod` through `scripts/ci-export-env.mjs`, logged in as the
+  read-only `spoon-ci` machine identity (Gitea secrets `INFISICAL_CLIENT_ID` and
+  `INFISICAL_CLIENT_SECRET`), and hands the result to the scripts as
+  `CI_ENV_FILE`. `scripts/export-env` refuses to run in CI.
+- A push to `main` builds the images, syncs the Convex env, deploys Convex, pushes
+  the images, and asks the VPS to redeploy `podman-spoon.service`
+  (`ssh [email protected] spoon`). A push to `main` is a production deploy.
 - Gitea image builds force `SPOON_AGENT_CONTAINER_RUNTIME=docker`; keep local
   Podman auto-detection out of CI image tagging/pushing.
 - CI must provide Convex deployment env for codegen, either

docs/server-deploy-changes.md

@@ -7,7 +7,7 @@ Most items have safe defaults; the **Required** ones are the only must-dos.
 ## Required
 
 1. **Build-time env for the Next image** — add to the build env file (the one CI /
-   `scripts/build-next-app` passes as build args; e.g. `DOTENV_PROD`):
+   `scripts/build-next-app` passes as build args, which CI reads from Infisical `prod`):
 
    ```
    NEXT_PUBLIC_SPOON_AGENT_WORKER_WS_URL=wss://worker.spoon.gbrown.org

scripts/ci-export-env.mjs

@@ -0,0 +1,62 @@
+#!/usr/bin/env node
+// ci-export-env.mjs [environment]
+//
+// CI's counterpart to scripts/export-env: prints an Infisical environment
+// (default `staging`) as dotenv on stdout, in the same quoting export-env uses.
+// It logs in as the `spoon-ci` machine identity through Universal Auth, using
+// INFISICAL_CLIENT_ID and INFISICAL_CLIENT_SECRET from the job's Gitea secrets,
+// and reads the project and host from .infisical.json. Plain Node, no CLI to
+// install, so it runs in every job whether or not Bun is set up.
+import { readFile } from 'node:fs/promises';
+
+const environment = process.argv[2] ?? 'staging';
+const { INFISICAL_CLIENT_ID: clientId, INFISICAL_CLIENT_SECRET: clientSecret } =
+  process.env;
+
+const fail = (message) => {
+  console.error(`ci-export-env: ${message}`);
+  process.exit(1);
+};
+
+if (!clientId || !clientSecret)
+  fail('INFISICAL_CLIENT_ID and INFISICAL_CLIENT_SECRET must be set.');
+
+const { workspaceId, domain } = JSON.parse(
+  await readFile(new URL('../.infisical.json', import.meta.url), 'utf8'),
+);
+
+const request = async (path, init) => {
+  const response = await fetch(`${domain}${path}`, init);
+  if (!response.ok)
+    fail(`${init?.method ?? 'GET'} ${path} returned ${response.status}.`);
+  return response.json();
+};
+
+const { accessToken } = await request('/v1/auth/universal-auth/login', {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json' },
+  body: JSON.stringify({ clientId, clientSecret }),
+});
+
+const query = new URLSearchParams({
+  workspaceId,
+  environment,
+  secretPath: '/',
+  expandSecretReferences: 'true',
+  include_imports: 'true',
+});
+const { secrets, imports = [] } = await request(`/v3/secrets/raw?${query}`, {
+  headers: { Authorization: `Bearer ${accessToken}` },
+});
+
+// Imported secrets first so the environment's own values win.
+const values = new Map();
+for (const { secretKey, secretValue } of [
+  ...imports.flatMap((folder) => folder.secrets),
+  ...secrets,
+])
+  values.set(secretKey, secretValue);
+
+const quote = (value) => `'${value.replaceAll("'", "'\\''")}'`;
+for (const [key, value] of [...values].sort(([a], [b]) => a.localeCompare(b)))
+  process.stdout.write(`${key}=${quote(value)}\n`);

scripts/ci-install

@@ -0,0 +1,14 @@
+#!/usr/bin/env bash
+# Install on a runner, and survive a half-written cache entry.
+#
+# Runners have failed twice with "Fail extracting tarball" on a different
+# package each time, on different machines, and passed on a re-run. The cache
+# is shared between jobs on a host, so a job that dies mid-write leaves an
+# entry the next job cannot read. Clearing it and installing again costs a
+# couple of minutes; a human re-running the job costs however long it takes
+# someone to notice.
+set -euo pipefail
+if bun install --frozen-lockfile; then exit 0; fi
+echo "Install failed. Clearing the bun cache and trying once more." >&2
+rm -rf "${BUN_INSTALL_CACHE_DIR:-$HOME/.bun/install/cache}"
+bun install --frozen-lockfile