Files
Panama/docs/superpowers/specs/2026-08-17-screen-intelligence-design.md
T
Gabriel Brown d96863b687 Convert British spellings to American across the repo
colour -> color, behaviour -> behavior, centre -> center, favourite ->
favorite, and about twenty other pairs, applied consistently across
comments, docs, error/UI copy, and a handful of QML identifiers that
used the British spelling as their actual name: SystemSettings'
serialiseValue/serialiseTable/normaliseGradient, Displays'
normaliseModes, Wallpaper's normalisePolicy, SettingsBackup's
serialiseHomeState, DateTime's ntpSynchronised property, Clipboard's
_normalise helper, and ShortcutCapture's cancelled signal (with its
onCancelled handler in ShortcutsPage.qml). Every call site and the two
tests that assert on the literal source text (settings-ownership and
settings-backup-live contracts) were updated in lockstep.

Left untouched: config/dot/espanso/match/packages/misspell-en/ is a
vendored third-party autocorrect dictionary -- its entries are typo
corrections, not our prose, and rewriting them would fight the
package's own purpose (and any future re-sync from upstream).

The already-American `favorites` property (Home page pinned
accessories) was never actually misspelled -- only nearby comments and
error strings said "favourites" -- so no data migration was needed
there.

Claude-Session: https://claude.ai/code/session_01E6TJUAh41HaP25MVHWkhRZ
2026-08-19 08:07:55 -04:00

4.3 KiB

Screen Intelligence Design

Purpose

Screen Intelligence makes text trapped in pixels feel like normal desktop content. It extends Panama's existing screenshot picker with a local Read action for a screen, window, or selection, then presents the result in a focused Prism glass sheet.

The feature is deliberately local-first. Screen pixels are captured into Panama's cache and processed by Tesseract and ZBar on the workstation. Nothing is uploaded unless the user explicitly chooses Search, Translate, or Open.

Interaction model

  • Super+Shift+S opens the existing capture picker in Selection + Read mode.
  • Print keeps opening the complete picker, where Screenshot, Record, and Read are peers.
  • Screen, Window, and Selection continue to use one shared target-selection model.
  • Committing a Read capture dismisses the picker before taking the final image, then opens a result sheet while recognition runs.
  • Escape dismisses the result and deletes its temporary image.
  • The result provides selectable text and explicit Copy, Search, and Translate actions.
  • A detected URL, QR code, or barcode is shown separately. Web URLs can be opened; all detected values can be copied.
  • A searchable desktop entry opens Read mode without requiring the shortcut.

Architecture

services/Capture.qml remains the owner of screen/window/selection geometry and frozen-frame behavior. It gains a third action state, intelligenceMode, and delegates committed Read geometry to services/ScreenIntelligence.qml only after the capture overlay has unmapped.

services/ScreenIntelligence.qml owns recognition state, process lifecycle, result actions, and cleanup. Its public boundary is:

  • refresh(): void
  • analyzeRegion(geometry: string, outputName: string): void
  • analyzeFile(path: string): void
  • copyText(): void
  • copyCode(): void
  • search(): void
  • translate(): void
  • openDetected(): void
  • close(): void

scripts/screen-intelligence is the deterministic machine boundary. probe reports engine availability. analyze-file IMAGE emits one JSON object containing OCR text, detected code data, and a user-safe error. It never evaluates UI-provided shell text.

modules/capture/IntelligenceResult.qml is a transient overlay, not an application window. It renders a quiet processing state, the selectable result, detected-content card, actions, and actionable failure guidance.

Recognition behavior

  • OCR uses the installed English Tesseract language pack.
  • Tesseract's automatic page-segmentation mode handles both short snippets and larger document regions.
  • ZBar recognition is additive. A capture can return text, a code, or both.
  • Empty recognition is not treated as a crash; the sheet explains that no readable text or code was found.
  • Missing engines produce a precise install command and keep the rest of the shell healthy.
  • Temporary captures live under Quickshell.cachePath(...) and are removed when dismissed or replaced.

Visual language

  • The result is a centered Prism glass sheet over a lightly dimmed desktop.
  • Tokyo Night Moon tokens are the only colors.
  • Processing uses a static status treatment; there is no spinner, pulse, shimmer, or idle animation.
  • OCR text is readable interface typography in a selectable editor, not a terminal-like dump.
  • The primary action is Copy. Network-leaving actions are visually secondary and clearly labeled.

Settings and discoverability

Panama Settings gains a real Screen Intelligence page. It shows engine readiness, the Super+Shift+S shortcut, the local-processing privacy boundary, and the exact package requirement if recognition is unavailable. The page contains no fake toggles.

The shortcut reference, Hyprland README, root README, desktop package list, and desktop entry remain in sync.

Verification

  • A generated image fixture must be recognized as literal text by the helper.
  • The live IPC contract must expose readiness, open Read mode, and route the Settings page.
  • The capture picker must construct with Screenshot, Record, and Read actions without QML warnings.
  • The result sheet must construct for processing, success, empty, and unavailable states.
  • Hyprland must register Super+Shift+S as Screen Intelligence and pass config verification.
  • The complete Quickshell contract suite must remain green.