106 lines
5.3 KiB
Markdown
106 lines
5.3 KiB
Markdown
# Screen Intelligence Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Add polished local OCR and code recognition to Panama's existing capture experience.
|
|
|
|
**Architecture:** Capture keeps ownership of geometry and delegates Read captures to a focused `ScreenIntelligence` singleton. A deterministic helper emits JSON from local Tesseract and ZBar, while a transient Quickshell result surface owns presentation and explicit follow-up actions.
|
|
|
|
**Tech Stack:** Quickshell 0.3.0, Qt 6 QML, Hyprland 0.56 Lua configuration, Bash, Tesseract 5, ZBar, grim, wl-clipboard
|
|
|
|
**Spec:** `docs/superpowers/specs/2026-08-17-screen-intelligence-design.md`
|
|
|
|
## Global Constraints
|
|
|
|
- Recognition is local until Search, Translate, or Open is explicitly selected.
|
|
- Reuse the existing Screen, Window, and Selection picker.
|
|
- Use Tokyo Night Moon and Prism tokens; no component-local colors.
|
|
- Do not add idle animation or polling.
|
|
- Temporary screen captures must be removed when dismissed or replaced.
|
|
- Missing recognition packages must degrade into clear guidance, not shell errors.
|
|
|
|
---
|
|
|
|
### Task 1: Recognition boundary
|
|
|
|
**Files:**
|
|
- Create: `config/dot/quickshell/scripts/screen-intelligence`
|
|
- Create: `tests/quickshell/screen-intelligence-helper-contract.sh`
|
|
- Modify: `setup/packages/hyprland-packages`
|
|
|
|
**Interfaces:**
|
|
- Produces: `screen-intelligence probe` and `screen-intelligence analyze-file IMAGE`, each emitting one JSON object.
|
|
- Consumes: `tesseract`, `zbarimg`, and `jq`.
|
|
|
|
- [x] Write a contract that generates a high-contrast `PANAMA SCREEN INTELLIGENCE` fixture and expects that literal text from `analyze-file`.
|
|
- [x] Run it and verify it fails because the helper does not exist.
|
|
- [x] Implement strict argument validation, readiness probing, OCR, code recognition, JSON encoding, and user-safe errors.
|
|
- [x] Add `tesseract`, `tesseract-langpack-eng`, and `zbar` to the Hyprland package set.
|
|
- [x] Run the helper contract to green with the real engine.
|
|
|
|
### Task 2: Recognition state and actions
|
|
|
|
**Files:**
|
|
- Create: `config/dot/quickshell/services/ScreenIntelligence.qml`
|
|
- Test: `tests/quickshell/screen-intelligence-contract.sh`
|
|
|
|
**Interfaces:**
|
|
- Produces: readiness properties, `phase`, `text`, detected-code data, `analyzeRegion`, `analyzeFile`, copy/search/translate/open actions, and `close`.
|
|
- Consumes: the Task 1 helper, `grim`, `wl-copy`, `xdg-open`, and `Quickshell.cachePath`.
|
|
|
|
- [x] Write a live IPC contract that expects the target, readiness JSON, fixture analysis, and cleanup.
|
|
- [x] Run it and verify the IPC target is missing.
|
|
- [x] Implement bounded process state, JSON parsing, explicit network actions, and temporary-file cleanup.
|
|
- [x] Wire a typed diagnostics and command IPC target in `shell.qml`.
|
|
- [x] Run the live contract to green.
|
|
|
|
### Task 3: Capture integration and result sheet
|
|
|
|
**Files:**
|
|
- Modify: `config/dot/quickshell/services/Capture.qml`
|
|
- Modify: `config/dot/quickshell/modules/capture/CaptureBar.qml`
|
|
- Create: `config/dot/quickshell/modules/capture/IntelligenceResult.qml`
|
|
- Modify: `config/dot/quickshell/shell.qml`
|
|
|
|
**Interfaces:**
|
|
- Produces: Screenshot, Record, and Read action selection plus the transient result overlay.
|
|
- Consumes: Task 2's `analyzeRegion` and state/action API.
|
|
|
|
- [x] Extend the live contract to open Selection + Read mode and inspect the exposed mode state.
|
|
- [x] Run it and verify the Read state is absent.
|
|
- [x] Add mutually exclusive action selection and defer recognition until the picker unmaps.
|
|
- [x] Build processing, success, empty, code, and unavailable result states with keyboard dismissal.
|
|
- [x] Restart Quickshell and inspect fresh logs for construction errors or warnings.
|
|
|
|
### Task 4: Settings, shortcut, launcher, and documentation
|
|
|
|
**Files:**
|
|
- Create: `config/dot/quickshell/modules/settings/ScreenIntelligencePage.qml`
|
|
- Modify: `config/dot/quickshell/modules/settings/SettingsSidebar.qml`
|
|
- Modify: `config/dot/quickshell/modules/settings/SettingsShell.qml`
|
|
- Modify: `config/dot/quickshell/modules/settings/qmldir`
|
|
- Modify: `config/dot/quickshell/services/ShellState.qml`
|
|
- Modify: `config/dot/hypr/keybinds.lua`
|
|
- Modify: `config/dot/hypr/rules.lua`
|
|
- Create: `config/local/share/applications/panama-screen-intelligence.desktop`
|
|
- Modify: `config/dot/hypr/README.md`
|
|
- Modify: `README.md`
|
|
- Modify: `tests/quickshell/settings-pages-contract.sh`
|
|
|
|
**Interfaces:**
|
|
- Produces: Settings destination `screen-intelligence`, `Super+Shift+S`, and a searchable desktop entry.
|
|
|
|
- [x] Extend Settings and keybinding contracts with the new page and shortcut.
|
|
- [x] Run them and verify the route and bind fail.
|
|
- [x] Build the readiness/privacy page and add it to Settings navigation.
|
|
- [x] Add the keybinding, overlay privacy rule, desktop entry, and documentation.
|
|
- [x] Link and validate the desktop entry, reload Hyprland, and run contracts to green.
|
|
|
|
### Task 5: Full-system verification
|
|
|
|
- [x] Run the helper and live Screen Intelligence contracts.
|
|
- [x] Run every Quickshell contract and `Hyprland --verify-config`.
|
|
- [x] Restart the live shell and inspect fresh logs.
|
|
- [ ] Exercise a real screen-region OCR capture and confirm Copy reaches clipboard history.
|
|
- [ ] Review the complete diff, commit the Panama snapshot, and push `main` to `origin`.
|