Gabriel Brown
7cd4131327
Super+D holds the microphone open, releasing it transcribes on the GPU and types the result wherever the cursor is. Roughly 150ms for a normal utterance once the model is resident, measured rather than hoped for. Getting there meant discarding two approaches. Fedora 44 cannot install any GPU-capable Whisper for Python -- openai-whisper needs a numba that needs an llvmlite that does not exist for 3.14, and faster-whisper needs a ctranslate2 nobody packaged. The whisper-cpp package IS built with HIP but ships libraries with no binary and no bindings, and hand-writing ctypes for a large by-value struct is a segfault waiting for a version bump. So a container, as suggested. Vulkan rather than ROCm, and upstream's image rather than one built here. ROCm is seven gigabytes and serves AMD alone; Vulkan compute runs on the AMD, Intel and NVIDIA machines this config is used on, in a twentieth of the space. The Vulkan tag already contains whisper-server, so there is no Containerfile to keep working -- an earlier draft of this commit had one, and it was strictly worse. Two bugs found by using it rather than by reading it. Whisper describes silence as the literal text "[BLANK_AUDIO]", and the first working version pasted that string into the clipboard; a transcription that is nothing but such markers is now discarded. And the server answers with a line per segment, which typed into a window is an Enter press -- sending the half-written message, submitting the form. Whitespace is collapsed to one line. Neither the image nor the model is installed by ./install. Together they are over two gigabytes that want the network, and Settings offers both as one action instead. Nothing starts at login either: whisper-server holds the model from the moment it starts, so the first press of the key is what brings it up. The contract pins both text bugs, that the server stays on loopback, and that it does not start at login. Reverting the [BLANK_AUDIO] guard did not fail it at first -- the check was still correct, it had simply stopped being called -- so it now checks the call site too. Claude-Session: https://claude.ai/code/session_01Q84axqUE5inJhf5Jz9CFy1
141 lines
7.2 KiB
Bash
Executable File
141 lines
7.2 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
|
|
# Dictation: hold a key, speak, and the words are typed where the cursor is.
|
|
#
|
|
# The parts worth pinning are the ones that put text into somebody's document,
|
|
# because a mistake there is not a feature failing -- it is the wrong thing
|
|
# arriving in the middle of a sentence, and both of these were real:
|
|
#
|
|
# * Whisper describes silence as the literal text "[BLANK_AUDIO]". The first
|
|
# working version of this dictated that string into the clipboard.
|
|
# * The server answers with one line per segment, and a newline typed into a
|
|
# window is an Enter press -- sending the half-written message, submitting
|
|
# the form, running the command.
|
|
#
|
|
# Plus the container: it transcribes whatever it is sent with no authentication,
|
|
# so it must not be reachable from the network, and it holds the model in memory
|
|
# once started, so it must not start at login.
|
|
|
|
set -uo pipefail
|
|
|
|
repo_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
|
helper="$repo_dir/config/dot/quickshell/scripts/panama-dictate"
|
|
quadlet="$repo_dir/config/containers/whisper/panama-whisper.container"
|
|
keybinds="$repo_dir/config/dot/hypr/keybinds.lua"
|
|
service="$repo_dir/config/dot/quickshell/services/Dictation.qml"
|
|
|
|
findings=()
|
|
note() { findings+=("$1"); }
|
|
|
|
[[ -x "$helper" ]] || { printf 'dictation contract: helper is missing or not executable\n' >&2; exit 1; }
|
|
|
|
# ── What gets typed ──────────────────────────────────────────────────────────
|
|
#
|
|
# is_speech and the whitespace collapse are exercised for real rather than
|
|
# grepped for, because both are judgements about text and a spelling of the
|
|
# regex is not what matters.
|
|
|
|
python3 - "$helper" <<'PROBE' || note 'the text handling did not behave as dictation requires'
|
|
import importlib.machinery, importlib.util, sys
|
|
spec = importlib.util.spec_from_loader("dictate",
|
|
importlib.machinery.SourceFileLoader("dictate", sys.argv[1]))
|
|
module = importlib.util.module_from_spec(spec)
|
|
spec.loader.exec_module(module)
|
|
|
|
# Whisper's own way of saying it heard no words. Typing these is the bug.
|
|
for marker in ("[BLANK_AUDIO]", "[ Silence ]", "(wind blowing)", "[MUSIC]",
|
|
"[BLANK_AUDIO] [BLANK_AUDIO]"):
|
|
assert not module.is_speech(marker), marker
|
|
|
|
# Real speech, including speech that merely contains brackets. Discarding these
|
|
# would be far worse than typing a stray marker.
|
|
for spoken in ("And so my fellow Americans", "commit the change (the small one) now",
|
|
"git log [main]", "a"):
|
|
assert module.is_speech(spoken), spoken
|
|
PROBE
|
|
|
|
# Testing is_speech in isolation is not enough, and this was found the honest
|
|
# way: reverting the guard in handle_stop left the probe above passing, because
|
|
# the function was still correct -- it had simply stopped being called. A
|
|
# transcription has to be checked before it is delivered.
|
|
grep -q 'not is_speech(text)' "$helper" \
|
|
|| note 'the non-speech check exists but nothing calls it before typing, so [BLANK_AUDIO] would be typed'
|
|
|
|
# The collapse itself: a transcription arriving as several lines has to leave as
|
|
# one, or the newlines are typed as Enter.
|
|
grep -q '" ".join(answer.split())' "$helper" \
|
|
|| note 'the transcription is not collapsed to one line, so a newline would be typed as Enter'
|
|
|
|
# ── How it gets there ────────────────────────────────────────────────────────
|
|
|
|
grep -q 'shutil.which("wtype")' "$helper" \
|
|
|| note 'dictation does not try to type the text'
|
|
grep -q 'shutil.which("wl-copy")' "$helper" \
|
|
|| note 'dictation has no clipboard fallback for applications that refuse synthetic keys'
|
|
|
|
# Typing must be tried FIRST. The clipboard works but overwrites whatever was
|
|
# copied and fills Panama's clipboard history with everything ever dictated.
|
|
typing_line="$(grep -n 'shutil.which("wtype")' "$helper" | head -1 | cut -d: -f1)"
|
|
clipboard_line="$(grep -n 'shutil.which("wl-copy")' "$helper" | head -1 | cut -d: -f1)"
|
|
[[ -n "$typing_line" && -n "$clipboard_line" && "$typing_line" -lt "$clipboard_line" ]] \
|
|
|| note 'the clipboard is tried before typing, so dictation would overwrite the clipboard by default'
|
|
|
|
declared="$(cat "$repo_dir"/setup/packages/* 2>/dev/null | sed 's/#.*//' | tr -d ' ' | grep -v '^$')"
|
|
grep -qix wtype <<<"$declared" \
|
|
|| note 'wtype is what types the text, and no package list installs it'
|
|
|
|
# ── The container ────────────────────────────────────────────────────────────
|
|
|
|
[[ -f "$quadlet" ]] || note 'the speech server has no quadlet'
|
|
if [[ -f "$quadlet" ]]; then
|
|
grep -qE '^PublishPort=127\.0\.0\.1:' "$quadlet" \
|
|
|| note 'the speech server is published beyond loopback, and it authenticates nothing'
|
|
grep -qE '^AddDevice=/dev/dri$' "$quadlet" \
|
|
|| note 'the GPU is not passed to the container, or is passed as a device number that differs per machine'
|
|
grep -q ':ro' "$quadlet" \
|
|
|| note 'the model is mounted writable, which nothing needs'
|
|
grep -q ':z' "$quadlet" \
|
|
|| note 'the model mount is not relabelled, so SELinux will deny it'
|
|
|
|
# No [Install]: whisper-server holds the model from the moment it starts.
|
|
grep -q '^\[Install\]' "$quadlet" \
|
|
&& note 'the speech server starts at login, holding the model in every session where nobody dictates'
|
|
|
|
# Vulkan is the whole reason for this image: it runs on the AMD, Intel and
|
|
# NVIDIA machines this config is used on. A ROCm tag would serve one of them.
|
|
grep -qE '^Image=.*vulkan' "$quadlet" \
|
|
|| note 'the image is not the Vulkan build, so it will not work on every machine this runs on'
|
|
fi
|
|
|
|
# ── The keybinds ─────────────────────────────────────────────────────────────
|
|
|
|
presses="$(grep -c 'dictate("start")' "$keybinds" || true)"
|
|
releases="$(grep -c 'dictate("stop")' "$keybinds" || true)"
|
|
(( presses == 1 && releases == 1 )) \
|
|
|| note "hold-to-talk needs one press bind and one release bind; found $presses and $releases"
|
|
|
|
grep -qE 'dictate\("stop"\)\)?,?$|release = true' "$keybinds" \
|
|
|| note 'the transcribe bind is not flagged release, so it would fire on press'
|
|
|
|
# A repeating press bind would restart the recording several times a second for
|
|
# as long as the key is held.
|
|
grep -A1 'dictate("start")' "$keybinds" | grep -q 'repeating' \
|
|
&& note 'the dictation press bind repeats, which would restart recording while the key is held'
|
|
|
|
grep -q 'dictate("cancel")' "$keybinds" \
|
|
|| note 'there is no way to abandon a recording without transcribing it'
|
|
|
|
# ── The page cannot claim more than it knows ─────────────────────────────────
|
|
|
|
grep -q '"status"' "$service" \
|
|
|| note 'the settings service never asks the helper what is installed'
|
|
|
|
if (( ${#findings[@]} > 0 )); then
|
|
mapfile -t findings < <(printf '%s\n' "${findings[@]}" | sort -u)
|
|
printf 'dictation contract: %d finding(s)\n' "${#findings[@]}" >&2
|
|
printf ' - %s\n' "${findings[@]}" >&2
|
|
exit 1
|
|
fi
|
|
|
|
printf 'dictation contract: PASS\n'
|