From c17c7a3a0dedc341c87cbf1f74ce0877f4081077 Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 14:48:14 -0400 Subject: [PATCH] feat(stt): stop sending anything about the running app to the LLM MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `config.llm.instruction` carried a formatting clause derived from the frontmost app: `AppKindPriming` recognized the app's bundle ID as a kind (terminal, code editor, Slack, Obsidian) and sent "Format the result as a shell command with no trailing period.", "… as Python code." (language inferred from the window title's filename), and so on. Remove the whole path rather than just the last link, so nothing is left dead for periphery to flag: - `AppKindPriming` and its tests, deleted. - `TranscriptionSteering.Fields.rewriteInstruction`, gone — `build` now renders exactly two fields, `conversation_context` and `keyterms_prompt`. - `LLMRewrite` encodes as `{}`. The `llm` block still rides on every request (it is what asks the service to rewrite at all), so the service's own default cleanup instruction now runs for every utterance. That is the same wire shape an unrecognized app already produced. - `TranscriptionContext.bundleID` and the `CapturedFocus` bundle-ID capture, removed — the clause was their only consumer. `windowTitle` stays: it anchors `KeyInjector`'s same-window paste separator. - `DictationLog`'s `llm_instruction` column, dropped. The steering and context test tables keep a case per app family that used to earn a clause, each pinned to `.empty`, so reintroducing app-kind steering fails there; the log's "omits every steering field" test pins `llm_instruction` never being written. Verified with `scripts/check.sh` (425 unit tests, 23 UI tests, sanitizers, leaks, periphery, linters) — green apart from a prettier reflow of AGENTS.md, applied here. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/project-guardrails/SKILL.md | 14 +- AGENTS.md | 110 ++++++------- BLURTENGINE.md | 8 +- README.md | 2 +- .../FocusCapture/FocusCapture.swift | 9 +- .../BlurtEngine/Pipeline/DictationLog.swift | 6 +- .../Pipeline/DictationSession.swift | 1 - Sources/BlurtEngine/STT/AppKindPriming.swift | 151 ------------------ .../STT/AssemblyAITranscriber.swift | 27 ++-- .../STT/TranscriptionContext.swift | 27 +--- .../STT/TranscriptionSteering.swift | 39 +++-- .../AppKindPrimingTests.swift | 129 --------------- .../AssemblyAITranscriberTests.swift | 34 ++-- .../BlurtEngineTests/DictationLogTests.swift | 27 ++-- .../TranscriptionContextTests.swift | 33 ++-- .../TranscriptionSteeringTests.swift | 69 +++----- 16 files changed, 179 insertions(+), 507 deletions(-) delete mode 100644 Sources/BlurtEngine/STT/AppKindPriming.swift delete mode 100644 Tests/BlurtEngineTests/AppKindPrimingTests.swift diff --git a/.claude/skills/project-guardrails/SKILL.md b/.claude/skills/project-guardrails/SKILL.md index 42671d3..8cecfd9 100644 --- a/.claude/skills/project-guardrails/SKILL.md +++ b/.claude/skills/project-guardrails/SKILL.md @@ -30,11 +30,15 @@ one, stop and ask the user first. This is the fast "don't" list; AGENTS.md's call. No on-device ASR/LLM, no model cache, no download UI. - **Never send `config.prompt`.** It takes a _description of the audio_, not instructions, and a custom value replaces the service's managed default - including its language steering. Formatting goes in `llm.instruction` - ("Format the result as markdown."), vocabulary in `keyterms_prompt`, the text - before the cursor in `conversation_context`. An imperative like "Transcribe - speech into markdown." in `prompt` is a measured no-op — that is exactly why - these three fields exist. + including its language steering. Vocabulary goes in `keyterms_prompt`, the + text before the cursor in `conversation_context`. An imperative like + "Transcribe speech into markdown." in `prompt` is a measured no-op — that is + exactly why those fields exist. +- **Never send anything about the frontmost app.** `AppKindPriming` recognized + the app's bundle ID as a kind (terminal, code editor, Slack, Obsidian) and + sent a formatting clause as `llm.instruction`; the whole path was removed. + The bundle ID isn't captured, and `llm` always goes out empty so the + service's default cleanup rewrite applies everywhere. - Don't reintroduce a "remove filler words (um, uh, like)" directive — the STT prompt doesn't act on it; it was deliberately dropped, and disfluency removal is the LLM rewrite's job. Same for a language directive: pinning to English diff --git a/AGENTS.md b/AGENTS.md index 83fd15c..93672bf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,10 +8,9 @@ the Claude-Code-specific tooling under `.claude/` (hooks, skills, subagents). Blurt is a macOS dictation app powered by [AssemblyAI](https://www.assemblyai.com). Tap or hold a trigger key, speak, and polished text is pasted into the focused app. Transcription is **one remote AssemblyAI dictation API call**: per-utterance steering rides along with the request — the text before -the cursor as `conversation_context`, the user's key terms as `keyterms_prompt`, and an app-kind -formatting clause recognized from the frontmost app — and the same request asks the service for its -server-side LLM cleanup rewrite (`config.llm`), so the text that comes back is already polished. The -user supplies their own API key. +the cursor as `conversation_context` and the user's key terms as `keyterms_prompt` — and the same +request asks the service for its server-side LLM cleanup rewrite (`config.llm`), so the text that +comes back is already polished. The user supplies their own API key. Four reflexes before you touch anything: @@ -53,7 +52,7 @@ Sources/BlurtEngine/ the engine (dependency-free Swift package) Injection/ KeyInjector (clipboard paste), SystemClipboard Permissions/ PermissionsChecker (mic + Accessibility) Pipeline/ DictationSession (actor) + phases, UI projections, geometry, log - STT/ AssemblyAITranscriber, TranscriptionSteering/Context, AppKindPriming, SyncSTTLimits + STT/ AssemblyAITranscriber, TranscriptionSteering/Context, SyncSTTLimits Update/ UpdateChecker (download-only) + the launch-check policy App/Blurt/ project.yml XcodeGen source of truth — Blurt.xcodeproj is GENERATED @@ -159,26 +158,27 @@ In Claude Code on the web, a `SessionStart` hook installs the portable linters a Each was tried the other way and reverted. If a task seems to require one, stop and ask first. (`.claude/skills/project-guardrails` is the compressed version of this list.) -| Don't | Because | -| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | -| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. | -| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. | -| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — request steering belongs in `TranscriptionSteering`. | -| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | -| Send `config.prompt` at all | The field takes a _description of the audio_, not instructions, and a custom value replaces the service's managed default **including its language steering**. Formatting goes in `llm.instruction`, vocabulary in `keyterms_prompt`, preceding text in `conversation_context`. | -| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection — and a custom prompt is what drops the managed default's language steering in the first place. | -| Put formatting instructions in `config.prompt` | Reshaping output is not something the STT prompt acts on, so _"Transcribe speech into markdown."_ was a measured no-op. `llm.instruction` (_"Format the result as markdown."_) is the lever that works. | -| Pack key terms into the prompt as `Keywords: a, b, c.` | The API documents `keyterms_prompt` for exactly this, and warns against packing keyword lists into the prompt. | -| Add a "remove filler words (um, uh, like)" clause | Not something the STT prompt acts on — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | -| Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | -| Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | -| Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | -| Add a self-replacing install or background auto-updater | Updates are download-only; `mxcl/AppUpdater` and its in-place updater were removed. The once-a-day launch _check_ (`AutomaticUpdateCheck`) is fine; installing for the user, or polling, is not. Extend `UpdateCheckModel`. | -| Hand-edit `Blurt.xcodeproj/project.pbxproj` | Generated from `project.yml`; `check.sh`'s drift check fails on any manual edit (a Claude PreToolUse hook also blocks it). | -| Redirect the post-build install away from `/Applications` | TCC won't register apps in DerivedData/`/tmp`, so permission toggles never appear. | -| Touch the real Keychain in tests | `APIKeyStore` is the production item — a test that writes it triggers Keychain prompts and corrupts the real item's ACL. Use an isolated service (see `KeychainStoreTests`) or `InMemoryAPIKeyStore`. | -| Add backwards-compat shims for removed types | Deleted types stay deleted — no deprecated re-exports. | +| Don't | Because | +| --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | +| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. | +| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. | +| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — request steering belongs in `TranscriptionSteering`. | +| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | +| Send `config.prompt` at all | The field takes a _description of the audio_, not instructions, and a custom value replaces the service's managed default **including its language steering**. Vocabulary goes in `keyterms_prompt`, preceding text in `conversation_context`. | +| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection — and a custom prompt is what drops the managed default's language steering in the first place. | +| Put formatting instructions in `config.prompt` | Reshaping output is not something the STT prompt acts on, so _"Transcribe speech into markdown."_ was a measured no-op. | +| Send anything about the frontmost app to the service | Removed on purpose. `AppKindPriming` recognized the app's bundle ID as a kind and sent a formatting clause as `llm.instruction`; the bundle ID is no longer captured, and `llm` always goes out empty so the service's default cleanup applies everywhere. | +| Pack key terms into the prompt as `Keywords: a, b, c.` | The API documents `keyterms_prompt` for exactly this, and warns against packing keyword lists into the prompt. | +| Add a "remove filler words (um, uh, like)" clause | Not something the STT prompt acts on — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | +| Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | +| Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | +| Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | +| Add a self-replacing install or background auto-updater | Updates are download-only; `mxcl/AppUpdater` and its in-place updater were removed. The once-a-day launch _check_ (`AutomaticUpdateCheck`) is fine; installing for the user, or polling, is not. Extend `UpdateCheckModel`. | +| Hand-edit `Blurt.xcodeproj/project.pbxproj` | Generated from `project.yml`; `check.sh`'s drift check fails on any manual edit (a Claude PreToolUse hook also blocks it). | +| Redirect the post-build install away from `/Applications` | TCC won't register apps in DerivedData/`/tmp`, so permission toggles never appear. | +| Touch the real Keychain in tests | `APIKeyStore` is the production item — a test that writes it triggers Keychain prompts and corrupts the real item's ACL. Use an isolated service (see `KeychainStoreTests`) or `InMemoryAPIKeyStore`. | +| Add backwards-compat shims for removed types | Deleted types stay deleted — no deprecated re-exports. | Release-side invariants (hardened runtime and a secure timestamp on every nested mach-o and embedded framework, or notarization rejects the build; roll-forward-only for a bad release) live in @@ -250,8 +250,8 @@ fields built per utterance by `TranscriptionSteering`: `conversation_context`, ` the `llm` block). No model header — the service pins the STT model server-side. **`prompt` is never sent** — see [Transcription steering](#transcription-steering). The `llm` block asks the service to run a cleanup rewrite (remove disfluencies, fix punctuation) over the verbatim transcript, all inside -the same request; its optional `instruction` is the app-kind formatting clause, and an empty block -selects the service's own default cleanup. The response carries both `text` (verbatim) and +the same request; it always goes out empty, which selects the service's own default cleanup +instruction. The response carries both `text` (verbatim) and `llm_response` (the rewrite); the transcriber returns the rewrite and falls back to `text` when `llm_response` is null — the rewrite is best-effort (5 s server-side budget), so a rewrite failure (`llm_error`) is a logged degradation, never a user-facing error. @@ -380,37 +380,40 @@ to right ⌘), so views must not re-declare `TriggerKey.rightCommand.rawValue` t ## Transcription steering -`Sources/BlurtEngine/STT/TranscriptionSteering.swift` renders the captured context into the three +`Sources/BlurtEngine/STT/TranscriptionSteering.swift` renders the captured context into the two request-customization fields, each with one job. It's unit-tested in `Tests/BlurtEngineTests/TranscriptionSteeringTests.swift`. -| Field | Carries | Cap | -| ---------------------- | ----------------------------------------------------------------- | ----------------------- | -| `conversation_context` | prior-cursor text, as a single turn | 4096 chars, clip head | -| `keyterms_prompt` | the user's key terms, verbatim | 2048 chars, whole terms | -| `llm.instruction` | the `AppKindPriming` formatting clause, or absent for the default | — | +| Field | Carries | Cap | +| ---------------------- | ----------------------------------- | ----------------------- | +| `conversation_context` | prior-cursor text, as a single turn | 4096 chars, clip head | +| `keyterms_prompt` | the user's key terms, verbatim | 2048 chars, whole terms | **`config.prompt` is never sent, and that is the whole point of this design.** The field takes a _description of the audio_ ("Cardiology consultation about chest pain symptoms."), not instructions — -transcription behavior is optimized out of the box — so the app-kind priming's old imperative form -(_"Transcribe speech into markdown."_) was aimed at a field that doesn't act on instructions and was a -no-op. Formatting now rides in `llm.instruction`, where an LLM actually rewrites the text -(_"Format the result as markdown."_ — for a code editor, the language is inferred from the window -title's filename: _"Format the result as Swift code."_), and vocabulary rides in `keyterms_prompt` -rather than being packed into the prompt as a `Keywords: a, b, c.` clause. Sending no prompt also -keeps the service's managed default, which a custom prompt replaces wholesale — **including its -language steering**, which is the mechanism behind the older finding that pinning the prompt to -English hurt non-English speech. - -The remaining focus context is **not** sent: app and field names render nowhere (real-world logs -showed them crowding the request — VS Code, for one, parks a screen-reader help announcement in the -focused field's description), the window title is read only to name a code editor's language, and -selected text is never priming because the paste replaces it. All of it is still captured — the -injector's separator logic consumes the prior text and window title — and reading stays -privacy-guarded: prior and selected text are skipped in secure fields, detected by AX role **or** -subrole and failing closed when the role can't be read, so a password is never read out of the field -at all. That guard, not the steering builder, is what keeps a password out of `conversation_context` -and off the dictation log. +transcription behavior is optimized out of the box — so an imperative like +_"Transcribe speech into markdown."_ was aimed at a field that doesn't act on instructions and was a +no-op. Vocabulary rides in `keyterms_prompt` rather than being packed into the prompt as a +`Keywords: a, b, c.` clause. Sending no prompt also keeps the service's managed default, which a +custom prompt replaces wholesale — **including its language steering**, which is the mechanism behind +the older finding that pinning the prompt to English hurt non-English speech. + +**Nothing describing the destination app is sent.** An earlier design recognized the frontmost app's +bundle ID as a _kind_ (terminal, code editor, Slack, Obsidian) and sent a matching formatting clause +as `llm.instruction`; that whole path — `AppKindPriming`, `TranscriptionContext.bundleID`, the +`CapturedFocus` bundle-ID capture, and the log's `llm_instruction` column — was removed. The `llm` +block now always goes out empty, so the service's own default cleanup rewrite runs for every +utterance regardless of where the text is going. + +The remaining focus context is **not** sent either: app and field names render nowhere (real-world +logs showed them crowding the request — VS Code, for one, parks a screen-reader help announcement in +the focused field's description), the window title is read only to anchor the injector's paste +separator, and selected text is never priming because the paste replaces it. All of it is still +captured — the injector's separator logic consumes the prior text and window title — and reading +stays privacy-guarded: prior and selected text are skipped in secure fields, detected by AX role +**or** subrole and failing closed when the role can't be read, so a password is never read out of the +field at all. That guard, not the steering builder, is what keeps a password out of +`conversation_context` and off the dictation log. `build(context:)` returns `.empty` when there's no usable context (or when nothing renders — an unrecognized app with no prior text and no key terms), and every empty field is **omitted** from the @@ -448,8 +451,7 @@ resolves. History: **`RecentDictations`** is an in-memory, newest-first ring shown in the ready window (never written to disk). **`DictationLog`** appends each completed dictation — the transcript plus the exact -steering fields sent (`conversation_context`, `keyterms_prompt`, `llm_instruction`, under the wire's -own names) — to +steering fields sent (`conversation_context` and `keyterms_prompt`, under the wire's own names) — to `~/Library/Logs/Blurt/dictations.jsonl` (`DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI — derived next to the URL so the label can't drift from the write target) — but **only** while developer mode is on; with it off, nothing is written. The Settings diff --git a/BLURTENGINE.md b/BLURTENGINE.md index dbb2ea5..26d39e1 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -59,7 +59,7 @@ press() ──▶ MicCapture.start() release() ──▶ MicCapture.s Key properties of the design, which your integration can rely on: - **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream. -- **Cleanup happens server-side.** The request's `llm` block asks the service for a cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; its optional `instruction` carries the app-kind formatting clause, and an empty block selects the service's own default cleanup. Recognition is primed separately by `conversation_context` and `keyterms_prompt` (built by `TranscriptionSteering` from the captured context); `config.prompt` is never sent. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. +- **Cleanup happens server-side.** The request's `llm` block asks the service for a cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; it always goes out empty, which selects the service's own default cleanup instruction. Recognition is primed separately by `conversation_context` and `keyterms_prompt` (built by `TranscriptionSteering` from the captured context); `config.prompt` is never sent, and neither is anything describing the destination app. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. - **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read. - **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400. @@ -152,15 +152,15 @@ The session calls `setTargetApp` at press time with the app that was frontmost w Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected: -- **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the bundle ID, window title, and key terms reach the prompt; the prior text and window title also steer the injector's paste separator, and nothing else is consumed. -- **`TranscriptionSteering.build(context:)`** renders that into the request's three customization fields: the prior-cursor text as the single `conversation_context` turn (clipped to 4096 chars keeping the tail, since the words nearest the cursor carry the continuity), the user's key terms as `keyterms_prompt` (whole terms fitted to 2048 chars total), and — when the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian) — the formatting clause as `llm.instruction`, e.g. "Format the result as a shell command with no trailing period." or, for a code editor, the language inferred from the window title's filename ("Format the result as Swift code."). **`config.prompt` is never sent**: it takes a description of the audio rather than instructions, and a custom value replaces the service's managed default including its language steering, which is why the app-kind clause moved to the `llm` block and the key terms to their own field. Nothing else is rendered — no app/field names, no selected text (the paste replaces it), and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default). Empty fields are omitted rather than sent as `[]`. Two further deliberate omissions: no language directive and no "remove filler words" clause (not something the STT prompt acts on — a no-op). Don't reintroduce either. +- **`TranscriptionContext`** carries the frontmost app name, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the prior text and key terms reach the request; the prior text and window title also steer the injector's paste separator, and nothing else is consumed. +- **`TranscriptionSteering.build(context:)`** renders that into the request's two customization fields: the prior-cursor text as the single `conversation_context` turn (clipped to 4096 chars keeping the tail, since the words nearest the cursor carry the continuity) and the user's key terms as `keyterms_prompt` (whole terms fitted to 2048 chars total). **`config.prompt` is never sent**: it takes a description of the audio rather than instructions, and a custom value replaces the service's managed default including its language steering, which is why the key terms ride in their own field. Nothing else is rendered — no app/field names, no selected text (the paste replaces it), and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default). Empty fields are omitted rather than sent as `[]`. Three further deliberate omissions: no language directive, no "remove filler words" clause (not something the STT prompt acts on — a no-op), and nothing identifying the destination app — an earlier `AppKindPriming` sent a bundle-ID-derived formatting clause as `llm.instruction`, and that whole path was removed. Don't reintroduce any of them. - **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere. For key storage, compose against **`APIKeyGateway`** — the injectable `current` / `save(_:)` / `hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. Two projections keep the surrounding UI out of your views: `Outcome.failureReport` classifies a failure as `.inline(message:)` (recoverable — show it beside the field) or `.alert(title:message:)` (a Keychain fault retyping can't fix), and **`APIKeyDisplay.resolve(key:)`** renders the stored key for an account row — masked tail, status and VoiceOver wording, and the connect-vs-rotate control titles. The mask reveals only the last `revealedTailLength` characters and, below `minimumLengthToMask`, none at all, so a short key can't be shown whole. Setup gating has a projection too: **`SetupReadiness.isReady(permissions:hasAPIKey:)`** is the "fully configured" rule (deliberately excluding the trigger key, which has a default), `SetupReadiness.pollInterval(isReady:)` is the permission-poll cadence (brisk during setup, coasting once ready), and `PermissionStatus.lostGrant(since:)` detects a permission revoked out from under a configured app. -Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact steering fields sent (`conversation_context`, `keyterms_prompt`, `llm_instruction`), never the context that wasn't sent — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section. +Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact steering fields sent (`conversation_context`, `keyterms_prompt`), never the context that wasn't sent — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section. ## Hotkey building blocks diff --git a/README.md b/README.md index 9dd68c0..f229f0b 100644 --- a/README.md +++ b/README.md @@ -166,7 +166,7 @@ Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dep Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM, live level meter; DX7/Juno-106 sound packs STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe - (STT + LLM rewrite) + TranscriptionSteering context/keyterms/format + (STT + LLM rewrite) + TranscriptionSteering context/keyterms Pipeline/ DictationSession actor: press/release/cancel commands, phase stream, auto-release before the API's recording cap Hotkey/ DictationKeyGate/Router: pure, unit-tested state machine for the diff --git a/Sources/BlurtEngine/FocusCapture/FocusCapture.swift b/Sources/BlurtEngine/FocusCapture/FocusCapture.swift index 21005ac..4724e42 100644 --- a/Sources/BlurtEngine/FocusCapture/FocusCapture.swift +++ b/Sources/BlurtEngine/FocusCapture/FocusCapture.swift @@ -4,20 +4,13 @@ import ApplicationServices struct CapturedFocus: Sendable { let pid: pid_t let processName: String? - /// The frontmost app's stable identity, feeding the prompt's app-kind - /// recognition (`AppKindPriming`) via `TranscriptionContext.bundleID`. - let bundleID: String? } enum FocusCapture { @MainActor static func captureFrontmost() -> CapturedFocus? { guard let app = NSWorkspace.shared.frontmostApplication else { return nil } - return CapturedFocus( - pid: app.processIdentifier, - processName: app.localizedName, - bundleID: app.bundleIdentifier - ) + return CapturedFocus(pid: app.processIdentifier, processName: app.localizedName) } static func runningApp(for captured: CapturedFocus) -> NSRunningApplication? { diff --git a/Sources/BlurtEngine/Pipeline/DictationLog.swift b/Sources/BlurtEngine/Pipeline/DictationLog.swift index 1974874..8441771 100644 --- a/Sources/BlurtEngine/Pipeline/DictationLog.swift +++ b/Sources/BlurtEngine/Pipeline/DictationLog.swift @@ -28,14 +28,12 @@ public enum DictationLog { /// an entry directly from a context. let conversationContext: [String] let keytermsPrompt: [String] - let llmInstruction: String? enum CodingKeys: String, CodingKey { case transcript case ts case conversationContext = "conversation_context" case keytermsPrompt = "keyterms_prompt" - case llmInstruction = "llm_instruction" } /// Mirrors `DictationConfig.encode(to:)`: an empty array omits its field, so @@ -50,7 +48,6 @@ public enum DictationLog { if !keytermsPrompt.isEmpty { try container.encode(keytermsPrompt, forKey: .keytermsPrompt) } - try container.encodeIfPresent(llmInstruction, forKey: .llmInstruction) } } @@ -118,8 +115,7 @@ public enum DictationLog { let entry = Entry( transcript: transcript, ts: now.formatted(timestampFormat), conversationContext: steering.conversationContext, - keytermsPrompt: steering.keyterms, - llmInstruction: steering.rewriteInstruction) + keytermsPrompt: steering.keyterms) guard var line = try? makeEncoder().encode(entry) else { return } line.append(0x0A) // '\n' diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index c5ed388..b3f278a 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -215,7 +215,6 @@ public actor DictationSession { let field = FocusCapture.captureFieldContext() let context = TranscriptionContext( appName: captured?.processName, - bundleID: captured?.bundleID, windowTitle: field.windowTitle, fieldLabel: field.fieldLabel, priorText: field.priorText, diff --git a/Sources/BlurtEngine/STT/AppKindPriming.swift b/Sources/BlurtEngine/STT/AppKindPriming.swift deleted file mode 100644 index 3620173..0000000 --- a/Sources/BlurtEngine/STT/AppKindPriming.swift +++ /dev/null @@ -1,151 +0,0 @@ -import Foundation - -/// The app-kind formatting instruction: recognizes what *kind* of app the -/// dictation targets (a terminal, a code editor, Slack, Obsidian) from the -/// frontmost app's bundle identifier and renders the one sentence -/// `TranscriptionSteering` sends as `config.llm.instruction` — "Format the -/// result as a shell command with no trailing period." / "… as Swift code." / -/// "… as a casual Slack message, using Slack emoji where they fit." / "… as -/// markdown." — telling the rewrite what shape of text the destination expects, -/// which the app's display name alone doesn't convey. -/// -/// For code editors the window title usually names the open file, so the -/// clause names the language inferred from that filename's extension -/// ("Format the result as Python code.") when one is recognizable, and stays -/// generic ("… as code.") otherwise. -/// -/// These are instructions to the **LLM that rewrites the finished transcript**, -/// not to the STT model. Reshaping output is not something `config.prompt` acts -/// on — that field takes a description of the audio — so a clause phrased as -/// "Transcribe speech into markdown." was a no-op wherever it landed. Keep the -/// imperative "Format the result as …" shape. -/// -/// Detection keys on bundle IDs, not display names: names are localized and -/// user-editable, while the bundle ID is the app's stable identity. An -/// unrecognized app contributes no clause — the request's `llm` block is then -/// empty and the service's own default cleanup instruction applies. Exercised by -/// `Tests/BlurtEngineTests/AppKindPrimingTests.swift`. -enum AppKindPriming { - /// The recognized destination families. Each renders one guidance sentence; - /// anything else contributes no clause. - enum Kind: Sendable, Equatable { - case terminal - case codeEditor - case slack - case obsidian - } - - /// Exact bundle-ID → kind matches for the recognized apps. - private static let kindsByBundleID: [String: Kind] = [ - // Terminals. - "com.apple.Terminal": .terminal, - "com.googlecode.iterm2": .terminal, - "dev.warp.Warp-Stable": .terminal, - "dev.warp.Warp-Preview": .terminal, - "com.mitchellh.ghostty": .terminal, - "net.kovidgoyal.kitty": .terminal, - "org.alacritty": .terminal, - "com.github.wez.wezterm": .terminal, - "co.zeit.hyper": .terminal, - // Code editors. Cursor ships under an opaque ToDesktop build id. - "com.microsoft.VSCode": .codeEditor, - "com.microsoft.VSCodeInsiders": .codeEditor, - "com.vscodium": .codeEditor, - "com.todesktop.230313mzl4w4u92": .codeEditor, - "com.exafunction.windsurf": .codeEditor, - "com.apple.dt.Xcode": .codeEditor, - "dev.zed.Zed": .codeEditor, - "dev.zed.Zed-Preview": .codeEditor, - "com.panic.Nova": .codeEditor, - "com.macromates.TextMate": .codeEditor, - // Chat and notes. - "com.tinyspeck.slackmacgap": .slack, - "md.obsidian": .obsidian, - ] - - /// Prefix matches for app families that ship many bundle IDs under one - /// vendor prefix (every JetBrains IDE, Sublime Text's versioned IDs). - private static let kindsByBundleIDPrefix: [(prefix: String, kind: Kind)] = [ - ("com.jetbrains.", .codeEditor), - ("com.sublimetext.", .codeEditor), - ] - - /// The kind `bundleID` identifies, or `nil` for an unrecognized (or absent) - /// bundle ID. - static func kind(ofBundleID bundleID: String?) -> Kind? { - guard let bundleID = bundleID.trimmedNonEmpty() else { return nil } - if let kind = kindsByBundleID[bundleID] { return kind } - return kindsByBundleIDPrefix.first { bundleID.hasPrefix($0.prefix) }?.kind - } - - /// The formatting instruction for the app `bundleID` identifies, or `nil` when - /// the app isn't recognized. `windowTitle` refines the code-editor clause - /// with the open file's language; the other kinds ignore it. - static func clause(bundleID: String?, windowTitle: String?) -> String? { - guard let kind = kind(ofBundleID: bundleID) else { return nil } - switch kind { - case .terminal: - // "Trailing period", not "terminal punctuation": in a clause about - // terminals the latter reads as the app, not the end of a sentence. - return "Format the result as a shell command with no trailing period." - case .codeEditor: - let subject = windowTitle.flatMap(language(inWindowTitle:)) ?? "code" - return "Format the result as \(subject)." - case .slack: - return "Format the result as a casual Slack message, using Slack emoji where they fit." - case .obsidian: - return "Format the result as markdown." - } - } - - /// What the open file says speech becomes, from the first token in `title` - /// that reads as a filename with a recognized extension - /// ("● main.py — blurt — Visual Studio Code" → "Python code"), or `nil` when - /// no token does. Editors lead their window titles with the open file, so - /// first match wins. - static func language(inWindowTitle title: String) -> String? { - for token in title.split(whereSeparator: \.isWhitespace) { - let name = token.trimmingCharacters(in: Self.filenameTrim) - // A leading-dot name (".zshrc") is a dotfile, not a base name + extension. - guard let dot = name.lastIndex(of: "."), dot != name.startIndex else { continue } - if let language = languagesByExtension[name[name.index(after: dot)...].lowercased()] { - return language - } - } - return nil - } - - /// Decoration editors wrap around the filename in a window title — dirty - /// markers, quotes, brackets, dash separators. - private static let filenameTrim = CharacterSet(charactersIn: "\"'`•●◆*()[]{}<>,;:—–-") - - /// Filename extension → what the clause says the result should be. Values - /// complete "Format the result as …", so languages carry a trailing "code" - /// while markup/data formats stand alone. Lowercased keys; lookups lowercase - /// the extension first. - private static let languagesByExtension: [String: String] = [ - "c": "C code", "h": "C code", - "cc": "C++ code", "cpp": "C++ code", "cxx": "C++ code", "hpp": "C++ code", - "cs": "C# code", - "css": "CSS", "scss": "CSS", - "go": "Go code", - "htm": "HTML", "html": "HTML", - "java": "Java code", - "cjs": "JavaScript code", "js": "JavaScript code", "jsx": "JavaScript code", "mjs": "JavaScript code", - "json": "JSON", - "kt": "Kotlin code", "kts": "Kotlin code", - "lua": "Lua code", - "m": "Objective-C code", "mm": "Objective-C code", - "markdown": "markdown", "md": "markdown", - "php": "PHP code", - "py": "Python code", "pyi": "Python code", - "rb": "Ruby code", - "rs": "Rust code", - "bash": "a shell script", "sh": "a shell script", "zsh": "a shell script", - "sql": "SQL", - "swift": "Swift code", - "toml": "TOML", - "ts": "TypeScript code", "tsx": "TypeScript code", - "yaml": "YAML", "yml": "YAML", - ] -} diff --git a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift index 6071e43..5fb1faf 100644 --- a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift +++ b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift @@ -106,12 +106,11 @@ public struct AssemblyAITranscriber: TranscriberProtocol { } /// Builds the JSON `config` part sent alongside the audio. Each steering field - /// is included only when it carries something — an empty array or a nil - /// instruction omits the field rather than stating an empty value, so the - /// service applies its own default. The `llm` block always rides along — see - /// `DictationConfig.llm`. Internal so tests can assert the steering wiring - /// without inspecting the multipart upload body (which `URLProtocol` mocks - /// can't observe reliably for `upload(from:)`). + /// is included only when it carries something — an empty array omits the field + /// rather than stating an empty value, so the service applies its own default. + /// The `llm` block always rides along — see `DictationConfig.llm`. Internal so + /// tests can assert the steering wiring without inspecting the multipart upload + /// body (which `URLProtocol` mocks can't observe reliably for `upload(from:)`). func makeConfigData(sampleRate: Int, steering: TranscriptionSteering.Fields) throws -> Data { try JSONEncoder().encode( DictationConfig( @@ -119,7 +118,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol { channels: 1, conversationContext: steering.conversationContext, keytermsPrompt: steering.keyterms, - llm: LLMRewrite(instruction: steering.rewriteInstruction) + llm: LLMRewrite() ) ) } @@ -212,10 +211,8 @@ public struct AssemblyAITranscriber: TranscriberProtocol { /// The user's key terms as the explicit vocabulary list, omitted when empty. let keytermsPrompt: [String] /// The rewrite request. Always present so the service runs the rewrite at - /// all; its `instruction` is the app-kind formatting clause, or absent to - /// select the service's default cleanup instruction. Per the API's - /// `instruction`-mode rules, output format and don't-answer-the-text - /// safeguards are enforced server-side either way. + /// all, and always empty so the instruction it runs is the service's own + /// default cleanup — Blurt sends nothing describing the destination app. let llm: LLMRewrite enum CodingKeys: String, CodingKey { case sampleRate = "sample_rate" @@ -244,9 +241,11 @@ public struct AssemblyAITranscriber: TranscriberProtocol { } } - private struct LLMRewrite: Encodable { - let instruction: String? - } + /// Encodes as `{}`: asking for the rewrite without steering it. An + /// `instruction` here once carried a formatting clause derived from the + /// frontmost app's bundle ID; that was removed, and the service's default + /// cleanup instruction is what runs now. + private struct LLMRewrite: Encodable {} private struct DictationResponse: Decodable { /// The verbatim transcript — always present, never altered by the LLM. diff --git a/Sources/BlurtEngine/STT/TranscriptionContext.swift b/Sources/BlurtEngine/STT/TranscriptionContext.swift index 7e637b5..331ec29 100644 --- a/Sources/BlurtEngine/STT/TranscriptionContext.swift +++ b/Sources/BlurtEngine/STT/TranscriptionContext.swift @@ -1,11 +1,10 @@ /// Per-utterance snapshot of where the dictation is going, gathered at /// dictation start from the focused app and field. `TranscriptionSteering.build` /// renders only the parts the request uses — the prior text (as -/// `conversation_context`), the key terms (as `keyterms_prompt`), the bundle ID -/// (via the `AppKindPriming` formatting clause in `llm.instruction`), and the -/// window title (the language refinement of that clause). The prior text and -/// window title also steer the injector's paste separator; nothing else is -/// consumed, and the rest of the context is neither sent nor logged. +/// `conversation_context`) and the key terms (as `keyterms_prompt`). Nothing +/// identifying the destination app is sent. The prior text and window title +/// also steer the injector's paste separator; nothing else is consumed, and the +/// rest of the context is neither sent nor logged. /// /// Every focus field is optional and best-effort: whatever couldn't be read is /// `nil`, and an entirely empty context customizes nothing (the server applies @@ -15,19 +14,9 @@ public struct TranscriptionContext: Sendable, Equatable { /// sent. public let appName: String? - /// The frontmost application's bundle identifier (e.g. - /// "com.tinyspeck.slackmacgap"). Never sent verbatim: it keys the - /// app-*kind* recognition (`AppKindPriming`) that selects the rewrite's - /// formatting instruction — terminal, code editor, Slack, Obsidian. - /// Preferred over `appName` for recognition because display names are - /// localized and user-editable while the bundle ID is stable. - public let bundleID: String? - /// The focused window's title (e.g. "main.py — blurt", a document name, a - /// Slack channel). Never sent verbatim. In a code editor it usually names the - /// open file, which is how the formatting clause learns the language ("… as - /// Swift code."); it also anchors the injector's same-window separator - /// fallback. + /// Slack channel). Never sent. Read only to anchor the injector's same-window + /// separator fallback. public let windowTitle: String? /// A short label for the focused field (placeholder/title/role, e.g. "To", @@ -54,7 +43,6 @@ public struct TranscriptionContext: Sendable, Equatable { public init( appName: String?, - bundleID: String? = nil, windowTitle: String? = nil, fieldLabel: String? = nil, priorText: String?, @@ -62,7 +50,6 @@ public struct TranscriptionContext: Sendable, Equatable { keyTerms: [String] = [] ) { self.appName = appName - self.bundleID = bundleID self.windowTitle = windowTitle self.fieldLabel = fieldLabel self.priorText = priorText @@ -74,7 +61,7 @@ public struct TranscriptionContext: Sendable, Equatable { /// are no key terms. public var isEmpty: Bool { keyTerms.isEmpty - && [appName, bundleID, windowTitle, fieldLabel, priorText, selectedText].allSatisfy { + && [appName, windowTitle, fieldLabel, priorText, selectedText].allSatisfy { $0.trimmedNonEmpty() == nil } } diff --git a/Sources/BlurtEngine/STT/TranscriptionSteering.swift b/Sources/BlurtEngine/STT/TranscriptionSteering.swift index 92fed52..f809f14 100644 --- a/Sources/BlurtEngine/STT/TranscriptionSteering.swift +++ b/Sources/BlurtEngine/STT/TranscriptionSteering.swift @@ -1,4 +1,4 @@ -/// Renders a `TranscriptionContext` into the three request-customization fields +/// Renders a `TranscriptionContext` into the two request-customization fields /// the dictation API accepts, each of which has one job (see /// `AssemblyAITranscriber` for the wire encoding): /// @@ -8,22 +8,24 @@ /// proper-noun consistency. /// - `keyterms_prompt` — the user's key terms, verbatim, as the explicit /// vocabulary list the field is for. -/// - `llm.instruction` — the app-kind formatting clause (`AppKindPriming`), -/// recognized from the frontmost app's bundle ID: markdown in Obsidian, Swift -/// code in a code editor (the language inferred from the window title's -/// filename), a shell command in a terminal, a casual message in Slack. /// /// **`config.prompt` is deliberately never sent.** The field takes a /// *description of the audio* ("Cardiology consultation about chest pain /// symptoms."), not instructions — transcription behavior is optimized out of /// the box, so an imperative like "Transcribe speech into markdown." was aimed -/// at a field that does not act on instructions. That is why the app-kind -/// priming now rides in `llm.instruction`, where an LLM actually rewrites the -/// text, and why the key terms moved to `keyterms_prompt` rather than being -/// packed into the prompt as a `Keywords:` clause. Sending no prompt also keeps -/// the service's managed default — a custom prompt replaces it wholesale, -/// including its language steering, which is the mechanism behind the earlier -/// finding that pinning the prompt to English hurt non-English speech. +/// at a field that does not act on instructions. That is also why the key terms +/// ride in `keyterms_prompt` rather than being packed into the prompt as a +/// `Keywords:` clause. Sending no prompt keeps the service's managed default — +/// a custom prompt replaces it wholesale, including its language steering, +/// which is the mechanism behind the earlier finding that pinning the prompt to +/// English hurt non-English speech. +/// +/// **Nothing describing the destination app is sent either.** An earlier +/// version recognized the frontmost app's bundle ID as a *kind* (terminal, code +/// editor, Slack, Obsidian) and sent a matching formatting clause as +/// `llm.instruction`. That is gone: the `llm` block now carries no instruction, +/// so the service's own default cleanup rewrite applies to every utterance +/// regardless of where the text is going. /// /// Blurt has no earlier turns to send: `conversation_context` carries exactly /// one entry, the prior-cursor text, or none. @@ -31,8 +33,8 @@ /// The rest of the captured context is not sent at all. The app name and field /// label render nowhere (real-world logs showed them crowding the request — VS /// Code parks a screen-reader help announcement in the focused field's -/// description), and the window title is read only to name a code editor's -/// language. Selected text is never priming: the paste replaces it, so +/// description), and the window title is read only to anchor the injector's +/// paste separator. Selected text is never priming: the paste replaces it, so /// conditioning the model on it would prime for text on its way out. /// /// Exercised by `Tests/BlurtEngineTests/TranscriptionSteeringTests.swift`. @@ -47,16 +49,13 @@ enum TranscriptionSteering { /// Explicit vocabulary to bias recognition toward, in the user's own /// spelling and capitalization. let keyterms: [String] - /// The rewrite instruction, or `nil` to let the service's default cleanup - /// instruction stand. - let rewriteInstruction: String? /// Nothing to customize — every field omitted, so the service applies its /// managed default prompt and its default cleanup rewrite. Also the value /// to compare against for "does this utterance customize anything?" — /// `Fields` is `Equatable`, so no separate emptiness predicate exists to /// drift from the fields themselves. - static let empty = Fields(conversationContext: [], keyterms: [], rewriteInstruction: nil) + static let empty = Fields(conversationContext: [], keyterms: []) } /// Cap the dictation API documents for `conversation_context`: 4096 characters @@ -84,9 +83,7 @@ enum TranscriptionSteering { guard let context, !context.isEmpty else { return .empty } return Fields( conversationContext: priorTurn(of: context).map { [$0] } ?? [], - keyterms: fittedKeyterms(context.keyTerms), - rewriteInstruction: AppKindPriming.clause( - bundleID: context.bundleID, windowTitle: context.windowTitle.trimmedNonEmpty())) + keyterms: fittedKeyterms(context.keyTerms)) } /// The prior-cursor text as one conversation turn, clipped to the field's cap diff --git a/Tests/BlurtEngineTests/AppKindPrimingTests.swift b/Tests/BlurtEngineTests/AppKindPrimingTests.swift deleted file mode 100644 index e491758..0000000 --- a/Tests/BlurtEngineTests/AppKindPrimingTests.swift +++ /dev/null @@ -1,129 +0,0 @@ -import Testing - -@testable import BlurtEngine - -@Suite("AppKindPriming") -struct AppKindPrimingTests { - // MARK: - Kind recognition - - /// One bundle-ID → kind expectation, tabled for per-case failure output like - /// `TranscriptionSteeringTests.Case`. - struct KindCase: Sendable, CustomTestStringConvertible { - let bundleID: String? - let expected: AppKindPriming.Kind? - var testDescription: String { bundleID ?? "nil" } - } - - static let kindCases: [KindCase] = [ - KindCase(bundleID: "com.apple.Terminal", expected: .terminal), - KindCase(bundleID: "com.googlecode.iterm2", expected: .terminal), - KindCase(bundleID: "com.mitchellh.ghostty", expected: .terminal), - KindCase(bundleID: "com.microsoft.VSCode", expected: .codeEditor), - KindCase(bundleID: "com.apple.dt.Xcode", expected: .codeEditor), - KindCase(bundleID: "com.todesktop.230313mzl4w4u92", expected: .codeEditor), - // Prefix families: any JetBrains IDE, any Sublime Text major version. - KindCase(bundleID: "com.jetbrains.pycharm", expected: .codeEditor), - KindCase(bundleID: "com.sublimetext.4", expected: .codeEditor), - KindCase(bundleID: "com.tinyspeck.slackmacgap", expected: .slack), - KindCase(bundleID: "md.obsidian", expected: .obsidian), - KindCase(bundleID: "com.apple.mail", expected: nil), - KindCase(bundleID: "", expected: nil), - KindCase(bundleID: " ", expected: nil), - KindCase(bundleID: nil, expected: nil), - ] - - @Test("bundle IDs map to their app kind", arguments: kindCases) - func kind(_ c: KindCase) { - #expect(AppKindPriming.kind(ofBundleID: c.bundleID) == c.expected) - } - - // MARK: - Language inference from window titles - - /// One window-title → language expectation. - struct LanguageCase: Sendable, CustomTestStringConvertible { - let title: String - let expected: String? - var testDescription: String { title } - } - - static let languageCases: [LanguageCase] = [ - LanguageCase(title: "main.py — blurt — Visual Studio Code", expected: "Python code"), - // VS Code prepends ● to the filename of a dirty buffer. - LanguageCase(title: "● api.ts — server", expected: "TypeScript code"), - LanguageCase(title: "blurt — TranscriptionPrompt.swift", expected: "Swift code"), - LanguageCase(title: "deploy.sh — infra", expected: "a shell script"), - // Extension matching is case-insensitive. - LanguageCase(title: "README.MD — notes", expected: "markdown"), - // First filename wins when several tokens carry extensions. - LanguageCase(title: "index.js next.config.ts", expected: "JavaScript code"), - // A dotfile has no base name + extension split. - LanguageCase(title: ".zshrc — dotfiles", expected: nil), - // Version numbers and hostnames are not filenames. - LanguageCase(title: "release v0.1.34 — dictation.assemblyai.com", expected: nil), - LanguageCase(title: "Untitled-1", expected: nil), - LanguageCase(title: "", expected: nil), - ] - - @Test("window titles yield the open file's language", arguments: languageCases) - func language(_ c: LanguageCase) { - #expect(AppKindPriming.language(inWindowTitle: c.title) == c.expected) - } - - // MARK: - Clause rendering - - @Test("a terminal renders the shell-command formatting instruction") - func terminalClause() { - #expect( - AppKindPriming.clause(bundleID: "com.apple.Terminal", windowTitle: nil) - == "Format the result as a shell command with no trailing period.") - } - - @Test("a code editor names the open file's language when the title carries one") - func codeEditorClauseWithLanguage() { - #expect( - AppKindPriming.clause(bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt") - == "Format the result as Python code.") - } - - @Test("a code editor stays generic when the title names no recognizable file") - func codeEditorClauseGeneric() { - #expect( - AppKindPriming.clause(bundleID: "com.apple.dt.Xcode", windowTitle: "Welcome to Xcode") - == "Format the result as code.") - } - - @Test("Slack renders the casual-message formatting instruction") - func slackClause() { - #expect( - AppKindPriming.clause(bundleID: "com.tinyspeck.slackmacgap", windowTitle: "#eng-backend") - == "Format the result as a casual Slack message, using Slack emoji where they fit.") - } - - @Test("Obsidian renders the markdown formatting instruction") - func obsidianClause() { - #expect( - AppKindPriming.clause(bundleID: "md.obsidian", windowTitle: "Meeting notes") - == "Format the result as markdown.") - } - - @Test("every clause reads as a rewrite instruction, not a transcription instruction") - func clausesAreRewriteInstructions() { - // These render into `config.llm.instruction`, which is an instruction to an - // LLM rewriting finished text. The Sync STT docs are explicit that - // `config.prompt` takes a *description of the audio* instead — so a clause - // that slipped back into "Transcribe speech into …" phrasing would be aimed - // at the wrong field, which is how the app-kind priming was silently a no-op - // before. - for bundleID in ["com.apple.Terminal", "com.microsoft.VSCode", "com.tinyspeck.slackmacgap", "md.obsidian"] { - let clause = AppKindPriming.clause(bundleID: bundleID, windowTitle: nil) - #expect(clause?.hasPrefix("Format the result as ") == true, "\(bundleID)") - #expect(clause?.contains("Transcribe") == false, "\(bundleID)") - } - } - - @Test("an unrecognized app contributes no clause") - func unrecognizedApp() { - #expect(AppKindPriming.clause(bundleID: "com.apple.mail", windowTitle: "Re: Q3") == nil) - #expect(AppKindPriming.clause(bundleID: nil, windowTitle: "main.py") == nil) - } -} diff --git a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift index b038127..5f641c2 100644 --- a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift +++ b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift @@ -158,11 +158,9 @@ struct HTTPClientTests { func configIncludesSteeringFields() throws { let object = try configObject( steering: TranscriptionSteering.Fields( - conversationContext: ["$ git status"], keyterms: ["Blurt", "AssemblyAI"], - rewriteInstruction: "Format the result as markdown.")) + conversationContext: ["$ git status"], keyterms: ["Blurt", "AssemblyAI"])) #expect(object["conversation_context"] as? [String] == ["$ git status"]) #expect(object["keyterms_prompt"] as? [String] == ["Blurt", "AssemblyAI"]) - #expect((object["llm"] as? [String: Any])?["instruction"] as? String == "Format the result as markdown.") #expect(object["sample_rate"] as? Int == 16_000) // The capture path is mono by construction; the declared geometry must agree. #expect(object["channels"] as? Int == 1) @@ -172,27 +170,33 @@ struct HTTPClientTests { func configNeverSendsPrompt() throws { // A custom `config.prompt` replaces the service's managed default *and* its // language steering, and the field wants a description of the audio rather - // than instructions — which is why formatting moved to `llm.instruction` and - // vocabulary to `keyterms_prompt`. Sending no prompt keeps the managed - // default, so this pins the field's absence for every steering shape. + // than instructions — which is why vocabulary rides in `keyterms_prompt`. + // Sending no prompt keeps the managed default, so this pins the field's + // absence for every steering shape. let shapes: [TranscriptionSteering.Fields] = [ .empty, - TranscriptionSteering.Fields( - conversationContext: ["Hi Sam,"], keyterms: ["Blurt"], - rewriteInstruction: "Format the result as markdown."), + TranscriptionSteering.Fields(conversationContext: ["Hi Sam,"], keyterms: ["Blurt"]), ] for steering in shapes { #expect(try configObject(steering: steering).keys.contains("prompt") == false) } } - @Test("config part requests the default cleanup rewrite when no formatting is needed") + @Test("config part always requests the unsteered default cleanup rewrite") func configRequestsDefaultRewrite() throws { - // `llm` must be present and empty when there is no app-kind clause: present - // so the service runs the rewrite at all, empty so the server-owned default - // cleanup instruction (and its guardrails) applies rather than a client-side - // copy. `isEmpty == true` also covers presence — false for a missing `llm`. - #expect((try configObject(steering: .empty)["llm"] as? [String: Any])?.isEmpty == true) + // `llm` must be present and empty for every utterance: present so the service + // runs the rewrite at all, empty so the server-owned default cleanup + // instruction (and its guardrails) applies rather than a client-side copy. + // Empty also means nothing about the destination app rides here, which is + // what an earlier `instruction` carried. `isEmpty == true` covers presence + // too — it is false for a missing `llm`. + let shapes: [TranscriptionSteering.Fields] = [ + .empty, + TranscriptionSteering.Fields(conversationContext: ["Hi Sam,"], keyterms: ["Blurt"]), + ] + for steering in shapes { + #expect((try configObject(steering: steering)["llm"] as? [String: Any])?.isEmpty == true) + } } @Test("config part omits the context and keyterms fields when they are empty") diff --git a/Tests/BlurtEngineTests/DictationLogTests.swift b/Tests/BlurtEngineTests/DictationLogTests.swift index 0b2b696..84187ad 100644 --- a/Tests/BlurtEngineTests/DictationLogTests.swift +++ b/Tests/BlurtEngineTests/DictationLogTests.swift @@ -14,12 +14,10 @@ private struct DecodedEntry: Decodable { private struct DecodedSteering: Decodable { let conversationContext: [String] let keytermsPrompt: [String] - let llmInstruction: String? enum CodingKeys: String, CodingKey { case conversationContext = "conversation_context" case keytermsPrompt = "keyterms_prompt" - case llmInstruction = "llm_instruction" } // A missing array decodes to empty rather than nil: whether a field was @@ -29,7 +27,6 @@ private struct DecodedSteering: Decodable { let container = try decoder.container(keyedBy: CodingKeys.self) conversationContext = try container.decodeIfPresent([String].self, forKey: .conversationContext) ?? [] keytermsPrompt = try container.decodeIfPresent([String].self, forKey: .keytermsPrompt) ?? [] - llmInstruction = try container.decodeIfPresent(String.self, forKey: .llmInstruction) } } @@ -110,18 +107,18 @@ struct DictationLogTests { func logsOnlyWhatWasSent() { let url = makeTempLogURL() let context = TranscriptionContext( - appName: "Obsidian", bundleID: "md.obsidian", windowTitle: "Grocery list", + appName: "Obsidian", windowTitle: "Grocery list", fieldLabel: "text entry area", priorText: "- milk", selectedText: "- bread") DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" // Everything the request carried is recorded, under the wire's own names… let decoded = try? JSONDecoder().decode(DecodedSteering.self, from: Data(line.utf8)) - #expect(decoded?.llmInstruction == "Format the result as markdown.") #expect(decoded?.conversationContext == ["- milk"]) // …and none of the captured-but-unsent context is. Values, not just keys: - // the entry must carry no trace of what stayed on the machine. Selected text - // is on this list because the paste replaces it, so it is never sent. - for unsent in ["Obsidian", "md.obsidian", "Grocery list", "text entry area", "- bread"] { + // the entry must carry no trace of what stayed on the machine. Two are + // load-bearing: selected text, which the paste replaces so it is never sent, + // and anything naming the destination app, which is never sent either. + for unsent in ["Obsidian", "Grocery list", "text entry area", "- bread"] { #expect(!line.contains(unsent)) } } @@ -130,7 +127,7 @@ struct DictationLogTests { func logsAssembledSteering() { let url = makeTempLogURL() let context = TranscriptionContext( - appName: "Obsidian", bundleID: "md.obsidian", windowTitle: "Grocery list", + appName: "Obsidian", windowTitle: "Grocery list", fieldLabel: "text entry area", priorText: "- milk", keyTerms: ["Blurt"]) DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" @@ -138,10 +135,9 @@ struct DictationLogTests { let sent = TranscriptionSteering.build(context: context) // The log is the corpus prompt iteration reads, so it has to agree with the // builder field-for-field rather than approximately. - #expect(decoded?.llmInstruction == sent.rewriteInstruction) #expect(decoded?.conversationContext == sent.conversationContext) #expect(decoded?.keytermsPrompt == sent.keyterms) - #expect(decoded?.llmInstruction == "Format the result as markdown.") + #expect(decoded?.conversationContext == ["- milk"]) #expect(decoded?.keytermsPrompt == ["Blurt"]) } @@ -151,7 +147,9 @@ struct DictationLogTests { DictationLog.write(transcript: "p", context: nil, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" // Absent, not `null` and not `[]` — the entry should read as "nothing was - // customized", matching the request, which omits these fields too. + // customized", matching the request, which omits these fields too. `prompt` + // and `llm_instruction` are never written for any context, so their absence + // here doubles as a pin against reintroducing either. #expect(!line.contains("\"prompt\"")) #expect(!line.contains("\"llm_instruction\"")) #expect(!line.contains("\"conversation_context\"")) @@ -210,8 +208,7 @@ struct DictationLogGateTests { // covered in `FocusCaptureTests`; this suite can only see contexts that // already cleared it. let context = TranscriptionContext( - appName: "Terminal", bundleID: "com.apple.Terminal", - windowTitle: "Vault", fieldLabel: "Command", + appName: "Terminal", windowTitle: "Vault", fieldLabel: "Command", priorText: "$ git", selectedText: nil) let offURL = makeTempLogURL() DictationLog.append( @@ -226,8 +223,8 @@ struct DictationLogGateTests { DictationLog.queue.sync {} #expect(!FileManager.default.fileExists(atPath: offURL.path)) let logged = readLog(onURL) - #expect(logged.contains("Format the result as a shell command with no trailing period.")) #expect(logged.contains("$ git")) + #expect(!logged.contains("Terminal")) #expect(!logged.contains("Vault")) #expect(!logged.contains("Command")) } diff --git a/Tests/BlurtEngineTests/TranscriptionContextTests.swift b/Tests/BlurtEngineTests/TranscriptionContextTests.swift index 0db647f..3550178 100644 --- a/Tests/BlurtEngineTests/TranscriptionContextTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionContextTests.swift @@ -7,7 +7,7 @@ import Testing /// The agreement with `TranscriptionSteering.build` is one-directional: /// `isEmpty == true` must always correspond to empty steering fields, while a /// non-empty context may still steer nothing — its signals can be carry-only -/// (app name, field label, or selected text from an unrecognized app). +/// (app name, window title, field label, or selected text). @Suite("TranscriptionContext") struct TranscriptionContextTests { @Test("both fields nil is empty") @@ -30,13 +30,6 @@ struct TranscriptionContextTests { #expect(!TranscriptionContext(appName: nil, priorText: "hello there").isEmpty) } - @Test("a bundle ID alone makes it non-empty (and produces a rewrite instruction)") - func bundleIDPresent() { - let context = TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil) - #expect(!context.isEmpty) - #expect(TranscriptionSteering.build(context: context).rewriteInstruction != nil) - } - @Test("real selected text makes it non-empty") func selectedTextPresent() { #expect(!TranscriptionContext(appName: nil, priorText: nil, selectedText: "highlighted").isEmpty) @@ -65,10 +58,8 @@ struct TranscriptionContextTests { #expect(TranscriptionSteering.build(context: context) == .empty) } - // Renderable signals (a recognized bundle ID, key terms, prior text) each - // steer at least one field… + // The two renderable signals (key terms, prior text) each steer a field… let renderable = [ - TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil), TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["Blurt"]), TranscriptionContext(appName: nil, priorText: "Hi Sam,"), ] @@ -78,11 +69,18 @@ struct TranscriptionContextTests { } // …while carry-only signals make the context non-empty (worth carrying for - // the injector) yet steer nothing. Selected text is the interesting one: the - // paste replaces it, so it is never priming. - let carryOnly = TranscriptionContext(appName: "Mail", priorText: nil, selectedText: "sel") - #expect(!carryOnly.isEmpty) - #expect(TranscriptionSteering.build(context: carryOnly) == .empty) + // the injector) yet steer nothing. Two are load-bearing: selected text, + // because the paste replaces it so it is never priming; and everything + // naming the destination app, which is deliberately not sent at all. + let carryOnly = [ + TranscriptionContext(appName: "Mail", priorText: nil, selectedText: "sel"), + TranscriptionContext(appName: "Code", windowTitle: "main.py — blurt", priorText: nil), + TranscriptionContext(appName: "Slack", fieldLabel: "Message", priorText: nil), + ] + for context in carryOnly { + #expect(!context.isEmpty) + #expect(TranscriptionSteering.build(context: context) == .empty) + } } @Test("Equatable compares every field") @@ -108,8 +106,5 @@ struct TranscriptionContextTests { #expect( TranscriptionContext(appName: "Notes", priorText: "x", keyTerms: ["a"]) != TranscriptionContext(appName: "Notes", priorText: "x", keyTerms: ["b"])) - #expect( - TranscriptionContext(appName: "Notes", bundleID: "com.a.b", priorText: "x") - != TranscriptionContext(appName: "Notes", bundleID: "com.c.d", priorText: "x")) } } diff --git a/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift b/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift index 5e57261..850b4ce 100644 --- a/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift @@ -10,10 +10,11 @@ struct TranscriptionSteeringTests { /// /// Each recognized signal has exactly one home, and the table pins both /// directions — what renders and what must never leak into the wrong field: - /// prior-cursor text → `conversation_context`, key terms → `keyterms_prompt`, - /// the app-kind formatting clause → `llm.instruction`. The remaining focus - /// signals (app name, field label, selected text) render nowhere; the window - /// title is read only to name a code editor's language. + /// prior-cursor text → `conversation_context`, key terms → `keyterms_prompt`. + /// Every other focus signal renders nowhere. That includes anything naming the + /// destination app: the table keeps a case per app family that once earned a + /// formatting clause, each pinned to `.empty`, so reintroducing app-kind + /// steering fails here. struct Case: Sendable, CustomTestStringConvertible { let name: String let context: TranscriptionContext? @@ -35,67 +36,45 @@ struct TranscriptionSteeringTests { appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", priorText: nil, selectedText: "the old plan"), expected: .empty), - Case( - name: "unrecognized bundle ID renders no instruction", - context: TranscriptionContext(appName: "Mail", bundleID: "com.apple.mail", priorText: nil), - expected: .empty), Case( name: "prior-cursor text becomes the single conversation-context turn", context: TranscriptionContext(appName: "Mail", priorText: "Hi Sam, thanks for"), expected: TranscriptionSteering.Fields( - conversationContext: ["Hi Sam, thanks for"], keyterms: [], rewriteInstruction: nil)), + conversationContext: ["Hi Sam, thanks for"], keyterms: [])), Case( name: "prior text is trimmed of surrounding whitespace", context: TranscriptionContext(appName: nil, priorText: " Hi Sam,\n "), - expected: TranscriptionSteering.Fields( - conversationContext: ["Hi Sam,"], keyterms: [], rewriteInstruction: nil)), + expected: TranscriptionSteering.Fields(conversationContext: ["Hi Sam,"], keyterms: [])), Case( name: "key terms become keyterms_prompt, not a prompt clause", context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["AssemblyAI", "Kubernetes"]), expected: TranscriptionSteering.Fields( - conversationContext: [], keyterms: ["AssemblyAI", "Kubernetes"], rewriteInstruction: nil)), - Case( - name: "terminal → shell-command formatting instruction", - context: TranscriptionContext(appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil), - expected: TranscriptionSteering.Fields( - conversationContext: [], keyterms: [], - rewriteInstruction: "Format the result as a shell command with no trailing period.")), + conversationContext: [], keyterms: ["AssemblyAI", "Kubernetes"])), Case( - name: "code editor names the window title's language", - context: TranscriptionContext( - appName: "Code", bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt", priorText: nil), - expected: TranscriptionSteering.Fields( - conversationContext: [], keyterms: [], - rewriteInstruction: "Format the result as Python code.")), + name: "a terminal steers nothing", + context: TranscriptionContext(appName: "Terminal", windowTitle: "zsh — 80×24", priorText: nil), + expected: .empty), Case( - name: "code editor with no recognizable filename stays generic", - context: TranscriptionContext( - appName: "Xcode", bundleID: "com.apple.dt.Xcode", windowTitle: "Welcome to Xcode", priorText: nil), - expected: TranscriptionSteering.Fields( - conversationContext: [], keyterms: [], rewriteInstruction: "Format the result as code.")), + name: "a code editor steers nothing, not even the open file's language", + context: TranscriptionContext(appName: "Code", windowTitle: "main.py — blurt", priorText: nil), + expected: .empty), Case( - name: "Slack → casual-message formatting instruction", - context: TranscriptionContext( - appName: "Slack", bundleID: "com.tinyspeck.slackmacgap", fieldLabel: "Message", priorText: nil), - expected: TranscriptionSteering.Fields( - conversationContext: [], keyterms: [], - rewriteInstruction: "Format the result as a casual Slack message, using Slack emoji where they fit.")), + name: "Slack steers nothing", + context: TranscriptionContext(appName: "Slack", fieldLabel: "Message", priorText: nil), + expected: .empty), Case( - name: "Obsidian → markdown formatting instruction", + name: "Obsidian steers nothing", context: TranscriptionContext( - appName: "Obsidian", bundleID: "md.obsidian", - windowTitle: "Grocery list - Cowork - Obsidian 1.12.7", fieldLabel: "text entry area", - priorText: nil), - expected: TranscriptionSteering.Fields( - conversationContext: [], keyterms: [], rewriteInstruction: "Format the result as markdown.")), + appName: "Obsidian", windowTitle: "Grocery list - Cowork - Obsidian 1.12.7", + fieldLabel: "text entry area", priorText: nil), + expected: .empty), Case( - name: "all three fields populate independently", + name: "both fields populate independently, and the app still renders nowhere", context: TranscriptionContext( - appName: "Terminal", bundleID: "com.apple.Terminal", windowTitle: "zsh — 80×24", + appName: "Terminal", windowTitle: "zsh — 80×24", priorText: "$ git status", selectedText: "modified: README.md", keyTerms: ["Blurt"]), expected: TranscriptionSteering.Fields( - conversationContext: ["$ git status"], keyterms: ["Blurt"], - rewriteInstruction: "Format the result as a shell command with no trailing period.")), + conversationContext: ["$ git status"], keyterms: ["Blurt"])), ] @Test("build maps focus context to the dictation steering fields", arguments: cases)