diff --git a/.claude/skills/project-guardrails/SKILL.md b/.claude/skills/project-guardrails/SKILL.md index 8dca901..42671d3 100644 --- a/.claude/skills/project-guardrails/SKILL.md +++ b/.claude/skills/project-guardrails/SKILL.md @@ -23,15 +23,22 @@ one, stop and ask the user first. This is the fast "don't" list; AGENTS.md's - **No streaming STT.** The AssemblyAI Sync API returns the full transcript in one response. Overlay goes "Transcribing…" → full text. -- **No separate LLM cleanup pass.** Cleanup rides in the Sync STT request's - `config.prompt` (`TranscriptionPrompt`). No LLM Gateway client, no +- **No separate LLM cleanup pass.** Cleanup rides in the dictation request's + server-side `llm` block (`TranscriptionSteering`). No LLM Gateway client, no `StylerProtocol`, no post-transcription styling stage. - **No local models / model downloads.** Transcription is a remote AssemblyAI call. No on-device ASR/LLM, no model cache, no download UI. -- Don't reintroduce a "remove filler words (um, uh, like)" directive in the - prompt — `universal-3-5-pro` ignores it; it was deliberately dropped. Same for - a language directive: pinning the prompt to English hurt non-English speech, so - language is left to the model's own detection. +- **Never send `config.prompt`.** It takes a _description of the audio_, not + instructions, and a custom value replaces the service's managed default + including its language steering. Formatting goes in `llm.instruction` + ("Format the result as markdown."), vocabulary in `keyterms_prompt`, the text + before the cursor in `conversation_context`. An imperative like "Transcribe + speech into markdown." in `prompt` is a measured no-op — that is exactly why + these three fields exist. +- Don't reintroduce a "remove filler words (um, uh, like)" directive — the STT + prompt doesn't act on it; it was deliberately dropped, and disfluency removal + is the LLM rewrite's job. Same for a language directive: pinning to English + hurt non-English speech, so language is left to the model's own detection. - **Injection is always a clipboard paste** (save → write → ⌘V → settle → restore), degrading to "left it on the clipboard" when the target is lost. No keystroke-by-keystroke typing path, no length threshold. diff --git a/AGENTS.md b/AGENTS.md index dc64a03..83fd15c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,11 +7,11 @@ the Claude-Code-specific tooling under `.claude/` (hooks, skills, subagents). Blurt is a macOS dictation app powered by [AssemblyAI](https://www.assemblyai.com). Tap or hold a trigger key, speak, and polished text is pasted into the focused app. Transcription is **one remote -AssemblyAI dictation API call**: a per-utterance `prompt` (a transcription directive plus contextual -priming built from the focused app/window/field and the user's key terms) rides along with the -request, and the same request asks the service for its server-side LLM cleanup rewrite -(`config.llm`), so the text that comes back is already polished. The user supplies their own API -key. +AssemblyAI dictation API call**: per-utterance steering rides along with the request — the text before +the cursor as `conversation_context`, the user's key terms as `keyterms_prompt`, and an app-kind +formatting clause recognized from the frontmost app — and the same request asks the service for its +server-side LLM cleanup rewrite (`config.llm`), so the text that comes back is already polished. The +user supplies their own API key. Four reflexes before you touch anything: @@ -53,7 +53,7 @@ Sources/BlurtEngine/ the engine (dependency-free Swift package) Injection/ KeyInjector (clipboard paste), SystemClipboard Permissions/ PermissionsChecker (mic + Accessibility) Pipeline/ DictationSession (actor) + phases, UI projections, geometry, log - STT/ AssemblyAITranscriber, TranscriptionPrompt/Context, SyncSTTLimits + STT/ AssemblyAITranscriber, TranscriptionSteering/Context, AppKindPriming, SyncSTTLimits Update/ UpdateChecker (download-only) + the launch-check policy App/Blurt/ project.yml XcodeGen source of truth — Blurt.xcodeproj is GENERATED @@ -159,23 +159,26 @@ In Claude Code on the web, a `SessionStart` hook installs the portable linters a Each was tried the other way and reverted. If a task seems to require one, stop and ask first. (`.claude/skills/project-guardrails` is the compressed version of this list.) -| Don't | Because | -| --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | -| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. | -| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. | -| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `TranscriptionPrompt`. | -| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | -| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection. | -| Add a "remove filler words (um, uh, like)" clause | Not in the STT model's trained instruction set — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | -| Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | -| Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | -| Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | -| Add a self-replacing install or background auto-updater | Updates are download-only; `mxcl/AppUpdater` and its in-place updater were removed. The once-a-day launch _check_ (`AutomaticUpdateCheck`) is fine; installing for the user, or polling, is not. Extend `UpdateCheckModel`. | -| Hand-edit `Blurt.xcodeproj/project.pbxproj` | Generated from `project.yml`; `check.sh`'s drift check fails on any manual edit (a Claude PreToolUse hook also blocks it). | -| Redirect the post-build install away from `/Applications` | TCC won't register apps in DerivedData/`/tmp`, so permission toggles never appear. | -| Touch the real Keychain in tests | `APIKeyStore` is the production item — a test that writes it triggers Keychain prompts and corrupts the real item's ACL. Use an isolated service (see `KeychainStoreTests`) or `InMemoryAPIKeyStore`. | -| Add backwards-compat shims for removed types | Deleted types stay deleted — no deprecated re-exports. | +| Don't | Because | +| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | +| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. | +| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. | +| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — request steering belongs in `TranscriptionSteering`. | +| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | +| Send `config.prompt` at all | The field takes a _description of the audio_, not instructions, and a custom value replaces the service's managed default **including its language steering**. Formatting goes in `llm.instruction`, vocabulary in `keyterms_prompt`, preceding text in `conversation_context`. | +| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection — and a custom prompt is what drops the managed default's language steering in the first place. | +| Put formatting instructions in `config.prompt` | Reshaping output is not something the STT prompt acts on, so _"Transcribe speech into markdown."_ was a measured no-op. `llm.instruction` (_"Format the result as markdown."_) is the lever that works. | +| Pack key terms into the prompt as `Keywords: a, b, c.` | The API documents `keyterms_prompt` for exactly this, and warns against packing keyword lists into the prompt. | +| Add a "remove filler words (um, uh, like)" clause | Not something the STT prompt acts on — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | +| Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | +| Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | +| Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | +| Add a self-replacing install or background auto-updater | Updates are download-only; `mxcl/AppUpdater` and its in-place updater were removed. The once-a-day launch _check_ (`AutomaticUpdateCheck`) is fine; installing for the user, or polling, is not. Extend `UpdateCheckModel`. | +| Hand-edit `Blurt.xcodeproj/project.pbxproj` | Generated from `project.yml`; `check.sh`'s drift check fails on any manual edit (a Claude PreToolUse hook also blocks it). | +| Redirect the post-build install away from `/Applications` | TCC won't register apps in DerivedData/`/tmp`, so permission toggles never appear. | +| Touch the real Keychain in tests | `APIKeyStore` is the production item — a test that writes it triggers Keychain prompts and corrupts the real item's ACL. Use an isolated service (see `KeychainStoreTests`) or `InMemoryAPIKeyStore`. | +| Add backwards-compat shims for removed types | Deleted types stay deleted — no deprecated re-exports. | Release-side invariants (hardened runtime and a secure timestamp on every nested mach-o and embedded framework, or notarization rejects the build; roll-forward-only for a bad release) live in @@ -242,11 +245,13 @@ the seam they inject. Implements `TranscriberProtocol` against AssemblyAI's **dictation** API: a single `POST https://dictation.assemblyai.com/transcribe` with the captured audio as a raw S16LE PCM blob -in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, `prompt`, and an -empty `llm` block). No model header — the service pins the STT model server-side. The `prompt` -(built per utterance by `TranscriptionPrompt`) steers _transcription_; the `llm` block asks the -service to run its default LLM cleanup rewrite (remove disfluencies, fix punctuation) over the -verbatim transcript, all inside the same request. The response carries both `text` (verbatim) and +in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, and the steering +fields built per utterance by `TranscriptionSteering`: `conversation_context`, `keyterms_prompt`, and +the `llm` block). No model header — the service pins the STT model server-side. **`prompt` is never +sent** — see [Transcription steering](#transcription-steering). The `llm` block asks the service to +run a cleanup rewrite (remove disfluencies, fix punctuation) over the verbatim transcript, all inside +the same request; its optional `instruction` is the app-kind formatting clause, and an empty block +selects the service's own default cleanup. The response carries both `text` (verbatim) and `llm_response` (the rewrite); the transcriber returns the rewrite and falls back to `text` when `llm_response` is null — the rewrite is best-effort (5 s server-side budget), so a rewrite failure (`llm_error`) is a logged degradation, never a user-facing error. @@ -373,30 +378,46 @@ the `@AppStorage(TriggerKeyStore.defaultsKey)` + `TriggerKey.fromPersisted` pair restating that pairing per view. The unset default belongs to `fromPersisted` (an absent keycode maps to right ⌘), so views must not re-declare `TriggerKey.rightCommand.rawValue` themselves. -## Transcription prompt - -`Sources/BlurtEngine/STT/TranscriptionPrompt.swift` builds the instruction passed as the dictation -request's `config.prompt` — it steers the _transcription_, not the LLM rewrite (that's the request's -separate `llm` block). It's unit-tested in `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. - -Every built prompt opens with the fixed `baseInstruction` — _"Transcribe without speaker labels, -audio event descriptions, or emotion markers."_ — a negative-exclusion clause that suppresses the -annotation markers (`[Speaker]`, `[door creaks]`, `[laughing]`) the model would otherwise paste into -the user's text. - -`build(context:)` wraps that pivot in _contextual priming_: prior-cursor text; the selected text (the -highlighted run the dictation will replace, so the model is primed on what's being rewritten — read -via `kAXSelectedTextAttribute`, skipped in secure fields, detected by AX role **or** subrole and -failing closed when the role can't be read, so a password can't reach the prompt); a topic hint from -the window title; a destination sentence from the app/field; and inline keyword boosting from the -user's key terms. It's phrased per AssemblyAI's Universal-3 Pro prompting guidance -(positive/authoritative wording, no "Don't"/"Avoid"/"Never") and stays under the dictation API's -documented 4096-character cap on `config.prompt` (`characterCap`). - -`build(context:)` returns `nil` when there's no usable context, and passing `prompt: nil` to the -transcriber omits the field so the server applies its own default. Two omissions are deliberate and -regression-tested — no language directive and no filler-word clause; see -[Settled decisions](#settled-decisions--dont-reintroduce-these). +## Transcription steering + +`Sources/BlurtEngine/STT/TranscriptionSteering.swift` renders the captured context into the three +request-customization fields, each with one job. It's unit-tested in +`Tests/BlurtEngineTests/TranscriptionSteeringTests.swift`. + +| Field | Carries | Cap | +| ---------------------- | ----------------------------------------------------------------- | ----------------------- | +| `conversation_context` | prior-cursor text, as a single turn | 4096 chars, clip head | +| `keyterms_prompt` | the user's key terms, verbatim | 2048 chars, whole terms | +| `llm.instruction` | the `AppKindPriming` formatting clause, or absent for the default | — | + +**`config.prompt` is never sent, and that is the whole point of this design.** The field takes a +_description of the audio_ ("Cardiology consultation about chest pain symptoms."), not instructions — +transcription behavior is optimized out of the box — so the app-kind priming's old imperative form +(_"Transcribe speech into markdown."_) was aimed at a field that doesn't act on instructions and was a +no-op. Formatting now rides in `llm.instruction`, where an LLM actually rewrites the text +(_"Format the result as markdown."_ — for a code editor, the language is inferred from the window +title's filename: _"Format the result as Swift code."_), and vocabulary rides in `keyterms_prompt` +rather than being packed into the prompt as a `Keywords: a, b, c.` clause. Sending no prompt also +keeps the service's managed default, which a custom prompt replaces wholesale — **including its +language steering**, which is the mechanism behind the older finding that pinning the prompt to +English hurt non-English speech. + +The remaining focus context is **not** sent: app and field names render nowhere (real-world logs +showed them crowding the request — VS Code, for one, parks a screen-reader help announcement in the +focused field's description), the window title is read only to name a code editor's language, and +selected text is never priming because the paste replaces it. All of it is still captured — the +injector's separator logic consumes the prior text and window title — and reading stays +privacy-guarded: prior and selected text are skipped in secure fields, detected by AX role **or** +subrole and failing closed when the role can't be read, so a password is never read out of the field +at all. That guard, not the steering builder, is what keeps a password out of `conversation_context` +and off the dictation log. + +`build(context:)` returns `.empty` when there's no usable context (or when nothing renders — an +unrecognized app with no prior text and no key terms), and every empty field is **omitted** from the +JSON rather than sent as `[]` or `null`, so the service applies its own defaults. Three omissions are +deliberate — no annotation-suppression clause (_"Transcribe without speaker labels, …"_ is already +part of the dictation service's own default prompt), no language directive, and no filler-word clause; +see [Settled decisions](#settled-decisions--dont-reintroduce-these). ## Settings, persistence, and cues @@ -426,7 +447,9 @@ pure edge detector deciding when the chimes fire; the AppKit `CueSoundPlayer` ju resolves. History: **`RecentDictations`** is an in-memory, newest-first ring shown in the ready window (never -written to disk). **`DictationLog`** appends each completed dictation with its context snapshot to +written to disk). **`DictationLog`** appends each completed dictation — the transcript plus the exact +steering fields sent (`conversation_context`, `keyterms_prompt`, `llm_instruction`, under the wire's +own names) — to `~/Library/Logs/Blurt/dictations.jsonl` (`DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI — derived next to the URL so the label can't drift from the write target) — but **only** while developer mode is on; with it off, nothing is written. The Settings diff --git a/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift b/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift index bd6027d..871f018 100644 --- a/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift +++ b/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift @@ -3,8 +3,8 @@ import SwiftUI /// The "Key Terms" section of the Settings window: a free-text /// area where the user lists comma-separated domain words (names, jargon, product -/// names). These are folded into every transcription's prompt as spelling priming -/// (see `KeyTermsStore` / `TranscriptionPrompt.build`), so the model favors those +/// names). These ride on every request as its `keyterms_prompt` vocabulary list +/// (see `KeyTermsStore` / `TranscriptionSteering.build`), so the model favors those /// spellings. Optional — it never gates setup; an empty list just sends no terms. struct KeyTermsStepView: View { /// Stored in UserDefaults so multiple settings windows/readers see edits live. diff --git a/BLURTENGINE.md b/BLURTENGINE.md index d0c2f9c..dbb2ea5 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -59,7 +59,7 @@ press() ──▶ MicCapture.start() release() ──▶ MicCapture.s Key properties of the design, which your integration can rely on: - **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream. -- **Cleanup happens server-side.** The request's empty `llm` block asks the service for its default cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; the per-utterance `config.prompt` (built by `TranscriptionPrompt` from the captured context) primes the _transcription_. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. +- **Cleanup happens server-side.** The request's `llm` block asks the service for a cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; its optional `instruction` carries the app-kind formatting clause, and an empty block selects the service's own default cleanup. Recognition is primed separately by `conversation_context` and `keyterms_prompt` (built by `TranscriptionSteering` from the captured context); `config.prompt` is never sent. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. - **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read. - **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400. @@ -152,15 +152,15 @@ The session calls `setTargetApp` at press time with the app that was frontmost w Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected: -- **`TranscriptionContext`** carries the frontmost app name, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. -- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block), opening with the fixed `baseInstruction` ("Transcribe without speaker labels, audio event descriptions, or emotion markers.") and staying under the API's 4096-character cap. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. +- **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the bundle ID, window title, and key terms reach the prompt; the prior text and window title also steer the injector's paste separator, and nothing else is consumed. +- **`TranscriptionSteering.build(context:)`** renders that into the request's three customization fields: the prior-cursor text as the single `conversation_context` turn (clipped to 4096 chars keeping the tail, since the words nearest the cursor carry the continuity), the user's key terms as `keyterms_prompt` (whole terms fitted to 2048 chars total), and — when the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian) — the formatting clause as `llm.instruction`, e.g. "Format the result as a shell command with no trailing period." or, for a code editor, the language inferred from the window title's filename ("Format the result as Swift code."). **`config.prompt` is never sent**: it takes a description of the audio rather than instructions, and a custom value replaces the service's managed default including its language steering, which is why the app-kind clause moved to the `llm` block and the key terms to their own field. Nothing else is rendered — no app/field names, no selected text (the paste replaces it), and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default). Empty fields are omitted rather than sent as `[]`. Two further deliberate omissions: no language directive and no "remove filler words" clause (not something the STT prompt acts on — a no-op). Don't reintroduce either. - **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere. For key storage, compose against **`APIKeyGateway`** — the injectable `current` / `save(_:)` / `hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. Two projections keep the surrounding UI out of your views: `Outcome.failureReport` classifies a failure as `.inline(message:)` (recoverable — show it beside the field) or `.alert(title:message:)` (a Keychain fault retyping can't fix), and **`APIKeyDisplay.resolve(key:)`** renders the stored key for an account row — masked tail, status and VoiceOver wording, and the connect-vs-rotate control titles. The mask reveals only the last `revealedTailLength` characters and, below `minimumLengthToMask`, none at all, so a short key can't be shown whole. Setup gating has a projection too: **`SetupReadiness.isReady(permissions:hasAPIKey:)`** is the "fully configured" rule (deliberately excluding the trigger key, which has a default), `SetupReadiness.pollInterval(isReady:)` is the permission-poll cadence (brisk during setup, coasting once ready), and `PermissionStatus.lostGrant(since:)` detects a permission revoked out from under a configured app. -Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) with its context snapshot — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section. +Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact steering fields sent (`conversation_context`, `keyterms_prompt`, `llm_instruction`), never the context that wasn't sent — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section. ## Hotkey building blocks @@ -195,7 +195,7 @@ Run `swift test` for the engine suites (`--filter DictationSessionTests` for one Each of these was tried the other way and reverted; the longer stories are in [AGENTS.md](AGENTS.md) and the source comments: - **No external SPM dependencies in the engine.** Foundation/Security/AVFoundation only. -- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and transcription steering belongs in `TranscriptionPrompt`. +- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and request steering belongs in `TranscriptionSteering`. - **No `AVAudioEngine`/`installTap` capture path.** Fresh `AVAudioRecorder` per session, resolved at record time. - **Paste is always clipboard-based** (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation for lost targets. - **No English-pinning or filler-word clauses in the prompt.** diff --git a/README.md b/README.md index 9075c6c..9dd68c0 100644 --- a/README.md +++ b/README.md @@ -166,14 +166,14 @@ Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dep Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM, live level meter; DX7/Juno-106 sound packs STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe - (STT + LLM rewrite) + TranscriptionPrompt contextual priming + (STT + LLM rewrite) + TranscriptionSteering context/keyterms/format Pipeline/ DictationSession actor: press/release/cancel commands, phase stream, auto-release before the API's recording cap Hotkey/ DictationKeyGate/Router: pure, unit-tested state machine for the lone-modifier trigger (tap vs hold vs combo) Injection/ KeyInjector: save clipboard → paste via synthesized ⌘V → restore - FocusCapture/ Accessibility reads of the focused app/window/field that prime - the transcription prompt + FocusCapture/ Accessibility reads of the focused app/window/field feeding the + request's steering fields, the log, and paste separators Config/, Update/ Keychain API-key store, key terms, download-only release check App/Blurt/ AppKit/SwiftUI shell (Xcode project generated by XcodeGen) diff --git a/Sources/BlurtEngine/Config/KeyTermsStore.swift b/Sources/BlurtEngine/Config/KeyTermsStore.swift index de9e223..c96488f 100644 --- a/Sources/BlurtEngine/Config/KeyTermsStore.swift +++ b/Sources/BlurtEngine/Config/KeyTermsStore.swift @@ -1,9 +1,9 @@ import Foundation /// Storage for the user's dictation "key terms" — a comma-separated list of -/// domain words (names, jargon, product names) that get folded into the dictation -/// request `prompt` as vocabulary priming, so the model is more likely to spell -/// them correctly (see `TranscriptionPrompt.build`). +/// domain words (names, jargon, product names) sent as the dictation request's +/// `keyterms_prompt` vocabulary list, so the model is more likely to spell +/// them correctly (see `TranscriptionSteering.build`). /// /// Unlike the API key these aren't secret, so they live in `UserDefaults` rather /// than the Keychain. The transcription pipeline reads the parsed list via @@ -37,7 +37,7 @@ public enum KeyTermsStore { } /// Pure parse of a comma-separated string into a clean term list. Exposed so - /// `TranscriptionPrompt` and tests can reuse the exact same rules. + /// `TranscriptionSteering` and tests can reuse the exact same rules. public static func parse(_ text: String?) -> [String] { guard let text else { return [] } var seen = Set() diff --git a/Sources/BlurtEngine/FocusCapture/FocusCapture.swift b/Sources/BlurtEngine/FocusCapture/FocusCapture.swift index 86d9da4..21005ac 100644 --- a/Sources/BlurtEngine/FocusCapture/FocusCapture.swift +++ b/Sources/BlurtEngine/FocusCapture/FocusCapture.swift @@ -4,6 +4,9 @@ import ApplicationServices struct CapturedFocus: Sendable { let pid: pid_t let processName: String? + /// The frontmost app's stable identity, feeding the prompt's app-kind + /// recognition (`AppKindPriming`) via `TranscriptionContext.bundleID`. + let bundleID: String? } enum FocusCapture { @@ -12,7 +15,8 @@ enum FocusCapture { guard let app = NSWorkspace.shared.frontmostApplication else { return nil } return CapturedFocus( pid: app.processIdentifier, - processName: app.localizedName + processName: app.localizedName, + bundleID: app.bundleIdentifier ) } @@ -20,17 +24,18 @@ enum FocusCapture { NSRunningApplication(processIdentifier: captured.pid) } - /// Accessibility-derived priming read from the system-wide focused UI element - /// at dictation start (see `TranscriptionContext`). Every field is - /// best-effort: any signal that can't be read is `nil`, and a fully-empty - /// result simply means less context, never an error. + /// Accessibility-derived focus context read from the system-wide focused UI + /// element at dictation start (see `TranscriptionContext` for what each + /// signal feeds). Every field is best-effort: any signal that can't be read + /// is `nil`, and a fully-empty result simply means less context, never an + /// error. struct FocusedFieldContext: Sendable { - /// Text immediately preceding the insertion point ("prior chunk context"). + /// Text immediately preceding the insertion point. let priorText: String? /// The text currently selected in the focused field — the dictation will - /// replace it, so it primes the model on what the utterance is about. + /// replace it. let selectedText: String? - /// The focused window's title — a dense topic hint. + /// The focused window's title (in a code editor it names the open file). let windowTitle: String? /// A short label for the focused field ("To", "Search", "Message"). let fieldLabel: String? @@ -49,8 +54,8 @@ enum FocusCapture { /// /// Secure text fields (password inputs) are detected by role **or** subrole and /// never have their contents read, so a typed password — selected or not — can't - /// leak into the STT prompt. The check fails closed: an unreadable role is - /// treated as secure, since it can't be shown not to be. + /// leak into the dictation log or the injector. The check fails closed: an + /// unreadable role is treated as secure, since it can't be shown not to be. /// /// Deliberately `nonisolated`: each read below is a synchronous cross-process /// IPC round trip into the frontmost app, and an unresponsive app blocks the diff --git a/Sources/BlurtEngine/Pipeline/DictationLog.swift b/Sources/BlurtEngine/Pipeline/DictationLog.swift index 15c87f5..1974874 100644 --- a/Sources/BlurtEngine/Pipeline/DictationLog.swift +++ b/Sources/BlurtEngine/Pipeline/DictationLog.swift @@ -1,31 +1,57 @@ import Foundation -/// Append-only JSONL log of completed transcripts at +/// Append-only JSONL log of completed dictations — the transcript that came +/// back and every request field that was sent to get it — at /// `~/Library/Logs/Blurt/dictations.jsonl`. Used to build a real-world /// corpus for prompt iteration. Written only while developer mode is switched /// on (`DeveloperModeStore` — the Settings window's Developer section, which /// also displays this path), so a user who never opts in has no dictation /// text on disk. public enum DictationLog { + /// One logged dictation: what came back (`transcript`), when, and exactly what + /// was sent to steer it. Fields carry the request's own wire names so a log + /// line reads as the request it describes. + /// + /// Only what was *sent* is recorded. The captured-but-unsent focus context — + /// app and field names, the window title, selected text — deliberately stays + /// off disk. Prior-cursor text is on the sent side of that line now that it + /// rides as `conversation_context`; what keeps a password out of it is + /// `FocusCapture`, which skips prior and selected text in secure fields + /// entirely (failing closed when the AX role can't be read), so it never + /// reaches a context in the first place. struct Entry: Encodable { let transcript: String let ts: String - /// Focused-app topic hint sent as context, when one was captured. - let app: String? - /// Focused-window title sent as a topic hint, when one was captured. - let window: String? - /// Focused-field label sent as context, when one was captured. - let field: String? - /// Text-before-cursor "prior chunk context" sent, when any was captured. - /// Lets you verify accessibility-tree prior-text reading actually fired. - let prior: String? - /// Selected text sent as context (the dictation replaced it), when any. - let selected: String? - /// The fully-assembled `config.prompt` sent to AssemblyAI for this - /// utterance. Built here from `context` (rather than threaded through from - /// the transcriber) so the log always reflects what was actually sent, - /// even for calls that construct an entry directly from a context. - let prompt: String? + /// The steering fields sent to AssemblyAI for this utterance. Built here + /// from `context` (rather than threaded through from the transcriber) so the + /// log always reflects what was actually sent, even for calls that construct + /// an entry directly from a context. + let conversationContext: [String] + let keytermsPrompt: [String] + let llmInstruction: String? + + enum CodingKeys: String, CodingKey { + case transcript + case ts + case conversationContext = "conversation_context" + case keytermsPrompt = "keyterms_prompt" + case llmInstruction = "llm_instruction" + } + + /// Mirrors `DictationConfig.encode(to:)`: an empty array omits its field, so + /// a line states only what the request actually carried. + func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + try container.encode(transcript, forKey: .transcript) + try container.encode(ts, forKey: .ts) + if !conversationContext.isEmpty { + try container.encode(conversationContext, forKey: .conversationContext) + } + if !keytermsPrompt.isEmpty { + try container.encode(keytermsPrompt, forKey: .keytermsPrompt) + } + try container.encodeIfPresent(llmInstruction, forKey: .llmInstruction) + } } /// Where the log lives. Public so the Settings window's Developer section @@ -88,11 +114,12 @@ public enum DictationLog { static func write( transcript: String, context: TranscriptionContext? = nil, to url: URL, now: Date ) { + let steering = TranscriptionSteering.build(context: context) let entry = Entry( transcript: transcript, ts: now.formatted(timestampFormat), - app: context?.appName, window: context?.windowTitle, field: context?.fieldLabel, - prior: context?.priorText, selected: context?.selectedText, - prompt: TranscriptionPrompt.build(context: context)) + conversationContext: steering.conversationContext, + keytermsPrompt: steering.keyterms, + llmInstruction: steering.rewriteInstruction) guard var line = try? makeEncoder().encode(entry) else { return } line.append(0x0A) // '\n' diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift index eb830eb..63005f8 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift @@ -39,8 +39,7 @@ extension DictationSession { // cancelled pipeline clear a *newer* press's stream: `cancel()` detaches this // task while it's parked in `firstValue`, a fresh `press()` installs its own // `contextStream`, and this task's resumption then nils that one out — so - // dictation #2 transcribes with `context: nil`, losing not just its priming but - // `baseInstruction`, and `[Speaker]`-style markers can reach the pasted text. + // dictation #2 transcribes with `context: nil`, silently losing its priming. // The window is microseconds, but the invariant is now local instead of // depending on scheduling. let stream = contextStream diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index b3f278a..c5ed388 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -215,6 +215,7 @@ public actor DictationSession { let field = FocusCapture.captureFieldContext() let context = TranscriptionContext( appName: captured?.processName, + bundleID: captured?.bundleID, windowTitle: field.windowTitle, fieldLabel: field.fieldLabel, priorText: field.priorText, diff --git a/Sources/BlurtEngine/STT/AppKindPriming.swift b/Sources/BlurtEngine/STT/AppKindPriming.swift new file mode 100644 index 0000000..3620173 --- /dev/null +++ b/Sources/BlurtEngine/STT/AppKindPriming.swift @@ -0,0 +1,151 @@ +import Foundation + +/// The app-kind formatting instruction: recognizes what *kind* of app the +/// dictation targets (a terminal, a code editor, Slack, Obsidian) from the +/// frontmost app's bundle identifier and renders the one sentence +/// `TranscriptionSteering` sends as `config.llm.instruction` — "Format the +/// result as a shell command with no trailing period." / "… as Swift code." / +/// "… as a casual Slack message, using Slack emoji where they fit." / "… as +/// markdown." — telling the rewrite what shape of text the destination expects, +/// which the app's display name alone doesn't convey. +/// +/// For code editors the window title usually names the open file, so the +/// clause names the language inferred from that filename's extension +/// ("Format the result as Python code.") when one is recognizable, and stays +/// generic ("… as code.") otherwise. +/// +/// These are instructions to the **LLM that rewrites the finished transcript**, +/// not to the STT model. Reshaping output is not something `config.prompt` acts +/// on — that field takes a description of the audio — so a clause phrased as +/// "Transcribe speech into markdown." was a no-op wherever it landed. Keep the +/// imperative "Format the result as …" shape. +/// +/// Detection keys on bundle IDs, not display names: names are localized and +/// user-editable, while the bundle ID is the app's stable identity. An +/// unrecognized app contributes no clause — the request's `llm` block is then +/// empty and the service's own default cleanup instruction applies. Exercised by +/// `Tests/BlurtEngineTests/AppKindPrimingTests.swift`. +enum AppKindPriming { + /// The recognized destination families. Each renders one guidance sentence; + /// anything else contributes no clause. + enum Kind: Sendable, Equatable { + case terminal + case codeEditor + case slack + case obsidian + } + + /// Exact bundle-ID → kind matches for the recognized apps. + private static let kindsByBundleID: [String: Kind] = [ + // Terminals. + "com.apple.Terminal": .terminal, + "com.googlecode.iterm2": .terminal, + "dev.warp.Warp-Stable": .terminal, + "dev.warp.Warp-Preview": .terminal, + "com.mitchellh.ghostty": .terminal, + "net.kovidgoyal.kitty": .terminal, + "org.alacritty": .terminal, + "com.github.wez.wezterm": .terminal, + "co.zeit.hyper": .terminal, + // Code editors. Cursor ships under an opaque ToDesktop build id. + "com.microsoft.VSCode": .codeEditor, + "com.microsoft.VSCodeInsiders": .codeEditor, + "com.vscodium": .codeEditor, + "com.todesktop.230313mzl4w4u92": .codeEditor, + "com.exafunction.windsurf": .codeEditor, + "com.apple.dt.Xcode": .codeEditor, + "dev.zed.Zed": .codeEditor, + "dev.zed.Zed-Preview": .codeEditor, + "com.panic.Nova": .codeEditor, + "com.macromates.TextMate": .codeEditor, + // Chat and notes. + "com.tinyspeck.slackmacgap": .slack, + "md.obsidian": .obsidian, + ] + + /// Prefix matches for app families that ship many bundle IDs under one + /// vendor prefix (every JetBrains IDE, Sublime Text's versioned IDs). + private static let kindsByBundleIDPrefix: [(prefix: String, kind: Kind)] = [ + ("com.jetbrains.", .codeEditor), + ("com.sublimetext.", .codeEditor), + ] + + /// The kind `bundleID` identifies, or `nil` for an unrecognized (or absent) + /// bundle ID. + static func kind(ofBundleID bundleID: String?) -> Kind? { + guard let bundleID = bundleID.trimmedNonEmpty() else { return nil } + if let kind = kindsByBundleID[bundleID] { return kind } + return kindsByBundleIDPrefix.first { bundleID.hasPrefix($0.prefix) }?.kind + } + + /// The formatting instruction for the app `bundleID` identifies, or `nil` when + /// the app isn't recognized. `windowTitle` refines the code-editor clause + /// with the open file's language; the other kinds ignore it. + static func clause(bundleID: String?, windowTitle: String?) -> String? { + guard let kind = kind(ofBundleID: bundleID) else { return nil } + switch kind { + case .terminal: + // "Trailing period", not "terminal punctuation": in a clause about + // terminals the latter reads as the app, not the end of a sentence. + return "Format the result as a shell command with no trailing period." + case .codeEditor: + let subject = windowTitle.flatMap(language(inWindowTitle:)) ?? "code" + return "Format the result as \(subject)." + case .slack: + return "Format the result as a casual Slack message, using Slack emoji where they fit." + case .obsidian: + return "Format the result as markdown." + } + } + + /// What the open file says speech becomes, from the first token in `title` + /// that reads as a filename with a recognized extension + /// ("● main.py — blurt — Visual Studio Code" → "Python code"), or `nil` when + /// no token does. Editors lead their window titles with the open file, so + /// first match wins. + static func language(inWindowTitle title: String) -> String? { + for token in title.split(whereSeparator: \.isWhitespace) { + let name = token.trimmingCharacters(in: Self.filenameTrim) + // A leading-dot name (".zshrc") is a dotfile, not a base name + extension. + guard let dot = name.lastIndex(of: "."), dot != name.startIndex else { continue } + if let language = languagesByExtension[name[name.index(after: dot)...].lowercased()] { + return language + } + } + return nil + } + + /// Decoration editors wrap around the filename in a window title — dirty + /// markers, quotes, brackets, dash separators. + private static let filenameTrim = CharacterSet(charactersIn: "\"'`•●◆*()[]{}<>,;:—–-") + + /// Filename extension → what the clause says the result should be. Values + /// complete "Format the result as …", so languages carry a trailing "code" + /// while markup/data formats stand alone. Lowercased keys; lookups lowercase + /// the extension first. + private static let languagesByExtension: [String: String] = [ + "c": "C code", "h": "C code", + "cc": "C++ code", "cpp": "C++ code", "cxx": "C++ code", "hpp": "C++ code", + "cs": "C# code", + "css": "CSS", "scss": "CSS", + "go": "Go code", + "htm": "HTML", "html": "HTML", + "java": "Java code", + "cjs": "JavaScript code", "js": "JavaScript code", "jsx": "JavaScript code", "mjs": "JavaScript code", + "json": "JSON", + "kt": "Kotlin code", "kts": "Kotlin code", + "lua": "Lua code", + "m": "Objective-C code", "mm": "Objective-C code", + "markdown": "markdown", "md": "markdown", + "php": "PHP code", + "py": "Python code", "pyi": "Python code", + "rb": "Ruby code", + "rs": "Rust code", + "bash": "a shell script", "sh": "a shell script", "zsh": "a shell script", + "sql": "SQL", + "swift": "Swift code", + "toml": "TOML", + "ts": "TypeScript code", "tsx": "TypeScript code", + "yaml": "YAML", "yml": "YAML", + ] +} diff --git a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift index dcd2353..6071e43 100644 --- a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift +++ b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift @@ -51,8 +51,8 @@ public struct AssemblyAITranscriber: TranscriberProtocol { guard let apiKey = apiKeyProvider(), !apiKey.isEmpty else { throw BlurtError.apiKeyMissing } - let prompt = TranscriptionPrompt.build(context: context) - let config = try makeConfigData(sampleRate: sampleRate, prompt: prompt) + let steering = TranscriptionSteering.build(context: context) + let config = try makeConfigData(sampleRate: sampleRate, steering: steering) let boundary = "blurt-\(UUID().uuidString)" var request = URLRequest(url: baseURL.appendingPathComponent("transcribe")) @@ -105,18 +105,21 @@ public struct AssemblyAITranscriber: TranscriberProtocol { "warm-up connect \(elapsedMs, format: .fixed(precision: 0), privacy: .public)ms") } - /// Builds the JSON `config` part sent alongside the audio. The context - /// `prompt` is included only when non-empty; a nil or blank prompt omits the - /// field so the server applies its default prompt. The `llm` block always - /// rides along — see `DictationConfig.llm`. Internal so tests can assert the - /// prompt wiring without inspecting the multipart upload body (which - /// `URLProtocol` mocks can't observe reliably for `upload(from:)`). - func makeConfigData(sampleRate: Int, prompt: String?) throws -> Data { + /// Builds the JSON `config` part sent alongside the audio. Each steering field + /// is included only when it carries something — an empty array or a nil + /// instruction omits the field rather than stating an empty value, so the + /// service applies its own default. The `llm` block always rides along — see + /// `DictationConfig.llm`. Internal so tests can assert the steering wiring + /// without inspecting the multipart upload body (which `URLProtocol` mocks + /// can't observe reliably for `upload(from:)`). + func makeConfigData(sampleRate: Int, steering: TranscriptionSteering.Fields) throws -> Data { try JSONEncoder().encode( DictationConfig( sampleRate: sampleRate, channels: 1, - prompt: prompt.trimmedNonEmpty() + conversationContext: steering.conversationContext, + keytermsPrompt: steering.keyterms, + llm: LLMRewrite(instruction: steering.rewriteInstruction) ) ) } @@ -194,28 +197,56 @@ public struct AssemblyAITranscriber: TranscriberProtocol { // MARK: - Wire types + /// The JSON `config` part. Note the absence of `prompt`: that field takes a + /// description of the audio rather than instructions, and a custom value + /// replaces the service's managed default (language steering included), so + /// Blurt sends none — see `TranscriptionSteering` for the full reasoning. private struct DictationConfig: Encodable { let sampleRate: Int let channels: Int - /// Custom transcription instruction. Encoded only when non-nil (the - /// synthesized `encode` uses `encodeIfPresent` for optionals), so omitting - /// it falls back to the server's default prompt. Steers *transcription*; - /// the cleanup rewrite is the `llm` block's job. - let prompt: String? - /// The rewrite request. An empty object selects the service's default - /// cleanup instruction; per the API's `instruction`-mode rules, output - /// format and don't-answer-the-text safeguards are enforced server-side, - /// so nothing rides along here. - let llm = LLMRewrite() + /// Turns preceding this utterance (Blurt sends at most one: the text before + /// the insertion point). Encoded only when non-nil — the synthesized + /// `encode` uses `encodeIfPresent` for optionals — so an utterance with no + /// prior text omits the field instead of sending `[]`. + let conversationContext: [String] + /// The user's key terms as the explicit vocabulary list, omitted when empty. + let keytermsPrompt: [String] + /// The rewrite request. Always present so the service runs the rewrite at + /// all; its `instruction` is the app-kind formatting clause, or absent to + /// select the service's default cleanup instruction. Per the API's + /// `instruction`-mode rules, output format and don't-answer-the-text + /// safeguards are enforced server-side either way. + let llm: LLMRewrite enum CodingKeys: String, CodingKey { case sampleRate = "sample_rate" case channels - case prompt + case conversationContext = "conversation_context" + case keytermsPrompt = "keyterms_prompt" case llm } + + /// Hand-written so an empty array *omits* its field rather than encoding + /// `[]`. The distinction is on the wire: `"keyterms_prompt": []` states an + /// empty vocabulary, while omission leaves the service to its own handling. + /// (Optional arrays would express this too, but read as "maybe no list" + /// where the truth is "a list, possibly empty".) + func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + try container.encode(sampleRate, forKey: .sampleRate) + try container.encode(channels, forKey: .channels) + if !conversationContext.isEmpty { + try container.encode(conversationContext, forKey: .conversationContext) + } + if !keytermsPrompt.isEmpty { + try container.encode(keytermsPrompt, forKey: .keytermsPrompt) + } + try container.encode(llm, forKey: .llm) + } } - private struct LLMRewrite: Encodable {} + private struct LLMRewrite: Encodable { + let instruction: String? + } private struct DictationResponse: Decodable { /// The verbatim transcript — always present, never altered by the LLM. diff --git a/Sources/BlurtEngine/STT/TranscriptionContext.swift b/Sources/BlurtEngine/STT/TranscriptionContext.swift index 5a5fc3e..7e637b5 100644 --- a/Sources/BlurtEngine/STT/TranscriptionContext.swift +++ b/Sources/BlurtEngine/STT/TranscriptionContext.swift @@ -1,45 +1,60 @@ -/// Per-utterance context the STT model is trained to use as *contextual* -/// priming — it improves recognition accuracy (vocabulary, continuity, -/// capitalization) without changing the output format. Gathered at dictation -/// start from the focused app and the text preceding the cursor, then rendered -/// into the request `prompt` by `TranscriptionPrompt.build`. +/// Per-utterance snapshot of where the dictation is going, gathered at +/// dictation start from the focused app and field. `TranscriptionSteering.build` +/// renders only the parts the request uses — the prior text (as +/// `conversation_context`), the key terms (as `keyterms_prompt`), the bundle ID +/// (via the `AppKindPriming` formatting clause in `llm.instruction`), and the +/// window title (the language refinement of that clause). The prior text and +/// window title also steer the injector's paste separator; nothing else is +/// consumed, and the rest of the context is neither sent nor logged. /// -/// Both fields are optional: whichever is available is used, and an entirely -/// empty context yields no prompt (the server applies its own default). +/// Every focus field is optional and best-effort: whatever couldn't be read is +/// `nil`, and an entirely empty context customizes nothing (the server applies +/// its own defaults). public struct TranscriptionContext: Sendable, Equatable { - /// The frontmost application's display name (e.g. "Slack", "Xcode"), passed - /// as a domain/topic hint so the model expects that app's vocabulary. + /// The frontmost application's display name (e.g. "Slack", "Xcode"). Never + /// sent. public let appName: String? - /// The focused window's title (e.g. "Re: Q3 pricing — Gmail", a document - /// name, a Slack channel). The densest topic hint available — usually packed - /// with the proper nouns and domain vocabulary the model would otherwise guess. + /// The frontmost application's bundle identifier (e.g. + /// "com.tinyspeck.slackmacgap"). Never sent verbatim: it keys the + /// app-*kind* recognition (`AppKindPriming`) that selects the rewrite's + /// formatting instruction — terminal, code editor, Slack, Obsidian. + /// Preferred over `appName` for recognition because display names are + /// localized and user-editable while the bundle ID is stable. + public let bundleID: String? + + /// The focused window's title (e.g. "main.py — blurt", a document name, a + /// Slack channel). Never sent verbatim. In a code editor it usually names the + /// open file, which is how the formatting clause learns the language ("… as + /// Swift code."); it also anchors the injector's same-window separator + /// fallback. public let windowTitle: String? /// A short label for the focused field (placeholder/title/role, e.g. "To", - /// "Subject", "Search", "Message"), passed so the model knows what *kind* of - /// text is expected — an email address, a search query, and prose should be - /// transcribed differently. + /// "Subject", "Search", "Message"). Never sent. public let fieldLabel: String? - /// Text immediately preceding the insertion point in the focused field, - /// passed as "prior chunk context" so the transcript continues naturally. + /// Text immediately preceding the insertion point in the focused field. Sent + /// as the single `conversation_context` turn, so the model knows what the + /// utterance continues; it also drives the injector's leading-separator + /// decision. Skipped entirely in secure fields by `FocusCapture`, so a + /// password never reaches this field. public let priorText: String? - /// The text currently selected in the focused field, when any. Dictating with - /// a selection replaces it (the paste overwrites the highlighted range), so - /// this is passed as priming for what the utterance is about — the vocabulary - /// and topic of the text being rewritten. + /// The text currently selected in the focused field, when any. Never sent: the + /// paste replaces it, so priming the model with it would condition the + /// transcription on text that is on its way out. public let selectedText: String? - /// User-configured domain vocabulary (names, jargon, product names) carried as - /// spelling priming so the model favors these spellings. Unlike the other + /// User-configured domain vocabulary (names, jargon, product names), sent as + /// `keyterms_prompt` so the model favors these spellings. Unlike the other /// fields this isn't per-utterance focus state — it's the same list every time, /// sourced from `KeyTermsStore`. public let keyTerms: [String] public init( appName: String?, + bundleID: String? = nil, windowTitle: String? = nil, fieldLabel: String? = nil, priorText: String?, @@ -47,6 +62,7 @@ public struct TranscriptionContext: Sendable, Equatable { keyTerms: [String] = [] ) { self.appName = appName + self.bundleID = bundleID self.windowTitle = windowTitle self.fieldLabel = fieldLabel self.priorText = priorText @@ -58,7 +74,7 @@ public struct TranscriptionContext: Sendable, Equatable { /// are no key terms. public var isEmpty: Bool { keyTerms.isEmpty - && [appName, windowTitle, fieldLabel, priorText, selectedText].allSatisfy { + && [appName, bundleID, windowTitle, fieldLabel, priorText, selectedText].allSatisfy { $0.trimmedNonEmpty() == nil } } diff --git a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift deleted file mode 100644 index 9339c54..0000000 --- a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift +++ /dev/null @@ -1,130 +0,0 @@ -/// Builds the instruction sent to the dictation API as the `prompt` -/// field of the request `config` (see `AssemblyAITranscriber`). The STT model -/// prepends this to its own system prompt. -/// -/// Every built prompt opens with a fixed `baseInstruction` — a plain-text -/// exclusion clause (see below) — and wraps it in -/// *contextual* priming: a topic hint built from the window title, a -/// destination sentence built from the focused app and field label, "prior -/// chunk context" (the text preceding the cursor), the selected text (which the -/// dictation replaces), and keyword boosting, all of which the model is -/// mid-trained to use for better recognition accuracy. -/// -/// On the directives in `baseInstruction`: a "remove filler words"-style -/// *content* reshaping is **not** in the model's trained instruction set, so it -/// is a no-op and is deliberately omitted (see the project memory note). A -/// language directive is likewise omitted — pinning the prompt to English hurt -/// non-English transcription, so language is left to the model's own detection. -/// The negative feature *exclusion* ("without speaker labels, …") is a trained -/// instruction-following type, so it does take effect — the exclusion -/// suppresses the annotation markers the model would otherwise emit (`[Speaker]`, -/// `[door creaks]`, `[laughing]`, …), which in a dictation product would be -/// pasted into the user's text as literal tokens. The list is trimmed to the -/// three annotation types a dictation user could plausibly trigger; the rarer -/// types (unclear-speech, censor, foreign-language, lyrics) are left out to keep -/// the negative clause short, matching the doc's brief negative examples. -/// -/// Output follows the trained format with `baseInstruction` as its pivot: the -/// prior-chunk context, the topic hint, and the destination sentence precede it -/// as the `{context}. {baseInstruction}` shape, and keyword boosting trails it -/// inline as `Keywords: a, b, c.` (per the mid-training instruction-type -/// reference). It stays under the API's `characterCap`: the contextual -/// blocks are clipped upstream in `FocusCapture`, and the key-terms clause is -/// fitted to the remaining budget here. Exercised by -/// `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. -enum TranscriptionPrompt { - /// The standing dictation instruction prepended to the model's system prompt - /// on every built prompt. A negative-exclusion clause (§5/§6) naming the - /// annotation feature types the model is trained to emit, so it suppresses - /// them. No language directive: pinning the prompt to English degraded - /// transcription for non-English speech, so the model is left to detect the - /// spoken language itself. - static let baseInstruction = - "Transcribe without speaker labels, audio event descriptions, or emotion markers." - - /// Hard cap the dictation API places on `config.prompt` ("max 4096 chars"); - /// a longer prompt risks failing the whole request, so `build` must never - /// exceed it. The contextual blocks are all clipped upstream in - /// `FocusCapture`; the user's key terms are the one unbounded input, so - /// `build` fits them to whatever budget remains. - static let characterCap = 4096 - - /// Renders `context` into a transcription prompt, or `nil` when there is no usable - /// context (the server then applies its own default prompt). - static func build(context: TranscriptionContext?) -> String? { - // `isEmpty` is the context type's own "no usable content" rule — the same - // predicate `DictationSession.performPress` gates on before yielding a - // context. Asking it here (rather than re-deriving the field-by-field test) - // keeps a newly added context signal from being silently dropped. - guard let context, !context.isEmpty else { return nil } - let prior = context.priorText.trimmedNonEmpty() ?? "" - let selected = context.selectedText.trimmedNonEmpty() ?? "" - let app = context.appName.trimmedNonEmpty() ?? "" - let window = context.windowTitle.trimmedNonEmpty() ?? "" - let field = context.fieldLabel.trimmedNonEmpty() ?? "" - let keyTerms = context.keyTerms - - // `baseInstruction` is the pivot of the trained format. Contextual priming - // sits *before* it; keyword boosting trails *after* it. The leading blocks, - // separated by blank lines, precede it: - // 1. the prior-chunk block (`Previous transcript:\n…`, its own paragraph), - // 2. the selected-text block (`Selected text:\n…`, what the dictation - // replaces — primes vocabulary/topic of the text being rewritten), - // 3. the location clause (topic hint + destination sentence). - var blocks: [String] = [] - if !prior.isEmpty { - blocks.append("Previous transcript:\n\(prior)") - } - if !selected.isEmpty { - blocks.append("Selected text:\n\(selected)") - } - let location = locationClause(app: app, window: window, field: field) - // The topic hint and `baseInstruction` share one line as the trained - // `{context}. {baseInstruction}` shape; with no topic it's the bare base. - let instruction = location.isEmpty ? baseInstruction : "\(location) \(baseInstruction)" - blocks.append(instruction) - var prompt = blocks.joined(separator: "\n\n") - if !keyTerms.isEmpty { - // Spelling priming: the user's domain vocabulary, boosted via the trained - // inline `Keywords: a, b, c.` form (Section 2.3) trailing the marker so the - // model favors these exact spellings for names/jargon it would guess at. - // The terms list is the one input with no upstream length cap, so include - // only as many whole terms as `characterCap` leaves room for, so a huge - // Settings list can't crowd out the instruction itself or balloon every - // request. - var included: [String] = [] - var remaining = characterCap - prompt.count - " Keywords: .".count - for term in keyTerms { - let cost = term.count + (included.isEmpty ? 0 : ", ".count) - guard cost <= remaining else { break } - included.append(term) - remaining -= cost - } - if !included.isEmpty { - prompt += " Keywords: \(included.joined(separator: ", "))." - } - } - return prompt - } - - /// The "where am I typing" priming clause, assembled from whichever of the - /// app / window / field signals are present (empty when none are). Two trained - /// shapes joined by a space: a topic hint built from the window title (the - /// richest vocabulary signal — `This is about "…".`, mid-training §2.1) leads, - /// and a destination sentence built from the app/field (`Dictated into …`) - /// trails it. Each sentence ends with a period so the clause joins cleanly - /// before `baseInstruction`. - private static func locationClause(app: String, window: String, field: String) -> String { - let topic = window.isEmpty ? "" : "This is about \"\(window)\"." - - let destination: String - switch (app.isEmpty, field.isEmpty) { - case (false, false): destination = "Dictated into \(app), in the \"\(field)\" field." - case (false, true): destination = "Dictated into \(app)." - case (true, false): destination = "Dictated in the \"\(field)\" field." - case (true, true): destination = "" - } - - return [topic, destination].filter { !$0.isEmpty }.joined(separator: " ") - } -} diff --git a/Sources/BlurtEngine/STT/TranscriptionSteering.swift b/Sources/BlurtEngine/STT/TranscriptionSteering.swift new file mode 100644 index 0000000..92fed52 --- /dev/null +++ b/Sources/BlurtEngine/STT/TranscriptionSteering.swift @@ -0,0 +1,112 @@ +/// Renders a `TranscriptionContext` into the three request-customization fields +/// the dictation API accepts, each of which has one job (see +/// `AssemblyAITranscriber` for the wire encoding): +/// +/// - `conversation_context` — the text immediately before the insertion point, +/// as a single turn. This is real left-context: it tells the model what the +/// utterance is continuing, which is what fixes mid-sentence casing and +/// proper-noun consistency. +/// - `keyterms_prompt` — the user's key terms, verbatim, as the explicit +/// vocabulary list the field is for. +/// - `llm.instruction` — the app-kind formatting clause (`AppKindPriming`), +/// recognized from the frontmost app's bundle ID: markdown in Obsidian, Swift +/// code in a code editor (the language inferred from the window title's +/// filename), a shell command in a terminal, a casual message in Slack. +/// +/// **`config.prompt` is deliberately never sent.** The field takes a +/// *description of the audio* ("Cardiology consultation about chest pain +/// symptoms."), not instructions — transcription behavior is optimized out of +/// the box, so an imperative like "Transcribe speech into markdown." was aimed +/// at a field that does not act on instructions. That is why the app-kind +/// priming now rides in `llm.instruction`, where an LLM actually rewrites the +/// text, and why the key terms moved to `keyterms_prompt` rather than being +/// packed into the prompt as a `Keywords:` clause. Sending no prompt also keeps +/// the service's managed default — a custom prompt replaces it wholesale, +/// including its language steering, which is the mechanism behind the earlier +/// finding that pinning the prompt to English hurt non-English speech. +/// +/// Blurt has no earlier turns to send: `conversation_context` carries exactly +/// one entry, the prior-cursor text, or none. +/// +/// The rest of the captured context is not sent at all. The app name and field +/// label render nowhere (real-world logs showed them crowding the request — VS +/// Code parks a screen-reader help announcement in the focused field's +/// description), and the window title is read only to name a code editor's +/// language. Selected text is never priming: the paste replaces it, so +/// conditioning the model on it would prime for text on its way out. +/// +/// Exercised by `Tests/BlurtEngineTests/TranscriptionSteeringTests.swift`. +enum TranscriptionSteering { + /// What one utterance sends beyond the audio and its geometry. Built here so + /// the transcriber and the dictation log describe the same request rather than + /// each deriving it. + struct Fields: Sendable, Equatable { + /// Turns preceding this utterance, oldest first. At most one entry (the + /// prior-cursor text). + let conversationContext: [String] + /// Explicit vocabulary to bias recognition toward, in the user's own + /// spelling and capitalization. + let keyterms: [String] + /// The rewrite instruction, or `nil` to let the service's default cleanup + /// instruction stand. + let rewriteInstruction: String? + + /// Nothing to customize — every field omitted, so the service applies its + /// managed default prompt and its default cleanup rewrite. Also the value + /// to compare against for "does this utterance customize anything?" — + /// `Fields` is `Equatable`, so no separate emptiness predicate exists to + /// drift from the fields themselves. + static let empty = Fields(conversationContext: [], keyterms: [], rewriteInstruction: nil) + } + + /// Cap the dictation API documents for `conversation_context`: 4096 characters + /// across all turns. Over-cap context is trimmed rather than rejected, so this + /// is a quality guard, not a request-failure guard — but the clip must keep the + /// *tail*, since the words nearest the insertion point are the ones carrying + /// continuity. `FocusCapture` already caps prior text far below this + /// (`maxPriorChars`, 320); the guard is here so a hand-built or future-widened + /// context can't silently exceed the field. + static let conversationContextCharacterCap = 4096 + + /// Cap the dictation API documents for `keyterms_prompt`: 2048 characters + /// totalled across every term. Key terms are the one input with no upstream + /// length limit — a Settings list of any size — so `build` includes only as + /// many whole leading terms as fit. + static let keytermsCharacterCap = 2048 + + /// Renders `context` into the fields to send. An absent or unusable context + /// yields `.empty`, which sends no customization at all. + static func build(context: TranscriptionContext?) -> Fields { + // `isEmpty` is the context type's own "no usable content" rule — the same + // predicate `DictationSession.performPress` gates on before yielding a + // context. Asking it here (rather than re-deriving the field-by-field test) + // keeps a newly added context signal from being silently dropped. + guard let context, !context.isEmpty else { return .empty } + return Fields( + conversationContext: priorTurn(of: context).map { [$0] } ?? [], + keyterms: fittedKeyterms(context.keyTerms), + rewriteInstruction: AppKindPriming.clause( + bundleID: context.bundleID, windowTitle: context.windowTitle.trimmedNonEmpty())) + } + + /// The prior-cursor text as one conversation turn, clipped to the field's cap + /// from the front so the text nearest the insertion point survives. + private static func priorTurn(of context: TranscriptionContext) -> String? { + guard let prior = context.priorText.trimmedNonEmpty() else { return nil } + guard prior.count > conversationContextCharacterCap else { return prior } + return String(prior.suffix(conversationContextCharacterCap)) + } + + /// As many whole leading terms as the total-length cap allows. Whole terms + /// only: half a proper noun biases the model toward a spelling nobody wants. + private static func fittedKeyterms(_ terms: [String]) -> [String] { + var included: [String] = [] + var remaining = keytermsCharacterCap + for term in terms { + guard term.count <= remaining else { break } + included.append(term) + remaining -= term.count + } + return included + } +} diff --git a/Tests/BlurtEngineTests/AppKindPrimingTests.swift b/Tests/BlurtEngineTests/AppKindPrimingTests.swift new file mode 100644 index 0000000..e491758 --- /dev/null +++ b/Tests/BlurtEngineTests/AppKindPrimingTests.swift @@ -0,0 +1,129 @@ +import Testing + +@testable import BlurtEngine + +@Suite("AppKindPriming") +struct AppKindPrimingTests { + // MARK: - Kind recognition + + /// One bundle-ID → kind expectation, tabled for per-case failure output like + /// `TranscriptionSteeringTests.Case`. + struct KindCase: Sendable, CustomTestStringConvertible { + let bundleID: String? + let expected: AppKindPriming.Kind? + var testDescription: String { bundleID ?? "nil" } + } + + static let kindCases: [KindCase] = [ + KindCase(bundleID: "com.apple.Terminal", expected: .terminal), + KindCase(bundleID: "com.googlecode.iterm2", expected: .terminal), + KindCase(bundleID: "com.mitchellh.ghostty", expected: .terminal), + KindCase(bundleID: "com.microsoft.VSCode", expected: .codeEditor), + KindCase(bundleID: "com.apple.dt.Xcode", expected: .codeEditor), + KindCase(bundleID: "com.todesktop.230313mzl4w4u92", expected: .codeEditor), + // Prefix families: any JetBrains IDE, any Sublime Text major version. + KindCase(bundleID: "com.jetbrains.pycharm", expected: .codeEditor), + KindCase(bundleID: "com.sublimetext.4", expected: .codeEditor), + KindCase(bundleID: "com.tinyspeck.slackmacgap", expected: .slack), + KindCase(bundleID: "md.obsidian", expected: .obsidian), + KindCase(bundleID: "com.apple.mail", expected: nil), + KindCase(bundleID: "", expected: nil), + KindCase(bundleID: " ", expected: nil), + KindCase(bundleID: nil, expected: nil), + ] + + @Test("bundle IDs map to their app kind", arguments: kindCases) + func kind(_ c: KindCase) { + #expect(AppKindPriming.kind(ofBundleID: c.bundleID) == c.expected) + } + + // MARK: - Language inference from window titles + + /// One window-title → language expectation. + struct LanguageCase: Sendable, CustomTestStringConvertible { + let title: String + let expected: String? + var testDescription: String { title } + } + + static let languageCases: [LanguageCase] = [ + LanguageCase(title: "main.py — blurt — Visual Studio Code", expected: "Python code"), + // VS Code prepends ● to the filename of a dirty buffer. + LanguageCase(title: "● api.ts — server", expected: "TypeScript code"), + LanguageCase(title: "blurt — TranscriptionPrompt.swift", expected: "Swift code"), + LanguageCase(title: "deploy.sh — infra", expected: "a shell script"), + // Extension matching is case-insensitive. + LanguageCase(title: "README.MD — notes", expected: "markdown"), + // First filename wins when several tokens carry extensions. + LanguageCase(title: "index.js next.config.ts", expected: "JavaScript code"), + // A dotfile has no base name + extension split. + LanguageCase(title: ".zshrc — dotfiles", expected: nil), + // Version numbers and hostnames are not filenames. + LanguageCase(title: "release v0.1.34 — dictation.assemblyai.com", expected: nil), + LanguageCase(title: "Untitled-1", expected: nil), + LanguageCase(title: "", expected: nil), + ] + + @Test("window titles yield the open file's language", arguments: languageCases) + func language(_ c: LanguageCase) { + #expect(AppKindPriming.language(inWindowTitle: c.title) == c.expected) + } + + // MARK: - Clause rendering + + @Test("a terminal renders the shell-command formatting instruction") + func terminalClause() { + #expect( + AppKindPriming.clause(bundleID: "com.apple.Terminal", windowTitle: nil) + == "Format the result as a shell command with no trailing period.") + } + + @Test("a code editor names the open file's language when the title carries one") + func codeEditorClauseWithLanguage() { + #expect( + AppKindPriming.clause(bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt") + == "Format the result as Python code.") + } + + @Test("a code editor stays generic when the title names no recognizable file") + func codeEditorClauseGeneric() { + #expect( + AppKindPriming.clause(bundleID: "com.apple.dt.Xcode", windowTitle: "Welcome to Xcode") + == "Format the result as code.") + } + + @Test("Slack renders the casual-message formatting instruction") + func slackClause() { + #expect( + AppKindPriming.clause(bundleID: "com.tinyspeck.slackmacgap", windowTitle: "#eng-backend") + == "Format the result as a casual Slack message, using Slack emoji where they fit.") + } + + @Test("Obsidian renders the markdown formatting instruction") + func obsidianClause() { + #expect( + AppKindPriming.clause(bundleID: "md.obsidian", windowTitle: "Meeting notes") + == "Format the result as markdown.") + } + + @Test("every clause reads as a rewrite instruction, not a transcription instruction") + func clausesAreRewriteInstructions() { + // These render into `config.llm.instruction`, which is an instruction to an + // LLM rewriting finished text. The Sync STT docs are explicit that + // `config.prompt` takes a *description of the audio* instead — so a clause + // that slipped back into "Transcribe speech into …" phrasing would be aimed + // at the wrong field, which is how the app-kind priming was silently a no-op + // before. + for bundleID in ["com.apple.Terminal", "com.microsoft.VSCode", "com.tinyspeck.slackmacgap", "md.obsidian"] { + let clause = AppKindPriming.clause(bundleID: bundleID, windowTitle: nil) + #expect(clause?.hasPrefix("Format the result as ") == true, "\(bundleID)") + #expect(clause?.contains("Transcribe") == false, "\(bundleID)") + } + } + + @Test("an unrecognized app contributes no clause") + func unrecognizedApp() { + #expect(AppKindPriming.clause(bundleID: "com.apple.mail", windowTitle: "Re: Q3") == nil) + #expect(AppKindPriming.clause(bundleID: nil, windowTitle: "main.py") == nil) + } +} diff --git a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift index 2da4581..b038127 100644 --- a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift +++ b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift @@ -52,14 +52,14 @@ struct HTTPClientTests { #expect(result == expected) } - @Test("transcriber succeeds with a real context (builds and sends a prompt)") + @Test("transcriber succeeds with a real context (builds and sends steering fields)") func transcribeWithContext() async throws { let transport = FakeHTTPTransport { request in guard request.url?.path.hasSuffix("/transcribe") == true else { return (404, Data()) } return (200, json(["text": "hello world"])) } - // A non-empty context exercises the TranscriptionPrompt.build path inside + // A non-empty context exercises the TranscriptionSteering.build path inside // transcribe() that the nil-context happy path skips. The fake can't observe // the multipart upload body, so this asserts the request still round-trips // cleanly rather than the wire contents (covered directly by makeConfigData). @@ -154,29 +154,55 @@ struct HTTPClientTests { } } - @Test("config part carries the built context prompt") - func configIncludesPrompt() throws { - let object = try configObject(prompt: "CONTEXT. Transcribe.") - #expect(object["prompt"] as? String == "CONTEXT. Transcribe.") + @Test("config part carries each steering field under its documented wire name") + func configIncludesSteeringFields() throws { + let object = try configObject( + steering: TranscriptionSteering.Fields( + conversationContext: ["$ git status"], keyterms: ["Blurt", "AssemblyAI"], + rewriteInstruction: "Format the result as markdown.")) + #expect(object["conversation_context"] as? [String] == ["$ git status"]) + #expect(object["keyterms_prompt"] as? [String] == ["Blurt", "AssemblyAI"]) + #expect((object["llm"] as? [String: Any])?["instruction"] as? String == "Format the result as markdown.") #expect(object["sample_rate"] as? Int == 16_000) // The capture path is mono by construction; the declared geometry must agree. #expect(object["channels"] as? Int == 1) } - @Test("config part always requests the default cleanup rewrite", arguments: ["CONTEXT. Transcribe.", nil]) - func configRequestsDefaultRewrite(prompt: String?) throws { - // `llm` must be present and empty on every request: present so the service - // runs the rewrite at all, empty so the server-owned default cleanup - // instruction (and its guardrails) applies rather than a client-side copy. - // `isEmpty == true` also covers presence — it is false for a missing `llm`. - #expect((try configObject(prompt: prompt)["llm"] as? [String: Any])?.isEmpty == true) + @Test("config part never sends a prompt — the managed default is what steers transcription") + func configNeverSendsPrompt() throws { + // A custom `config.prompt` replaces the service's managed default *and* its + // language steering, and the field wants a description of the audio rather + // than instructions — which is why formatting moved to `llm.instruction` and + // vocabulary to `keyterms_prompt`. Sending no prompt keeps the managed + // default, so this pins the field's absence for every steering shape. + let shapes: [TranscriptionSteering.Fields] = [ + .empty, + TranscriptionSteering.Fields( + conversationContext: ["Hi Sam,"], keyterms: ["Blurt"], + rewriteInstruction: "Format the result as markdown."), + ] + for steering in shapes { + #expect(try configObject(steering: steering).keys.contains("prompt") == false) + } + } + + @Test("config part requests the default cleanup rewrite when no formatting is needed") + func configRequestsDefaultRewrite() throws { + // `llm` must be present and empty when there is no app-kind clause: present + // so the service runs the rewrite at all, empty so the server-owned default + // cleanup instruction (and its guardrails) applies rather than a client-side + // copy. `isEmpty == true` also covers presence — false for a missing `llm`. + #expect((try configObject(steering: .empty)["llm"] as? [String: Any])?.isEmpty == true) } - @Test( - "config part omits the prompt field when there is no usable context", - arguments: [nil, " \n"]) - func configOmitsPrompt(prompt: String?) throws { - #expect(try configObject(prompt: prompt).keys.contains("prompt") == false) + @Test("config part omits the context and keyterms fields when they are empty") + func configOmitsEmptySteeringFields() throws { + // An empty array is not the same as an absent field: sending + // `"keyterms_prompt": []` states an empty vocabulary where omission lets the + // service apply its own handling. + let object = try configObject(steering: .empty) + #expect(object.keys.contains("conversation_context") == false) + #expect(object.keys.contains("keyterms_prompt") == false) } @Test("the multipart body frames the audio and config parts the dictation API expects") @@ -306,9 +332,9 @@ struct HTTPClientTests { /// config assertion below wants, since `makeConfigData` returns raw JSON. /// A part that isn't a JSON object at all fails here rather than turning every /// downstream assertion into a silent nil-compare. - private func configObject(prompt: String?) throws -> [String: Any] { + private func configObject(steering: TranscriptionSteering.Fields) throws -> [String: Any] { let config = try makeTranscriber(apiKey: "test-key") - .makeConfigData(sampleRate: 16_000, prompt: prompt) + .makeConfigData(sampleRate: 16_000, steering: steering) return try #require(JSONSerialization.jsonObject(with: config) as? [String: Any]) } diff --git a/Tests/BlurtEngineTests/DictationLogTests.swift b/Tests/BlurtEngineTests/DictationLogTests.swift index 163fabb..0b2b696 100644 --- a/Tests/BlurtEngineTests/DictationLogTests.swift +++ b/Tests/BlurtEngineTests/DictationLogTests.swift @@ -8,15 +8,29 @@ private struct DecodedEntry: Decodable { let ts: String } -/// Decodes the optional focus-context fields so tests can assert they're -/// threaded from the `TranscriptionContext` onto disk. -private struct DecodedContext: Decodable { - let app: String? - let window: String? - let field: String? - let prior: String? - let selected: String? - let prompt: String? +/// Decodes the steering fields so tests can assert the exact request +/// customization that was sent is what lands on disk, under the same wire names +/// the request uses. +private struct DecodedSteering: Decodable { + let conversationContext: [String] + let keytermsPrompt: [String] + let llmInstruction: String? + + enum CodingKeys: String, CodingKey { + case conversationContext = "conversation_context" + case keytermsPrompt = "keyterms_prompt" + case llmInstruction = "llm_instruction" + } + + // A missing array decodes to empty rather than nil: whether a field was + // *omitted* is asserted against the raw line, so nothing here needs to tell + // absent from empty. + init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + conversationContext = try container.decodeIfPresent([String].self, forKey: .conversationContext) ?? [] + keytermsPrompt = try container.decodeIfPresent([String].self, forKey: .keytermsPrompt) ?? [] + llmInstruction = try container.decodeIfPresent(String.self, forKey: .llmInstruction) + } } /// Each test gets a fresh empty file in a unique temp directory so the host's real @@ -92,50 +106,56 @@ struct DictationLogTests { #expect(transcript < ts) } - @Test("threads focus context (incl. selected text) onto disk") - func logsContext() { + @Test("logs only what was sent — context captured but never sent stays off disk") + func logsOnlyWhatWasSent() { let url = makeTempLogURL() let context = TranscriptionContext( - appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", - priorText: "Hi Sam,", selectedText: "the old plan") + appName: "Obsidian", bundleID: "md.obsidian", windowTitle: "Grocery list", + fieldLabel: "text entry area", priorText: "- milk", selectedText: "- bread") DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) - #expect(decoded?.app == "Mail") - #expect(decoded?.window == "Re: Q3 pricing") - #expect(decoded?.field == "Body") - #expect(decoded?.prior == "Hi Sam,") - #expect(decoded?.selected == "the old plan") - } - - @Test("omits the selected field when nothing is selected") - func omitsSelectedWhenAbsent() { - let url = makeTempLogURL() - DictationLog.write(transcript: "p", context: nil, to: url, now: Date()) - let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - // `Encodable` synthesis uses `encodeIfPresent`, so a nil field is absent - // rather than `"selected":null`. - #expect(!line.contains("selected")) + // Everything the request carried is recorded, under the wire's own names… + let decoded = try? JSONDecoder().decode(DecodedSteering.self, from: Data(line.utf8)) + #expect(decoded?.llmInstruction == "Format the result as markdown.") + #expect(decoded?.conversationContext == ["- milk"]) + // …and none of the captured-but-unsent context is. Values, not just keys: + // the entry must carry no trace of what stayed on the machine. Selected text + // is on this list because the paste replaces it, so it is never sent. + for unsent in ["Obsidian", "md.obsidian", "Grocery list", "text entry area", "- bread"] { + #expect(!line.contains(unsent)) + } } - @Test("logs the same assembled prompt the transcriber sends") - func logsAssembledPrompt() { + @Test("logs the same steering fields the transcriber sends") + func logsAssembledSteering() { let url = makeTempLogURL() let context = TranscriptionContext( - appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", - priorText: "Hi Sam,", selectedText: "the old plan") + appName: "Obsidian", bundleID: "md.obsidian", windowTitle: "Grocery list", + fieldLabel: "text entry area", priorText: "- milk", keyTerms: ["Blurt"]) DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) - #expect(decoded?.prompt == TranscriptionPrompt.build(context: context)) + let decoded = try? JSONDecoder().decode(DecodedSteering.self, from: Data(line.utf8)) + let sent = TranscriptionSteering.build(context: context) + // The log is the corpus prompt iteration reads, so it has to agree with the + // builder field-for-field rather than approximately. + #expect(decoded?.llmInstruction == sent.rewriteInstruction) + #expect(decoded?.conversationContext == sent.conversationContext) + #expect(decoded?.keytermsPrompt == sent.keyterms) + #expect(decoded?.llmInstruction == "Format the result as markdown.") + #expect(decoded?.keytermsPrompt == ["Blurt"]) } - @Test("omits the prompt field when there is no context to build one") - func omitsPromptWhenNoContext() { + @Test("omits every steering field when there is no context to build one") + func omitsSteeringWhenNoContext() { let url = makeTempLogURL() DictationLog.write(transcript: "p", context: nil, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" + // Absent, not `null` and not `[]` — the entry should read as "nothing was + // customized", matching the request, which omits these fields too. #expect(!line.contains("\"prompt\"")) + #expect(!line.contains("\"llm_instruction\"")) + #expect(!line.contains("\"conversation_context\"")) + #expect(!line.contains("\"keyterms_prompt\"")) } @Test("survives unicode in transcript field") @@ -178,11 +198,21 @@ struct DictationLogGateTests { @Test("the gate is checked before the context is touched, for both settings") func gateAppliesToContextualEntries() { - // The pipeline always passes the captured context, which is the part carrying - // prior text and the assembled prompt. Off must persist none of it. + // The pipeline always passes the captured context — the input the logged + // steering fields are built from. Off must persist nothing at all; on + // persists what was sent, and still never the context that wasn't (the + // window title and field label stay off disk either way). + // + // Prior-cursor text *is* sent now (as `conversation_context`), so it is on + // disk for a user who opted in. What keeps a password out of it is upstream, + // in `FocusCapture`: prior and selected text are skipped entirely in secure + // fields, failing closed when the AX role can't be read. That guard is + // covered in `FocusCaptureTests`; this suite can only see contexts that + // already cleared it. let context = TranscriptionContext( - appName: "1Password", windowTitle: "Vault", fieldLabel: "Password", - priorText: "hunter2", selectedText: nil) + appName: "Terminal", bundleID: "com.apple.Terminal", + windowTitle: "Vault", fieldLabel: "Command", + priorText: "$ git", selectedText: nil) let offURL = makeTempLogURL() DictationLog.append( transcript: "p", context: context, @@ -195,7 +225,11 @@ struct DictationLogGateTests { DictationLog.queue.sync {} #expect(!FileManager.default.fileExists(atPath: offURL.path)) - #expect(readLog(onURL).contains("hunter2")) + let logged = readLog(onURL) + #expect(logged.contains("Format the result as a shell command with no trailing period.")) + #expect(logged.contains("$ git")) + #expect(!logged.contains("Vault")) + #expect(!logged.contains("Command")) } } diff --git a/Tests/BlurtEngineTests/TranscriptionContextTests.swift b/Tests/BlurtEngineTests/TranscriptionContextTests.swift index deef980..0db647f 100644 --- a/Tests/BlurtEngineTests/TranscriptionContextTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionContextTests.swift @@ -3,9 +3,11 @@ import Testing @testable import BlurtEngine /// `TranscriptionContext.isEmpty` is the gate `FocusCapture`/`DictationSession` -/// use to decide whether a context is worth sending as priming. It mirrors the -/// emptiness logic in `TranscriptionPrompt.build`, so the two must agree: -/// `isEmpty == true` should always correspond to `build` returning `nil`. +/// use to decide whether a context is worth carrying at all (steering AND log). +/// The agreement with `TranscriptionSteering.build` is one-directional: +/// `isEmpty == true` must always correspond to empty steering fields, while a +/// non-empty context may still steer nothing — its signals can be carry-only +/// (app name, field label, or selected text from an unrecognized app). @Suite("TranscriptionContext") struct TranscriptionContextTests { @Test("both fields nil is empty") @@ -28,6 +30,13 @@ struct TranscriptionContextTests { #expect(!TranscriptionContext(appName: nil, priorText: "hello there").isEmpty) } + @Test("a bundle ID alone makes it non-empty (and produces a rewrite instruction)") + func bundleIDPresent() { + let context = TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil) + #expect(!context.isEmpty) + #expect(TranscriptionSteering.build(context: context).rewriteInstruction != nil) + } + @Test("real selected text makes it non-empty") func selectedTextPresent() { #expect(!TranscriptionContext(appName: nil, priorText: nil, selectedText: "highlighted").isEmpty) @@ -38,32 +47,42 @@ struct TranscriptionContextTests { #expect(TranscriptionContext(appName: nil, priorText: nil, selectedText: " \n").isEmpty) } - @Test("key terms alone make it non-empty (and produce a prompt)") + @Test("key terms alone make it non-empty (and produce keyterms)") func keyTermsPresent() { let context = TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["Blurt"]) #expect(!context.isEmpty) - #expect(TranscriptionPrompt.build(context: context) != nil) + #expect(TranscriptionSteering.build(context: context).keyterms == ["Blurt"]) } - @Test("emptiness agrees with TranscriptionPrompt.build returning nil") - func agreesWithPromptBuild() { + @Test("an empty context always corresponds to empty steering fields") + func agreesWithSteeringBuild() { let empties = [ TranscriptionContext(appName: nil, priorText: nil), TranscriptionContext(appName: " ", priorText: "\n"), ] for context in empties { #expect(context.isEmpty) - #expect(TranscriptionPrompt.build(context: context) == nil) + #expect(TranscriptionSteering.build(context: context) == .empty) } - let nonEmpties = [ - TranscriptionContext(appName: "Mail", priorText: nil), - TranscriptionContext(appName: nil, priorText: nil, selectedText: "selected"), + // Renderable signals (a recognized bundle ID, key terms, prior text) each + // steer at least one field… + let renderable = [ + TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil), + TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["Blurt"]), + TranscriptionContext(appName: nil, priorText: "Hi Sam,"), ] - for context in nonEmpties { + for context in renderable { #expect(!context.isEmpty) - #expect(TranscriptionPrompt.build(context: context) != nil) + #expect(TranscriptionSteering.build(context: context) != .empty) } + + // …while carry-only signals make the context non-empty (worth carrying for + // the injector) yet steer nothing. Selected text is the interesting one: the + // paste replaces it, so it is never priming. + let carryOnly = TranscriptionContext(appName: "Mail", priorText: nil, selectedText: "sel") + #expect(!carryOnly.isEmpty) + #expect(TranscriptionSteering.build(context: carryOnly) == .empty) } @Test("Equatable compares every field") @@ -89,5 +108,8 @@ struct TranscriptionContextTests { #expect( TranscriptionContext(appName: "Notes", priorText: "x", keyTerms: ["a"]) != TranscriptionContext(appName: "Notes", priorText: "x", keyTerms: ["b"])) + #expect( + TranscriptionContext(appName: "Notes", bundleID: "com.a.b", priorText: "x") + != TranscriptionContext(appName: "Notes", bundleID: "com.c.d", priorText: "x")) } } diff --git a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift deleted file mode 100644 index 260465a..0000000 --- a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift +++ /dev/null @@ -1,147 +0,0 @@ -import Testing - -@testable import BlurtEngine - -@Suite("TranscriptionPrompt") -struct TranscriptionPromptTests { - /// The standing plain-text exclusion clause that every built prompt carries - /// (see `TranscriptionPrompt.baseInstruction`). Kept here as the single source - /// of truth so the expectations below read clearly. - static let base = - "Transcribe without speaker labels, audio event descriptions, or emotion markers." - - /// One `build(context:)` → prompt expectation. Parameterizing these (rather - /// than a `@Test` apiece) keeps the whole context→prompt contract in one - /// readable table and gives per-case failure output. - struct Case: Sendable, CustomTestStringConvertible { - let name: String - let context: TranscriptionContext? - let expected: String? - var testDescription: String { name } - } - - static let cases: [Case] = [ - Case(name: "nil context → no prompt (server default)", context: nil, expected: nil), - Case( - name: "empty context → no prompt", - context: TranscriptionContext(appName: nil, priorText: nil), expected: nil), - Case( - name: "whitespace-only context → no prompt", - context: TranscriptionContext(appName: " ", priorText: "\n"), expected: nil), - Case( - name: "app only → destination sentence", - context: TranscriptionContext(appName: "Slack", priorText: nil), - expected: "Dictated into Slack. \(base)"), - Case( - name: "prior only → Previous transcript framing", - context: TranscriptionContext(appName: nil, priorText: "and then the build finished"), - expected: "Previous transcript:\nand then the build finished\n\n\(base)"), - Case( - name: "app + window → topic hint leads, destination trails", - context: TranscriptionContext(appName: "Mail", windowTitle: "Re: Q3 pricing", priorText: nil), - expected: "This is about \"Re: Q3 pricing\". Dictated into Mail. \(base)"), - Case( - name: "window only → bare topic hint", - context: TranscriptionContext(appName: nil, windowTitle: "Untitled.txt", priorText: nil), - expected: "This is about \"Untitled.txt\". \(base)"), - Case( - name: "field only → destination sentence", - context: TranscriptionContext(appName: nil, fieldLabel: "Search", priorText: nil), - expected: "Dictated in the \"Search\" field. \(base)"), - Case( - name: "app + field without window → destination names both", - context: TranscriptionContext(appName: "Slack", fieldLabel: "Message", priorText: nil), - expected: "Dictated into Slack, in the \"Message\" field. \(base)"), - Case( - name: "all four signals combine", - context: TranscriptionContext( - appName: "Slack", windowTitle: "#eng-backend", fieldLabel: "Message", priorText: "thanks for"), - expected: - "Previous transcript:\nthanks for\n\nThis is about \"#eng-backend\". Dictated into Slack, in the \"Message\" field. \(base)" - ), - Case( - name: "prior + app combine", - context: TranscriptionContext(appName: "Mail", priorText: "Dear Sam,"), - expected: "Previous transcript:\nDear Sam,\n\nDictated into Mail. \(base)"), - Case( - name: "prior + app are trimmed", - context: TranscriptionContext(appName: " Notes ", priorText: " hello "), - expected: "Previous transcript:\nhello\n\nDictated into Notes. \(base)"), - Case( - name: "selected only → Selected text framing", - context: TranscriptionContext(appName: nil, priorText: nil, selectedText: "the quarterly numbers"), - expected: "Selected text:\nthe quarterly numbers\n\n\(base)"), - Case( - name: "selected follows prior as its own block", - context: TranscriptionContext(appName: nil, priorText: "as we discussed,", selectedText: "the old plan"), - expected: "Previous transcript:\nas we discussed,\n\nSelected text:\nthe old plan\n\n\(base)"), - Case( - name: "selected + location + prior combine", - context: TranscriptionContext( - appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", - priorText: "Hi Sam,", selectedText: "let's push the date"), - expected: - "Previous transcript:\nHi Sam,\n\nSelected text:\nlet's push the date\n\nThis is about \"Re: Q3 pricing\". Dictated into Mail, in the \"Body\" field. \(base)" - ), - Case( - name: "blank selected adds no block", - context: TranscriptionContext(appName: "Notes", priorText: nil, selectedText: " \n"), - expected: "Dictated into Notes. \(base)"), - Case( - name: "selected sits between prior and keyword boost", - context: TranscriptionContext( - appName: "Slack", priorText: "thanks for", selectedText: "the draft", keyTerms: ["Blurt"]), - expected: - "Previous transcript:\nthanks for\n\nSelected text:\nthe draft\n\nDictated into Slack. \(base) Keywords: Blurt." - ), - Case( - name: "key terms only → inline keyword boost", - context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["AssemblyAI", "Kubernetes"]), - expected: "\(base) Keywords: AssemblyAI, Kubernetes."), - Case( - name: "key terms trail base alongside focus context", - context: TranscriptionContext(appName: "Slack", priorText: nil, keyTerms: ["Blurt"]), - expected: "Dictated into Slack. \(base) Keywords: Blurt."), - Case( - name: "empty key terms add no clause", - context: TranscriptionContext(appName: "Notes", priorText: nil, keyTerms: []), - expected: "Dictated into Notes. \(base)"), - ] - - @Test("build maps focus context to the transcription prompt", arguments: cases) - func build(_ c: Case) { - #expect(TranscriptionPrompt.build(context: c.context) == c.expected) - } - - @Test("built prompt fits within the API's 4096-character cap for capped prior text") - func withinCap() { - let longPrior = String(repeating: "word ", count: 200) - let prompt = TranscriptionPrompt.build( - context: TranscriptionContext(appName: "Xcode", priorText: longPrior)) - #expect((prompt?.count ?? 0) <= TranscriptionPrompt.characterCap) - } - - @Test("the keyword clause is omitted entirely when not even the first term fits") - func keyTermsOmittedWhenNoneFit() { - // A single term longer than the whole cap leaves no budget for even one - // keyword: the clause (and its "Keywords:" scaffolding) must be dropped - // whole, not emitted empty or dangling. - let huge = String(repeating: "k", count: TranscriptionPrompt.characterCap) - let prompt = TranscriptionPrompt.build( - context: TranscriptionContext(appName: "Xcode", priorText: nil, keyTerms: [huge])) - #expect(prompt == "Dictated into Xcode. \(Self.base)") - } - - @Test("an oversized key-terms list is fitted to the cap, keeping whole leading terms") - func keyTermsFittedToCap() throws { - // Key terms are the one input with no upstream length cap; a huge Settings - // list must not push the prompt over the API cap (which fails the request). - let terms = (0..<2000).map { "term\($0)" } - let prompt = TranscriptionPrompt.build( - context: TranscriptionContext(appName: "Xcode", priorText: nil, keyTerms: terms)) - let built = try #require(prompt) - #expect(built.count <= TranscriptionPrompt.characterCap) - #expect(built.contains(" Keywords: term0, term1")) - #expect(built.hasSuffix(".")) - } -} diff --git a/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift b/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift new file mode 100644 index 0000000..5e57261 --- /dev/null +++ b/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift @@ -0,0 +1,159 @@ +import Testing + +@testable import BlurtEngine + +@Suite("TranscriptionSteering") +struct TranscriptionSteeringTests { + /// One `build(context:)` → steering-fields expectation. Parameterizing these + /// (rather than a `@Test` apiece) keeps the whole context→wire contract in one + /// readable table and gives per-case failure output. + /// + /// Each recognized signal has exactly one home, and the table pins both + /// directions — what renders and what must never leak into the wrong field: + /// prior-cursor text → `conversation_context`, key terms → `keyterms_prompt`, + /// the app-kind formatting clause → `llm.instruction`. The remaining focus + /// signals (app name, field label, selected text) render nowhere; the window + /// title is read only to name a code editor's language. + struct Case: Sendable, CustomTestStringConvertible { + let name: String + let context: TranscriptionContext? + let expected: TranscriptionSteering.Fields + var testDescription: String { name } + } + + static let cases: [Case] = [ + Case(name: "nil context → nothing to steer with", context: nil, expected: .empty), + Case( + name: "empty context → nothing to steer with", + context: TranscriptionContext(appName: nil, priorText: nil), expected: .empty), + Case( + name: "whitespace-only context → nothing to steer with", + context: TranscriptionContext(appName: " ", priorText: "\n"), expected: .empty), + Case( + name: "app name and field label render nowhere", + context: TranscriptionContext( + appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", + priorText: nil, selectedText: "the old plan"), + expected: .empty), + Case( + name: "unrecognized bundle ID renders no instruction", + context: TranscriptionContext(appName: "Mail", bundleID: "com.apple.mail", priorText: nil), + expected: .empty), + Case( + name: "prior-cursor text becomes the single conversation-context turn", + context: TranscriptionContext(appName: "Mail", priorText: "Hi Sam, thanks for"), + expected: TranscriptionSteering.Fields( + conversationContext: ["Hi Sam, thanks for"], keyterms: [], rewriteInstruction: nil)), + Case( + name: "prior text is trimmed of surrounding whitespace", + context: TranscriptionContext(appName: nil, priorText: " Hi Sam,\n "), + expected: TranscriptionSteering.Fields( + conversationContext: ["Hi Sam,"], keyterms: [], rewriteInstruction: nil)), + Case( + name: "key terms become keyterms_prompt, not a prompt clause", + context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["AssemblyAI", "Kubernetes"]), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: ["AssemblyAI", "Kubernetes"], rewriteInstruction: nil)), + Case( + name: "terminal → shell-command formatting instruction", + context: TranscriptionContext(appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: [], + rewriteInstruction: "Format the result as a shell command with no trailing period.")), + Case( + name: "code editor names the window title's language", + context: TranscriptionContext( + appName: "Code", bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt", priorText: nil), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: [], + rewriteInstruction: "Format the result as Python code.")), + Case( + name: "code editor with no recognizable filename stays generic", + context: TranscriptionContext( + appName: "Xcode", bundleID: "com.apple.dt.Xcode", windowTitle: "Welcome to Xcode", priorText: nil), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: [], rewriteInstruction: "Format the result as code.")), + Case( + name: "Slack → casual-message formatting instruction", + context: TranscriptionContext( + appName: "Slack", bundleID: "com.tinyspeck.slackmacgap", fieldLabel: "Message", priorText: nil), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: [], + rewriteInstruction: "Format the result as a casual Slack message, using Slack emoji where they fit.")), + Case( + name: "Obsidian → markdown formatting instruction", + context: TranscriptionContext( + appName: "Obsidian", bundleID: "md.obsidian", + windowTitle: "Grocery list - Cowork - Obsidian 1.12.7", fieldLabel: "text entry area", + priorText: nil), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: [], rewriteInstruction: "Format the result as markdown.")), + Case( + name: "all three fields populate independently", + context: TranscriptionContext( + appName: "Terminal", bundleID: "com.apple.Terminal", windowTitle: "zsh — 80×24", + priorText: "$ git status", selectedText: "modified: README.md", keyTerms: ["Blurt"]), + expected: TranscriptionSteering.Fields( + conversationContext: ["$ git status"], keyterms: ["Blurt"], + rewriteInstruction: "Format the result as a shell command with no trailing period.")), + ] + + @Test("build maps focus context to the dictation steering fields", arguments: cases) + func build(_ c: Case) { + #expect(TranscriptionSteering.build(context: c.context) == c.expected) + } + + // MARK: - Selected text never becomes context + + @Test("selected text stays out of conversation context — the paste replaces it") + func selectedTextIsNotContext() { + // Selected text is about to be *overwritten* by the paste, so priming the + // model with it would condition the transcription on text that is on its way + // out. Only the text before the insertion point is real left-context. + let fields = TranscriptionSteering.build( + context: TranscriptionContext( + appName: nil, priorText: "keep this", selectedText: "REPLACED")) + #expect(fields.conversationContext == ["keep this"]) + } + + // MARK: - Documented caps + + @Test("an oversized key-terms list is fitted to the keyterms cap, keeping whole leading terms") + func keyTermsFittedToCap() { + // Key terms are the one unbounded input (a Settings list of any length), and + // the field's cap is the total across all terms — so a huge list must be cut + // to whole terms rather than pushing the request over the documented limit. + let terms = (0..<2000).map { "term\($0)" } + let fields = TranscriptionSteering.build( + context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: terms)) + #expect(fields.keyterms.first == "term0") + #expect(fields.keyterms.count < terms.count) + #expect(fields.keyterms.reduce(0) { $0 + $1.count } <= TranscriptionSteering.keytermsCharacterCap) + } + + @Test("a single key term larger than the cap drops the field entirely") + func keyTermsOmittedWhenNoneFit() { + let huge = String(repeating: "k", count: TranscriptionSteering.keytermsCharacterCap + 1) + let fields = TranscriptionSteering.build( + context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: [huge])) + #expect(fields.keyterms.isEmpty) + } + + @Test("over-long prior text is clipped to the cap, keeping the text nearest the cursor") + func priorTextClippedToCapKeepingTail() { + // The words immediately before the insertion point are the ones that carry + // continuity, so a clip must drop the *head* — the opposite of how the key + // terms list is fitted. (FocusCapture already caps prior text far below this; + // the guard is here so a hand-built or future-widened context can't exceed + // the field's documented limit.) + let long = + String(repeating: "a", count: 100) + + String(repeating: "b", count: TranscriptionSteering.conversationContextCharacterCap) + let fields = TranscriptionSteering.build( + context: TranscriptionContext(appName: nil, priorText: long)) + let turn = fields.conversationContext.first ?? "" + #expect(turn.count == TranscriptionSteering.conversationContextCharacterCap) + #expect(turn.hasSuffix("b")) + #expect(!turn.contains("a")) + } +}