Skip to content
19 changes: 13 additions & 6 deletions .claude/skills/project-guardrails/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,22 @@ one, stop and ask the user first. This is the fast "don't" list; AGENTS.md's

- **No streaming STT.** The AssemblyAI Sync API returns the full transcript in
one response. Overlay goes "Transcribing…" → full text.
- **No separate LLM cleanup pass.** Cleanup rides in the Sync STT request's
`config.prompt` (`TranscriptionPrompt`). No LLM Gateway client, no
- **No separate LLM cleanup pass.** Cleanup rides in the dictation request's
server-side `llm` block (`TranscriptionSteering`). No LLM Gateway client, no
`StylerProtocol`, no post-transcription styling stage.
- **No local models / model downloads.** Transcription is a remote AssemblyAI
call. No on-device ASR/LLM, no model cache, no download UI.
- Don't reintroduce a "remove filler words (um, uh, like)" directive in the
prompt — `universal-3-5-pro` ignores it; it was deliberately dropped. Same for
a language directive: pinning the prompt to English hurt non-English speech, so
language is left to the model's own detection.
- **Never send `config.prompt`.** It takes a _description of the audio_, not
instructions, and a custom value replaces the service's managed default
including its language steering. Formatting goes in `llm.instruction`
("Format the result as markdown."), vocabulary in `keyterms_prompt`, the text
before the cursor in `conversation_context`. An imperative like "Transcribe
speech into markdown." in `prompt` is a measured no-op — that is exactly why
these three fields exist.
- Don't reintroduce a "remove filler words (um, uh, like)" directive — the STT
prompt doesn't act on it; it was deliberately dropped, and disfluency removal
is the LLM rewrite's job. Same for a language directive: pinning to English
hurt non-English speech, so language is left to the model's own detection.
- **Injection is always a clipboard paste** (save → write → ⌘V → settle →
restore), degrading to "left it on the clipboard" when the target is lost. No
keystroke-by-keystroke typing path, no length threshold.
Expand Down
129 changes: 76 additions & 53 deletions AGENTS.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ import SwiftUI

/// The "Key Terms" section of the Settings window: a free-text
/// area where the user lists comma-separated domain words (names, jargon, product
/// names). These are folded into every transcription's prompt as spelling priming
/// (see `KeyTermsStore` / `TranscriptionPrompt.build`), so the model favors those
/// names). These ride on every request as its `keyterms_prompt` vocabulary list
/// (see `KeyTermsStore` / `TranscriptionSteering.build`), so the model favors those
/// spellings. Optional — it never gates setup; an empty list just sends no terms.
struct KeyTermsStepView: View {
/// Stored in UserDefaults so multiple settings windows/readers see edits live.
Expand Down
10 changes: 5 additions & 5 deletions BLURTENGINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ press() ──▶ MicCapture.start() release() ──▶ MicCapture.s
Key properties of the design, which your integration can rely on:

- **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream.
- **Cleanup happens server-side.** The request's empty `llm` block asks the service for its default cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; the per-utterance `config.prompt` (built by `TranscriptionPrompt` from the captured context) primes the _transcription_. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one.
- **Cleanup happens server-side.** The request's `llm` block asks the service for a cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; its optional `instruction` carries the app-kind formatting clause, and an empty block selects the service's own default cleanup. Recognition is primed separately by `conversation_context` and `keyterms_prompt` (built by `TranscriptionSteering` from the captured context); `config.prompt` is never sent. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one.
- **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read.
- **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400.

Expand Down Expand Up @@ -152,15 +152,15 @@ The session calls `setTargetApp` at press time with the app that was frontmost w

Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected:

- **`TranscriptionContext`** carries the frontmost app name, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript.
- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block), opening with the fixed `baseInstruction` ("Transcribe without speaker labels, audio event descriptions, or emotion markers.") and staying under the API's 4096-character cap. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either.
- **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the bundle ID, window title, and key terms reach the prompt; the prior text and window title also steer the injector's paste separator, and nothing else is consumed.
- **`TranscriptionSteering.build(context:)`** renders that into the request's three customization fields: the prior-cursor text as the single `conversation_context` turn (clipped to 4096 chars keeping the tail, since the words nearest the cursor carry the continuity), the user's key terms as `keyterms_prompt` (whole terms fitted to 2048 chars total), and — when the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian) — the formatting clause as `llm.instruction`, e.g. "Format the result as a shell command with no trailing period." or, for a code editor, the language inferred from the window title's filename ("Format the result as Swift code."). **`config.prompt` is never sent**: it takes a description of the audio rather than instructions, and a custom value replaces the service's managed default including its language steering, which is why the app-kind clause moved to the `llm` block and the key terms to their own field. Nothing else is rendered — no app/field names, no selected text (the paste replaces it), and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default). Empty fields are omitted rather than sent as `[]`. Two further deliberate omissions: no language directive and no "remove filler words" clause (not something the STT prompt acts on — a no-op). Don't reintroduce either.
- **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere.

For key storage, compose against **`APIKeyGateway`** — the injectable `current` / `save(_:)` / `hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. Two projections keep the surrounding UI out of your views: `Outcome.failureReport` classifies a failure as `.inline(message:)` (recoverable — show it beside the field) or `.alert(title:message:)` (a Keychain fault retyping can't fix), and **`APIKeyDisplay.resolve(key:)`** renders the stored key for an account row — masked tail, status and VoiceOver wording, and the connect-vs-rotate control titles. The mask reveals only the last `revealedTailLength` characters and, below `minimumLengthToMask`, none at all, so a short key can't be shown whole.

Setup gating has a projection too: **`SetupReadiness.isReady(permissions:hasAPIKey:)`** is the "fully configured" rule (deliberately excluding the trigger key, which has a default), `SetupReadiness.pollInterval(isReady:)` is the permission-poll cadence (brisk during setup, coasting once ready), and `PermissionStatus.lostGrant(since:)` detects a permission revoked out from under a configured app.

Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) with its context snapshot — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section.
Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact steering fields sent (`conversation_context`, `keyterms_prompt`, `llm_instruction`), never the context that wasn't sent — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section.

## Hotkey building blocks

Expand Down Expand Up @@ -195,7 +195,7 @@ Run `swift test` for the engine suites (`--filter DictationSessionTests` for one
Each of these was tried the other way and reverted; the longer stories are in [AGENTS.md](AGENTS.md) and the source comments:

- **No external SPM dependencies in the engine.** Foundation/Security/AVFoundation only.
- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and transcription steering belongs in `TranscriptionPrompt`.
- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and request steering belongs in `TranscriptionSteering`.
- **No `AVAudioEngine`/`installTap` capture path.** Fresh `AVAudioRecorder` per session, resolved at record time.
- **Paste is always clipboard-based** (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation for lost targets.
- **No English-pinning or filler-word clauses in the prompt.**
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,14 +166,14 @@ Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dep
Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM,
live level meter; DX7/Juno-106 sound packs
STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe
(STT + LLM rewrite) + TranscriptionPrompt contextual priming
(STT + LLM rewrite) + TranscriptionSteering context/keyterms/format
Pipeline/ DictationSession actor: press/release/cancel commands, phase
stream, auto-release before the API's recording cap
Hotkey/ DictationKeyGate/Router: pure, unit-tested state machine for the
lone-modifier trigger (tap vs hold vs combo)
Injection/ KeyInjector: save clipboard → paste via synthesized ⌘V → restore
FocusCapture/ Accessibility reads of the focused app/window/field that prime
the transcription prompt
FocusCapture/ Accessibility reads of the focused app/window/field feeding the
request's steering fields, the log, and paste separators
Config/, Update/ Keychain API-key store, key terms, download-only release check

App/Blurt/ AppKit/SwiftUI shell (Xcode project generated by XcodeGen)
Expand Down
8 changes: 4 additions & 4 deletions Sources/BlurtEngine/Config/KeyTermsStore.swift
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
import Foundation

/// Storage for the user's dictation "key terms" — a comma-separated list of
/// domain words (names, jargon, product names) that get folded into the dictation
/// request `prompt` as vocabulary priming, so the model is more likely to spell
/// them correctly (see `TranscriptionPrompt.build`).
/// domain words (names, jargon, product names) sent as the dictation request's
/// `keyterms_prompt` vocabulary list, so the model is more likely to spell
/// them correctly (see `TranscriptionSteering.build`).
///
/// Unlike the API key these aren't secret, so they live in `UserDefaults` rather
/// than the Keychain. The transcription pipeline reads the parsed list via
Expand Down Expand Up @@ -37,7 +37,7 @@ public enum KeyTermsStore {
}

/// Pure parse of a comma-separated string into a clean term list. Exposed so
/// `TranscriptionPrompt` and tests can reuse the exact same rules.
/// `TranscriptionSteering` and tests can reuse the exact same rules.
public static func parse(_ text: String?) -> [String] {
guard let text else { return [] }
var seen = Set<String>()
Expand Down
25 changes: 15 additions & 10 deletions Sources/BlurtEngine/FocusCapture/FocusCapture.swift
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ import ApplicationServices
struct CapturedFocus: Sendable {
let pid: pid_t
let processName: String?
/// The frontmost app's stable identity, feeding the prompt's app-kind
/// recognition (`AppKindPriming`) via `TranscriptionContext.bundleID`.
let bundleID: String?
}

enum FocusCapture {
Expand All @@ -12,25 +15,27 @@ enum FocusCapture {
guard let app = NSWorkspace.shared.frontmostApplication else { return nil }
return CapturedFocus(
pid: app.processIdentifier,
processName: app.localizedName
processName: app.localizedName,
bundleID: app.bundleIdentifier
)
}

static func runningApp(for captured: CapturedFocus) -> NSRunningApplication? {
NSRunningApplication(processIdentifier: captured.pid)
}

/// Accessibility-derived priming read from the system-wide focused UI element
/// at dictation start (see `TranscriptionContext`). Every field is
/// best-effort: any signal that can't be read is `nil`, and a fully-empty
/// result simply means less context, never an error.
/// Accessibility-derived focus context read from the system-wide focused UI
/// element at dictation start (see `TranscriptionContext` for what each
/// signal feeds). Every field is best-effort: any signal that can't be read
/// is `nil`, and a fully-empty result simply means less context, never an
/// error.
struct FocusedFieldContext: Sendable {
/// Text immediately preceding the insertion point ("prior chunk context").
/// Text immediately preceding the insertion point.
let priorText: String?
/// The text currently selected in the focused field — the dictation will
/// replace it, so it primes the model on what the utterance is about.
/// replace it.
let selectedText: String?
/// The focused window's title a dense topic hint.
/// The focused window's title (in a code editor it names the open file).
let windowTitle: String?
/// A short label for the focused field ("To", "Search", "Message").
let fieldLabel: String?
Expand All @@ -49,8 +54,8 @@ enum FocusCapture {
///
/// Secure text fields (password inputs) are detected by role **or** subrole and
/// never have their contents read, so a typed password — selected or not — can't
/// leak into the STT prompt. The check fails closed: an unreadable role is
/// treated as secure, since it can't be shown not to be.
/// leak into the dictation log or the injector. The check fails closed: an
/// unreadable role is treated as secure, since it can't be shown not to be.
///
/// Deliberately `nonisolated`: each read below is a synchronous cross-process
/// IPC round trip into the frontmost app, and an unresponsive app blocks the
Expand Down
Loading