Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions .claude/skills/project-guardrails/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,15 @@ one, stop and ask the user first. This is the fast "don't" list; AGENTS.md's
call. No on-device ASR/LLM, no model cache, no download UI.
- **Never send `config.prompt`.** It takes a _description of the audio_, not
instructions, and a custom value replaces the service's managed default
including its language steering. Formatting goes in `llm.instruction`
("Format the result as markdown."), vocabulary in `keyterms_prompt`, the text
before the cursor in `conversation_context`. An imperative like "Transcribe
speech into markdown." in `prompt` is a measured no-op — that is exactly why
these three fields exist.
including its language steering. Vocabulary goes in `keyterms_prompt`, the
text before the cursor in `conversation_context`. An imperative like
"Transcribe speech into markdown." in `prompt` is a measured no-op — that is
exactly why those fields exist.
- **Never send anything about the frontmost app.** `AppKindPriming` recognized
the app's bundle ID as a kind (terminal, code editor, Slack, Obsidian) and
sent a formatting clause as `llm.instruction`; the whole path was removed.
The bundle ID isn't captured, and `llm` always goes out empty so the
service's default cleanup rewrite applies everywhere.
- Don't reintroduce a "remove filler words (um, uh, like)" directive — the STT
prompt doesn't act on it; it was deliberately dropped, and disfluency removal
is the LLM rewrite's job. Same for a language directive: pinning to English
Expand Down
110 changes: 56 additions & 54 deletions AGENTS.md

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions BLURTENGINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ press() ──▶ MicCapture.start() release() ──▶ MicCapture.s
Key properties of the design, which your integration can rely on:

- **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream.
- **Cleanup happens server-side.** The request's `llm` block asks the service for a cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; its optional `instruction` carries the app-kind formatting clause, and an empty block selects the service's own default cleanup. Recognition is primed separately by `conversation_context` and `keyterms_prompt` (built by `TranscriptionSteering` from the captured context); `config.prompt` is never sent. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one.
- **Cleanup happens server-side.** The request's `llm` block asks the service for a cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; it always goes out empty, which selects the service's own default cleanup instruction. Recognition is primed separately by `conversation_context` and `keyterms_prompt` (built by `TranscriptionSteering` from the captured context); `config.prompt` is never sent, and neither is anything describing the destination app. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one.
- **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read.
- **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400.

Expand Down Expand Up @@ -152,15 +152,15 @@ The session calls `setTargetApp` at press time with the app that was frontmost w

Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected:

- **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the bundle ID, window title, and key terms reach the prompt; the prior text and window title also steer the injector's paste separator, and nothing else is consumed.
- **`TranscriptionSteering.build(context:)`** renders that into the request's three customization fields: the prior-cursor text as the single `conversation_context` turn (clipped to 4096 chars keeping the tail, since the words nearest the cursor carry the continuity), the user's key terms as `keyterms_prompt` (whole terms fitted to 2048 chars total), and — when the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian) — the formatting clause as `llm.instruction`, e.g. "Format the result as a shell command with no trailing period." or, for a code editor, the language inferred from the window title's filename ("Format the result as Swift code."). **`config.prompt` is never sent**: it takes a description of the audio rather than instructions, and a custom value replaces the service's managed default including its language steering, which is why the app-kind clause moved to the `llm` block and the key terms to their own field. Nothing else is rendered — no app/field names, no selected text (the paste replaces it), and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default). Empty fields are omitted rather than sent as `[]`. Two further deliberate omissions: no language directive and no "remove filler words" clause (not something the STT prompt acts on — a no-op). Don't reintroduce either.
- **`TranscriptionContext`** carries the frontmost app name, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the prior text and key terms reach the request; the prior text and window title also steer the injector's paste separator, and nothing else is consumed.
- **`TranscriptionSteering.build(context:)`** renders that into the request's two customization fields: the prior-cursor text as the single `conversation_context` turn (clipped to 4096 chars keeping the tail, since the words nearest the cursor carry the continuity) and the user's key terms as `keyterms_prompt` (whole terms fitted to 2048 chars total). **`config.prompt` is never sent**: it takes a description of the audio rather than instructions, and a custom value replaces the service's managed default including its language steering, which is why the key terms ride in their own field. Nothing else is rendered — no app/field names, no selected text (the paste replaces it), and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default). Empty fields are omitted rather than sent as `[]`. Three further deliberate omissions: no language directive, no "remove filler words" clause (not something the STT prompt acts on — a no-op), and nothing identifying the destination app — an earlier `AppKindPriming` sent a bundle-ID-derived formatting clause as `llm.instruction`, and that whole path was removed. Don't reintroduce any of them.
- **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere.

For key storage, compose against **`APIKeyGateway`** — the injectable `current` / `save(_:)` / `hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. Two projections keep the surrounding UI out of your views: `Outcome.failureReport` classifies a failure as `.inline(message:)` (recoverable — show it beside the field) or `.alert(title:message:)` (a Keychain fault retyping can't fix), and **`APIKeyDisplay.resolve(key:)`** renders the stored key for an account row — masked tail, status and VoiceOver wording, and the connect-vs-rotate control titles. The mask reveals only the last `revealedTailLength` characters and, below `minimumLengthToMask`, none at all, so a short key can't be shown whole.

Setup gating has a projection too: **`SetupReadiness.isReady(permissions:hasAPIKey:)`** is the "fully configured" rule (deliberately excluding the trigger key, which has a default), `SetupReadiness.pollInterval(isReady:)` is the permission-poll cadence (brisk during setup, coasting once ready), and `PermissionStatus.lostGrant(since:)` detects a permission revoked out from under a configured app.

Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact steering fields sent (`conversation_context`, `keyterms_prompt`, `llm_instruction`), never the context that wasn't sent — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section.
Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact steering fields sent (`conversation_context`, `keyterms_prompt`), never the context that wasn't sent — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section.

## Hotkey building blocks

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dep
Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM,
live level meter; DX7/Juno-106 sound packs
STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe
(STT + LLM rewrite) + TranscriptionSteering context/keyterms/format
(STT + LLM rewrite) + TranscriptionSteering context/keyterms
Pipeline/ DictationSession actor: press/release/cancel commands, phase
stream, auto-release before the API's recording cap
Hotkey/ DictationKeyGate/Router: pure, unit-tested state machine for the
Expand Down
9 changes: 1 addition & 8 deletions Sources/BlurtEngine/FocusCapture/FocusCapture.swift
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,13 @@ import ApplicationServices
struct CapturedFocus: Sendable {
let pid: pid_t
let processName: String?
/// The frontmost app's stable identity, feeding the prompt's app-kind
/// recognition (`AppKindPriming`) via `TranscriptionContext.bundleID`.
let bundleID: String?
}

enum FocusCapture {
@MainActor
static func captureFrontmost() -> CapturedFocus? {
guard let app = NSWorkspace.shared.frontmostApplication else { return nil }
return CapturedFocus(
pid: app.processIdentifier,
processName: app.localizedName,
bundleID: app.bundleIdentifier
)
return CapturedFocus(pid: app.processIdentifier, processName: app.localizedName)
}

static func runningApp(for captured: CapturedFocus) -> NSRunningApplication? {
Expand Down
6 changes: 1 addition & 5 deletions Sources/BlurtEngine/Pipeline/DictationLog.swift
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,12 @@ public enum DictationLog {
/// an entry directly from a context.
let conversationContext: [String]
let keytermsPrompt: [String]
let llmInstruction: String?

enum CodingKeys: String, CodingKey {
case transcript
case ts
case conversationContext = "conversation_context"
case keytermsPrompt = "keyterms_prompt"
case llmInstruction = "llm_instruction"
}

/// Mirrors `DictationConfig.encode(to:)`: an empty array omits its field, so
Expand All @@ -50,7 +48,6 @@ public enum DictationLog {
if !keytermsPrompt.isEmpty {
try container.encode(keytermsPrompt, forKey: .keytermsPrompt)
}
try container.encodeIfPresent(llmInstruction, forKey: .llmInstruction)
}
}

Expand Down Expand Up @@ -118,8 +115,7 @@ public enum DictationLog {
let entry = Entry(
transcript: transcript, ts: now.formatted(timestampFormat),
conversationContext: steering.conversationContext,
keytermsPrompt: steering.keyterms,
llmInstruction: steering.rewriteInstruction)
keytermsPrompt: steering.keyterms)
guard var line = try? makeEncoder().encode(entry) else { return }
line.append(0x0A) // '\n'

Expand Down
1 change: 0 additions & 1 deletion Sources/BlurtEngine/Pipeline/DictationSession.swift
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,6 @@ public actor DictationSession {
let field = FocusCapture.captureFieldContext()
let context = TranscriptionContext(
appName: captured?.processName,
bundleID: captured?.bundleID,
windowTitle: field.windowTitle,
fieldLabel: field.fieldLabel,
priorText: field.priorText,
Expand Down
Loading