From 90f0d1d71d23e1260f57c44fd26c4d7eeda107b1 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 28 Jul 2026 21:04:42 +0000 Subject: [PATCH 1/8] feat(stt): add app-kind guidance to the dictation prompt Recognize what kind of app the dictation targets from the frontmost app's bundle identifier and add one destination-specific priming sentence to the dictation request's config.prompt: - terminals ("You are dictating into a terminal: expect shell commands, program names, flags, and file paths.") - code editors, naming the language inferred from the window title's filename ("You are writing Python in a code editor: ...") - Slack ("... casual tone and emoji are expected.") - Obsidian ("You are writing a Markdown note in Obsidian: ...") New pure AppKindPriming type owns the bundle-ID -> kind table (exact matches plus JetBrains/Sublime prefix families), the filename-extension -> language map, and the clause wording (positive phrasing per the Universal-3 Pro prompting guidance). TranscriptionContext gains a bundleID field, captured alongside the process name in FocusCapture and counted by isEmpty; TranscriptionPrompt places the guidance after the existing destination sentence, before baseInstruction. Unrecognized apps add nothing, so existing prompts are unchanged. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01R28wj6vXZDW3iS7FWpHT1U --- AGENTS.md | 9 +- BLURTENGINE.md | 4 +- .../FocusCapture/FocusCapture.swift | 6 +- .../Pipeline/DictationSession.swift | 1 + Sources/BlurtEngine/STT/AppKindPriming.swift | 141 ++++++++++++++++++ .../STT/TranscriptionContext.swift | 12 +- .../BlurtEngine/STT/TranscriptionPrompt.swift | 16 +- .../AppKindPrimingTests.swift | 114 ++++++++++++++ .../TranscriptionContextTests.swift | 10 ++ .../TranscriptionPromptTests.swift | 24 +++ 10 files changed, 327 insertions(+), 10 deletions(-) create mode 100644 Sources/BlurtEngine/STT/AppKindPriming.swift create mode 100644 Tests/BlurtEngineTests/AppKindPrimingTests.swift diff --git a/AGENTS.md b/AGENTS.md index dc64a03..5676b8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -53,7 +53,7 @@ Sources/BlurtEngine/ the engine (dependency-free Swift package) Injection/ KeyInjector (clipboard paste), SystemClipboard Permissions/ PermissionsChecker (mic + Accessibility) Pipeline/ DictationSession (actor) + phases, UI projections, geometry, log - STT/ AssemblyAITranscriber, TranscriptionPrompt/Context, SyncSTTLimits + STT/ AssemblyAITranscriber, TranscriptionPrompt/Context, AppKindPriming, SyncSTTLimits Update/ UpdateChecker (download-only) + the launch-check policy App/Blurt/ project.yml XcodeGen source of truth — Blurt.xcodeproj is GENERATED @@ -388,8 +388,11 @@ the user's text. highlighted run the dictation will replace, so the model is primed on what's being rewritten — read via `kAXSelectedTextAttribute`, skipped in secure fields, detected by AX role **or** subrole and failing closed when the role can't be read, so a password can't reach the prompt); a topic hint from -the window title; a destination sentence from the app/field; and inline keyword boosting from the -user's key terms. It's phrased per AssemblyAI's Universal-3 Pro prompting guidance +the window title; a destination sentence from the app/field; an app-kind guidance sentence +(`AppKindPriming` — the frontmost app's bundle ID is recognized as a terminal, code editor, Slack, +or Obsidian, and a code editor's clause names the language inferred from the window title's +filename, so the model expects the right shape of text: shell commands, identifiers, casual chat +with emoji, or Markdown); and inline keyword boosting from the user's key terms. It's phrased per AssemblyAI's Universal-3 Pro prompting guidance (positive/authoritative wording, no "Don't"/"Avoid"/"Never") and stays under the dictation API's documented 4096-character cap on `config.prompt` (`characterCap`). diff --git a/BLURTENGINE.md b/BLURTENGINE.md index d0c2f9c..ae656da 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -152,8 +152,8 @@ The session calls `setTargetApp` at press time with the app that was frontmost w Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected: -- **`TranscriptionContext`** carries the frontmost app name, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. -- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block), opening with the fixed `baseInstruction` ("Transcribe without speaker labels, audio event descriptions, or emotion markers.") and staying under the API's 4096-character cap. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. +- **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. +- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block), opening with the fixed `baseInstruction` ("Transcribe without speaker labels, audio event descriptions, or emotion markers.") and staying under the API's 4096-character cap. When the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian), the prompt adds one app-kind guidance sentence, e.g. "You are dictating into a terminal: expect shell commands, program names, flags, and file paths."; for code editors it names the language inferred from the window title's filename ("You are writing Python …"). Unrecognized apps add nothing. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. - **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere. For key storage, compose against **`APIKeyGateway`** — the injectable `current` / `save(_:)` / `hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. Two projections keep the surrounding UI out of your views: `Outcome.failureReport` classifies a failure as `.inline(message:)` (recoverable — show it beside the field) or `.alert(title:message:)` (a Keychain fault retyping can't fix), and **`APIKeyDisplay.resolve(key:)`** renders the stored key for an account row — masked tail, status and VoiceOver wording, and the connect-vs-rotate control titles. The mask reveals only the last `revealedTailLength` characters and, below `minimumLengthToMask`, none at all, so a short key can't be shown whole. diff --git a/Sources/BlurtEngine/FocusCapture/FocusCapture.swift b/Sources/BlurtEngine/FocusCapture/FocusCapture.swift index 86d9da4..6fe058a 100644 --- a/Sources/BlurtEngine/FocusCapture/FocusCapture.swift +++ b/Sources/BlurtEngine/FocusCapture/FocusCapture.swift @@ -4,6 +4,9 @@ import ApplicationServices struct CapturedFocus: Sendable { let pid: pid_t let processName: String? + /// The frontmost app's stable identity, feeding the prompt's app-kind + /// recognition (`AppKindPriming`) via `TranscriptionContext.bundleID`. + let bundleID: String? } enum FocusCapture { @@ -12,7 +15,8 @@ enum FocusCapture { guard let app = NSWorkspace.shared.frontmostApplication else { return nil } return CapturedFocus( pid: app.processIdentifier, - processName: app.localizedName + processName: app.localizedName, + bundleID: app.bundleIdentifier ) } diff --git a/Sources/BlurtEngine/Pipeline/DictationSession.swift b/Sources/BlurtEngine/Pipeline/DictationSession.swift index b3f278a..c5ed388 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession.swift @@ -215,6 +215,7 @@ public actor DictationSession { let field = FocusCapture.captureFieldContext() let context = TranscriptionContext( appName: captured?.processName, + bundleID: captured?.bundleID, windowTitle: field.windowTitle, fieldLabel: field.fieldLabel, priorText: field.priorText, diff --git a/Sources/BlurtEngine/STT/AppKindPriming.swift b/Sources/BlurtEngine/STT/AppKindPriming.swift new file mode 100644 index 0000000..d67ef4a --- /dev/null +++ b/Sources/BlurtEngine/STT/AppKindPriming.swift @@ -0,0 +1,141 @@ +import Foundation + +/// App-kind guidance for the transcription prompt: recognizes what *kind* of +/// app the dictation targets (a terminal, a code editor, Slack, Obsidian) from +/// the frontmost app's bundle identifier and renders one priming sentence for +/// `TranscriptionPrompt` to place before `baseInstruction`. The sentence tells +/// the model what shape of text the destination expects — shell commands in a +/// terminal, identifiers and symbols in an editor, casual chat in Slack, +/// Markdown in Obsidian — which the app's display name alone doesn't convey. +/// +/// For code editors the window title usually names the open file, so the +/// clause names the language inferred from that filename's extension ("You are +/// writing Python …") when one is recognizable, and stays generic otherwise. +/// +/// Detection keys on bundle IDs, not display names: names are localized and +/// user-editable, while the bundle ID is the app's stable identity. An +/// unrecognized app contributes no clause — the prompt simply falls back to +/// the existing destination sentence built from the app name. Wording follows +/// the same Universal-3 Pro prompting guidance as the rest of the prompt +/// (positive/authoritative phrasing, no negations). Exercised by +/// `Tests/BlurtEngineTests/AppKindPrimingTests.swift`. +enum AppKindPriming { + /// The recognized destination families. Each renders one guidance sentence; + /// anything else contributes no clause. + enum Kind: Sendable, Equatable { + case terminal + case codeEditor + case slack + case obsidian + } + + /// Exact bundle-ID → kind matches for the recognized apps. + private static let kindsByBundleID: [String: Kind] = [ + // Terminals. + "com.apple.Terminal": .terminal, + "com.googlecode.iterm2": .terminal, + "dev.warp.Warp-Stable": .terminal, + "dev.warp.Warp-Preview": .terminal, + "com.mitchellh.ghostty": .terminal, + "net.kovidgoyal.kitty": .terminal, + "org.alacritty": .terminal, + "com.github.wez.wezterm": .terminal, + "co.zeit.hyper": .terminal, + // Code editors. Cursor ships under an opaque ToDesktop build id. + "com.microsoft.VSCode": .codeEditor, + "com.microsoft.VSCodeInsiders": .codeEditor, + "com.vscodium": .codeEditor, + "com.todesktop.230313mzl4w4u92": .codeEditor, + "com.exafunction.windsurf": .codeEditor, + "com.apple.dt.Xcode": .codeEditor, + "dev.zed.Zed": .codeEditor, + "dev.zed.Zed-Preview": .codeEditor, + "com.panic.Nova": .codeEditor, + "com.macromates.TextMate": .codeEditor, + // Chat and notes. + "com.tinyspeck.slackmacgap": .slack, + "md.obsidian": .obsidian, + ] + + /// Prefix matches for app families that ship many bundle IDs under one + /// vendor prefix (every JetBrains IDE, Sublime Text's versioned IDs). + private static let kindsByBundleIDPrefix: [(prefix: String, kind: Kind)] = [ + ("com.jetbrains.", .codeEditor), + ("com.sublimetext.", .codeEditor), + ] + + /// The kind `bundleID` identifies, or `nil` for an unrecognized (or absent) + /// bundle ID. + static func kind(ofBundleID bundleID: String?) -> Kind? { + guard let bundleID = bundleID.trimmedNonEmpty() else { return nil } + if let kind = kindsByBundleID[bundleID] { return kind } + return kindsByBundleIDPrefix.first { bundleID.hasPrefix($0.prefix) }?.kind + } + + /// The guidance sentence for the app `bundleID` identifies, or `nil` when + /// the app isn't recognized. `windowTitle` refines the code-editor clause + /// with the open file's language; the other kinds ignore it. + static func clause(bundleID: String?, windowTitle: String?) -> String? { + guard let kind = kind(ofBundleID: bundleID) else { return nil } + switch kind { + case .terminal: + return "You are dictating into a terminal: expect shell commands, program names, flags, and file paths." + case .codeEditor: + let subject = windowTitle.flatMap(language(inWindowTitle:)) ?? "code" + return "You are writing \(subject) in a code editor: expect code identifiers, symbols, and technical terms." + case .slack: + return "You are writing a Slack message: casual tone and emoji are expected." + case .obsidian: + return "You are writing a Markdown note in Obsidian: Markdown syntax is expected." + } + } + + /// The language of the first token in `title` that reads as a filename with + /// a recognized extension ("● main.py — blurt — Visual Studio Code" → + /// "Python"), or `nil` when no token does. Editors lead their window titles + /// with the open file, so first match wins. + static func language(inWindowTitle title: String) -> String? { + for token in title.split(whereSeparator: \.isWhitespace) { + let name = token.trimmingCharacters(in: Self.filenameTrim) + // A leading-dot name (".zshrc") is a dotfile, not a base name + extension. + guard let dot = name.lastIndex(of: "."), dot != name.startIndex else { continue } + if let language = languagesByExtension[name[name.index(after: dot)...].lowercased()] { + return language + } + } + return nil + } + + /// Decoration editors wrap around the filename in a window title — dirty + /// markers, quotes, brackets, dash separators. + private static let filenameTrim = CharacterSet(charactersIn: "\"'`•●◆*()[]{}<>,;:—–-") + + /// Filename extension → how the clause names what's being written. Values + /// complete "You are writing … in a code editor", so most are bare language + /// names. Lowercased keys; lookups lowercase the extension first. + private static let languagesByExtension: [String: String] = [ + "c": "C", "h": "C", + "cc": "C++", "cpp": "C++", "cxx": "C++", "hpp": "C++", + "cs": "C#", + "css": "CSS", "scss": "CSS", + "go": "Go", + "htm": "HTML", "html": "HTML", + "java": "Java", + "cjs": "JavaScript", "js": "JavaScript", "jsx": "JavaScript", "mjs": "JavaScript", + "json": "JSON", + "kt": "Kotlin", "kts": "Kotlin", + "lua": "Lua", + "m": "Objective-C", "mm": "Objective-C", + "markdown": "Markdown", "md": "Markdown", + "php": "PHP", + "py": "Python", "pyi": "Python", + "rb": "Ruby", + "rs": "Rust", + "bash": "a shell script", "sh": "a shell script", "zsh": "a shell script", + "sql": "SQL", + "swift": "Swift", + "toml": "TOML", + "ts": "TypeScript", "tsx": "TypeScript", + "yaml": "YAML", "yml": "YAML", + ] +} diff --git a/Sources/BlurtEngine/STT/TranscriptionContext.swift b/Sources/BlurtEngine/STT/TranscriptionContext.swift index 5a5fc3e..c3735ef 100644 --- a/Sources/BlurtEngine/STT/TranscriptionContext.swift +++ b/Sources/BlurtEngine/STT/TranscriptionContext.swift @@ -11,6 +11,14 @@ public struct TranscriptionContext: Sendable, Equatable { /// as a domain/topic hint so the model expects that app's vocabulary. public let appName: String? + /// The frontmost application's bundle identifier (e.g. + /// "com.tinyspeck.slackmacgap"). Never rendered verbatim: it keys the + /// app-*kind* recognition (`AppKindPriming`) that adds destination-specific + /// guidance — terminal, code editor, Slack, Obsidian — to the prompt. + /// Preferred over `appName` for recognition because display names are + /// localized and user-editable while the bundle ID is stable. + public let bundleID: String? + /// The focused window's title (e.g. "Re: Q3 pricing — Gmail", a document /// name, a Slack channel). The densest topic hint available — usually packed /// with the proper nouns and domain vocabulary the model would otherwise guess. @@ -40,6 +48,7 @@ public struct TranscriptionContext: Sendable, Equatable { public init( appName: String?, + bundleID: String? = nil, windowTitle: String? = nil, fieldLabel: String? = nil, priorText: String?, @@ -47,6 +56,7 @@ public struct TranscriptionContext: Sendable, Equatable { keyTerms: [String] = [] ) { self.appName = appName + self.bundleID = bundleID self.windowTitle = windowTitle self.fieldLabel = fieldLabel self.priorText = priorText @@ -58,7 +68,7 @@ public struct TranscriptionContext: Sendable, Equatable { /// are no key terms. public var isEmpty: Bool { keyTerms.isEmpty - && [appName, windowTitle, fieldLabel, priorText, selectedText].allSatisfy { + && [appName, bundleID, windowTitle, fieldLabel, priorText, selectedText].allSatisfy { $0.trimmedNonEmpty() == nil } } diff --git a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift index 9339c54..9d72972 100644 --- a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift +++ b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift @@ -5,7 +5,9 @@ /// Every built prompt opens with a fixed `baseInstruction` — a plain-text /// exclusion clause (see below) — and wraps it in /// *contextual* priming: a topic hint built from the window title, a -/// destination sentence built from the focused app and field label, "prior +/// destination sentence built from the focused app and field label, an +/// app-kind guidance sentence recognized from the frontmost app's bundle ID +/// (`AppKindPriming` — terminal, code editor, Slack, Obsidian), "prior /// chunk context" (the text preceding the cursor), the selected text (which the /// dictation replaces), and keyword boosting, all of which the model is /// mid-trained to use for better recognition accuracy. @@ -70,7 +72,8 @@ enum TranscriptionPrompt { // 1. the prior-chunk block (`Previous transcript:\n…`, its own paragraph), // 2. the selected-text block (`Selected text:\n…`, what the dictation // replaces — primes vocabulary/topic of the text being rewritten), - // 3. the location clause (topic hint + destination sentence). + // 3. the location clause (topic hint + destination sentence + app-kind + // guidance). var blocks: [String] = [] if !prior.isEmpty { blocks.append("Previous transcript:\n\(prior)") @@ -78,7 +81,14 @@ enum TranscriptionPrompt { if !selected.isEmpty { blocks.append("Selected text:\n\(selected)") } - let location = locationClause(app: app, window: window, field: field) + // App-kind guidance ("You are dictating into a terminal …") follows the + // destination sentence: recognized from the bundle ID, refined by the + // window title (a code editor's title names the open file, hence the + // language). Unrecognized apps add nothing. + let guidance = AppKindPriming.clause(bundleID: context.bundleID, windowTitle: window) ?? "" + let location = [locationClause(app: app, window: window, field: field), guidance] + .filter { !$0.isEmpty } + .joined(separator: " ") // The topic hint and `baseInstruction` share one line as the trained // `{context}. {baseInstruction}` shape; with no topic it's the bare base. let instruction = location.isEmpty ? baseInstruction : "\(location) \(baseInstruction)" diff --git a/Tests/BlurtEngineTests/AppKindPrimingTests.swift b/Tests/BlurtEngineTests/AppKindPrimingTests.swift new file mode 100644 index 0000000..6146aba --- /dev/null +++ b/Tests/BlurtEngineTests/AppKindPrimingTests.swift @@ -0,0 +1,114 @@ +import Testing + +@testable import BlurtEngine + +@Suite("AppKindPriming") +struct AppKindPrimingTests { + // MARK: - Kind recognition + + /// One bundle-ID → kind expectation, tabled for per-case failure output like + /// `TranscriptionPromptTests.Case`. + struct KindCase: Sendable, CustomTestStringConvertible { + let bundleID: String? + let expected: AppKindPriming.Kind? + var testDescription: String { bundleID ?? "nil" } + } + + static let kindCases: [KindCase] = [ + KindCase(bundleID: "com.apple.Terminal", expected: .terminal), + KindCase(bundleID: "com.googlecode.iterm2", expected: .terminal), + KindCase(bundleID: "com.mitchellh.ghostty", expected: .terminal), + KindCase(bundleID: "com.microsoft.VSCode", expected: .codeEditor), + KindCase(bundleID: "com.apple.dt.Xcode", expected: .codeEditor), + KindCase(bundleID: "com.todesktop.230313mzl4w4u92", expected: .codeEditor), + // Prefix families: any JetBrains IDE, any Sublime Text major version. + KindCase(bundleID: "com.jetbrains.pycharm", expected: .codeEditor), + KindCase(bundleID: "com.sublimetext.4", expected: .codeEditor), + KindCase(bundleID: "com.tinyspeck.slackmacgap", expected: .slack), + KindCase(bundleID: "md.obsidian", expected: .obsidian), + KindCase(bundleID: "com.apple.mail", expected: nil), + KindCase(bundleID: "", expected: nil), + KindCase(bundleID: " ", expected: nil), + KindCase(bundleID: nil, expected: nil), + ] + + @Test("bundle IDs map to their app kind", arguments: kindCases) + func kind(_ c: KindCase) { + #expect(AppKindPriming.kind(ofBundleID: c.bundleID) == c.expected) + } + + // MARK: - Language inference from window titles + + /// One window-title → language expectation. + struct LanguageCase: Sendable, CustomTestStringConvertible { + let title: String + let expected: String? + var testDescription: String { title } + } + + static let languageCases: [LanguageCase] = [ + LanguageCase(title: "main.py — blurt — Visual Studio Code", expected: "Python"), + // VS Code prepends ● to the filename of a dirty buffer. + LanguageCase(title: "● api.ts — server", expected: "TypeScript"), + LanguageCase(title: "blurt — TranscriptionPrompt.swift", expected: "Swift"), + LanguageCase(title: "deploy.sh — infra", expected: "a shell script"), + // Extension matching is case-insensitive. + LanguageCase(title: "README.MD — notes", expected: "Markdown"), + // First filename wins when several tokens carry extensions. + LanguageCase(title: "index.js next.config.ts", expected: "JavaScript"), + // A dotfile has no base name + extension split. + LanguageCase(title: ".zshrc — dotfiles", expected: nil), + // Version numbers and hostnames are not filenames. + LanguageCase(title: "release v0.1.34 — dictation.assemblyai.com", expected: nil), + LanguageCase(title: "Untitled-1", expected: nil), + LanguageCase(title: "", expected: nil), + ] + + @Test("window titles yield the open file's language", arguments: languageCases) + func language(_ c: LanguageCase) { + #expect(AppKindPriming.language(inWindowTitle: c.title) == c.expected) + } + + // MARK: - Clause rendering + + @Test("a terminal renders the command-line clause") + func terminalClause() { + #expect( + AppKindPriming.clause(bundleID: "com.apple.Terminal", windowTitle: nil) + == "You are dictating into a terminal: expect shell commands, program names, flags, and file paths.") + } + + @Test("a code editor names the open file's language when the title carries one") + func codeEditorClauseWithLanguage() { + #expect( + AppKindPriming.clause(bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt") + == "You are writing Python in a code editor: expect code identifiers, symbols, and technical terms.") + } + + @Test("a code editor stays generic when the title names no recognizable file") + func codeEditorClauseGeneric() { + #expect( + AppKindPriming.clause(bundleID: "com.apple.dt.Xcode", windowTitle: "Welcome to Xcode") + == "You are writing code in a code editor: expect code identifiers, symbols, and technical terms.") + } + + @Test("Slack renders the casual-chat clause") + func slackClause() { + #expect( + AppKindPriming.clause(bundleID: "com.tinyspeck.slackmacgap", windowTitle: "#eng-backend") + == "You are writing a Slack message: casual tone and emoji are expected.") + } + + @Test("Obsidian renders the Markdown clause") + func obsidianClause() { + #expect( + AppKindPriming.clause(bundleID: "md.obsidian", windowTitle: "Meeting notes") + == "You are writing a Markdown note in Obsidian: Markdown syntax is expected.") + } + + @Test("an unrecognized app contributes no clause") + func unrecognizedApp() { + #expect(AppKindPriming.clause(bundleID: "com.apple.mail", windowTitle: "Re: Q3") == nil) + #expect(AppKindPriming.clause(bundleID: nil, windowTitle: "main.py") == nil) + } +} diff --git a/Tests/BlurtEngineTests/TranscriptionContextTests.swift b/Tests/BlurtEngineTests/TranscriptionContextTests.swift index deef980..624ad30 100644 --- a/Tests/BlurtEngineTests/TranscriptionContextTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionContextTests.swift @@ -28,6 +28,13 @@ struct TranscriptionContextTests { #expect(!TranscriptionContext(appName: nil, priorText: "hello there").isEmpty) } + @Test("a bundle ID alone makes it non-empty (and produces a prompt)") + func bundleIDPresent() { + let context = TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil) + #expect(!context.isEmpty) + #expect(TranscriptionPrompt.build(context: context) != nil) + } + @Test("real selected text makes it non-empty") func selectedTextPresent() { #expect(!TranscriptionContext(appName: nil, priorText: nil, selectedText: "highlighted").isEmpty) @@ -89,5 +96,8 @@ struct TranscriptionContextTests { #expect( TranscriptionContext(appName: "Notes", priorText: "x", keyTerms: ["a"]) != TranscriptionContext(appName: "Notes", priorText: "x", keyTerms: ["b"])) + #expect( + TranscriptionContext(appName: "Notes", bundleID: "com.a.b", priorText: "x") + != TranscriptionContext(appName: "Notes", bundleID: "com.c.d", priorText: "x")) } } diff --git a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift index 260465a..6380d27 100644 --- a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift @@ -94,6 +94,30 @@ struct TranscriptionPromptTests { expected: "Previous transcript:\nthanks for\n\nSelected text:\nthe draft\n\nDictated into Slack. \(base) Keywords: Blurt." ), + Case( + name: "recognized bundle ID → app-kind guidance after the destination", + context: TranscriptionContext( + appName: "Slack", bundleID: "com.tinyspeck.slackmacgap", fieldLabel: "Message", priorText: nil), + expected: + "Dictated into Slack, in the \"Message\" field. You are writing a Slack message: casual tone and emoji are expected. \(base)" + ), + Case( + name: "bundle ID alone → guidance still built", + context: TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil), + expected: + "You are dictating into a terminal: expect shell commands, program names, flags, and file paths. \(base)" + ), + Case( + name: "code-editor guidance names the window title's language", + context: TranscriptionContext( + appName: "Code", bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt", priorText: nil), + expected: + "This is about \"main.py — blurt\". Dictated into Code. You are writing Python in a code editor: expect code identifiers, symbols, and technical terms. \(base)" + ), + Case( + name: "unrecognized bundle ID adds no guidance", + context: TranscriptionContext(appName: "Mail", bundleID: "com.apple.mail", priorText: nil), + expected: "Dictated into Mail. \(base)"), Case( name: "key terms only → inline keyword boost", context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["AssemblyAI", "Kubernetes"]), From 7c2978ec324cc751d8e0df8400df2f99cb190cb8 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 28 Jul 2026 21:27:23 +0000 Subject: [PATCH 2/8] feat(log): record the bundle ID behind the logged dictation prompt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DictationLog already writes the fully-assembled config.prompt with every entry, so the new app-kind guidance reaches the corpus automatically. Complete the context snapshot by also logging the frontmost app's bundle identifier — the input AppKindPriming keys on — so the log shows why a prompt carried (or lacked) a guidance sentence, and pin with a test that the guidance actually lands in the logged prompt. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01R28wj6vXZDW3iS7FWpHT1U --- .../BlurtEngine/Pipeline/DictationLog.swift | 7 ++++++- .../BlurtEngineTests/DictationLogTests.swift | 20 +++++++++++++++++-- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/Sources/BlurtEngine/Pipeline/DictationLog.swift b/Sources/BlurtEngine/Pipeline/DictationLog.swift index 15c87f5..c4b48e3 100644 --- a/Sources/BlurtEngine/Pipeline/DictationLog.swift +++ b/Sources/BlurtEngine/Pipeline/DictationLog.swift @@ -12,6 +12,10 @@ public enum DictationLog { let ts: String /// Focused-app topic hint sent as context, when one was captured. let app: String? + /// Focused-app bundle identifier, when one was captured. The input the + /// prompt's app-kind guidance (`AppKindPriming`) keys on, logged so the + /// corpus shows *why* a prompt carried (or lacked) a guidance sentence. + let bundle: String? /// Focused-window title sent as a topic hint, when one was captured. let window: String? /// Focused-field label sent as context, when one was captured. @@ -90,7 +94,8 @@ public enum DictationLog { ) { let entry = Entry( transcript: transcript, ts: now.formatted(timestampFormat), - app: context?.appName, window: context?.windowTitle, field: context?.fieldLabel, + app: context?.appName, bundle: context?.bundleID, + window: context?.windowTitle, field: context?.fieldLabel, prior: context?.priorText, selected: context?.selectedText, prompt: TranscriptionPrompt.build(context: context)) guard var line = try? makeEncoder().encode(entry) else { return } diff --git a/Tests/BlurtEngineTests/DictationLogTests.swift b/Tests/BlurtEngineTests/DictationLogTests.swift index 163fabb..ce8207b 100644 --- a/Tests/BlurtEngineTests/DictationLogTests.swift +++ b/Tests/BlurtEngineTests/DictationLogTests.swift @@ -12,6 +12,7 @@ private struct DecodedEntry: Decodable { /// threaded from the `TranscriptionContext` onto disk. private struct DecodedContext: Decodable { let app: String? + let bundle: String? let window: String? let field: String? let prior: String? @@ -96,12 +97,13 @@ struct DictationLogTests { func logsContext() { let url = makeTempLogURL() let context = TranscriptionContext( - appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", - priorText: "Hi Sam,", selectedText: "the old plan") + appName: "Mail", bundleID: "com.apple.mail", windowTitle: "Re: Q3 pricing", + fieldLabel: "Body", priorText: "Hi Sam,", selectedText: "the old plan") DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) #expect(decoded?.app == "Mail") + #expect(decoded?.bundle == "com.apple.mail") #expect(decoded?.window == "Re: Q3 pricing") #expect(decoded?.field == "Body") #expect(decoded?.prior == "Hi Sam,") @@ -130,6 +132,20 @@ struct DictationLogTests { #expect(decoded?.prompt == TranscriptionPrompt.build(context: context)) } + @Test("the logged prompt carries the app-kind guidance the bundle ID selected") + func logsPromptWithAppKindGuidance() { + let url = makeTempLogURL() + let context = TranscriptionContext( + appName: "Slack", bundleID: "com.tinyspeck.slackmacgap", fieldLabel: "Message", + priorText: nil) + DictationLog.write(transcript: "p", context: context, to: url, now: Date()) + let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" + let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) + // The exact wording is TranscriptionPromptTests' contract; here the point is + // that what lands on disk includes the guidance actually sent for this app. + #expect(decoded?.prompt?.contains("You are writing a Slack message") == true) + } + @Test("omits the prompt field when there is no context to build one") func omitsPromptWhenNoContext() { let url = makeTempLogURL() From 57b657ca46d3703c14ec6cff5f76846cc476a4f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 29 Jul 2026 00:54:33 +0000 Subject: [PATCH 3/8] feat(stt): drop the annotation-suppression clause from the prompt The dictation service's own default prompt already carries "Transcribe without speaker labels, audio event descriptions, or emotion markers.", so restating it as baseInstruction on every request only spent budget from the 4096-character prompt cap. The built prompt is now contextual priming only: prior text, selected text, the location clause (topic + destination + app-kind guidance), and trailing keyword boosting. A context that renders no text (e.g. its only signal is an unrecognized bundle ID) now collapses to nil so the server default still applies, pinned by new test cases alongside the rewritten prompt expectations. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01R28wj6vXZDW3iS7FWpHT1U --- AGENTS.md | 17 ++-- BLURTENGINE.md | 2 +- .../Pipeline/DictationSession+Pipeline.swift | 3 +- Sources/BlurtEngine/STT/AppKindPriming.swift | 4 +- .../BlurtEngine/STT/TranscriptionPrompt.swift | 92 +++++++++---------- .../TranscriptionPromptTests.swift | 73 +++++++++------ 6 files changed, 97 insertions(+), 94 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5676b8e..487f76f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -379,12 +379,12 @@ to right ⌘), so views must not re-declare `TriggerKey.rightCommand.rawValue` t request's `config.prompt` — it steers the _transcription_, not the LLM rewrite (that's the request's separate `llm` block). It's unit-tested in `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. -Every built prompt opens with the fixed `baseInstruction` — _"Transcribe without speaker labels, -audio event descriptions, or emotion markers."_ — a negative-exclusion clause that suppresses the -annotation markers (`[Speaker]`, `[door creaks]`, `[laughing]`) the model would otherwise paste into -the user's text. +The built prompt is _contextual priming only_ — no standing instruction opens it. The +annotation-suppression clause that once did (_"Transcribe without speaker labels, audio event +descriptions, or emotion markers."_) is already part of the dictation service's own default prompt, +so restating it client-side only spent budget from the 4096-character cap. -`build(context:)` wraps that pivot in _contextual priming_: prior-cursor text; the selected text (the +`build(context:)` assembles: prior-cursor text; the selected text (the highlighted run the dictation will replace, so the model is primed on what's being rewritten — read via `kAXSelectedTextAttribute`, skipped in secure fields, detected by AX role **or** subrole and failing closed when the role can't be read, so a password can't reach the prompt); a topic hint from @@ -396,9 +396,10 @@ with emoji, or Markdown); and inline keyword boosting from the user's key terms. (positive/authoritative wording, no "Don't"/"Avoid"/"Never") and stays under the dictation API's documented 4096-character cap on `config.prompt` (`characterCap`). -`build(context:)` returns `nil` when there's no usable context, and passing `prompt: nil` to the -transcriber omits the field so the server applies its own default. Two omissions are deliberate and -regression-tested — no language directive and no filler-word clause; see +`build(context:)` returns `nil` when there's no usable context (or when the context renders no +text, e.g. only an unrecognized bundle ID), and passing `prompt: nil` to the transcriber omits the +field so the server applies its own default. Two omissions are deliberate and regression-tested — +no language directive and no filler-word clause; see [Settled decisions](#settled-decisions--dont-reintroduce-these). ## Settings, persistence, and cues diff --git a/BLURTENGINE.md b/BLURTENGINE.md index ae656da..1ed4847 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -153,7 +153,7 @@ The session calls `setTargetApp` at press time with the app that was frontmost w Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected: - **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. -- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block), opening with the fixed `baseInstruction` ("Transcribe without speaker labels, audio event descriptions, or emotion markers.") and staying under the API's 4096-character cap. When the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian), the prompt adds one app-kind guidance sentence, e.g. "You are dictating into a terminal: expect shell commands, program names, flags, and file paths."; for code editors it names the language inferred from the window title's filename ("You are writing Python …"). Unrecognized apps add nothing. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. +- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block). The prompt is contextual priming only, staying under the API's 4096-character cap; the standing annotation-suppression clause ("Transcribe without speaker labels, …") is part of the service's own default prompt and is not restated. When the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian), the prompt adds one app-kind guidance sentence, e.g. "You are dictating into a terminal: expect shell commands, program names, flags, and file paths."; for code editors it names the language inferred from the window title's filename ("You are writing Python …"). Unrecognized apps add nothing. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. - **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere. For key storage, compose against **`APIKeyGateway`** — the injectable `current` / `save(_:)` / `hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. Two projections keep the surrounding UI out of your views: `Outcome.failureReport` classifies a failure as `.inline(message:)` (recoverable — show it beside the field) or `.alert(title:message:)` (a Keychain fault retyping can't fix), and **`APIKeyDisplay.resolve(key:)`** renders the stored key for an account row — masked tail, status and VoiceOver wording, and the connect-vs-rotate control titles. The mask reveals only the last `revealedTailLength` characters and, below `minimumLengthToMask`, none at all, so a short key can't be shown whole. diff --git a/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift b/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift index eb830eb..63005f8 100644 --- a/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift +++ b/Sources/BlurtEngine/Pipeline/DictationSession+Pipeline.swift @@ -39,8 +39,7 @@ extension DictationSession { // cancelled pipeline clear a *newer* press's stream: `cancel()` detaches this // task while it's parked in `firstValue`, a fresh `press()` installs its own // `contextStream`, and this task's resumption then nils that one out — so - // dictation #2 transcribes with `context: nil`, losing not just its priming but - // `baseInstruction`, and `[Speaker]`-style markers can reach the pasted text. + // dictation #2 transcribes with `context: nil`, silently losing its priming. // The window is microseconds, but the invariant is now local instead of // depending on scheduling. let stream = contextStream diff --git a/Sources/BlurtEngine/STT/AppKindPriming.swift b/Sources/BlurtEngine/STT/AppKindPriming.swift index d67ef4a..0fe0a78 100644 --- a/Sources/BlurtEngine/STT/AppKindPriming.swift +++ b/Sources/BlurtEngine/STT/AppKindPriming.swift @@ -3,8 +3,8 @@ import Foundation /// App-kind guidance for the transcription prompt: recognizes what *kind* of /// app the dictation targets (a terminal, a code editor, Slack, Obsidian) from /// the frontmost app's bundle identifier and renders one priming sentence for -/// `TranscriptionPrompt` to place before `baseInstruction`. The sentence tells -/// the model what shape of text the destination expects — shell commands in a +/// `TranscriptionPrompt` to place after the destination sentence. The sentence +/// tells the model what shape of text the destination expects — shell commands in a /// terminal, identifiers and symbols in an editor, casual chat in Slack, /// Markdown in Obsidian — which the app's display name alone doesn't convey. /// diff --git a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift index 9d72972..67b63f8 100644 --- a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift +++ b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift @@ -2,48 +2,35 @@ /// field of the request `config` (see `AssemblyAITranscriber`). The STT model /// prepends this to its own system prompt. /// -/// Every built prompt opens with a fixed `baseInstruction` — a plain-text -/// exclusion clause (see below) — and wraps it in -/// *contextual* priming: a topic hint built from the window title, a -/// destination sentence built from the focused app and field label, an -/// app-kind guidance sentence recognized from the frontmost app's bundle ID -/// (`AppKindPriming` — terminal, code editor, Slack, Obsidian), "prior -/// chunk context" (the text preceding the cursor), the selected text (which the -/// dictation replaces), and keyword boosting, all of which the model is -/// mid-trained to use for better recognition accuracy. +/// The built prompt is *contextual priming only*: a topic hint built from the +/// window title, a destination sentence built from the focused app and field +/// label, an app-kind guidance sentence recognized from the frontmost app's +/// bundle ID (`AppKindPriming` — terminal, code editor, Slack, Obsidian), +/// "prior chunk context" (the text preceding the cursor), the selected text +/// (which the dictation replaces), and keyword boosting, all of which the +/// model is mid-trained to use for better recognition accuracy. /// -/// On the directives in `baseInstruction`: a "remove filler words"-style -/// *content* reshaping is **not** in the model's trained instruction set, so it -/// is a no-op and is deliberately omitted (see the project memory note). A -/// language directive is likewise omitted — pinning the prompt to English hurt -/// non-English transcription, so language is left to the model's own detection. -/// The negative feature *exclusion* ("without speaker labels, …") is a trained -/// instruction-following type, so it does take effect — the exclusion -/// suppresses the annotation markers the model would otherwise emit (`[Speaker]`, -/// `[door creaks]`, `[laughing]`, …), which in a dictation product would be -/// pasted into the user's text as literal tokens. The list is trimmed to the -/// three annotation types a dictation user could plausibly trigger; the rarer -/// types (unclear-speech, censor, foreign-language, lyrics) are left out to keep -/// the negative clause short, matching the doc's brief negative examples. +/// Three standing directives are deliberately *absent*: +/// - No annotation-suppression clause ("Transcribe without speaker labels, +/// audio event descriptions, or emotion markers."): the dictation service +/// already includes it in its own default prompt, so restating it here only +/// spent budget from `characterCap`. +/// - No "remove filler words"-style *content* reshaping: not in the model's +/// trained instruction set, so it is a no-op and is deliberately omitted +/// (see the project memory note); disfluency removal is the server-side LLM +/// rewrite's job. +/// - No language directive: pinning the prompt to English hurt non-English +/// transcription, so language is left to the model's own detection. /// -/// Output follows the trained format with `baseInstruction` as its pivot: the -/// prior-chunk context, the topic hint, and the destination sentence precede it -/// as the `{context}. {baseInstruction}` shape, and keyword boosting trails it -/// inline as `Keywords: a, b, c.` (per the mid-training instruction-type -/// reference). It stays under the API's `characterCap`: the contextual -/// blocks are clipped upstream in `FocusCapture`, and the key-terms clause is -/// fitted to the remaining budget here. Exercised by +/// Output follows the trained format: the prior-chunk context and the selected +/// text lead as their own paragraphs, the location clause (topic hint + +/// destination sentence + app-kind guidance) follows, and keyword boosting +/// trails inline as `Keywords: a, b, c.` (per the mid-training +/// instruction-type reference). It stays under the API's `characterCap`: the +/// contextual blocks are clipped upstream in `FocusCapture`, and the key-terms +/// clause is fitted to the remaining budget here. Exercised by /// `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. enum TranscriptionPrompt { - /// The standing dictation instruction prepended to the model's system prompt - /// on every built prompt. A negative-exclusion clause (§5/§6) naming the - /// annotation feature types the model is trained to emit, so it suppresses - /// them. No language directive: pinning the prompt to English degraded - /// transcription for non-English speech, so the model is left to detect the - /// spoken language itself. - static let baseInstruction = - "Transcribe without speaker labels, audio event descriptions, or emotion markers." - /// Hard cap the dictation API places on `config.prompt` ("max 4096 chars"); /// a longer prompt risks failing the whole request, so `build` must never /// exceed it. The contextual blocks are all clipped upstream in @@ -66,9 +53,8 @@ enum TranscriptionPrompt { let field = context.fieldLabel.trimmedNonEmpty() ?? "" let keyTerms = context.keyTerms - // `baseInstruction` is the pivot of the trained format. Contextual priming - // sits *before* it; keyword boosting trails *after* it. The leading blocks, - // separated by blank lines, precede it: + // The priming blocks, separated by blank lines; keyword boosting trails + // the last one inline: // 1. the prior-chunk block (`Previous transcript:\n…`, its own paragraph), // 2. the selected-text block (`Selected text:\n…`, what the dictation // replaces — primes vocabulary/topic of the text being rewritten), @@ -89,21 +75,21 @@ enum TranscriptionPrompt { let location = [locationClause(app: app, window: window, field: field), guidance] .filter { !$0.isEmpty } .joined(separator: " ") - // The topic hint and `baseInstruction` share one line as the trained - // `{context}. {baseInstruction}` shape; with no topic it's the bare base. - let instruction = location.isEmpty ? baseInstruction : "\(location) \(baseInstruction)" - blocks.append(instruction) + if !location.isEmpty { + blocks.append(location) + } var prompt = blocks.joined(separator: "\n\n") if !keyTerms.isEmpty { // Spelling priming: the user's domain vocabulary, boosted via the trained - // inline `Keywords: a, b, c.` form (Section 2.3) trailing the marker so the + // inline `Keywords: a, b, c.` form (Section 2.3) trailing the context so the // model favors these exact spellings for names/jargon it would guess at. // The terms list is the one input with no upstream length cap, so include // only as many whole terms as `characterCap` leaves room for, so a huge - // Settings list can't crowd out the instruction itself or balloon every + // Settings list can't crowd out the priming itself or balloon every // request. + let scaffold = prompt.isEmpty ? "Keywords: ." : " Keywords: ." var included: [String] = [] - var remaining = characterCap - prompt.count - " Keywords: .".count + var remaining = characterCap - prompt.count - scaffold.count for term in keyTerms { let cost = term.count + (included.isEmpty ? 0 : ", ".count) guard cost <= remaining else { break } @@ -111,10 +97,14 @@ enum TranscriptionPrompt { remaining -= cost } if !included.isEmpty { - prompt += " Keywords: \(included.joined(separator: ", "))." + let clause = "Keywords: \(included.joined(separator: ", "))." + prompt = prompt.isEmpty ? clause : "\(prompt) \(clause)" } } - return prompt + // A context can be non-empty yet render nothing — e.g. its only signal is + // a bundle ID no app-kind guidance recognizes. Return nil rather than an + // empty prompt so the server default still applies. + return prompt.trimmedNonEmpty() } /// The "where am I typing" priming clause, assembled from whichever of the @@ -123,7 +113,7 @@ enum TranscriptionPrompt { /// richest vocabulary signal — `This is about "…".`, mid-training §2.1) leads, /// and a destination sentence built from the app/field (`Dictated into …`) /// trails it. Each sentence ends with a period so the clause joins cleanly - /// before `baseInstruction`. + /// with the app-kind guidance that may follow it. private static func locationClause(app: String, window: String, field: String) -> String { let topic = window.isEmpty ? "" : "This is about \"\(window)\"." diff --git a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift index 6380d27..34470e6 100644 --- a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift @@ -4,15 +4,14 @@ import Testing @Suite("TranscriptionPrompt") struct TranscriptionPromptTests { - /// The standing plain-text exclusion clause that every built prompt carries - /// (see `TranscriptionPrompt.baseInstruction`). Kept here as the single source - /// of truth so the expectations below read clearly. - static let base = - "Transcribe without speaker labels, audio event descriptions, or emotion markers." - /// One `build(context:)` → prompt expectation. Parameterizing these (rather /// than a `@Test` apiece) keeps the whole context→prompt contract in one /// readable table and gives per-case failure output. + /// + /// The prompt is contextual priming only — no standing instruction opens it. + /// The annotation-suppression clause ("Transcribe without speaker labels, …") + /// is part of the dictation service's own default prompt, so a case's + /// expectation is exactly its context rendered, nothing more. struct Case: Sendable, CustomTestStringConvertible { let name: String let context: TranscriptionContext? @@ -31,105 +30,109 @@ struct TranscriptionPromptTests { Case( name: "app only → destination sentence", context: TranscriptionContext(appName: "Slack", priorText: nil), - expected: "Dictated into Slack. \(base)"), + expected: "Dictated into Slack."), Case( name: "prior only → Previous transcript framing", context: TranscriptionContext(appName: nil, priorText: "and then the build finished"), - expected: "Previous transcript:\nand then the build finished\n\n\(base)"), + expected: "Previous transcript:\nand then the build finished"), Case( name: "app + window → topic hint leads, destination trails", context: TranscriptionContext(appName: "Mail", windowTitle: "Re: Q3 pricing", priorText: nil), - expected: "This is about \"Re: Q3 pricing\". Dictated into Mail. \(base)"), + expected: "This is about \"Re: Q3 pricing\". Dictated into Mail."), Case( name: "window only → bare topic hint", context: TranscriptionContext(appName: nil, windowTitle: "Untitled.txt", priorText: nil), - expected: "This is about \"Untitled.txt\". \(base)"), + expected: "This is about \"Untitled.txt\"."), Case( name: "field only → destination sentence", context: TranscriptionContext(appName: nil, fieldLabel: "Search", priorText: nil), - expected: "Dictated in the \"Search\" field. \(base)"), + expected: "Dictated in the \"Search\" field."), Case( name: "app + field without window → destination names both", context: TranscriptionContext(appName: "Slack", fieldLabel: "Message", priorText: nil), - expected: "Dictated into Slack, in the \"Message\" field. \(base)"), + expected: "Dictated into Slack, in the \"Message\" field."), Case( name: "all four signals combine", context: TranscriptionContext( appName: "Slack", windowTitle: "#eng-backend", fieldLabel: "Message", priorText: "thanks for"), expected: - "Previous transcript:\nthanks for\n\nThis is about \"#eng-backend\". Dictated into Slack, in the \"Message\" field. \(base)" + "Previous transcript:\nthanks for\n\nThis is about \"#eng-backend\". Dictated into Slack, in the \"Message\" field." ), Case( name: "prior + app combine", context: TranscriptionContext(appName: "Mail", priorText: "Dear Sam,"), - expected: "Previous transcript:\nDear Sam,\n\nDictated into Mail. \(base)"), + expected: "Previous transcript:\nDear Sam,\n\nDictated into Mail."), Case( name: "prior + app are trimmed", context: TranscriptionContext(appName: " Notes ", priorText: " hello "), - expected: "Previous transcript:\nhello\n\nDictated into Notes. \(base)"), + expected: "Previous transcript:\nhello\n\nDictated into Notes."), Case( name: "selected only → Selected text framing", context: TranscriptionContext(appName: nil, priorText: nil, selectedText: "the quarterly numbers"), - expected: "Selected text:\nthe quarterly numbers\n\n\(base)"), + expected: "Selected text:\nthe quarterly numbers"), Case( name: "selected follows prior as its own block", context: TranscriptionContext(appName: nil, priorText: "as we discussed,", selectedText: "the old plan"), - expected: "Previous transcript:\nas we discussed,\n\nSelected text:\nthe old plan\n\n\(base)"), + expected: "Previous transcript:\nas we discussed,\n\nSelected text:\nthe old plan"), Case( name: "selected + location + prior combine", context: TranscriptionContext( appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", priorText: "Hi Sam,", selectedText: "let's push the date"), expected: - "Previous transcript:\nHi Sam,\n\nSelected text:\nlet's push the date\n\nThis is about \"Re: Q3 pricing\". Dictated into Mail, in the \"Body\" field. \(base)" + "Previous transcript:\nHi Sam,\n\nSelected text:\nlet's push the date\n\nThis is about \"Re: Q3 pricing\". Dictated into Mail, in the \"Body\" field." ), Case( name: "blank selected adds no block", context: TranscriptionContext(appName: "Notes", priorText: nil, selectedText: " \n"), - expected: "Dictated into Notes. \(base)"), + expected: "Dictated into Notes."), Case( name: "selected sits between prior and keyword boost", context: TranscriptionContext( appName: "Slack", priorText: "thanks for", selectedText: "the draft", keyTerms: ["Blurt"]), expected: - "Previous transcript:\nthanks for\n\nSelected text:\nthe draft\n\nDictated into Slack. \(base) Keywords: Blurt." + "Previous transcript:\nthanks for\n\nSelected text:\nthe draft\n\nDictated into Slack. Keywords: Blurt." ), Case( name: "recognized bundle ID → app-kind guidance after the destination", context: TranscriptionContext( appName: "Slack", bundleID: "com.tinyspeck.slackmacgap", fieldLabel: "Message", priorText: nil), expected: - "Dictated into Slack, in the \"Message\" field. You are writing a Slack message: casual tone and emoji are expected. \(base)" + "Dictated into Slack, in the \"Message\" field. You are writing a Slack message: casual tone and emoji are expected." ), Case( name: "bundle ID alone → guidance still built", context: TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil), expected: - "You are dictating into a terminal: expect shell commands, program names, flags, and file paths. \(base)" + "You are dictating into a terminal: expect shell commands, program names, flags, and file paths." ), Case( name: "code-editor guidance names the window title's language", context: TranscriptionContext( appName: "Code", bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt", priorText: nil), expected: - "This is about \"main.py — blurt\". Dictated into Code. You are writing Python in a code editor: expect code identifiers, symbols, and technical terms. \(base)" + "This is about \"main.py — blurt\". Dictated into Code. You are writing Python in a code editor: expect code identifiers, symbols, and technical terms." ), Case( name: "unrecognized bundle ID adds no guidance", context: TranscriptionContext(appName: "Mail", bundleID: "com.apple.mail", priorText: nil), - expected: "Dictated into Mail. \(base)"), + expected: "Dictated into Mail."), + Case( + name: "unrecognized bundle ID with no other signal → no prompt", + context: TranscriptionContext(appName: nil, bundleID: "com.example.mystery", priorText: nil), + expected: nil), Case( - name: "key terms only → inline keyword boost", + name: "key terms only → bare keyword boost", context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["AssemblyAI", "Kubernetes"]), - expected: "\(base) Keywords: AssemblyAI, Kubernetes."), + expected: "Keywords: AssemblyAI, Kubernetes."), Case( - name: "key terms trail base alongside focus context", + name: "key terms trail the focus context inline", context: TranscriptionContext(appName: "Slack", priorText: nil, keyTerms: ["Blurt"]), - expected: "Dictated into Slack. \(base) Keywords: Blurt."), + expected: "Dictated into Slack. Keywords: Blurt."), Case( name: "empty key terms add no clause", context: TranscriptionContext(appName: "Notes", priorText: nil, keyTerms: []), - expected: "Dictated into Notes. \(base)"), + expected: "Dictated into Notes."), ] @Test("build maps focus context to the transcription prompt", arguments: cases) @@ -153,7 +156,17 @@ struct TranscriptionPromptTests { let huge = String(repeating: "k", count: TranscriptionPrompt.characterCap) let prompt = TranscriptionPrompt.build( context: TranscriptionContext(appName: "Xcode", priorText: nil, keyTerms: [huge])) - #expect(prompt == "Dictated into Xcode. \(Self.base)") + #expect(prompt == "Dictated into Xcode.") + } + + @Test("a key-terms-only prompt whose first term doesn't fit yields no prompt at all") + func keyTermsOnlyNoneFit() { + // With no other context and no term fitting the cap, nothing renders — and + // an empty prompt must collapse to nil so the server default applies. + let huge = String(repeating: "k", count: TranscriptionPrompt.characterCap) + let prompt = TranscriptionPrompt.build( + context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: [huge])) + #expect(prompt == nil) } @Test("an oversized key-terms list is fitted to the cap, keeping whole leading terms") From 49e643a2e0900fedb03b1b5f4f12c9f3d9380204 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 29 Jul 2026 01:19:55 +0000 Subject: [PATCH 4/8] feat(stt): send only the app-kind instruction and key terms as the prompt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Real-world dictation logs showed the contextual blocks crowding the instruction — a VS Code dictation carried the Monaco screen-reader help announcement as its "field", plus topic and destination sentences ahead of the actual instruction. Strip config.prompt down to two optional clauses: the app-kind instruction from AppKindPriming, reworded to the "Transcribe speech into ..." form ("... into markdown." in Obsidian, "... into Swift code." in a code editor, shell commands in a terminal, a casual Slack message with emoji in Slack), and the trailing "Keywords: ..." boost fitted to the 4096-character cap. The rest of the focus capture (app/field names, window title, prior and selected text) no longer renders into the prompt but is still captured: it feeds the dictation log, the injector's separator logic, and the code-editor language refinement. A context whose signals render nothing now yields no prompt at all, so the server default applies. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01R28wj6vXZDW3iS7FWpHT1U --- AGENTS.md | 47 +++--- BLURTENGINE.md | 4 +- README.md | 6 +- .../FocusCapture/FocusCapture.swift | 19 +-- .../BlurtEngine/Pipeline/DictationLog.swift | 11 +- Sources/BlurtEngine/STT/AppKindPriming.swift | 85 +++++----- .../STT/TranscriptionContext.swift | 48 +++--- .../BlurtEngine/STT/TranscriptionPrompt.swift | 126 +++++---------- .../AppKindPrimingTests.swift | 26 ++-- .../BlurtEngineTests/DictationLogTests.swift | 13 +- .../TranscriptionContextTests.swift | 25 ++- .../TranscriptionPromptTests.swift | 146 ++++++------------ 12 files changed, 233 insertions(+), 323 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 487f76f..f200bd8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,8 +7,8 @@ the Claude-Code-specific tooling under `.claude/` (hooks, skills, subagents). Blurt is a macOS dictation app powered by [AssemblyAI](https://www.assemblyai.com). Tap or hold a trigger key, speak, and polished text is pasted into the focused app. Transcription is **one remote -AssemblyAI dictation API call**: a per-utterance `prompt` (a transcription directive plus contextual -priming built from the focused app/window/field and the user's key terms) rides along with the +AssemblyAI dictation API call**: a per-utterance `prompt` (an app-kind transcription instruction +recognized from the frontmost app, plus the user's key terms) rides along with the request, and the same request asks the service for its server-side LLM cleanup rewrite (`config.llm`), so the text that comes back is already polished. The user supplies their own API key. @@ -379,27 +379,28 @@ to right ⌘), so views must not re-declare `TriggerKey.rightCommand.rawValue` t request's `config.prompt` — it steers the _transcription_, not the LLM rewrite (that's the request's separate `llm` block). It's unit-tested in `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. -The built prompt is _contextual priming only_ — no standing instruction opens it. The -annotation-suppression clause that once did (_"Transcribe without speaker labels, audio event -descriptions, or emotion markers."_) is already part of the dictation service's own default prompt, -so restating it client-side only spent budget from the 4096-character cap. - -`build(context:)` assembles: prior-cursor text; the selected text (the -highlighted run the dictation will replace, so the model is primed on what's being rewritten — read -via `kAXSelectedTextAttribute`, skipped in secure fields, detected by AX role **or** subrole and -failing closed when the role can't be read, so a password can't reach the prompt); a topic hint from -the window title; a destination sentence from the app/field; an app-kind guidance sentence -(`AppKindPriming` — the frontmost app's bundle ID is recognized as a terminal, code editor, Slack, -or Obsidian, and a code editor's clause names the language inferred from the window title's -filename, so the model expects the right shape of text: shell commands, identifiers, casual chat -with emoji, or Markdown); and inline keyword boosting from the user's key terms. It's phrased per AssemblyAI's Universal-3 Pro prompting guidance -(positive/authoritative wording, no "Don't"/"Avoid"/"Never") and stays under the dictation API's -documented 4096-character cap on `config.prompt` (`characterCap`). - -`build(context:)` returns `nil` when there's no usable context (or when the context renders no -text, e.g. only an unrecognized bundle ID), and passing `prompt: nil` to the transcriber omits the -field so the server applies its own default. Two omissions are deliberate and regression-tested — -no language directive and no filler-word clause; see +The built prompt is deliberately minimal — two clauses, each optional. `AppKindPriming` recognizes +the frontmost app's bundle ID as a terminal, code editor, Slack, or Obsidian and renders the +app-kind instruction (_"Transcribe speech into markdown."_ — for a code editor, the language is +inferred from the window title's filename: _"Transcribe speech into Swift code."_); the user's key +terms trail it as inline keyword boosting (`Keywords: a, b, c.`), fitted to the dictation API's +documented 4096-character cap on `config.prompt` (`characterCap`). Wording is phrased per +AssemblyAI's Universal-3 Pro prompting guidance (positive/authoritative, no +"Don't"/"Avoid"/"Never"). + +The rest of the captured focus context — window title, app and field names, prior-cursor text, the +selected text — is deliberately **not** rendered into the prompt: real-world logs showed it +crowding the instruction (VS Code, for one, parks a screen-reader help announcement in the focused +field's description). It is still captured, feeding the dictation log, the injector's separator +logic, and the code-editor language refinement — and reading it stays privacy-guarded (prior and +selected text are skipped in secure fields, detected by AX role **or** subrole and failing closed +when the role can't be read, so a password can't reach the log or leave the machine). + +`build(context:)` returns `nil` when there's no usable context (or when nothing renders — focus +signals from an unrecognized app, no key terms), and passing `prompt: nil` to the transcriber omits +the field so the server applies its own default. Three omissions are deliberate — no +annotation-suppression clause (_"Transcribe without speaker labels, …"_ is already part of the +dictation service's own default prompt), no language directive, and no filler-word clause; see [Settled decisions](#settled-decisions--dont-reintroduce-these). ## Settings, persistence, and cues diff --git a/BLURTENGINE.md b/BLURTENGINE.md index 1ed4847..c1143f0 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -152,8 +152,8 @@ The session calls `setTargetApp` at press time with the app that was frontmost w Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected: -- **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. -- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block). The prompt is contextual priming only, staying under the API's 4096-character cap; the standing annotation-suppression clause ("Transcribe without speaker labels, …") is part of the service's own default prompt and is not restated. When the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian), the prompt adds one app-kind guidance sentence, e.g. "You are dictating into a terminal: expect shell commands, program names, flags, and file paths."; for code editors it names the language inferred from the window title's filename ("You are writing Python …"). Unrecognized apps add nothing. An empty context yields `nil`, which omits the field so the server applies its own default. Two deliberate omissions, both regression-tested: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. +- **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the bundle ID, window title, and key terms reach the prompt; the rest feeds the dictation log and the injector's separator logic. +- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block). The prompt is deliberately minimal: when the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian), it is that one app-kind instruction, e.g. "Transcribe speech into shell commands." or, for a code editor, the language inferred from the window title's filename ("Transcribe speech into Swift code."); the user's key terms trail it as `Keywords: a, b, c.`, fitted to the API's 4096-character cap. Nothing else is rendered — no window/app/field/prior/selected context, and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default prompt). A context that renders nothing yields `nil`, which omits the field so the server applies its own default. Two further deliberate omissions: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. - **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere. For key storage, compose against **`APIKeyGateway`** — the injectable `current` / `save(_:)` / `hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. Two projections keep the surrounding UI out of your views: `Outcome.failureReport` classifies a failure as `.inline(message:)` (recoverable — show it beside the field) or `.alert(title:message:)` (a Keychain fault retyping can't fix), and **`APIKeyDisplay.resolve(key:)`** renders the stored key for an account row — masked tail, status and VoiceOver wording, and the connect-vs-rotate control titles. The mask reveals only the last `revealedTailLength` characters and, below `minimumLengthToMask`, none at all, so a short key can't be shown whole. diff --git a/README.md b/README.md index 9075c6c..864bb48 100644 --- a/README.md +++ b/README.md @@ -166,14 +166,14 @@ Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dep Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM, live level meter; DX7/Juno-106 sound packs STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe - (STT + LLM rewrite) + TranscriptionPrompt contextual priming + (STT + LLM rewrite) + TranscriptionPrompt app-kind instruction Pipeline/ DictationSession actor: press/release/cancel commands, phase stream, auto-release before the API's recording cap Hotkey/ DictationKeyGate/Router: pure, unit-tested state machine for the lone-modifier trigger (tap vs hold vs combo) Injection/ KeyInjector: save clipboard → paste via synthesized ⌘V → restore - FocusCapture/ Accessibility reads of the focused app/window/field that prime - the transcription prompt + FocusCapture/ Accessibility reads of the focused app/window/field feeding the + prompt's app-kind instruction, the log, and paste separators Config/, Update/ Keychain API-key store, key terms, download-only release check App/Blurt/ AppKit/SwiftUI shell (Xcode project generated by XcodeGen) diff --git a/Sources/BlurtEngine/FocusCapture/FocusCapture.swift b/Sources/BlurtEngine/FocusCapture/FocusCapture.swift index 6fe058a..21005ac 100644 --- a/Sources/BlurtEngine/FocusCapture/FocusCapture.swift +++ b/Sources/BlurtEngine/FocusCapture/FocusCapture.swift @@ -24,17 +24,18 @@ enum FocusCapture { NSRunningApplication(processIdentifier: captured.pid) } - /// Accessibility-derived priming read from the system-wide focused UI element - /// at dictation start (see `TranscriptionContext`). Every field is - /// best-effort: any signal that can't be read is `nil`, and a fully-empty - /// result simply means less context, never an error. + /// Accessibility-derived focus context read from the system-wide focused UI + /// element at dictation start (see `TranscriptionContext` for what each + /// signal feeds). Every field is best-effort: any signal that can't be read + /// is `nil`, and a fully-empty result simply means less context, never an + /// error. struct FocusedFieldContext: Sendable { - /// Text immediately preceding the insertion point ("prior chunk context"). + /// Text immediately preceding the insertion point. let priorText: String? /// The text currently selected in the focused field — the dictation will - /// replace it, so it primes the model on what the utterance is about. + /// replace it. let selectedText: String? - /// The focused window's title — a dense topic hint. + /// The focused window's title (in a code editor it names the open file). let windowTitle: String? /// A short label for the focused field ("To", "Search", "Message"). let fieldLabel: String? @@ -53,8 +54,8 @@ enum FocusCapture { /// /// Secure text fields (password inputs) are detected by role **or** subrole and /// never have their contents read, so a typed password — selected or not — can't - /// leak into the STT prompt. The check fails closed: an unreadable role is - /// treated as secure, since it can't be shown not to be. + /// leak into the dictation log or the injector. The check fails closed: an + /// unreadable role is treated as secure, since it can't be shown not to be. /// /// Deliberately `nonisolated`: each read below is a synchronous cross-process /// IPC round trip into the frontmost app, and an unresponsive app blocks the diff --git a/Sources/BlurtEngine/Pipeline/DictationLog.swift b/Sources/BlurtEngine/Pipeline/DictationLog.swift index c4b48e3..ea1ded8 100644 --- a/Sources/BlurtEngine/Pipeline/DictationLog.swift +++ b/Sources/BlurtEngine/Pipeline/DictationLog.swift @@ -10,20 +10,21 @@ public enum DictationLog { struct Entry: Encodable { let transcript: String let ts: String - /// Focused-app topic hint sent as context, when one was captured. + /// Focused-app display name, when one was captured. let app: String? /// Focused-app bundle identifier, when one was captured. The input the /// prompt's app-kind guidance (`AppKindPriming`) keys on, logged so the /// corpus shows *why* a prompt carried (or lacked) a guidance sentence. let bundle: String? - /// Focused-window title sent as a topic hint, when one was captured. + /// Focused-window title, when one was captured (in a code editor it names + /// the open file, which is what refines the prompt's instruction). let window: String? /// Focused-field label sent as context, when one was captured. let field: String? - /// Text-before-cursor "prior chunk context" sent, when any was captured. - /// Lets you verify accessibility-tree prior-text reading actually fired. + /// Text before the cursor at press time, when any was captured. Lets you + /// verify accessibility-tree prior-text reading actually fired. let prior: String? - /// Selected text sent as context (the dictation replaced it), when any. + /// Selected text at press time (the dictation replaced it), when any. let selected: String? /// The fully-assembled `config.prompt` sent to AssemblyAI for this /// utterance. Built here from `context` (rather than threaded through from diff --git a/Sources/BlurtEngine/STT/AppKindPriming.swift b/Sources/BlurtEngine/STT/AppKindPriming.swift index 0fe0a78..af93021 100644 --- a/Sources/BlurtEngine/STT/AppKindPriming.swift +++ b/Sources/BlurtEngine/STT/AppKindPriming.swift @@ -1,23 +1,24 @@ import Foundation -/// App-kind guidance for the transcription prompt: recognizes what *kind* of -/// app the dictation targets (a terminal, a code editor, Slack, Obsidian) from -/// the frontmost app's bundle identifier and renders one priming sentence for -/// `TranscriptionPrompt` to place after the destination sentence. The sentence -/// tells the model what shape of text the destination expects — shell commands in a -/// terminal, identifiers and symbols in an editor, casual chat in Slack, -/// Markdown in Obsidian — which the app's display name alone doesn't convey. +/// The app-kind transcription instruction: recognizes what *kind* of app the +/// dictation targets (a terminal, a code editor, Slack, Obsidian) from the +/// frontmost app's bundle identifier and renders the one instruction sentence +/// `TranscriptionPrompt` sends — "Transcribe speech into shell commands." / +/// "… into Swift code." / "… into a casual Slack message with emoji." / +/// "… into markdown." — telling the model what shape of text the destination +/// expects, which the app's display name alone doesn't convey. /// /// For code editors the window title usually names the open file, so the -/// clause names the language inferred from that filename's extension ("You are -/// writing Python …") when one is recognizable, and stays generic otherwise. +/// clause names the language inferred from that filename's extension +/// ("Transcribe speech into Python code.") when one is recognizable, and stays +/// generic ("… into code.") otherwise. /// /// Detection keys on bundle IDs, not display names: names are localized and /// user-editable, while the bundle ID is the app's stable identity. An -/// unrecognized app contributes no clause — the prompt simply falls back to -/// the existing destination sentence built from the app name. Wording follows -/// the same Universal-3 Pro prompting guidance as the rest of the prompt -/// (positive/authoritative phrasing, no negations). Exercised by +/// unrecognized app contributes no clause — the request then carries no +/// instruction and the service's own default prompt applies. Wording follows +/// Universal-3 Pro prompting guidance (positive/authoritative phrasing, no +/// negations). Exercised by /// `Tests/BlurtEngineTests/AppKindPrimingTests.swift`. enum AppKindPriming { /// The recognized destination families. Each renders one guidance sentence; @@ -72,28 +73,29 @@ enum AppKindPriming { return kindsByBundleIDPrefix.first { bundleID.hasPrefix($0.prefix) }?.kind } - /// The guidance sentence for the app `bundleID` identifies, or `nil` when + /// The instruction sentence for the app `bundleID` identifies, or `nil` when /// the app isn't recognized. `windowTitle` refines the code-editor clause /// with the open file's language; the other kinds ignore it. static func clause(bundleID: String?, windowTitle: String?) -> String? { guard let kind = kind(ofBundleID: bundleID) else { return nil } switch kind { case .terminal: - return "You are dictating into a terminal: expect shell commands, program names, flags, and file paths." + return "Transcribe speech into shell commands." case .codeEditor: let subject = windowTitle.flatMap(language(inWindowTitle:)) ?? "code" - return "You are writing \(subject) in a code editor: expect code identifiers, symbols, and technical terms." + return "Transcribe speech into \(subject)." case .slack: - return "You are writing a Slack message: casual tone and emoji are expected." + return "Transcribe speech into a casual Slack message with emoji." case .obsidian: - return "You are writing a Markdown note in Obsidian: Markdown syntax is expected." + return "Transcribe speech into markdown." } } - /// The language of the first token in `title` that reads as a filename with - /// a recognized extension ("● main.py — blurt — Visual Studio Code" → - /// "Python"), or `nil` when no token does. Editors lead their window titles - /// with the open file, so first match wins. + /// What the open file says speech becomes, from the first token in `title` + /// that reads as a filename with a recognized extension + /// ("● main.py — blurt — Visual Studio Code" → "Python code"), or `nil` when + /// no token does. Editors lead their window titles with the open file, so + /// first match wins. static func language(inWindowTitle title: String) -> String? { for token in title.split(whereSeparator: \.isWhitespace) { let name = token.trimmingCharacters(in: Self.filenameTrim) @@ -110,32 +112,33 @@ enum AppKindPriming { /// markers, quotes, brackets, dash separators. private static let filenameTrim = CharacterSet(charactersIn: "\"'`•●◆*()[]{}<>,;:—–-") - /// Filename extension → how the clause names what's being written. Values - /// complete "You are writing … in a code editor", so most are bare language - /// names. Lowercased keys; lookups lowercase the extension first. + /// Filename extension → what the clause says speech becomes. Values complete + /// "Transcribe speech into …", so languages carry a trailing "code" while + /// markup/data formats stand alone. Lowercased keys; lookups lowercase the + /// extension first. private static let languagesByExtension: [String: String] = [ - "c": "C", "h": "C", - "cc": "C++", "cpp": "C++", "cxx": "C++", "hpp": "C++", - "cs": "C#", + "c": "C code", "h": "C code", + "cc": "C++ code", "cpp": "C++ code", "cxx": "C++ code", "hpp": "C++ code", + "cs": "C# code", "css": "CSS", "scss": "CSS", - "go": "Go", + "go": "Go code", "htm": "HTML", "html": "HTML", - "java": "Java", - "cjs": "JavaScript", "js": "JavaScript", "jsx": "JavaScript", "mjs": "JavaScript", + "java": "Java code", + "cjs": "JavaScript code", "js": "JavaScript code", "jsx": "JavaScript code", "mjs": "JavaScript code", "json": "JSON", - "kt": "Kotlin", "kts": "Kotlin", - "lua": "Lua", - "m": "Objective-C", "mm": "Objective-C", - "markdown": "Markdown", "md": "Markdown", - "php": "PHP", - "py": "Python", "pyi": "Python", - "rb": "Ruby", - "rs": "Rust", + "kt": "Kotlin code", "kts": "Kotlin code", + "lua": "Lua code", + "m": "Objective-C code", "mm": "Objective-C code", + "markdown": "markdown", "md": "markdown", + "php": "PHP code", + "py": "Python code", "pyi": "Python code", + "rb": "Ruby code", + "rs": "Rust code", "bash": "a shell script", "sh": "a shell script", "zsh": "a shell script", "sql": "SQL", - "swift": "Swift", + "swift": "Swift code", "toml": "TOML", - "ts": "TypeScript", "tsx": "TypeScript", + "ts": "TypeScript code", "tsx": "TypeScript code", "yaml": "YAML", "yml": "YAML", ] } diff --git a/Sources/BlurtEngine/STT/TranscriptionContext.swift b/Sources/BlurtEngine/STT/TranscriptionContext.swift index c3735ef..f3a334b 100644 --- a/Sources/BlurtEngine/STT/TranscriptionContext.swift +++ b/Sources/BlurtEngine/STT/TranscriptionContext.swift @@ -1,43 +1,45 @@ -/// Per-utterance context the STT model is trained to use as *contextual* -/// priming — it improves recognition accuracy (vocabulary, continuity, -/// capitalization) without changing the output format. Gathered at dictation -/// start from the focused app and the text preceding the cursor, then rendered -/// into the request `prompt` by `TranscriptionPrompt.build`. +/// Per-utterance snapshot of where the dictation is going, gathered at +/// dictation start from the focused app and field. `TranscriptionPrompt.build` +/// renders only the parts of it the request prompt uses — the bundle ID (via +/// the `AppKindPriming` app-kind instruction), the window title (the language +/// refinement of that instruction), and the key terms — while the rest rides +/// along for the dictation log and the injector's separator logic. /// -/// Both fields are optional: whichever is available is used, and an entirely -/// empty context yields no prompt (the server applies its own default). +/// Every focus field is optional and best-effort: whatever couldn't be read is +/// `nil`, and an entirely empty context yields no prompt (the server applies +/// its own default). public struct TranscriptionContext: Sendable, Equatable { - /// The frontmost application's display name (e.g. "Slack", "Xcode"), passed - /// as a domain/topic hint so the model expects that app's vocabulary. + /// The frontmost application's display name (e.g. "Slack", "Xcode"). Not + /// rendered into the prompt; recorded in the dictation log. public let appName: String? /// The frontmost application's bundle identifier (e.g. /// "com.tinyspeck.slackmacgap"). Never rendered verbatim: it keys the - /// app-*kind* recognition (`AppKindPriming`) that adds destination-specific - /// guidance — terminal, code editor, Slack, Obsidian — to the prompt. + /// app-*kind* recognition (`AppKindPriming`) that selects the prompt's + /// transcription instruction — terminal, code editor, Slack, Obsidian. /// Preferred over `appName` for recognition because display names are /// localized and user-editable while the bundle ID is stable. public let bundleID: String? - /// The focused window's title (e.g. "Re: Q3 pricing — Gmail", a document - /// name, a Slack channel). The densest topic hint available — usually packed - /// with the proper nouns and domain vocabulary the model would otherwise guess. + /// The focused window's title (e.g. "main.py — blurt", a document name, a + /// Slack channel). In a code editor it usually names the open file, which is + /// how the app-kind instruction learns the language ("… into Swift code."); + /// it also anchors the injector's same-window separator fallback. public let windowTitle: String? /// A short label for the focused field (placeholder/title/role, e.g. "To", - /// "Subject", "Search", "Message"), passed so the model knows what *kind* of - /// text is expected — an email address, a search query, and prose should be - /// transcribed differently. + /// "Subject", "Search", "Message"). Not rendered into the prompt; recorded + /// in the dictation log. public let fieldLabel: String? - /// Text immediately preceding the insertion point in the focused field, - /// passed as "prior chunk context" so the transcript continues naturally. + /// Text immediately preceding the insertion point in the focused field. Not + /// rendered into the prompt; it drives the injector's leading-separator + /// decision and is recorded in the dictation log. public let priorText: String? - /// The text currently selected in the focused field, when any. Dictating with - /// a selection replaces it (the paste overwrites the highlighted range), so - /// this is passed as priming for what the utterance is about — the vocabulary - /// and topic of the text being rewritten. + /// The text currently selected in the focused field, when any (the paste + /// will replace it). Not rendered into the prompt; recorded in the + /// dictation log. public let selectedText: String? /// User-configured domain vocabulary (names, jargon, product names) carried as diff --git a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift index 67b63f8..3b11b33 100644 --- a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift +++ b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift @@ -2,40 +2,38 @@ /// field of the request `config` (see `AssemblyAITranscriber`). The STT model /// prepends this to its own system prompt. /// -/// The built prompt is *contextual priming only*: a topic hint built from the -/// window title, a destination sentence built from the focused app and field -/// label, an app-kind guidance sentence recognized from the frontmost app's -/// bundle ID (`AppKindPriming` — terminal, code editor, Slack, Obsidian), -/// "prior chunk context" (the text preceding the cursor), the selected text -/// (which the dictation replaces), and keyword boosting, all of which the -/// model is mid-trained to use for better recognition accuracy. +/// The prompt is deliberately minimal — two clauses, each optional: +/// - the app-kind transcription instruction (`AppKindPriming`), recognized +/// from the frontmost app's bundle ID: "Transcribe speech into markdown." +/// in Obsidian, "Transcribe speech into Swift code." in a code editor (the +/// language inferred from the window title's filename), shell commands in a +/// terminal, a casual Slack message in Slack; +/// - inline keyword boosting trailing it (`Keywords: a, b, c.`, the trained +/// §2.3 form) from the user's key terms, fitted to `characterCap`. /// -/// Three standing directives are deliberately *absent*: -/// - No annotation-suppression clause ("Transcribe without speaker labels, +/// The rest of the captured focus context — window title, app and field +/// names, prior-cursor text, selected text — is deliberately **not** rendered: +/// real-world logs showed it crowding the instruction (VS Code, for one, parks +/// a screen-reader help announcement in the focused field's description). It +/// is still captured, feeding the dictation log, the injector's separator +/// logic, and the code-editor language refinement above. Also deliberately +/// absent: +/// - the annotation-suppression clause ("Transcribe without speaker labels, /// audio event descriptions, or emotion markers."): the dictation service -/// already includes it in its own default prompt, so restating it here only -/// spent budget from `characterCap`. -/// - No "remove filler words"-style *content* reshaping: not in the model's -/// trained instruction set, so it is a no-op and is deliberately omitted -/// (see the project memory note); disfluency removal is the server-side LLM -/// rewrite's job. -/// - No language directive: pinning the prompt to English hurt non-English +/// already includes it in its own default prompt; +/// - a "remove filler words"-style *content* reshaping: not in the model's +/// trained instruction set, so it is a no-op (see the project memory note); +/// disfluency removal is the server-side LLM rewrite's job; +/// - a language directive: pinning the prompt to English hurt non-English /// transcription, so language is left to the model's own detection. /// -/// Output follows the trained format: the prior-chunk context and the selected -/// text lead as their own paragraphs, the location clause (topic hint + -/// destination sentence + app-kind guidance) follows, and keyword boosting -/// trails inline as `Keywords: a, b, c.` (per the mid-training -/// instruction-type reference). It stays under the API's `characterCap`: the -/// contextual blocks are clipped upstream in `FocusCapture`, and the key-terms -/// clause is fitted to the remaining budget here. Exercised by -/// `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. +/// Exercised by `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. enum TranscriptionPrompt { /// Hard cap the dictation API places on `config.prompt` ("max 4096 chars"); /// a longer prompt risks failing the whole request, so `build` must never - /// exceed it. The contextual blocks are all clipped upstream in - /// `FocusCapture`; the user's key terms are the one unbounded input, so - /// `build` fits them to whatever budget remains. + /// exceed it. The instruction clause is a bounded sentence; the user's key + /// terms are the one unbounded input, so `build` fits them to whatever + /// budget remains. static let characterCap = 4096 /// Renders `context` into a transcription prompt, or `nil` when there is no usable @@ -46,47 +44,18 @@ enum TranscriptionPrompt { // context. Asking it here (rather than re-deriving the field-by-field test) // keeps a newly added context signal from being silently dropped. guard let context, !context.isEmpty else { return nil } - let prior = context.priorText.trimmedNonEmpty() ?? "" - let selected = context.selectedText.trimmedNonEmpty() ?? "" - let app = context.appName.trimmedNonEmpty() ?? "" - let window = context.windowTitle.trimmedNonEmpty() ?? "" - let field = context.fieldLabel.trimmedNonEmpty() ?? "" + var prompt = + AppKindPriming.clause( + bundleID: context.bundleID, windowTitle: context.windowTitle.trimmedNonEmpty()) ?? "" let keyTerms = context.keyTerms - - // The priming blocks, separated by blank lines; keyword boosting trails - // the last one inline: - // 1. the prior-chunk block (`Previous transcript:\n…`, its own paragraph), - // 2. the selected-text block (`Selected text:\n…`, what the dictation - // replaces — primes vocabulary/topic of the text being rewritten), - // 3. the location clause (topic hint + destination sentence + app-kind - // guidance). - var blocks: [String] = [] - if !prior.isEmpty { - blocks.append("Previous transcript:\n\(prior)") - } - if !selected.isEmpty { - blocks.append("Selected text:\n\(selected)") - } - // App-kind guidance ("You are dictating into a terminal …") follows the - // destination sentence: recognized from the bundle ID, refined by the - // window title (a code editor's title names the open file, hence the - // language). Unrecognized apps add nothing. - let guidance = AppKindPriming.clause(bundleID: context.bundleID, windowTitle: window) ?? "" - let location = [locationClause(app: app, window: window, field: field), guidance] - .filter { !$0.isEmpty } - .joined(separator: " ") - if !location.isEmpty { - blocks.append(location) - } - var prompt = blocks.joined(separator: "\n\n") if !keyTerms.isEmpty { // Spelling priming: the user's domain vocabulary, boosted via the trained - // inline `Keywords: a, b, c.` form (Section 2.3) trailing the context so the - // model favors these exact spellings for names/jargon it would guess at. - // The terms list is the one input with no upstream length cap, so include - // only as many whole terms as `characterCap` leaves room for, so a huge - // Settings list can't crowd out the priming itself or balloon every - // request. + // inline `Keywords: a, b, c.` form (Section 2.3) trailing the instruction + // so the model favors these exact spellings for names/jargon it would + // guess at. The terms list is the one input with no upstream length cap, + // so include only as many whole terms as `characterCap` leaves room for, + // so a huge Settings list can't crowd out the instruction itself or + // balloon every request. let scaffold = prompt.isEmpty ? "Keywords: ." : " Keywords: ." var included: [String] = [] var remaining = characterCap - prompt.count - scaffold.count @@ -101,30 +70,9 @@ enum TranscriptionPrompt { prompt = prompt.isEmpty ? clause : "\(prompt) \(clause)" } } - // A context can be non-empty yet render nothing — e.g. its only signal is - // a bundle ID no app-kind guidance recognizes. Return nil rather than an - // empty prompt so the server default still applies. + // A context can be non-empty yet render nothing — focus signals from an + // unrecognized app, no key terms. Return nil rather than an empty prompt + // so the server default still applies. return prompt.trimmedNonEmpty() } - - /// The "where am I typing" priming clause, assembled from whichever of the - /// app / window / field signals are present (empty when none are). Two trained - /// shapes joined by a space: a topic hint built from the window title (the - /// richest vocabulary signal — `This is about "…".`, mid-training §2.1) leads, - /// and a destination sentence built from the app/field (`Dictated into …`) - /// trails it. Each sentence ends with a period so the clause joins cleanly - /// with the app-kind guidance that may follow it. - private static func locationClause(app: String, window: String, field: String) -> String { - let topic = window.isEmpty ? "" : "This is about \"\(window)\"." - - let destination: String - switch (app.isEmpty, field.isEmpty) { - case (false, false): destination = "Dictated into \(app), in the \"\(field)\" field." - case (false, true): destination = "Dictated into \(app)." - case (true, false): destination = "Dictated in the \"\(field)\" field." - case (true, true): destination = "" - } - - return [topic, destination].filter { !$0.isEmpty }.joined(separator: " ") - } } diff --git a/Tests/BlurtEngineTests/AppKindPrimingTests.swift b/Tests/BlurtEngineTests/AppKindPrimingTests.swift index 6146aba..2199c9c 100644 --- a/Tests/BlurtEngineTests/AppKindPrimingTests.swift +++ b/Tests/BlurtEngineTests/AppKindPrimingTests.swift @@ -47,15 +47,15 @@ struct AppKindPrimingTests { } static let languageCases: [LanguageCase] = [ - LanguageCase(title: "main.py — blurt — Visual Studio Code", expected: "Python"), + LanguageCase(title: "main.py — blurt — Visual Studio Code", expected: "Python code"), // VS Code prepends ● to the filename of a dirty buffer. - LanguageCase(title: "● api.ts — server", expected: "TypeScript"), - LanguageCase(title: "blurt — TranscriptionPrompt.swift", expected: "Swift"), + LanguageCase(title: "● api.ts — server", expected: "TypeScript code"), + LanguageCase(title: "blurt — TranscriptionPrompt.swift", expected: "Swift code"), LanguageCase(title: "deploy.sh — infra", expected: "a shell script"), // Extension matching is case-insensitive. - LanguageCase(title: "README.MD — notes", expected: "Markdown"), + LanguageCase(title: "README.MD — notes", expected: "markdown"), // First filename wins when several tokens carry extensions. - LanguageCase(title: "index.js next.config.ts", expected: "JavaScript"), + LanguageCase(title: "index.js next.config.ts", expected: "JavaScript code"), // A dotfile has no base name + extension split. LanguageCase(title: ".zshrc — dotfiles", expected: nil), // Version numbers and hostnames are not filenames. @@ -71,39 +71,39 @@ struct AppKindPrimingTests { // MARK: - Clause rendering - @Test("a terminal renders the command-line clause") + @Test("a terminal renders the shell-commands instruction") func terminalClause() { #expect( AppKindPriming.clause(bundleID: "com.apple.Terminal", windowTitle: nil) - == "You are dictating into a terminal: expect shell commands, program names, flags, and file paths.") + == "Transcribe speech into shell commands.") } @Test("a code editor names the open file's language when the title carries one") func codeEditorClauseWithLanguage() { #expect( AppKindPriming.clause(bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt") - == "You are writing Python in a code editor: expect code identifiers, symbols, and technical terms.") + == "Transcribe speech into Python code.") } @Test("a code editor stays generic when the title names no recognizable file") func codeEditorClauseGeneric() { #expect( AppKindPriming.clause(bundleID: "com.apple.dt.Xcode", windowTitle: "Welcome to Xcode") - == "You are writing code in a code editor: expect code identifiers, symbols, and technical terms.") + == "Transcribe speech into code.") } - @Test("Slack renders the casual-chat clause") + @Test("Slack renders the casual-message instruction") func slackClause() { #expect( AppKindPriming.clause(bundleID: "com.tinyspeck.slackmacgap", windowTitle: "#eng-backend") - == "You are writing a Slack message: casual tone and emoji are expected.") + == "Transcribe speech into a casual Slack message with emoji.") } - @Test("Obsidian renders the Markdown clause") + @Test("Obsidian renders the markdown instruction") func obsidianClause() { #expect( AppKindPriming.clause(bundleID: "md.obsidian", windowTitle: "Meeting notes") - == "You are writing a Markdown note in Obsidian: Markdown syntax is expected.") + == "Transcribe speech into markdown.") } @Test("an unrecognized app contributes no clause") diff --git a/Tests/BlurtEngineTests/DictationLogTests.swift b/Tests/BlurtEngineTests/DictationLogTests.swift index ce8207b..e5a6882 100644 --- a/Tests/BlurtEngineTests/DictationLogTests.swift +++ b/Tests/BlurtEngineTests/DictationLogTests.swift @@ -124,16 +124,17 @@ struct DictationLogTests { func logsAssembledPrompt() { let url = makeTempLogURL() let context = TranscriptionContext( - appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", - priorText: "Hi Sam,", selectedText: "the old plan") + appName: "Obsidian", bundleID: "md.obsidian", windowTitle: "Grocery list", + fieldLabel: "text entry area", priorText: "- milk", keyTerms: ["Blurt"]) DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) #expect(decoded?.prompt == TranscriptionPrompt.build(context: context)) + #expect(decoded?.prompt == "Transcribe speech into markdown. Keywords: Blurt.") } - @Test("the logged prompt carries the app-kind guidance the bundle ID selected") - func logsPromptWithAppKindGuidance() { + @Test("the logged prompt carries the app-kind instruction the bundle ID selected") + func logsPromptWithAppKindInstruction() { let url = makeTempLogURL() let context = TranscriptionContext( appName: "Slack", bundleID: "com.tinyspeck.slackmacgap", fieldLabel: "Message", @@ -142,8 +143,8 @@ struct DictationLogTests { let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) // The exact wording is TranscriptionPromptTests' contract; here the point is - // that what lands on disk includes the guidance actually sent for this app. - #expect(decoded?.prompt?.contains("You are writing a Slack message") == true) + // that what lands on disk includes the instruction actually sent for this app. + #expect(decoded?.prompt?.contains("Slack message") == true) } @Test("omits the prompt field when there is no context to build one") diff --git a/Tests/BlurtEngineTests/TranscriptionContextTests.swift b/Tests/BlurtEngineTests/TranscriptionContextTests.swift index 624ad30..127325c 100644 --- a/Tests/BlurtEngineTests/TranscriptionContextTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionContextTests.swift @@ -3,9 +3,11 @@ import Testing @testable import BlurtEngine /// `TranscriptionContext.isEmpty` is the gate `FocusCapture`/`DictationSession` -/// use to decide whether a context is worth sending as priming. It mirrors the -/// emptiness logic in `TranscriptionPrompt.build`, so the two must agree: -/// `isEmpty == true` should always correspond to `build` returning `nil`. +/// use to decide whether a context is worth carrying at all (prompt AND log). +/// The agreement with `TranscriptionPrompt.build` is one-directional: +/// `isEmpty == true` must always correspond to `build` returning `nil`, while a +/// non-empty context may still build no prompt — its signals can be log-only +/// (app/window/field/prior/selected from an unrecognized app). @Suite("TranscriptionContext") struct TranscriptionContextTests { @Test("both fields nil is empty") @@ -52,7 +54,7 @@ struct TranscriptionContextTests { #expect(TranscriptionPrompt.build(context: context) != nil) } - @Test("emptiness agrees with TranscriptionPrompt.build returning nil") + @Test("an empty context always corresponds to build returning nil") func agreesWithPromptBuild() { let empties = [ TranscriptionContext(appName: nil, priorText: nil), @@ -63,14 +65,21 @@ struct TranscriptionContextTests { #expect(TranscriptionPrompt.build(context: context) == nil) } - let nonEmpties = [ - TranscriptionContext(appName: "Mail", priorText: nil), - TranscriptionContext(appName: nil, priorText: nil, selectedText: "selected"), + // Renderable signals (a recognized bundle ID, key terms) build a prompt… + let renderable = [ + TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil), + TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["Blurt"]), ] - for context in nonEmpties { + for context in renderable { #expect(!context.isEmpty) #expect(TranscriptionPrompt.build(context: context) != nil) } + + // …while log-only signals make the context non-empty (worth carrying for + // the dictation log and the injector) yet build no prompt. + let logOnly = TranscriptionContext(appName: "Mail", priorText: "Hi Sam,", selectedText: "sel") + #expect(!logOnly.isEmpty) + #expect(TranscriptionPrompt.build(context: logOnly) == nil) } @Test("Equatable compares every field") diff --git a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift index 34470e6..90ceda1 100644 --- a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift @@ -8,10 +8,11 @@ struct TranscriptionPromptTests { /// than a `@Test` apiece) keeps the whole context→prompt contract in one /// readable table and gives per-case failure output. /// - /// The prompt is contextual priming only — no standing instruction opens it. - /// The annotation-suppression clause ("Transcribe without speaker labels, …") - /// is part of the dictation service's own default prompt, so a case's - /// expectation is exactly its context rendered, nothing more. + /// The prompt is the app-kind instruction plus trailing keyword boosting — + /// nothing else. The other focus signals (app name, window title, field + /// label, prior text, selected text) are captured for the dictation log and + /// the injector, and must never surface in the prompt; the cases below pin + /// both directions. struct Case: Sendable, CustomTestStringConvertible { let name: String let context: TranscriptionContext? @@ -28,111 +29,60 @@ struct TranscriptionPromptTests { name: "whitespace-only context → no prompt", context: TranscriptionContext(appName: " ", priorText: "\n"), expected: nil), Case( - name: "app only → destination sentence", - context: TranscriptionContext(appName: "Slack", priorText: nil), - expected: "Dictated into Slack."), - Case( - name: "prior only → Previous transcript framing", - context: TranscriptionContext(appName: nil, priorText: "and then the build finished"), - expected: "Previous transcript:\nand then the build finished"), - Case( - name: "app + window → topic hint leads, destination trails", - context: TranscriptionContext(appName: "Mail", windowTitle: "Re: Q3 pricing", priorText: nil), - expected: "This is about \"Re: Q3 pricing\". Dictated into Mail."), - Case( - name: "window only → bare topic hint", - context: TranscriptionContext(appName: nil, windowTitle: "Untitled.txt", priorText: nil), - expected: "This is about \"Untitled.txt\"."), - Case( - name: "field only → destination sentence", - context: TranscriptionContext(appName: nil, fieldLabel: "Search", priorText: nil), - expected: "Dictated in the \"Search\" field."), - Case( - name: "app + field without window → destination names both", - context: TranscriptionContext(appName: "Slack", fieldLabel: "Message", priorText: nil), - expected: "Dictated into Slack, in the \"Message\" field."), - Case( - name: "all four signals combine", + name: "focus signals alone render nothing — log-only, not prompt", context: TranscriptionContext( - appName: "Slack", windowTitle: "#eng-backend", fieldLabel: "Message", priorText: "thanks for"), - expected: - "Previous transcript:\nthanks for\n\nThis is about \"#eng-backend\". Dictated into Slack, in the \"Message\" field." - ), - Case( - name: "prior + app combine", - context: TranscriptionContext(appName: "Mail", priorText: "Dear Sam,"), - expected: "Previous transcript:\nDear Sam,\n\nDictated into Mail."), - Case( - name: "prior + app are trimmed", - context: TranscriptionContext(appName: " Notes ", priorText: " hello "), - expected: "Previous transcript:\nhello\n\nDictated into Notes."), + appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", + priorText: "Hi Sam,", selectedText: "the old plan"), + expected: nil), Case( - name: "selected only → Selected text framing", - context: TranscriptionContext(appName: nil, priorText: nil, selectedText: "the quarterly numbers"), - expected: "Selected text:\nthe quarterly numbers"), + name: "unrecognized bundle ID renders nothing", + context: TranscriptionContext(appName: "Mail", bundleID: "com.apple.mail", priorText: nil), + expected: nil), Case( - name: "selected follows prior as its own block", - context: TranscriptionContext(appName: nil, priorText: "as we discussed,", selectedText: "the old plan"), - expected: "Previous transcript:\nas we discussed,\n\nSelected text:\nthe old plan"), + name: "terminal → shell-commands instruction", + context: TranscriptionContext(appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil), + expected: "Transcribe speech into shell commands."), Case( - name: "selected + location + prior combine", + name: "code editor names the window title's language", context: TranscriptionContext( - appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", - priorText: "Hi Sam,", selectedText: "let's push the date"), - expected: - "Previous transcript:\nHi Sam,\n\nSelected text:\nlet's push the date\n\nThis is about \"Re: Q3 pricing\". Dictated into Mail, in the \"Body\" field." - ), - Case( - name: "blank selected adds no block", - context: TranscriptionContext(appName: "Notes", priorText: nil, selectedText: " \n"), - expected: "Dictated into Notes."), + appName: "Code", bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt", priorText: nil), + expected: "Transcribe speech into Python code."), Case( - name: "selected sits between prior and keyword boost", + name: "code editor with no recognizable filename stays generic", context: TranscriptionContext( - appName: "Slack", priorText: "thanks for", selectedText: "the draft", keyTerms: ["Blurt"]), - expected: - "Previous transcript:\nthanks for\n\nSelected text:\nthe draft\n\nDictated into Slack. Keywords: Blurt." - ), + appName: "Xcode", bundleID: "com.apple.dt.Xcode", windowTitle: "Welcome to Xcode", priorText: nil), + expected: "Transcribe speech into code."), Case( - name: "recognized bundle ID → app-kind guidance after the destination", + name: "Slack → casual-message instruction", context: TranscriptionContext( appName: "Slack", bundleID: "com.tinyspeck.slackmacgap", fieldLabel: "Message", priorText: nil), - expected: - "Dictated into Slack, in the \"Message\" field. You are writing a Slack message: casual tone and emoji are expected." - ), - Case( - name: "bundle ID alone → guidance still built", - context: TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil), - expected: - "You are dictating into a terminal: expect shell commands, program names, flags, and file paths." - ), + expected: "Transcribe speech into a casual Slack message with emoji."), Case( - name: "code-editor guidance names the window title's language", + name: "Obsidian → markdown instruction, window/field stay out", context: TranscriptionContext( - appName: "Code", bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt", priorText: nil), - expected: - "This is about \"main.py — blurt\". Dictated into Code. You are writing Python in a code editor: expect code identifiers, symbols, and technical terms." - ), + appName: "Obsidian", bundleID: "md.obsidian", + windowTitle: "Grocery list - Cowork - Obsidian 1.12.7", fieldLabel: "text entry area", + priorText: nil), + expected: "Transcribe speech into markdown."), Case( - name: "unrecognized bundle ID adds no guidance", - context: TranscriptionContext(appName: "Mail", bundleID: "com.apple.mail", priorText: nil), - expected: "Dictated into Mail."), - Case( - name: "unrecognized bundle ID with no other signal → no prompt", - context: TranscriptionContext(appName: nil, bundleID: "com.example.mystery", priorText: nil), - expected: nil), + name: "prior/selected text never precede the instruction", + context: TranscriptionContext( + appName: "Terminal", bundleID: "com.apple.Terminal", windowTitle: "zsh — 80×24", + priorText: "$ git status", selectedText: "modified: README.md"), + expected: "Transcribe speech into shell commands."), Case( name: "key terms only → bare keyword boost", context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["AssemblyAI", "Kubernetes"]), expected: "Keywords: AssemblyAI, Kubernetes."), Case( - name: "key terms trail the focus context inline", - context: TranscriptionContext(appName: "Slack", priorText: nil, keyTerms: ["Blurt"]), - expected: "Dictated into Slack. Keywords: Blurt."), + name: "key terms trail the instruction inline", + context: TranscriptionContext( + appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil, keyTerms: ["Blurt"]), + expected: "Transcribe speech into shell commands. Keywords: Blurt."), Case( name: "empty key terms add no clause", - context: TranscriptionContext(appName: "Notes", priorText: nil, keyTerms: []), - expected: "Dictated into Notes."), + context: TranscriptionContext(appName: nil, bundleID: "md.obsidian", priorText: nil, keyTerms: []), + expected: "Transcribe speech into markdown."), ] @Test("build maps focus context to the transcription prompt", arguments: cases) @@ -140,14 +90,6 @@ struct TranscriptionPromptTests { #expect(TranscriptionPrompt.build(context: c.context) == c.expected) } - @Test("built prompt fits within the API's 4096-character cap for capped prior text") - func withinCap() { - let longPrior = String(repeating: "word ", count: 200) - let prompt = TranscriptionPrompt.build( - context: TranscriptionContext(appName: "Xcode", priorText: longPrior)) - #expect((prompt?.count ?? 0) <= TranscriptionPrompt.characterCap) - } - @Test("the keyword clause is omitted entirely when not even the first term fits") func keyTermsOmittedWhenNoneFit() { // A single term longer than the whole cap leaves no budget for even one @@ -155,13 +97,14 @@ struct TranscriptionPromptTests { // whole, not emitted empty or dangling. let huge = String(repeating: "k", count: TranscriptionPrompt.characterCap) let prompt = TranscriptionPrompt.build( - context: TranscriptionContext(appName: "Xcode", priorText: nil, keyTerms: [huge])) - #expect(prompt == "Dictated into Xcode.") + context: TranscriptionContext( + appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil, keyTerms: [huge])) + #expect(prompt == "Transcribe speech into shell commands.") } @Test("a key-terms-only prompt whose first term doesn't fit yields no prompt at all") func keyTermsOnlyNoneFit() { - // With no other context and no term fitting the cap, nothing renders — and + // With no instruction and no term fitting the cap, nothing renders — and // an empty prompt must collapse to nil so the server default applies. let huge = String(repeating: "k", count: TranscriptionPrompt.characterCap) let prompt = TranscriptionPrompt.build( @@ -175,7 +118,8 @@ struct TranscriptionPromptTests { // list must not push the prompt over the API cap (which fails the request). let terms = (0..<2000).map { "term\($0)" } let prompt = TranscriptionPrompt.build( - context: TranscriptionContext(appName: "Xcode", priorText: nil, keyTerms: terms)) + context: TranscriptionContext( + appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil, keyTerms: terms)) let built = try #require(prompt) #expect(built.count <= TranscriptionPrompt.characterCap) #expect(built.contains(" Keywords: term0, term1")) From 15e1718102dc3a56244d9993c94c38f9fa503734 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 29 Jul 2026 01:41:30 +0000 Subject: [PATCH 5/8] =?UTF-8?q?feat(log):=20record=20only=20what=20was=20s?= =?UTF-8?q?ent=20=E2=80=94=20transcript=20and=20prompt?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dictation log now writes each entry as the transcript that came back, the timestamp, and the exact config.prompt that was sent — nothing else. The raw focus context (app and field names, window title, prior/selected text) is not sent to the service, so it stays off disk entirely, even with developer mode on; the gate test now pins that prior text never reaches the log. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01R28wj6vXZDW3iS7FWpHT1U --- AGENTS.md | 12 +-- BLURTENGINE.md | 4 +- .../BlurtEngine/Pipeline/DictationLog.swift | 27 ++----- .../STT/TranscriptionContext.swift | 15 ++-- .../BlurtEngineTests/DictationLogTests.swift | 76 +++++++------------ 5 files changed, 50 insertions(+), 84 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f200bd8..e4db4ba 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -391,10 +391,11 @@ AssemblyAI's Universal-3 Pro prompting guidance (positive/authoritative, no The rest of the captured focus context — window title, app and field names, prior-cursor text, the selected text — is deliberately **not** rendered into the prompt: real-world logs showed it crowding the instruction (VS Code, for one, parks a screen-reader help announcement in the focused -field's description). It is still captured, feeding the dictation log, the injector's separator -logic, and the code-editor language refinement — and reading it stays privacy-guarded (prior and -selected text are skipped in secure fields, detected by AX role **or** subrole and failing closed -when the role can't be read, so a password can't reach the log or leave the machine). +field's description). It is still captured — the injector's separator logic +consumes the prior text and window title, and the code-editor language refinement reads the window +title — but none of it is written to the dictation log, and reading it stays privacy-guarded (prior +and selected text are skipped in secure fields, detected by AX role **or** subrole and failing +closed when the role can't be read, so a password is never read out of the field at all). `build(context:)` returns `nil` when there's no usable context (or when nothing renders — focus signals from an unrecognized app, no key terms), and passing `prompt: nil` to the transcriber omits @@ -431,7 +432,8 @@ pure edge detector deciding when the chimes fire; the AppKit `CueSoundPlayer` ju resolves. History: **`RecentDictations`** is an in-memory, newest-first ring shown in the ready window (never -written to disk). **`DictationLog`** appends each completed dictation with its context snapshot to +written to disk). **`DictationLog`** appends each completed dictation — the transcript plus the exact +`config.prompt` sent — to `~/Library/Logs/Blurt/dictations.jsonl` (`DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI — derived next to the URL so the label can't drift from the write target) — but **only** while developer mode is on; with it off, nothing is written. The Settings diff --git a/BLURTENGINE.md b/BLURTENGINE.md index c1143f0..ab92d0d 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -152,7 +152,7 @@ The session calls `setTargetApp` at press time with the app that was frontmost w Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected: -- **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the bundle ID, window title, and key terms reach the prompt; the rest feeds the dictation log and the injector's separator logic. +- **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the bundle ID, window title, and key terms reach the prompt; the prior text and window title also steer the injector's paste separator, and nothing else is consumed. - **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block). The prompt is deliberately minimal: when the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian), it is that one app-kind instruction, e.g. "Transcribe speech into shell commands." or, for a code editor, the language inferred from the window title's filename ("Transcribe speech into Swift code."); the user's key terms trail it as `Keywords: a, b, c.`, fitted to the API's 4096-character cap. Nothing else is rendered — no window/app/field/prior/selected context, and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default prompt). A context that renders nothing yields `nil`, which omits the field so the server applies its own default. Two further deliberate omissions: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. - **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere. @@ -160,7 +160,7 @@ For key storage, compose against **`APIKeyGateway`** — the injectable `current Setup gating has a projection too: **`SetupReadiness.isReady(permissions:hasAPIKey:)`** is the "fully configured" rule (deliberately excluding the trigger key, which has a default), `SetupReadiness.pollInterval(isReady:)` is the permission-poll cadence (brisk during setup, coasting once ready), and `PermissionStatus.lostGrant(since:)` detects a permission revoked out from under a configured app. -Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) with its context snapshot — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section. +Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact `config.prompt` sent, never the raw focus context — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section. ## Hotkey building blocks diff --git a/Sources/BlurtEngine/Pipeline/DictationLog.swift b/Sources/BlurtEngine/Pipeline/DictationLog.swift index ea1ded8..0ed5d5a 100644 --- a/Sources/BlurtEngine/Pipeline/DictationLog.swift +++ b/Sources/BlurtEngine/Pipeline/DictationLog.swift @@ -1,31 +1,21 @@ import Foundation -/// Append-only JSONL log of completed transcripts at +/// Append-only JSONL log of completed dictations — the transcript that came +/// back and the exact `config.prompt` that was sent — at /// `~/Library/Logs/Blurt/dictations.jsonl`. Used to build a real-world /// corpus for prompt iteration. Written only while developer mode is switched /// on (`DeveloperModeStore` — the Settings window's Developer section, which /// also displays this path), so a user who never opts in has no dictation /// text on disk. public enum DictationLog { + /// One logged dictation: what came back (`transcript`), when, and exactly + /// what was sent (`prompt`). Nothing else — the raw focus context (app and + /// field names, window title, prior/selected text) deliberately stays off + /// disk: it isn't sent to the service, so it doesn't belong in a log of the + /// exchange. struct Entry: Encodable { let transcript: String let ts: String - /// Focused-app display name, when one was captured. - let app: String? - /// Focused-app bundle identifier, when one was captured. The input the - /// prompt's app-kind guidance (`AppKindPriming`) keys on, logged so the - /// corpus shows *why* a prompt carried (or lacked) a guidance sentence. - let bundle: String? - /// Focused-window title, when one was captured (in a code editor it names - /// the open file, which is what refines the prompt's instruction). - let window: String? - /// Focused-field label sent as context, when one was captured. - let field: String? - /// Text before the cursor at press time, when any was captured. Lets you - /// verify accessibility-tree prior-text reading actually fired. - let prior: String? - /// Selected text at press time (the dictation replaced it), when any. - let selected: String? /// The fully-assembled `config.prompt` sent to AssemblyAI for this /// utterance. Built here from `context` (rather than threaded through from /// the transcriber) so the log always reflects what was actually sent, @@ -95,9 +85,6 @@ public enum DictationLog { ) { let entry = Entry( transcript: transcript, ts: now.formatted(timestampFormat), - app: context?.appName, bundle: context?.bundleID, - window: context?.windowTitle, field: context?.fieldLabel, - prior: context?.priorText, selected: context?.selectedText, prompt: TranscriptionPrompt.build(context: context)) guard var line = try? makeEncoder().encode(entry) else { return } line.append(0x0A) // '\n' diff --git a/Sources/BlurtEngine/STT/TranscriptionContext.swift b/Sources/BlurtEngine/STT/TranscriptionContext.swift index f3a334b..0fac7a5 100644 --- a/Sources/BlurtEngine/STT/TranscriptionContext.swift +++ b/Sources/BlurtEngine/STT/TranscriptionContext.swift @@ -2,15 +2,16 @@ /// dictation start from the focused app and field. `TranscriptionPrompt.build` /// renders only the parts of it the request prompt uses — the bundle ID (via /// the `AppKindPriming` app-kind instruction), the window title (the language -/// refinement of that instruction), and the key terms — while the rest rides -/// along for the dictation log and the injector's separator logic. +/// refinement of that instruction), and the key terms. The prior text and +/// window title also steer the injector's paste separator; nothing else is +/// consumed, and none of the raw context is sent or logged. /// /// Every focus field is optional and best-effort: whatever couldn't be read is /// `nil`, and an entirely empty context yields no prompt (the server applies /// its own default). public struct TranscriptionContext: Sendable, Equatable { /// The frontmost application's display name (e.g. "Slack", "Xcode"). Not - /// rendered into the prompt; recorded in the dictation log. + /// rendered into the prompt. public let appName: String? /// The frontmost application's bundle identifier (e.g. @@ -28,18 +29,16 @@ public struct TranscriptionContext: Sendable, Equatable { public let windowTitle: String? /// A short label for the focused field (placeholder/title/role, e.g. "To", - /// "Subject", "Search", "Message"). Not rendered into the prompt; recorded - /// in the dictation log. + /// "Subject", "Search", "Message"). Not rendered into the prompt. public let fieldLabel: String? /// Text immediately preceding the insertion point in the focused field. Not /// rendered into the prompt; it drives the injector's leading-separator - /// decision and is recorded in the dictation log. + /// decision. public let priorText: String? /// The text currently selected in the focused field, when any (the paste - /// will replace it). Not rendered into the prompt; recorded in the - /// dictation log. + /// will replace it). Not rendered into the prompt. public let selectedText: String? /// User-configured domain vocabulary (names, jargon, product names) carried as diff --git a/Tests/BlurtEngineTests/DictationLogTests.swift b/Tests/BlurtEngineTests/DictationLogTests.swift index e5a6882..a41e927 100644 --- a/Tests/BlurtEngineTests/DictationLogTests.swift +++ b/Tests/BlurtEngineTests/DictationLogTests.swift @@ -8,15 +8,9 @@ private struct DecodedEntry: Decodable { let ts: String } -/// Decodes the optional focus-context fields so tests can assert they're -/// threaded from the `TranscriptionContext` onto disk. -private struct DecodedContext: Decodable { - let app: String? - let bundle: String? - let window: String? - let field: String? - let prior: String? - let selected: String? +/// Decodes the optional prompt field so tests can assert the exact +/// `config.prompt` that was sent is what lands on disk. +private struct DecodedPrompt: Decodable { let prompt: String? } @@ -93,31 +87,22 @@ struct DictationLogTests { #expect(transcript < ts) } - @Test("threads focus context (incl. selected text) onto disk") - func logsContext() { + @Test("logs only what was sent — raw focus context stays off disk") + func logsOnlyWhatWasSent() { let url = makeTempLogURL() let context = TranscriptionContext( - appName: "Mail", bundleID: "com.apple.mail", windowTitle: "Re: Q3 pricing", - fieldLabel: "Body", priorText: "Hi Sam,", selectedText: "the old plan") + appName: "Obsidian", bundleID: "md.obsidian", windowTitle: "Grocery list", + fieldLabel: "text entry area", priorText: "- milk", selectedText: "- bread") DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) - #expect(decoded?.app == "Mail") - #expect(decoded?.bundle == "com.apple.mail") - #expect(decoded?.window == "Re: Q3 pricing") - #expect(decoded?.field == "Body") - #expect(decoded?.prior == "Hi Sam,") - #expect(decoded?.selected == "the old plan") - } - - @Test("omits the selected field when nothing is selected") - func omitsSelectedWhenAbsent() { - let url = makeTempLogURL() - DictationLog.write(transcript: "p", context: nil, to: url, now: Date()) - let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - // `Encodable` synthesis uses `encodeIfPresent`, so a nil field is absent - // rather than `"selected":null`. - #expect(!line.contains("selected")) + // The prompt that was sent is recorded… + let decoded = try? JSONDecoder().decode(DecodedPrompt.self, from: Data(line.utf8)) + #expect(decoded?.prompt == "Transcribe speech into markdown.") + // …and none of the captured-but-unsent context is. Values, not just keys: + // the entry must carry no trace of what stayed on the machine. + for unsent in ["Obsidian", "md.obsidian", "Grocery list", "text entry area", "- milk", "- bread"] { + #expect(!line.contains(unsent)) + } } @Test("logs the same assembled prompt the transcriber sends") @@ -128,30 +113,18 @@ struct DictationLogTests { fieldLabel: "text entry area", priorText: "- milk", keyTerms: ["Blurt"]) DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) + let decoded = try? JSONDecoder().decode(DecodedPrompt.self, from: Data(line.utf8)) #expect(decoded?.prompt == TranscriptionPrompt.build(context: context)) #expect(decoded?.prompt == "Transcribe speech into markdown. Keywords: Blurt.") } - @Test("the logged prompt carries the app-kind instruction the bundle ID selected") - func logsPromptWithAppKindInstruction() { - let url = makeTempLogURL() - let context = TranscriptionContext( - appName: "Slack", bundleID: "com.tinyspeck.slackmacgap", fieldLabel: "Message", - priorText: nil) - DictationLog.write(transcript: "p", context: context, to: url, now: Date()) - let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - let decoded = try? JSONDecoder().decode(DecodedContext.self, from: Data(line.utf8)) - // The exact wording is TranscriptionPromptTests' contract; here the point is - // that what lands on disk includes the instruction actually sent for this app. - #expect(decoded?.prompt?.contains("Slack message") == true) - } - @Test("omits the prompt field when there is no context to build one") func omitsPromptWhenNoContext() { let url = makeTempLogURL() DictationLog.write(transcript: "p", context: nil, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" + // `Encodable` synthesis uses `encodeIfPresent`, so a nil prompt is absent + // rather than `"prompt":null`. #expect(!line.contains("\"prompt\"")) } @@ -195,10 +168,13 @@ struct DictationLogGateTests { @Test("the gate is checked before the context is touched, for both settings") func gateAppliesToContextualEntries() { - // The pipeline always passes the captured context, which is the part carrying - // prior text and the assembled prompt. Off must persist none of it. + // The pipeline always passes the captured context — the input the logged + // prompt is built from. Off must persist nothing at all; on persists the + // prompt but still never the raw context (prior text stays off disk even + // for a user who opted in). let context = TranscriptionContext( - appName: "1Password", windowTitle: "Vault", fieldLabel: "Password", + appName: "Terminal", bundleID: "com.apple.Terminal", + windowTitle: "Vault", fieldLabel: "Password", priorText: "hunter2", selectedText: nil) let offURL = makeTempLogURL() DictationLog.append( @@ -212,7 +188,9 @@ struct DictationLogGateTests { DictationLog.queue.sync {} #expect(!FileManager.default.fileExists(atPath: offURL.path)) - #expect(readLog(onURL).contains("hunter2")) + let logged = readLog(onURL) + #expect(logged.contains("Transcribe speech into shell commands.")) + #expect(!logged.contains("hunter2")) } } From 4c9a7e5a6b58466593e62c5692da6cbba767cc75 Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Tue, 28 Jul 2026 22:13:48 -0400 Subject: [PATCH 6/8] feat(stt): steer dictation with the fields the API documents for it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `config.prompt` was carrying two things it cannot act on. The Sync STT reference is explicit that the field takes a *description of the audio* ("Cardiology consultation about chest pain symptoms."), not instructions — transcription behavior is optimized out of the box — so the app-kind imperative ("Transcribe speech into markdown.") was aimed at a field that does not reshape output. That is the same category as the filler-removal clause dropped earlier for being a no-op; it was never a wording problem. The docs likewise warn against packing keyword lists into the prompt. Stop sending `prompt` and use the three fields that each do one job: - `conversation_context` — the text before the cursor, as a single turn. Real left-context, so the model knows what the utterance continues. This was captured all along and thrown away at the request boundary. - `keyterms_prompt` — the user's key terms verbatim, replacing the inline `Keywords: a, b, c.` clause, refitted to the field's 2048-char total. - `llm.instruction` — the app-kind clause, reworded from "Transcribe speech into X" to "Format the result as X" now that it addresses the LLM that rewrites the finished transcript rather than the STT decoder. Sending no prompt also keeps the service's managed default, which a custom value replaces wholesale *including its language steering* — the mechanism behind the earlier finding that pinning the prompt to English hurt non-English speech. `TranscriptionPrompt` becomes `TranscriptionSteering`, returning all three fields so the transcriber and the dictation log describe one request instead of each deriving it. Empty fields are omitted rather than sent as `[]`. The log now records every field that was sent, under the wire's own names. Prior-cursor text is on the sent side of that line, so it reaches `~/Library/Logs/Blurt/dictations.jsonl` for a user who turned developer mode on. What keeps a password out of it is unchanged and upstream: FocusCapture skips prior and selected text in secure fields, failing closed when the AX role can't be read. Selected text is still never sent — the paste replaces it, so priming on it would condition the model on text on its way out. Verified with `swift test` (435 tests, 68 suites); the full `scripts/check.sh` has not been run on this change. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/project-guardrails/SKILL.md | 19 ++- AGENTS.md | 106 +++++++----- .../Blurt/Wizard/Steps/KeyTermsStepView.swift | 4 +- BLURTENGINE.md | 8 +- README.md | 4 +- .../BlurtEngine/Config/KeyTermsStore.swift | 8 +- .../BlurtEngine/Pipeline/DictationLog.swift | 58 +++++-- Sources/BlurtEngine/STT/AppKindPriming.swift | 49 +++--- .../STT/AssemblyAITranscriber.swift | 75 ++++++--- .../STT/TranscriptionContext.swift | 51 +++--- .../BlurtEngine/STT/TranscriptionPrompt.swift | 78 --------- .../STT/TranscriptionSteering.swift | 114 +++++++++++++ .../AppKindPrimingTests.swift | 33 +++- .../AssemblyAITranscriberTests.swift | 66 +++++--- .../BlurtEngineTests/DictationLogTests.swift | 91 +++++++--- .../TranscriptionContextTests.swift | 41 ++--- .../TranscriptionPromptTests.swift | 128 -------------- .../TranscriptionSteeringTests.swift | 159 ++++++++++++++++++ 18 files changed, 671 insertions(+), 421 deletions(-) delete mode 100644 Sources/BlurtEngine/STT/TranscriptionPrompt.swift create mode 100644 Sources/BlurtEngine/STT/TranscriptionSteering.swift delete mode 100644 Tests/BlurtEngineTests/TranscriptionPromptTests.swift create mode 100644 Tests/BlurtEngineTests/TranscriptionSteeringTests.swift diff --git a/.claude/skills/project-guardrails/SKILL.md b/.claude/skills/project-guardrails/SKILL.md index 8dca901..42671d3 100644 --- a/.claude/skills/project-guardrails/SKILL.md +++ b/.claude/skills/project-guardrails/SKILL.md @@ -23,15 +23,22 @@ one, stop and ask the user first. This is the fast "don't" list; AGENTS.md's - **No streaming STT.** The AssemblyAI Sync API returns the full transcript in one response. Overlay goes "Transcribing…" → full text. -- **No separate LLM cleanup pass.** Cleanup rides in the Sync STT request's - `config.prompt` (`TranscriptionPrompt`). No LLM Gateway client, no +- **No separate LLM cleanup pass.** Cleanup rides in the dictation request's + server-side `llm` block (`TranscriptionSteering`). No LLM Gateway client, no `StylerProtocol`, no post-transcription styling stage. - **No local models / model downloads.** Transcription is a remote AssemblyAI call. No on-device ASR/LLM, no model cache, no download UI. -- Don't reintroduce a "remove filler words (um, uh, like)" directive in the - prompt — `universal-3-5-pro` ignores it; it was deliberately dropped. Same for - a language directive: pinning the prompt to English hurt non-English speech, so - language is left to the model's own detection. +- **Never send `config.prompt`.** It takes a _description of the audio_, not + instructions, and a custom value replaces the service's managed default + including its language steering. Formatting goes in `llm.instruction` + ("Format the result as markdown."), vocabulary in `keyterms_prompt`, the text + before the cursor in `conversation_context`. An imperative like "Transcribe + speech into markdown." in `prompt` is a measured no-op — that is exactly why + these three fields exist. +- Don't reintroduce a "remove filler words (um, uh, like)" directive — the STT + prompt doesn't act on it; it was deliberately dropped, and disfluency removal + is the LLM rewrite's job. Same for a language directive: pinning to English + hurt non-English speech, so language is left to the model's own detection. - **Injection is always a clipboard paste** (save → write → ⌘V → settle → restore), degrading to "left it on the clipboard" when the target is lost. No keystroke-by-keystroke typing path, no length threshold. diff --git a/AGENTS.md b/AGENTS.md index e4db4ba..db70738 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,11 +7,11 @@ the Claude-Code-specific tooling under `.claude/` (hooks, skills, subagents). Blurt is a macOS dictation app powered by [AssemblyAI](https://www.assemblyai.com). Tap or hold a trigger key, speak, and polished text is pasted into the focused app. Transcription is **one remote -AssemblyAI dictation API call**: a per-utterance `prompt` (an app-kind transcription instruction -recognized from the frontmost app, plus the user's key terms) rides along with the -request, and the same request asks the service for its server-side LLM cleanup rewrite -(`config.llm`), so the text that comes back is already polished. The user supplies their own API -key. +AssemblyAI dictation API call**: per-utterance steering rides along with the request — the text before +the cursor as `conversation_context`, the user's key terms as `keyterms_prompt`, and an app-kind +formatting clause recognized from the frontmost app — and the same request asks the service for its +server-side LLM cleanup rewrite (`config.llm`), so the text that comes back is already polished. The +user supplies their own API key. Four reflexes before you touch anything: @@ -53,7 +53,7 @@ Sources/BlurtEngine/ the engine (dependency-free Swift package) Injection/ KeyInjector (clipboard paste), SystemClipboard Permissions/ PermissionsChecker (mic + Accessibility) Pipeline/ DictationSession (actor) + phases, UI projections, geometry, log - STT/ AssemblyAITranscriber, TranscriptionPrompt/Context, AppKindPriming, SyncSTTLimits + STT/ AssemblyAITranscriber, TranscriptionSteering/Context, AppKindPriming, SyncSTTLimits Update/ UpdateChecker (download-only) + the launch-check policy App/Blurt/ project.yml XcodeGen source of truth — Blurt.xcodeproj is GENERATED @@ -164,10 +164,13 @@ Each was tried the other way and reverted. If a task seems to require one, stop | Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | | Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. | | Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. | -| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `TranscriptionPrompt`. | +| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — request steering belongs in `TranscriptionSteering`. | | Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | -| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection. | -| Add a "remove filler words (um, uh, like)" clause | Not in the STT model's trained instruction set — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | +| Send `config.prompt` at all | The field takes a _description of the audio_, not instructions, and a custom value replaces the service's managed default **including its language steering**. Formatting goes in `llm.instruction`, vocabulary in `keyterms_prompt`, preceding text in `conversation_context`. | +| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection — and a custom prompt is what drops the managed default's language steering in the first place. | +| Put formatting instructions in `config.prompt` | Reshaping output is not something the STT prompt acts on, so _"Transcribe speech into markdown."_ was a measured no-op. `llm.instruction` (_"Format the result as markdown."_) is the lever that works. | +| Pack key terms into the prompt as `Keywords: a, b, c.` | The API documents `keyterms_prompt` for exactly this, and warns against packing keyword lists into the prompt. | +| Add a "remove filler words (um, uh, like)" clause | Not something the STT prompt acts on — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | | Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | | Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | | Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | @@ -242,11 +245,13 @@ the seam they inject. Implements `TranscriberProtocol` against AssemblyAI's **dictation** API: a single `POST https://dictation.assemblyai.com/transcribe` with the captured audio as a raw S16LE PCM blob -in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, `prompt`, and an -empty `llm` block). No model header — the service pins the STT model server-side. The `prompt` -(built per utterance by `TranscriptionPrompt`) steers _transcription_; the `llm` block asks the -service to run its default LLM cleanup rewrite (remove disfluencies, fix punctuation) over the -verbatim transcript, all inside the same request. The response carries both `text` (verbatim) and +in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, and the steering +fields built per utterance by `TranscriptionSteering`: `conversation_context`, `keyterms_prompt`, and +the `llm` block). No model header — the service pins the STT model server-side. **`prompt` is never +sent** — see [Transcription steering](#transcription-steering). The `llm` block asks the service to +run a cleanup rewrite (remove disfluencies, fix punctuation) over the verbatim transcript, all inside +the same request; its optional `instruction` is the app-kind formatting clause, and an empty block +selects the service's own default cleanup. The response carries both `text` (verbatim) and `llm_response` (the rewrite); the transcriber returns the rewrite and falls back to `text` when `llm_response` is null — the rewrite is best-effort (5 s server-side budget), so a rewrite failure (`llm_error`) is a logged degradation, never a user-facing error. @@ -373,36 +378,46 @@ the `@AppStorage(TriggerKeyStore.defaultsKey)` + `TriggerKey.fromPersisted` pair restating that pairing per view. The unset default belongs to `fromPersisted` (an absent keycode maps to right ⌘), so views must not re-declare `TriggerKey.rightCommand.rawValue` themselves. -## Transcription prompt - -`Sources/BlurtEngine/STT/TranscriptionPrompt.swift` builds the instruction passed as the dictation -request's `config.prompt` — it steers the _transcription_, not the LLM rewrite (that's the request's -separate `llm` block). It's unit-tested in `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. - -The built prompt is deliberately minimal — two clauses, each optional. `AppKindPriming` recognizes -the frontmost app's bundle ID as a terminal, code editor, Slack, or Obsidian and renders the -app-kind instruction (_"Transcribe speech into markdown."_ — for a code editor, the language is -inferred from the window title's filename: _"Transcribe speech into Swift code."_); the user's key -terms trail it as inline keyword boosting (`Keywords: a, b, c.`), fitted to the dictation API's -documented 4096-character cap on `config.prompt` (`characterCap`). Wording is phrased per -AssemblyAI's Universal-3 Pro prompting guidance (positive/authoritative, no -"Don't"/"Avoid"/"Never"). - -The rest of the captured focus context — window title, app and field names, prior-cursor text, the -selected text — is deliberately **not** rendered into the prompt: real-world logs showed it -crowding the instruction (VS Code, for one, parks a screen-reader help announcement in the focused -field's description). It is still captured — the injector's separator logic -consumes the prior text and window title, and the code-editor language refinement reads the window -title — but none of it is written to the dictation log, and reading it stays privacy-guarded (prior -and selected text are skipped in secure fields, detected by AX role **or** subrole and failing -closed when the role can't be read, so a password is never read out of the field at all). - -`build(context:)` returns `nil` when there's no usable context (or when nothing renders — focus -signals from an unrecognized app, no key terms), and passing `prompt: nil` to the transcriber omits -the field so the server applies its own default. Three omissions are deliberate — no -annotation-suppression clause (_"Transcribe without speaker labels, …"_ is already part of the -dictation service's own default prompt), no language directive, and no filler-word clause; see -[Settled decisions](#settled-decisions--dont-reintroduce-these). +## Transcription steering + +`Sources/BlurtEngine/STT/TranscriptionSteering.swift` renders the captured context into the three +request-customization fields, each with one job. It's unit-tested in +`Tests/BlurtEngineTests/TranscriptionSteeringTests.swift`. + +| Field | Carries | Cap | +| ---------------------- | -------------------------------------------------------------- | ----------------------- | +| `conversation_context` | prior-cursor text, as a single turn | 4096 chars, clip head | +| `keyterms_prompt` | the user's key terms, verbatim | 2048 chars, whole terms | +| `llm.instruction` | the `AppKindPriming` formatting clause, or absent for the default | — | + +**`config.prompt` is never sent, and that is the whole point of this design.** The field takes a +_description of the audio_ ("Cardiology consultation about chest pain symptoms."), not instructions — +transcription behavior is optimized out of the box — so the app-kind priming's old imperative form +(_"Transcribe speech into markdown."_) was aimed at a field that doesn't act on instructions and was a +no-op. Formatting now rides in `llm.instruction`, where an LLM actually rewrites the text +(_"Format the result as markdown."_ — for a code editor, the language is inferred from the window +title's filename: _"Format the result as Swift code."_), and vocabulary rides in `keyterms_prompt` +rather than being packed into the prompt as a `Keywords: a, b, c.` clause. Sending no prompt also +keeps the service's managed default, which a custom prompt replaces wholesale — **including its +language steering**, which is the mechanism behind the older finding that pinning the prompt to +English hurt non-English speech. + +The remaining focus context is **not** sent: app and field names render nowhere (real-world logs +showed them crowding the request — VS Code, for one, parks a screen-reader help announcement in the +focused field's description), the window title is read only to name a code editor's language, and +selected text is never priming because the paste replaces it. All of it is still captured — the +injector's separator logic consumes the prior text and window title — and reading stays +privacy-guarded: prior and selected text are skipped in secure fields, detected by AX role **or** +subrole and failing closed when the role can't be read, so a password is never read out of the field +at all. That guard, not the steering builder, is what keeps a password out of `conversation_context` +and off the dictation log. + +`build(context:)` returns `.empty` when there's no usable context (or when nothing renders — an +unrecognized app with no prior text and no key terms), and every empty field is **omitted** from the +JSON rather than sent as `[]` or `null`, so the service applies its own defaults. Three omissions are +deliberate — no annotation-suppression clause (_"Transcribe without speaker labels, …"_ is already +part of the dictation service's own default prompt), no language directive, and no filler-word clause; +see [Settled decisions](#settled-decisions--dont-reintroduce-these). ## Settings, persistence, and cues @@ -433,7 +448,8 @@ resolves. History: **`RecentDictations`** is an in-memory, newest-first ring shown in the ready window (never written to disk). **`DictationLog`** appends each completed dictation — the transcript plus the exact -`config.prompt` sent — to +steering fields sent (`conversation_context`, `keyterms_prompt`, `llm_instruction`, under the wire's +own names) — to `~/Library/Logs/Blurt/dictations.jsonl` (`DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI — derived next to the URL so the label can't drift from the write target) — but **only** while developer mode is on; with it off, nothing is written. The Settings diff --git a/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift b/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift index bd6027d..871f018 100644 --- a/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift +++ b/App/Blurt/Blurt/Wizard/Steps/KeyTermsStepView.swift @@ -3,8 +3,8 @@ import SwiftUI /// The "Key Terms" section of the Settings window: a free-text /// area where the user lists comma-separated domain words (names, jargon, product -/// names). These are folded into every transcription's prompt as spelling priming -/// (see `KeyTermsStore` / `TranscriptionPrompt.build`), so the model favors those +/// names). These ride on every request as its `keyterms_prompt` vocabulary list +/// (see `KeyTermsStore` / `TranscriptionSteering.build`), so the model favors those /// spellings. Optional — it never gates setup; an empty list just sends no terms. struct KeyTermsStepView: View { /// Stored in UserDefaults so multiple settings windows/readers see edits live. diff --git a/BLURTENGINE.md b/BLURTENGINE.md index ab92d0d..dbb2ea5 100644 --- a/BLURTENGINE.md +++ b/BLURTENGINE.md @@ -59,7 +59,7 @@ press() ──▶ MicCapture.start() release() ──▶ MicCapture.s Key properties of the design, which your integration can rely on: - **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream. -- **Cleanup happens server-side.** The request's empty `llm` block asks the service for its default cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; the per-utterance `config.prompt` (built by `TranscriptionPrompt` from the captured context) primes the _transcription_. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. +- **Cleanup happens server-side.** The request's `llm` block asks the service for a cleanup rewrite (remove disfluencies, fix punctuation), applied to the verbatim transcript inside the same call; its optional `instruction` carries the app-kind formatting clause, and an empty block selects the service's own default cleanup. Recognition is primed separately by `conversation_context` and `keyterms_prompt` (built by `TranscriptionSteering` from the captured context); `config.prompt` is never sent. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one. - **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read. - **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400. @@ -153,14 +153,14 @@ The session calls `setTargetApp` at press time with the app that was frontmost w Recognition quality comes from per-utterance priming, assembled automatically inside `press()` — hosts don't call these APIs directly, but should know what's collected: - **`TranscriptionContext`** carries the frontmost app name and bundle identifier, window title, focused-field label, the text before the caret, the selected text (which a paste will replace), and the user's key terms. It's captured via Accessibility at press time (skipped in secure fields), off the hot path — and consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`), so a hung read is abandoned rather than stalling the transcript. Only the bundle ID, window title, and key terms reach the prompt; the prior text and window title also steer the injector's paste separator, and nothing else is consumed. -- **`TranscriptionPrompt.build(context:)`** renders that into the dictation request's `config.prompt` (transcription steering only — the cleanup rewrite is the separate `llm` block). The prompt is deliberately minimal: when the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian), it is that one app-kind instruction, e.g. "Transcribe speech into shell commands." or, for a code editor, the language inferred from the window title's filename ("Transcribe speech into Swift code."); the user's key terms trail it as `Keywords: a, b, c.`, fitted to the API's 4096-character cap. Nothing else is rendered — no window/app/field/prior/selected context, and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default prompt). A context that renders nothing yields `nil`, which omits the field so the server applies its own default. Two further deliberate omissions: no language directive (pinning to English hurt non-English speech) and no "remove filler words" clause (not in the model's trained instruction set — a no-op). Don't reintroduce either. +- **`TranscriptionSteering.build(context:)`** renders that into the request's three customization fields: the prior-cursor text as the single `conversation_context` turn (clipped to 4096 chars keeping the tail, since the words nearest the cursor carry the continuity), the user's key terms as `keyterms_prompt` (whole terms fitted to 2048 chars total), and — when the bundle ID identifies a recognized destination kind (`AppKindPriming` — terminals, code editors, Slack, Obsidian) — the formatting clause as `llm.instruction`, e.g. "Format the result as a shell command with no trailing period." or, for a code editor, the language inferred from the window title's filename ("Format the result as Swift code."). **`config.prompt` is never sent**: it takes a description of the audio rather than instructions, and a custom value replaces the service's managed default including its language steering, which is why the app-kind clause moved to the `llm` block and the key terms to their own field. Nothing else is rendered — no app/field names, no selected text (the paste replaces it), and no standing annotation-suppression clause ("Transcribe without speaker labels, …" is part of the service's own default). Empty fields are omitted rather than sent as `[]`. Two further deliberate omissions: no language directive and no "remove filler words" clause (not something the STT prompt acts on — a no-op). Don't reintroduce either. - **`KeyTermsStore`** persists the user's domain vocabulary (names, jargon) in `UserDefaults`; `DictationSession` re-reads it at every press via its `keyTermsProvider` closure, so Settings edits apply to the next utterance without rebuilding the session. Pass your own provider to source terms from elsewhere. For key storage, compose against **`APIKeyGateway`** — the injectable `current` / `save(_:)` / `hasKey` seam over the key store. `ProductionAPIKeyStore` forwards to the Keychain-backed `APIKeyStore`; `InMemoryAPIKeyStore` is a ready-made in-memory conformance for tests and harnesses (Blurt's XCUITest runs use it so the real Keychain item is never touched, and its `hasKey` backs the session's `readinessCheck`). For a settings UI, **`APIKeySubmission`** wraps the gateway with the validate-then-save flow (`submit(_:)` → valid / invalid / unreachable / saveFailed, via `APIKeyValidator`): it saves only a key AssemblyAI actively accepts, so an unverified key never persists. Two projections keep the surrounding UI out of your views: `Outcome.failureReport` classifies a failure as `.inline(message:)` (recoverable — show it beside the field) or `.alert(title:message:)` (a Keychain fault retyping can't fix), and **`APIKeyDisplay.resolve(key:)`** renders the stored key for an account row — masked tail, status and VoiceOver wording, and the connect-vs-rotate control titles. The mask reveals only the last `revealedTailLength` characters and, below `minimumLengthToMask`, none at all, so a short key can't be shown whole. Setup gating has a projection too: **`SetupReadiness.isReady(permissions:hasAPIKey:)`** is the "fully configured" rule (deliberately excluding the trigger key, which has a default), `SetupReadiness.pollInterval(isReady:)` is the permission-poll cadence (brisk during setup, coasting once ready), and `PermissionStatus.lostGrant(since:)` detects a permission revoked out from under a configured app. -Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact `config.prompt` sent, never the raw focus context — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section. +Each completed dictation is appended to **`DictationLog`** (a local JSONL history at `~/Library/Logs/Blurt/dictations.jsonl` — `DictationLog.defaultURL`, or `defaultDisplayPath` for the home-abbreviated form to show in UI) — each entry is the transcript plus the exact steering fields sent (`conversation_context`, `keyterms_prompt`, `llm_instruction`), never the context that wasn't sent — but only while developer mode is switched on. **`DeveloperModeStore`** persists that opt-in in `UserDefaults` (`BlurtDeveloperMode`, off by default); with it off, nothing is written to disk. Blurt surfaces the switch (and the log path) in the Settings window's Developer section. ## Hotkey building blocks @@ -195,7 +195,7 @@ Run `swift test` for the engine suites (`--filter DictationSessionTests` for one Each of these was tried the other way and reverted; the longer stories are in [AGENTS.md](AGENTS.md) and the source comments: - **No external SPM dependencies in the engine.** Foundation/Security/AVFoundation only. -- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and transcription steering belongs in `TranscriptionPrompt`. +- **No streaming STT, no local models, no client-side LLM cleanup pass.** One dictation request per utterance is the architecture; the cleanup rewrite is server-side (the request's `llm` block), and request steering belongs in `TranscriptionSteering`. - **No `AVAudioEngine`/`installTap` capture path.** Fresh `AVAudioRecorder` per session, resolved at record time. - **Paste is always clipboard-based** (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation for lost targets. - **No English-pinning or filler-word clauses in the prompt.** diff --git a/README.md b/README.md index 864bb48..9dd68c0 100644 --- a/README.md +++ b/README.md @@ -166,14 +166,14 @@ Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dep Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM, live level meter; DX7/Juno-106 sound packs STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe - (STT + LLM rewrite) + TranscriptionPrompt app-kind instruction + (STT + LLM rewrite) + TranscriptionSteering context/keyterms/format Pipeline/ DictationSession actor: press/release/cancel commands, phase stream, auto-release before the API's recording cap Hotkey/ DictationKeyGate/Router: pure, unit-tested state machine for the lone-modifier trigger (tap vs hold vs combo) Injection/ KeyInjector: save clipboard → paste via synthesized ⌘V → restore FocusCapture/ Accessibility reads of the focused app/window/field feeding the - prompt's app-kind instruction, the log, and paste separators + request's steering fields, the log, and paste separators Config/, Update/ Keychain API-key store, key terms, download-only release check App/Blurt/ AppKit/SwiftUI shell (Xcode project generated by XcodeGen) diff --git a/Sources/BlurtEngine/Config/KeyTermsStore.swift b/Sources/BlurtEngine/Config/KeyTermsStore.swift index de9e223..c96488f 100644 --- a/Sources/BlurtEngine/Config/KeyTermsStore.swift +++ b/Sources/BlurtEngine/Config/KeyTermsStore.swift @@ -1,9 +1,9 @@ import Foundation /// Storage for the user's dictation "key terms" — a comma-separated list of -/// domain words (names, jargon, product names) that get folded into the dictation -/// request `prompt` as vocabulary priming, so the model is more likely to spell -/// them correctly (see `TranscriptionPrompt.build`). +/// domain words (names, jargon, product names) sent as the dictation request's +/// `keyterms_prompt` vocabulary list, so the model is more likely to spell +/// them correctly (see `TranscriptionSteering.build`). /// /// Unlike the API key these aren't secret, so they live in `UserDefaults` rather /// than the Keychain. The transcription pipeline reads the parsed list via @@ -37,7 +37,7 @@ public enum KeyTermsStore { } /// Pure parse of a comma-separated string into a clean term list. Exposed so - /// `TranscriptionPrompt` and tests can reuse the exact same rules. + /// `TranscriptionSteering` and tests can reuse the exact same rules. public static func parse(_ text: String?) -> [String] { guard let text else { return [] } var seen = Set() diff --git a/Sources/BlurtEngine/Pipeline/DictationLog.swift b/Sources/BlurtEngine/Pipeline/DictationLog.swift index 0ed5d5a..1974874 100644 --- a/Sources/BlurtEngine/Pipeline/DictationLog.swift +++ b/Sources/BlurtEngine/Pipeline/DictationLog.swift @@ -1,26 +1,57 @@ import Foundation /// Append-only JSONL log of completed dictations — the transcript that came -/// back and the exact `config.prompt` that was sent — at +/// back and every request field that was sent to get it — at /// `~/Library/Logs/Blurt/dictations.jsonl`. Used to build a real-world /// corpus for prompt iteration. Written only while developer mode is switched /// on (`DeveloperModeStore` — the Settings window's Developer section, which /// also displays this path), so a user who never opts in has no dictation /// text on disk. public enum DictationLog { - /// One logged dictation: what came back (`transcript`), when, and exactly - /// what was sent (`prompt`). Nothing else — the raw focus context (app and - /// field names, window title, prior/selected text) deliberately stays off - /// disk: it isn't sent to the service, so it doesn't belong in a log of the - /// exchange. + /// One logged dictation: what came back (`transcript`), when, and exactly what + /// was sent to steer it. Fields carry the request's own wire names so a log + /// line reads as the request it describes. + /// + /// Only what was *sent* is recorded. The captured-but-unsent focus context — + /// app and field names, the window title, selected text — deliberately stays + /// off disk. Prior-cursor text is on the sent side of that line now that it + /// rides as `conversation_context`; what keeps a password out of it is + /// `FocusCapture`, which skips prior and selected text in secure fields + /// entirely (failing closed when the AX role can't be read), so it never + /// reaches a context in the first place. struct Entry: Encodable { let transcript: String let ts: String - /// The fully-assembled `config.prompt` sent to AssemblyAI for this - /// utterance. Built here from `context` (rather than threaded through from - /// the transcriber) so the log always reflects what was actually sent, - /// even for calls that construct an entry directly from a context. - let prompt: String? + /// The steering fields sent to AssemblyAI for this utterance. Built here + /// from `context` (rather than threaded through from the transcriber) so the + /// log always reflects what was actually sent, even for calls that construct + /// an entry directly from a context. + let conversationContext: [String] + let keytermsPrompt: [String] + let llmInstruction: String? + + enum CodingKeys: String, CodingKey { + case transcript + case ts + case conversationContext = "conversation_context" + case keytermsPrompt = "keyterms_prompt" + case llmInstruction = "llm_instruction" + } + + /// Mirrors `DictationConfig.encode(to:)`: an empty array omits its field, so + /// a line states only what the request actually carried. + func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + try container.encode(transcript, forKey: .transcript) + try container.encode(ts, forKey: .ts) + if !conversationContext.isEmpty { + try container.encode(conversationContext, forKey: .conversationContext) + } + if !keytermsPrompt.isEmpty { + try container.encode(keytermsPrompt, forKey: .keytermsPrompt) + } + try container.encodeIfPresent(llmInstruction, forKey: .llmInstruction) + } } /// Where the log lives. Public so the Settings window's Developer section @@ -83,9 +114,12 @@ public enum DictationLog { static func write( transcript: String, context: TranscriptionContext? = nil, to url: URL, now: Date ) { + let steering = TranscriptionSteering.build(context: context) let entry = Entry( transcript: transcript, ts: now.formatted(timestampFormat), - prompt: TranscriptionPrompt.build(context: context)) + conversationContext: steering.conversationContext, + keytermsPrompt: steering.keyterms, + llmInstruction: steering.rewriteInstruction) guard var line = try? makeEncoder().encode(entry) else { return } line.append(0x0A) // '\n' diff --git a/Sources/BlurtEngine/STT/AppKindPriming.swift b/Sources/BlurtEngine/STT/AppKindPriming.swift index af93021..3620173 100644 --- a/Sources/BlurtEngine/STT/AppKindPriming.swift +++ b/Sources/BlurtEngine/STT/AppKindPriming.swift @@ -1,24 +1,29 @@ import Foundation -/// The app-kind transcription instruction: recognizes what *kind* of app the +/// The app-kind formatting instruction: recognizes what *kind* of app the /// dictation targets (a terminal, a code editor, Slack, Obsidian) from the -/// frontmost app's bundle identifier and renders the one instruction sentence -/// `TranscriptionPrompt` sends — "Transcribe speech into shell commands." / -/// "… into Swift code." / "… into a casual Slack message with emoji." / -/// "… into markdown." — telling the model what shape of text the destination -/// expects, which the app's display name alone doesn't convey. +/// frontmost app's bundle identifier and renders the one sentence +/// `TranscriptionSteering` sends as `config.llm.instruction` — "Format the +/// result as a shell command with no trailing period." / "… as Swift code." / +/// "… as a casual Slack message, using Slack emoji where they fit." / "… as +/// markdown." — telling the rewrite what shape of text the destination expects, +/// which the app's display name alone doesn't convey. /// /// For code editors the window title usually names the open file, so the /// clause names the language inferred from that filename's extension -/// ("Transcribe speech into Python code.") when one is recognizable, and stays -/// generic ("… into code.") otherwise. +/// ("Format the result as Python code.") when one is recognizable, and stays +/// generic ("… as code.") otherwise. +/// +/// These are instructions to the **LLM that rewrites the finished transcript**, +/// not to the STT model. Reshaping output is not something `config.prompt` acts +/// on — that field takes a description of the audio — so a clause phrased as +/// "Transcribe speech into markdown." was a no-op wherever it landed. Keep the +/// imperative "Format the result as …" shape. /// /// Detection keys on bundle IDs, not display names: names are localized and /// user-editable, while the bundle ID is the app's stable identity. An -/// unrecognized app contributes no clause — the request then carries no -/// instruction and the service's own default prompt applies. Wording follows -/// Universal-3 Pro prompting guidance (positive/authoritative phrasing, no -/// negations). Exercised by +/// unrecognized app contributes no clause — the request's `llm` block is then +/// empty and the service's own default cleanup instruction applies. Exercised by /// `Tests/BlurtEngineTests/AppKindPrimingTests.swift`. enum AppKindPriming { /// The recognized destination families. Each renders one guidance sentence; @@ -73,21 +78,23 @@ enum AppKindPriming { return kindsByBundleIDPrefix.first { bundleID.hasPrefix($0.prefix) }?.kind } - /// The instruction sentence for the app `bundleID` identifies, or `nil` when + /// The formatting instruction for the app `bundleID` identifies, or `nil` when /// the app isn't recognized. `windowTitle` refines the code-editor clause /// with the open file's language; the other kinds ignore it. static func clause(bundleID: String?, windowTitle: String?) -> String? { guard let kind = kind(ofBundleID: bundleID) else { return nil } switch kind { case .terminal: - return "Transcribe speech into shell commands." + // "Trailing period", not "terminal punctuation": in a clause about + // terminals the latter reads as the app, not the end of a sentence. + return "Format the result as a shell command with no trailing period." case .codeEditor: let subject = windowTitle.flatMap(language(inWindowTitle:)) ?? "code" - return "Transcribe speech into \(subject)." + return "Format the result as \(subject)." case .slack: - return "Transcribe speech into a casual Slack message with emoji." + return "Format the result as a casual Slack message, using Slack emoji where they fit." case .obsidian: - return "Transcribe speech into markdown." + return "Format the result as markdown." } } @@ -112,10 +119,10 @@ enum AppKindPriming { /// markers, quotes, brackets, dash separators. private static let filenameTrim = CharacterSet(charactersIn: "\"'`•●◆*()[]{}<>,;:—–-") - /// Filename extension → what the clause says speech becomes. Values complete - /// "Transcribe speech into …", so languages carry a trailing "code" while - /// markup/data formats stand alone. Lowercased keys; lookups lowercase the - /// extension first. + /// Filename extension → what the clause says the result should be. Values + /// complete "Format the result as …", so languages carry a trailing "code" + /// while markup/data formats stand alone. Lowercased keys; lookups lowercase + /// the extension first. private static let languagesByExtension: [String: String] = [ "c": "C code", "h": "C code", "cc": "C++ code", "cpp": "C++ code", "cxx": "C++ code", "hpp": "C++ code", diff --git a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift index dcd2353..6071e43 100644 --- a/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift +++ b/Sources/BlurtEngine/STT/AssemblyAITranscriber.swift @@ -51,8 +51,8 @@ public struct AssemblyAITranscriber: TranscriberProtocol { guard let apiKey = apiKeyProvider(), !apiKey.isEmpty else { throw BlurtError.apiKeyMissing } - let prompt = TranscriptionPrompt.build(context: context) - let config = try makeConfigData(sampleRate: sampleRate, prompt: prompt) + let steering = TranscriptionSteering.build(context: context) + let config = try makeConfigData(sampleRate: sampleRate, steering: steering) let boundary = "blurt-\(UUID().uuidString)" var request = URLRequest(url: baseURL.appendingPathComponent("transcribe")) @@ -105,18 +105,21 @@ public struct AssemblyAITranscriber: TranscriberProtocol { "warm-up connect \(elapsedMs, format: .fixed(precision: 0), privacy: .public)ms") } - /// Builds the JSON `config` part sent alongside the audio. The context - /// `prompt` is included only when non-empty; a nil or blank prompt omits the - /// field so the server applies its default prompt. The `llm` block always - /// rides along — see `DictationConfig.llm`. Internal so tests can assert the - /// prompt wiring without inspecting the multipart upload body (which - /// `URLProtocol` mocks can't observe reliably for `upload(from:)`). - func makeConfigData(sampleRate: Int, prompt: String?) throws -> Data { + /// Builds the JSON `config` part sent alongside the audio. Each steering field + /// is included only when it carries something — an empty array or a nil + /// instruction omits the field rather than stating an empty value, so the + /// service applies its own default. The `llm` block always rides along — see + /// `DictationConfig.llm`. Internal so tests can assert the steering wiring + /// without inspecting the multipart upload body (which `URLProtocol` mocks + /// can't observe reliably for `upload(from:)`). + func makeConfigData(sampleRate: Int, steering: TranscriptionSteering.Fields) throws -> Data { try JSONEncoder().encode( DictationConfig( sampleRate: sampleRate, channels: 1, - prompt: prompt.trimmedNonEmpty() + conversationContext: steering.conversationContext, + keytermsPrompt: steering.keyterms, + llm: LLMRewrite(instruction: steering.rewriteInstruction) ) ) } @@ -194,28 +197,56 @@ public struct AssemblyAITranscriber: TranscriberProtocol { // MARK: - Wire types + /// The JSON `config` part. Note the absence of `prompt`: that field takes a + /// description of the audio rather than instructions, and a custom value + /// replaces the service's managed default (language steering included), so + /// Blurt sends none — see `TranscriptionSteering` for the full reasoning. private struct DictationConfig: Encodable { let sampleRate: Int let channels: Int - /// Custom transcription instruction. Encoded only when non-nil (the - /// synthesized `encode` uses `encodeIfPresent` for optionals), so omitting - /// it falls back to the server's default prompt. Steers *transcription*; - /// the cleanup rewrite is the `llm` block's job. - let prompt: String? - /// The rewrite request. An empty object selects the service's default - /// cleanup instruction; per the API's `instruction`-mode rules, output - /// format and don't-answer-the-text safeguards are enforced server-side, - /// so nothing rides along here. - let llm = LLMRewrite() + /// Turns preceding this utterance (Blurt sends at most one: the text before + /// the insertion point). Encoded only when non-nil — the synthesized + /// `encode` uses `encodeIfPresent` for optionals — so an utterance with no + /// prior text omits the field instead of sending `[]`. + let conversationContext: [String] + /// The user's key terms as the explicit vocabulary list, omitted when empty. + let keytermsPrompt: [String] + /// The rewrite request. Always present so the service runs the rewrite at + /// all; its `instruction` is the app-kind formatting clause, or absent to + /// select the service's default cleanup instruction. Per the API's + /// `instruction`-mode rules, output format and don't-answer-the-text + /// safeguards are enforced server-side either way. + let llm: LLMRewrite enum CodingKeys: String, CodingKey { case sampleRate = "sample_rate" case channels - case prompt + case conversationContext = "conversation_context" + case keytermsPrompt = "keyterms_prompt" case llm } + + /// Hand-written so an empty array *omits* its field rather than encoding + /// `[]`. The distinction is on the wire: `"keyterms_prompt": []` states an + /// empty vocabulary, while omission leaves the service to its own handling. + /// (Optional arrays would express this too, but read as "maybe no list" + /// where the truth is "a list, possibly empty".) + func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + try container.encode(sampleRate, forKey: .sampleRate) + try container.encode(channels, forKey: .channels) + if !conversationContext.isEmpty { + try container.encode(conversationContext, forKey: .conversationContext) + } + if !keytermsPrompt.isEmpty { + try container.encode(keytermsPrompt, forKey: .keytermsPrompt) + } + try container.encode(llm, forKey: .llm) + } } - private struct LLMRewrite: Encodable {} + private struct LLMRewrite: Encodable { + let instruction: String? + } private struct DictationResponse: Decodable { /// The verbatim transcript — always present, never altered by the LLM. diff --git a/Sources/BlurtEngine/STT/TranscriptionContext.swift b/Sources/BlurtEngine/STT/TranscriptionContext.swift index 0fac7a5..7e637b5 100644 --- a/Sources/BlurtEngine/STT/TranscriptionContext.swift +++ b/Sources/BlurtEngine/STT/TranscriptionContext.swift @@ -1,48 +1,53 @@ /// Per-utterance snapshot of where the dictation is going, gathered at -/// dictation start from the focused app and field. `TranscriptionPrompt.build` -/// renders only the parts of it the request prompt uses — the bundle ID (via -/// the `AppKindPriming` app-kind instruction), the window title (the language -/// refinement of that instruction), and the key terms. The prior text and +/// dictation start from the focused app and field. `TranscriptionSteering.build` +/// renders only the parts the request uses — the prior text (as +/// `conversation_context`), the key terms (as `keyterms_prompt`), the bundle ID +/// (via the `AppKindPriming` formatting clause in `llm.instruction`), and the +/// window title (the language refinement of that clause). The prior text and /// window title also steer the injector's paste separator; nothing else is -/// consumed, and none of the raw context is sent or logged. +/// consumed, and the rest of the context is neither sent nor logged. /// /// Every focus field is optional and best-effort: whatever couldn't be read is -/// `nil`, and an entirely empty context yields no prompt (the server applies -/// its own default). +/// `nil`, and an entirely empty context customizes nothing (the server applies +/// its own defaults). public struct TranscriptionContext: Sendable, Equatable { - /// The frontmost application's display name (e.g. "Slack", "Xcode"). Not - /// rendered into the prompt. + /// The frontmost application's display name (e.g. "Slack", "Xcode"). Never + /// sent. public let appName: String? /// The frontmost application's bundle identifier (e.g. - /// "com.tinyspeck.slackmacgap"). Never rendered verbatim: it keys the - /// app-*kind* recognition (`AppKindPriming`) that selects the prompt's - /// transcription instruction — terminal, code editor, Slack, Obsidian. + /// "com.tinyspeck.slackmacgap"). Never sent verbatim: it keys the + /// app-*kind* recognition (`AppKindPriming`) that selects the rewrite's + /// formatting instruction — terminal, code editor, Slack, Obsidian. /// Preferred over `appName` for recognition because display names are /// localized and user-editable while the bundle ID is stable. public let bundleID: String? /// The focused window's title (e.g. "main.py — blurt", a document name, a - /// Slack channel). In a code editor it usually names the open file, which is - /// how the app-kind instruction learns the language ("… into Swift code."); - /// it also anchors the injector's same-window separator fallback. + /// Slack channel). Never sent verbatim. In a code editor it usually names the + /// open file, which is how the formatting clause learns the language ("… as + /// Swift code."); it also anchors the injector's same-window separator + /// fallback. public let windowTitle: String? /// A short label for the focused field (placeholder/title/role, e.g. "To", - /// "Subject", "Search", "Message"). Not rendered into the prompt. + /// "Subject", "Search", "Message"). Never sent. public let fieldLabel: String? - /// Text immediately preceding the insertion point in the focused field. Not - /// rendered into the prompt; it drives the injector's leading-separator - /// decision. + /// Text immediately preceding the insertion point in the focused field. Sent + /// as the single `conversation_context` turn, so the model knows what the + /// utterance continues; it also drives the injector's leading-separator + /// decision. Skipped entirely in secure fields by `FocusCapture`, so a + /// password never reaches this field. public let priorText: String? - /// The text currently selected in the focused field, when any (the paste - /// will replace it). Not rendered into the prompt. + /// The text currently selected in the focused field, when any. Never sent: the + /// paste replaces it, so priming the model with it would condition the + /// transcription on text that is on its way out. public let selectedText: String? - /// User-configured domain vocabulary (names, jargon, product names) carried as - /// spelling priming so the model favors these spellings. Unlike the other + /// User-configured domain vocabulary (names, jargon, product names), sent as + /// `keyterms_prompt` so the model favors these spellings. Unlike the other /// fields this isn't per-utterance focus state — it's the same list every time, /// sourced from `KeyTermsStore`. public let keyTerms: [String] diff --git a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift b/Sources/BlurtEngine/STT/TranscriptionPrompt.swift deleted file mode 100644 index 3b11b33..0000000 --- a/Sources/BlurtEngine/STT/TranscriptionPrompt.swift +++ /dev/null @@ -1,78 +0,0 @@ -/// Builds the instruction sent to the dictation API as the `prompt` -/// field of the request `config` (see `AssemblyAITranscriber`). The STT model -/// prepends this to its own system prompt. -/// -/// The prompt is deliberately minimal — two clauses, each optional: -/// - the app-kind transcription instruction (`AppKindPriming`), recognized -/// from the frontmost app's bundle ID: "Transcribe speech into markdown." -/// in Obsidian, "Transcribe speech into Swift code." in a code editor (the -/// language inferred from the window title's filename), shell commands in a -/// terminal, a casual Slack message in Slack; -/// - inline keyword boosting trailing it (`Keywords: a, b, c.`, the trained -/// §2.3 form) from the user's key terms, fitted to `characterCap`. -/// -/// The rest of the captured focus context — window title, app and field -/// names, prior-cursor text, selected text — is deliberately **not** rendered: -/// real-world logs showed it crowding the instruction (VS Code, for one, parks -/// a screen-reader help announcement in the focused field's description). It -/// is still captured, feeding the dictation log, the injector's separator -/// logic, and the code-editor language refinement above. Also deliberately -/// absent: -/// - the annotation-suppression clause ("Transcribe without speaker labels, -/// audio event descriptions, or emotion markers."): the dictation service -/// already includes it in its own default prompt; -/// - a "remove filler words"-style *content* reshaping: not in the model's -/// trained instruction set, so it is a no-op (see the project memory note); -/// disfluency removal is the server-side LLM rewrite's job; -/// - a language directive: pinning the prompt to English hurt non-English -/// transcription, so language is left to the model's own detection. -/// -/// Exercised by `Tests/BlurtEngineTests/TranscriptionPromptTests.swift`. -enum TranscriptionPrompt { - /// Hard cap the dictation API places on `config.prompt` ("max 4096 chars"); - /// a longer prompt risks failing the whole request, so `build` must never - /// exceed it. The instruction clause is a bounded sentence; the user's key - /// terms are the one unbounded input, so `build` fits them to whatever - /// budget remains. - static let characterCap = 4096 - - /// Renders `context` into a transcription prompt, or `nil` when there is no usable - /// context (the server then applies its own default prompt). - static func build(context: TranscriptionContext?) -> String? { - // `isEmpty` is the context type's own "no usable content" rule — the same - // predicate `DictationSession.performPress` gates on before yielding a - // context. Asking it here (rather than re-deriving the field-by-field test) - // keeps a newly added context signal from being silently dropped. - guard let context, !context.isEmpty else { return nil } - var prompt = - AppKindPriming.clause( - bundleID: context.bundleID, windowTitle: context.windowTitle.trimmedNonEmpty()) ?? "" - let keyTerms = context.keyTerms - if !keyTerms.isEmpty { - // Spelling priming: the user's domain vocabulary, boosted via the trained - // inline `Keywords: a, b, c.` form (Section 2.3) trailing the instruction - // so the model favors these exact spellings for names/jargon it would - // guess at. The terms list is the one input with no upstream length cap, - // so include only as many whole terms as `characterCap` leaves room for, - // so a huge Settings list can't crowd out the instruction itself or - // balloon every request. - let scaffold = prompt.isEmpty ? "Keywords: ." : " Keywords: ." - var included: [String] = [] - var remaining = characterCap - prompt.count - scaffold.count - for term in keyTerms { - let cost = term.count + (included.isEmpty ? 0 : ", ".count) - guard cost <= remaining else { break } - included.append(term) - remaining -= cost - } - if !included.isEmpty { - let clause = "Keywords: \(included.joined(separator: ", "))." - prompt = prompt.isEmpty ? clause : "\(prompt) \(clause)" - } - } - // A context can be non-empty yet render nothing — focus signals from an - // unrecognized app, no key terms. Return nil rather than an empty prompt - // so the server default still applies. - return prompt.trimmedNonEmpty() - } -} diff --git a/Sources/BlurtEngine/STT/TranscriptionSteering.swift b/Sources/BlurtEngine/STT/TranscriptionSteering.swift new file mode 100644 index 0000000..486df5b --- /dev/null +++ b/Sources/BlurtEngine/STT/TranscriptionSteering.swift @@ -0,0 +1,114 @@ +/// Renders a `TranscriptionContext` into the three request-customization fields +/// the dictation API accepts, each of which has one job (see +/// `AssemblyAITranscriber` for the wire encoding): +/// +/// - `conversation_context` — the text immediately before the insertion point, +/// as a single turn. This is real left-context: it tells the model what the +/// utterance is continuing, which is what fixes mid-sentence casing and +/// proper-noun consistency. +/// - `keyterms_prompt` — the user's key terms, verbatim, as the explicit +/// vocabulary list the field is for. +/// - `llm.instruction` — the app-kind formatting clause (`AppKindPriming`), +/// recognized from the frontmost app's bundle ID: markdown in Obsidian, Swift +/// code in a code editor (the language inferred from the window title's +/// filename), a shell command in a terminal, a casual message in Slack. +/// +/// **`config.prompt` is deliberately never sent.** The field takes a +/// *description of the audio* ("Cardiology consultation about chest pain +/// symptoms."), not instructions — transcription behavior is optimized out of +/// the box, so an imperative like "Transcribe speech into markdown." was aimed +/// at a field that does not act on instructions. That is why the app-kind +/// priming now rides in `llm.instruction`, where an LLM actually rewrites the +/// text, and why the key terms moved to `keyterms_prompt` rather than being +/// packed into the prompt as a `Keywords:` clause. Sending no prompt also keeps +/// the service's managed default — a custom prompt replaces it wholesale, +/// including its language steering, which is the mechanism behind the earlier +/// finding that pinning the prompt to English hurt non-English speech. +/// +/// Blurt has no earlier turns to send: `conversation_context` carries exactly +/// one entry, the prior-cursor text, or none. +/// +/// The rest of the captured context is not sent at all. The app name and field +/// label render nowhere (real-world logs showed them crowding the request — VS +/// Code parks a screen-reader help announcement in the focused field's +/// description), and the window title is read only to name a code editor's +/// language. Selected text is never priming: the paste replaces it, so +/// conditioning the model on it would prime for text on its way out. +/// +/// Exercised by `Tests/BlurtEngineTests/TranscriptionSteeringTests.swift`. +enum TranscriptionSteering { + /// What one utterance sends beyond the audio and its geometry. Built here so + /// the transcriber and the dictation log describe the same request rather than + /// each deriving it. + struct Fields: Sendable, Equatable { + /// Turns preceding this utterance, oldest first. At most one entry (the + /// prior-cursor text). + let conversationContext: [String] + /// Explicit vocabulary to bias recognition toward, in the user's own + /// spelling and capitalization. + let keyterms: [String] + /// The rewrite instruction, or `nil` to let the service's default cleanup + /// instruction stand. + let rewriteInstruction: String? + + /// Nothing to customize — every field omitted, so the service applies its + /// managed default prompt and its default cleanup rewrite. + static let empty = Fields(conversationContext: [], keyterms: [], rewriteInstruction: nil) + + /// True when this utterance customizes nothing. + var isEmpty: Bool { + conversationContext.isEmpty && keyterms.isEmpty && rewriteInstruction == nil + } + } + + /// Cap the dictation API documents for `conversation_context`: 4096 characters + /// across all turns. Over-cap context is trimmed rather than rejected, so this + /// is a quality guard, not a request-failure guard — but the clip must keep the + /// *tail*, since the words nearest the insertion point are the ones carrying + /// continuity. `FocusCapture` already caps prior text far below this + /// (`maxPriorChars`, 320); the guard is here so a hand-built or future-widened + /// context can't silently exceed the field. + static let conversationContextCharacterCap = 4096 + + /// Cap the dictation API documents for `keyterms_prompt`: 2048 characters + /// totalled across every term. Key terms are the one input with no upstream + /// length limit — a Settings list of any size — so `build` includes only as + /// many whole leading terms as fit. + static let keytermsCharacterCap = 2048 + + /// Renders `context` into the fields to send. An absent or unusable context + /// yields `.empty`, which sends no customization at all. + static func build(context: TranscriptionContext?) -> Fields { + // `isEmpty` is the context type's own "no usable content" rule — the same + // predicate `DictationSession.performPress` gates on before yielding a + // context. Asking it here (rather than re-deriving the field-by-field test) + // keeps a newly added context signal from being silently dropped. + guard let context, !context.isEmpty else { return .empty } + return Fields( + conversationContext: priorTurn(of: context).map { [$0] } ?? [], + keyterms: fittedKeyterms(context.keyTerms), + rewriteInstruction: AppKindPriming.clause( + bundleID: context.bundleID, windowTitle: context.windowTitle.trimmedNonEmpty())) + } + + /// The prior-cursor text as one conversation turn, clipped to the field's cap + /// from the front so the text nearest the insertion point survives. + private static func priorTurn(of context: TranscriptionContext) -> String? { + guard let prior = context.priorText.trimmedNonEmpty() else { return nil } + guard prior.count > conversationContextCharacterCap else { return prior } + return String(prior.suffix(conversationContextCharacterCap)) + } + + /// As many whole leading terms as the total-length cap allows. Whole terms + /// only: half a proper noun biases the model toward a spelling nobody wants. + private static func fittedKeyterms(_ terms: [String]) -> [String] { + var included: [String] = [] + var remaining = keytermsCharacterCap + for term in terms { + guard term.count <= remaining else { break } + included.append(term) + remaining -= term.count + } + return included + } +} diff --git a/Tests/BlurtEngineTests/AppKindPrimingTests.swift b/Tests/BlurtEngineTests/AppKindPrimingTests.swift index 2199c9c..e491758 100644 --- a/Tests/BlurtEngineTests/AppKindPrimingTests.swift +++ b/Tests/BlurtEngineTests/AppKindPrimingTests.swift @@ -7,7 +7,7 @@ struct AppKindPrimingTests { // MARK: - Kind recognition /// One bundle-ID → kind expectation, tabled for per-case failure output like - /// `TranscriptionPromptTests.Case`. + /// `TranscriptionSteeringTests.Case`. struct KindCase: Sendable, CustomTestStringConvertible { let bundleID: String? let expected: AppKindPriming.Kind? @@ -71,39 +71,54 @@ struct AppKindPrimingTests { // MARK: - Clause rendering - @Test("a terminal renders the shell-commands instruction") + @Test("a terminal renders the shell-command formatting instruction") func terminalClause() { #expect( AppKindPriming.clause(bundleID: "com.apple.Terminal", windowTitle: nil) - == "Transcribe speech into shell commands.") + == "Format the result as a shell command with no trailing period.") } @Test("a code editor names the open file's language when the title carries one") func codeEditorClauseWithLanguage() { #expect( AppKindPriming.clause(bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt") - == "Transcribe speech into Python code.") + == "Format the result as Python code.") } @Test("a code editor stays generic when the title names no recognizable file") func codeEditorClauseGeneric() { #expect( AppKindPriming.clause(bundleID: "com.apple.dt.Xcode", windowTitle: "Welcome to Xcode") - == "Transcribe speech into code.") + == "Format the result as code.") } - @Test("Slack renders the casual-message instruction") + @Test("Slack renders the casual-message formatting instruction") func slackClause() { #expect( AppKindPriming.clause(bundleID: "com.tinyspeck.slackmacgap", windowTitle: "#eng-backend") - == "Transcribe speech into a casual Slack message with emoji.") + == "Format the result as a casual Slack message, using Slack emoji where they fit.") } - @Test("Obsidian renders the markdown instruction") + @Test("Obsidian renders the markdown formatting instruction") func obsidianClause() { #expect( AppKindPriming.clause(bundleID: "md.obsidian", windowTitle: "Meeting notes") - == "Transcribe speech into markdown.") + == "Format the result as markdown.") + } + + @Test("every clause reads as a rewrite instruction, not a transcription instruction") + func clausesAreRewriteInstructions() { + // These render into `config.llm.instruction`, which is an instruction to an + // LLM rewriting finished text. The Sync STT docs are explicit that + // `config.prompt` takes a *description of the audio* instead — so a clause + // that slipped back into "Transcribe speech into …" phrasing would be aimed + // at the wrong field, which is how the app-kind priming was silently a no-op + // before. + for bundleID in ["com.apple.Terminal", "com.microsoft.VSCode", "com.tinyspeck.slackmacgap", "md.obsidian"] { + let clause = AppKindPriming.clause(bundleID: bundleID, windowTitle: nil) + #expect(clause?.hasPrefix("Format the result as ") == true, "\(bundleID)") + #expect(clause?.contains("Transcribe") == false, "\(bundleID)") + } } @Test("an unrecognized app contributes no clause") diff --git a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift index 2da4581..b038127 100644 --- a/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift +++ b/Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift @@ -52,14 +52,14 @@ struct HTTPClientTests { #expect(result == expected) } - @Test("transcriber succeeds with a real context (builds and sends a prompt)") + @Test("transcriber succeeds with a real context (builds and sends steering fields)") func transcribeWithContext() async throws { let transport = FakeHTTPTransport { request in guard request.url?.path.hasSuffix("/transcribe") == true else { return (404, Data()) } return (200, json(["text": "hello world"])) } - // A non-empty context exercises the TranscriptionPrompt.build path inside + // A non-empty context exercises the TranscriptionSteering.build path inside // transcribe() that the nil-context happy path skips. The fake can't observe // the multipart upload body, so this asserts the request still round-trips // cleanly rather than the wire contents (covered directly by makeConfigData). @@ -154,29 +154,55 @@ struct HTTPClientTests { } } - @Test("config part carries the built context prompt") - func configIncludesPrompt() throws { - let object = try configObject(prompt: "CONTEXT. Transcribe.") - #expect(object["prompt"] as? String == "CONTEXT. Transcribe.") + @Test("config part carries each steering field under its documented wire name") + func configIncludesSteeringFields() throws { + let object = try configObject( + steering: TranscriptionSteering.Fields( + conversationContext: ["$ git status"], keyterms: ["Blurt", "AssemblyAI"], + rewriteInstruction: "Format the result as markdown.")) + #expect(object["conversation_context"] as? [String] == ["$ git status"]) + #expect(object["keyterms_prompt"] as? [String] == ["Blurt", "AssemblyAI"]) + #expect((object["llm"] as? [String: Any])?["instruction"] as? String == "Format the result as markdown.") #expect(object["sample_rate"] as? Int == 16_000) // The capture path is mono by construction; the declared geometry must agree. #expect(object["channels"] as? Int == 1) } - @Test("config part always requests the default cleanup rewrite", arguments: ["CONTEXT. Transcribe.", nil]) - func configRequestsDefaultRewrite(prompt: String?) throws { - // `llm` must be present and empty on every request: present so the service - // runs the rewrite at all, empty so the server-owned default cleanup - // instruction (and its guardrails) applies rather than a client-side copy. - // `isEmpty == true` also covers presence — it is false for a missing `llm`. - #expect((try configObject(prompt: prompt)["llm"] as? [String: Any])?.isEmpty == true) + @Test("config part never sends a prompt — the managed default is what steers transcription") + func configNeverSendsPrompt() throws { + // A custom `config.prompt` replaces the service's managed default *and* its + // language steering, and the field wants a description of the audio rather + // than instructions — which is why formatting moved to `llm.instruction` and + // vocabulary to `keyterms_prompt`. Sending no prompt keeps the managed + // default, so this pins the field's absence for every steering shape. + let shapes: [TranscriptionSteering.Fields] = [ + .empty, + TranscriptionSteering.Fields( + conversationContext: ["Hi Sam,"], keyterms: ["Blurt"], + rewriteInstruction: "Format the result as markdown."), + ] + for steering in shapes { + #expect(try configObject(steering: steering).keys.contains("prompt") == false) + } + } + + @Test("config part requests the default cleanup rewrite when no formatting is needed") + func configRequestsDefaultRewrite() throws { + // `llm` must be present and empty when there is no app-kind clause: present + // so the service runs the rewrite at all, empty so the server-owned default + // cleanup instruction (and its guardrails) applies rather than a client-side + // copy. `isEmpty == true` also covers presence — false for a missing `llm`. + #expect((try configObject(steering: .empty)["llm"] as? [String: Any])?.isEmpty == true) } - @Test( - "config part omits the prompt field when there is no usable context", - arguments: [nil, " \n"]) - func configOmitsPrompt(prompt: String?) throws { - #expect(try configObject(prompt: prompt).keys.contains("prompt") == false) + @Test("config part omits the context and keyterms fields when they are empty") + func configOmitsEmptySteeringFields() throws { + // An empty array is not the same as an absent field: sending + // `"keyterms_prompt": []` states an empty vocabulary where omission lets the + // service apply its own handling. + let object = try configObject(steering: .empty) + #expect(object.keys.contains("conversation_context") == false) + #expect(object.keys.contains("keyterms_prompt") == false) } @Test("the multipart body frames the audio and config parts the dictation API expects") @@ -306,9 +332,9 @@ struct HTTPClientTests { /// config assertion below wants, since `makeConfigData` returns raw JSON. /// A part that isn't a JSON object at all fails here rather than turning every /// downstream assertion into a silent nil-compare. - private func configObject(prompt: String?) throws -> [String: Any] { + private func configObject(steering: TranscriptionSteering.Fields) throws -> [String: Any] { let config = try makeTranscriber(apiKey: "test-key") - .makeConfigData(sampleRate: 16_000, prompt: prompt) + .makeConfigData(sampleRate: 16_000, steering: steering) return try #require(JSONSerialization.jsonObject(with: config) as? [String: Any]) } diff --git a/Tests/BlurtEngineTests/DictationLogTests.swift b/Tests/BlurtEngineTests/DictationLogTests.swift index a41e927..0b2b696 100644 --- a/Tests/BlurtEngineTests/DictationLogTests.swift +++ b/Tests/BlurtEngineTests/DictationLogTests.swift @@ -8,10 +8,29 @@ private struct DecodedEntry: Decodable { let ts: String } -/// Decodes the optional prompt field so tests can assert the exact -/// `config.prompt` that was sent is what lands on disk. -private struct DecodedPrompt: Decodable { - let prompt: String? +/// Decodes the steering fields so tests can assert the exact request +/// customization that was sent is what lands on disk, under the same wire names +/// the request uses. +private struct DecodedSteering: Decodable { + let conversationContext: [String] + let keytermsPrompt: [String] + let llmInstruction: String? + + enum CodingKeys: String, CodingKey { + case conversationContext = "conversation_context" + case keytermsPrompt = "keyterms_prompt" + case llmInstruction = "llm_instruction" + } + + // A missing array decodes to empty rather than nil: whether a field was + // *omitted* is asserted against the raw line, so nothing here needs to tell + // absent from empty. + init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + conversationContext = try container.decodeIfPresent([String].self, forKey: .conversationContext) ?? [] + keytermsPrompt = try container.decodeIfPresent([String].self, forKey: .keytermsPrompt) ?? [] + llmInstruction = try container.decodeIfPresent(String.self, forKey: .llmInstruction) + } } /// Each test gets a fresh empty file in a unique temp directory so the host's real @@ -87,7 +106,7 @@ struct DictationLogTests { #expect(transcript < ts) } - @Test("logs only what was sent — raw focus context stays off disk") + @Test("logs only what was sent — context captured but never sent stays off disk") func logsOnlyWhatWasSent() { let url = makeTempLogURL() let context = TranscriptionContext( @@ -95,37 +114,48 @@ struct DictationLogTests { fieldLabel: "text entry area", priorText: "- milk", selectedText: "- bread") DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - // The prompt that was sent is recorded… - let decoded = try? JSONDecoder().decode(DecodedPrompt.self, from: Data(line.utf8)) - #expect(decoded?.prompt == "Transcribe speech into markdown.") + // Everything the request carried is recorded, under the wire's own names… + let decoded = try? JSONDecoder().decode(DecodedSteering.self, from: Data(line.utf8)) + #expect(decoded?.llmInstruction == "Format the result as markdown.") + #expect(decoded?.conversationContext == ["- milk"]) // …and none of the captured-but-unsent context is. Values, not just keys: - // the entry must carry no trace of what stayed on the machine. - for unsent in ["Obsidian", "md.obsidian", "Grocery list", "text entry area", "- milk", "- bread"] { + // the entry must carry no trace of what stayed on the machine. Selected text + // is on this list because the paste replaces it, so it is never sent. + for unsent in ["Obsidian", "md.obsidian", "Grocery list", "text entry area", "- bread"] { #expect(!line.contains(unsent)) } } - @Test("logs the same assembled prompt the transcriber sends") - func logsAssembledPrompt() { + @Test("logs the same steering fields the transcriber sends") + func logsAssembledSteering() { let url = makeTempLogURL() let context = TranscriptionContext( appName: "Obsidian", bundleID: "md.obsidian", windowTitle: "Grocery list", fieldLabel: "text entry area", priorText: "- milk", keyTerms: ["Blurt"]) DictationLog.write(transcript: "p", context: context, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - let decoded = try? JSONDecoder().decode(DecodedPrompt.self, from: Data(line.utf8)) - #expect(decoded?.prompt == TranscriptionPrompt.build(context: context)) - #expect(decoded?.prompt == "Transcribe speech into markdown. Keywords: Blurt.") + let decoded = try? JSONDecoder().decode(DecodedSteering.self, from: Data(line.utf8)) + let sent = TranscriptionSteering.build(context: context) + // The log is the corpus prompt iteration reads, so it has to agree with the + // builder field-for-field rather than approximately. + #expect(decoded?.llmInstruction == sent.rewriteInstruction) + #expect(decoded?.conversationContext == sent.conversationContext) + #expect(decoded?.keytermsPrompt == sent.keyterms) + #expect(decoded?.llmInstruction == "Format the result as markdown.") + #expect(decoded?.keytermsPrompt == ["Blurt"]) } - @Test("omits the prompt field when there is no context to build one") - func omitsPromptWhenNoContext() { + @Test("omits every steering field when there is no context to build one") + func omitsSteeringWhenNoContext() { let url = makeTempLogURL() DictationLog.write(transcript: "p", context: nil, to: url, now: Date()) let line = readLog(url).split(separator: "\n").first.map(String.init) ?? "" - // `Encodable` synthesis uses `encodeIfPresent`, so a nil prompt is absent - // rather than `"prompt":null`. + // Absent, not `null` and not `[]` — the entry should read as "nothing was + // customized", matching the request, which omits these fields too. #expect(!line.contains("\"prompt\"")) + #expect(!line.contains("\"llm_instruction\"")) + #expect(!line.contains("\"conversation_context\"")) + #expect(!line.contains("\"keyterms_prompt\"")) } @Test("survives unicode in transcript field") @@ -169,13 +199,20 @@ struct DictationLogGateTests { @Test("the gate is checked before the context is touched, for both settings") func gateAppliesToContextualEntries() { // The pipeline always passes the captured context — the input the logged - // prompt is built from. Off must persist nothing at all; on persists the - // prompt but still never the raw context (prior text stays off disk even - // for a user who opted in). + // steering fields are built from. Off must persist nothing at all; on + // persists what was sent, and still never the context that wasn't (the + // window title and field label stay off disk either way). + // + // Prior-cursor text *is* sent now (as `conversation_context`), so it is on + // disk for a user who opted in. What keeps a password out of it is upstream, + // in `FocusCapture`: prior and selected text are skipped entirely in secure + // fields, failing closed when the AX role can't be read. That guard is + // covered in `FocusCaptureTests`; this suite can only see contexts that + // already cleared it. let context = TranscriptionContext( appName: "Terminal", bundleID: "com.apple.Terminal", - windowTitle: "Vault", fieldLabel: "Password", - priorText: "hunter2", selectedText: nil) + windowTitle: "Vault", fieldLabel: "Command", + priorText: "$ git", selectedText: nil) let offURL = makeTempLogURL() DictationLog.append( transcript: "p", context: context, @@ -189,8 +226,10 @@ struct DictationLogGateTests { DictationLog.queue.sync {} #expect(!FileManager.default.fileExists(atPath: offURL.path)) let logged = readLog(onURL) - #expect(logged.contains("Transcribe speech into shell commands.")) - #expect(!logged.contains("hunter2")) + #expect(logged.contains("Format the result as a shell command with no trailing period.")) + #expect(logged.contains("$ git")) + #expect(!logged.contains("Vault")) + #expect(!logged.contains("Command")) } } diff --git a/Tests/BlurtEngineTests/TranscriptionContextTests.swift b/Tests/BlurtEngineTests/TranscriptionContextTests.swift index 127325c..76c6de0 100644 --- a/Tests/BlurtEngineTests/TranscriptionContextTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionContextTests.swift @@ -3,11 +3,11 @@ import Testing @testable import BlurtEngine /// `TranscriptionContext.isEmpty` is the gate `FocusCapture`/`DictationSession` -/// use to decide whether a context is worth carrying at all (prompt AND log). -/// The agreement with `TranscriptionPrompt.build` is one-directional: -/// `isEmpty == true` must always correspond to `build` returning `nil`, while a -/// non-empty context may still build no prompt — its signals can be log-only -/// (app/window/field/prior/selected from an unrecognized app). +/// use to decide whether a context is worth carrying at all (steering AND log). +/// The agreement with `TranscriptionSteering.build` is one-directional: +/// `isEmpty == true` must always correspond to empty steering fields, while a +/// non-empty context may still steer nothing — its signals can be carry-only +/// (app name, field label, or selected text from an unrecognized app). @Suite("TranscriptionContext") struct TranscriptionContextTests { @Test("both fields nil is empty") @@ -30,11 +30,11 @@ struct TranscriptionContextTests { #expect(!TranscriptionContext(appName: nil, priorText: "hello there").isEmpty) } - @Test("a bundle ID alone makes it non-empty (and produces a prompt)") + @Test("a bundle ID alone makes it non-empty (and produces a rewrite instruction)") func bundleIDPresent() { let context = TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil) #expect(!context.isEmpty) - #expect(TranscriptionPrompt.build(context: context) != nil) + #expect(TranscriptionSteering.build(context: context).rewriteInstruction != nil) } @Test("real selected text makes it non-empty") @@ -47,39 +47,42 @@ struct TranscriptionContextTests { #expect(TranscriptionContext(appName: nil, priorText: nil, selectedText: " \n").isEmpty) } - @Test("key terms alone make it non-empty (and produce a prompt)") + @Test("key terms alone make it non-empty (and produce keyterms)") func keyTermsPresent() { let context = TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["Blurt"]) #expect(!context.isEmpty) - #expect(TranscriptionPrompt.build(context: context) != nil) + #expect(TranscriptionSteering.build(context: context).keyterms == ["Blurt"]) } - @Test("an empty context always corresponds to build returning nil") - func agreesWithPromptBuild() { + @Test("an empty context always corresponds to empty steering fields") + func agreesWithSteeringBuild() { let empties = [ TranscriptionContext(appName: nil, priorText: nil), TranscriptionContext(appName: " ", priorText: "\n"), ] for context in empties { #expect(context.isEmpty) - #expect(TranscriptionPrompt.build(context: context) == nil) + #expect(TranscriptionSteering.build(context: context).isEmpty) } - // Renderable signals (a recognized bundle ID, key terms) build a prompt… + // Renderable signals (a recognized bundle ID, key terms, prior text) each + // steer at least one field… let renderable = [ TranscriptionContext(appName: nil, bundleID: "com.apple.Terminal", priorText: nil), TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["Blurt"]), + TranscriptionContext(appName: nil, priorText: "Hi Sam,"), ] for context in renderable { #expect(!context.isEmpty) - #expect(TranscriptionPrompt.build(context: context) != nil) + #expect(!TranscriptionSteering.build(context: context).isEmpty) } - // …while log-only signals make the context non-empty (worth carrying for - // the dictation log and the injector) yet build no prompt. - let logOnly = TranscriptionContext(appName: "Mail", priorText: "Hi Sam,", selectedText: "sel") - #expect(!logOnly.isEmpty) - #expect(TranscriptionPrompt.build(context: logOnly) == nil) + // …while carry-only signals make the context non-empty (worth carrying for + // the injector) yet steer nothing. Selected text is the interesting one: the + // paste replaces it, so it is never priming. + let carryOnly = TranscriptionContext(appName: "Mail", priorText: nil, selectedText: "sel") + #expect(!carryOnly.isEmpty) + #expect(TranscriptionSteering.build(context: carryOnly).isEmpty) } @Test("Equatable compares every field") diff --git a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift b/Tests/BlurtEngineTests/TranscriptionPromptTests.swift deleted file mode 100644 index 90ceda1..0000000 --- a/Tests/BlurtEngineTests/TranscriptionPromptTests.swift +++ /dev/null @@ -1,128 +0,0 @@ -import Testing - -@testable import BlurtEngine - -@Suite("TranscriptionPrompt") -struct TranscriptionPromptTests { - /// One `build(context:)` → prompt expectation. Parameterizing these (rather - /// than a `@Test` apiece) keeps the whole context→prompt contract in one - /// readable table and gives per-case failure output. - /// - /// The prompt is the app-kind instruction plus trailing keyword boosting — - /// nothing else. The other focus signals (app name, window title, field - /// label, prior text, selected text) are captured for the dictation log and - /// the injector, and must never surface in the prompt; the cases below pin - /// both directions. - struct Case: Sendable, CustomTestStringConvertible { - let name: String - let context: TranscriptionContext? - let expected: String? - var testDescription: String { name } - } - - static let cases: [Case] = [ - Case(name: "nil context → no prompt (server default)", context: nil, expected: nil), - Case( - name: "empty context → no prompt", - context: TranscriptionContext(appName: nil, priorText: nil), expected: nil), - Case( - name: "whitespace-only context → no prompt", - context: TranscriptionContext(appName: " ", priorText: "\n"), expected: nil), - Case( - name: "focus signals alone render nothing — log-only, not prompt", - context: TranscriptionContext( - appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", - priorText: "Hi Sam,", selectedText: "the old plan"), - expected: nil), - Case( - name: "unrecognized bundle ID renders nothing", - context: TranscriptionContext(appName: "Mail", bundleID: "com.apple.mail", priorText: nil), - expected: nil), - Case( - name: "terminal → shell-commands instruction", - context: TranscriptionContext(appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil), - expected: "Transcribe speech into shell commands."), - Case( - name: "code editor names the window title's language", - context: TranscriptionContext( - appName: "Code", bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt", priorText: nil), - expected: "Transcribe speech into Python code."), - Case( - name: "code editor with no recognizable filename stays generic", - context: TranscriptionContext( - appName: "Xcode", bundleID: "com.apple.dt.Xcode", windowTitle: "Welcome to Xcode", priorText: nil), - expected: "Transcribe speech into code."), - Case( - name: "Slack → casual-message instruction", - context: TranscriptionContext( - appName: "Slack", bundleID: "com.tinyspeck.slackmacgap", fieldLabel: "Message", priorText: nil), - expected: "Transcribe speech into a casual Slack message with emoji."), - Case( - name: "Obsidian → markdown instruction, window/field stay out", - context: TranscriptionContext( - appName: "Obsidian", bundleID: "md.obsidian", - windowTitle: "Grocery list - Cowork - Obsidian 1.12.7", fieldLabel: "text entry area", - priorText: nil), - expected: "Transcribe speech into markdown."), - Case( - name: "prior/selected text never precede the instruction", - context: TranscriptionContext( - appName: "Terminal", bundleID: "com.apple.Terminal", windowTitle: "zsh — 80×24", - priorText: "$ git status", selectedText: "modified: README.md"), - expected: "Transcribe speech into shell commands."), - Case( - name: "key terms only → bare keyword boost", - context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["AssemblyAI", "Kubernetes"]), - expected: "Keywords: AssemblyAI, Kubernetes."), - Case( - name: "key terms trail the instruction inline", - context: TranscriptionContext( - appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil, keyTerms: ["Blurt"]), - expected: "Transcribe speech into shell commands. Keywords: Blurt."), - Case( - name: "empty key terms add no clause", - context: TranscriptionContext(appName: nil, bundleID: "md.obsidian", priorText: nil, keyTerms: []), - expected: "Transcribe speech into markdown."), - ] - - @Test("build maps focus context to the transcription prompt", arguments: cases) - func build(_ c: Case) { - #expect(TranscriptionPrompt.build(context: c.context) == c.expected) - } - - @Test("the keyword clause is omitted entirely when not even the first term fits") - func keyTermsOmittedWhenNoneFit() { - // A single term longer than the whole cap leaves no budget for even one - // keyword: the clause (and its "Keywords:" scaffolding) must be dropped - // whole, not emitted empty or dangling. - let huge = String(repeating: "k", count: TranscriptionPrompt.characterCap) - let prompt = TranscriptionPrompt.build( - context: TranscriptionContext( - appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil, keyTerms: [huge])) - #expect(prompt == "Transcribe speech into shell commands.") - } - - @Test("a key-terms-only prompt whose first term doesn't fit yields no prompt at all") - func keyTermsOnlyNoneFit() { - // With no instruction and no term fitting the cap, nothing renders — and - // an empty prompt must collapse to nil so the server default applies. - let huge = String(repeating: "k", count: TranscriptionPrompt.characterCap) - let prompt = TranscriptionPrompt.build( - context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: [huge])) - #expect(prompt == nil) - } - - @Test("an oversized key-terms list is fitted to the cap, keeping whole leading terms") - func keyTermsFittedToCap() throws { - // Key terms are the one input with no upstream length cap; a huge Settings - // list must not push the prompt over the API cap (which fails the request). - let terms = (0..<2000).map { "term\($0)" } - let prompt = TranscriptionPrompt.build( - context: TranscriptionContext( - appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil, keyTerms: terms)) - let built = try #require(prompt) - #expect(built.count <= TranscriptionPrompt.characterCap) - #expect(built.contains(" Keywords: term0, term1")) - #expect(built.hasSuffix(".")) - } -} diff --git a/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift b/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift new file mode 100644 index 0000000..5e57261 --- /dev/null +++ b/Tests/BlurtEngineTests/TranscriptionSteeringTests.swift @@ -0,0 +1,159 @@ +import Testing + +@testable import BlurtEngine + +@Suite("TranscriptionSteering") +struct TranscriptionSteeringTests { + /// One `build(context:)` → steering-fields expectation. Parameterizing these + /// (rather than a `@Test` apiece) keeps the whole context→wire contract in one + /// readable table and gives per-case failure output. + /// + /// Each recognized signal has exactly one home, and the table pins both + /// directions — what renders and what must never leak into the wrong field: + /// prior-cursor text → `conversation_context`, key terms → `keyterms_prompt`, + /// the app-kind formatting clause → `llm.instruction`. The remaining focus + /// signals (app name, field label, selected text) render nowhere; the window + /// title is read only to name a code editor's language. + struct Case: Sendable, CustomTestStringConvertible { + let name: String + let context: TranscriptionContext? + let expected: TranscriptionSteering.Fields + var testDescription: String { name } + } + + static let cases: [Case] = [ + Case(name: "nil context → nothing to steer with", context: nil, expected: .empty), + Case( + name: "empty context → nothing to steer with", + context: TranscriptionContext(appName: nil, priorText: nil), expected: .empty), + Case( + name: "whitespace-only context → nothing to steer with", + context: TranscriptionContext(appName: " ", priorText: "\n"), expected: .empty), + Case( + name: "app name and field label render nowhere", + context: TranscriptionContext( + appName: "Mail", windowTitle: "Re: Q3 pricing", fieldLabel: "Body", + priorText: nil, selectedText: "the old plan"), + expected: .empty), + Case( + name: "unrecognized bundle ID renders no instruction", + context: TranscriptionContext(appName: "Mail", bundleID: "com.apple.mail", priorText: nil), + expected: .empty), + Case( + name: "prior-cursor text becomes the single conversation-context turn", + context: TranscriptionContext(appName: "Mail", priorText: "Hi Sam, thanks for"), + expected: TranscriptionSteering.Fields( + conversationContext: ["Hi Sam, thanks for"], keyterms: [], rewriteInstruction: nil)), + Case( + name: "prior text is trimmed of surrounding whitespace", + context: TranscriptionContext(appName: nil, priorText: " Hi Sam,\n "), + expected: TranscriptionSteering.Fields( + conversationContext: ["Hi Sam,"], keyterms: [], rewriteInstruction: nil)), + Case( + name: "key terms become keyterms_prompt, not a prompt clause", + context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: ["AssemblyAI", "Kubernetes"]), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: ["AssemblyAI", "Kubernetes"], rewriteInstruction: nil)), + Case( + name: "terminal → shell-command formatting instruction", + context: TranscriptionContext(appName: "Terminal", bundleID: "com.apple.Terminal", priorText: nil), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: [], + rewriteInstruction: "Format the result as a shell command with no trailing period.")), + Case( + name: "code editor names the window title's language", + context: TranscriptionContext( + appName: "Code", bundleID: "com.microsoft.VSCode", windowTitle: "main.py — blurt", priorText: nil), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: [], + rewriteInstruction: "Format the result as Python code.")), + Case( + name: "code editor with no recognizable filename stays generic", + context: TranscriptionContext( + appName: "Xcode", bundleID: "com.apple.dt.Xcode", windowTitle: "Welcome to Xcode", priorText: nil), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: [], rewriteInstruction: "Format the result as code.")), + Case( + name: "Slack → casual-message formatting instruction", + context: TranscriptionContext( + appName: "Slack", bundleID: "com.tinyspeck.slackmacgap", fieldLabel: "Message", priorText: nil), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: [], + rewriteInstruction: "Format the result as a casual Slack message, using Slack emoji where they fit.")), + Case( + name: "Obsidian → markdown formatting instruction", + context: TranscriptionContext( + appName: "Obsidian", bundleID: "md.obsidian", + windowTitle: "Grocery list - Cowork - Obsidian 1.12.7", fieldLabel: "text entry area", + priorText: nil), + expected: TranscriptionSteering.Fields( + conversationContext: [], keyterms: [], rewriteInstruction: "Format the result as markdown.")), + Case( + name: "all three fields populate independently", + context: TranscriptionContext( + appName: "Terminal", bundleID: "com.apple.Terminal", windowTitle: "zsh — 80×24", + priorText: "$ git status", selectedText: "modified: README.md", keyTerms: ["Blurt"]), + expected: TranscriptionSteering.Fields( + conversationContext: ["$ git status"], keyterms: ["Blurt"], + rewriteInstruction: "Format the result as a shell command with no trailing period.")), + ] + + @Test("build maps focus context to the dictation steering fields", arguments: cases) + func build(_ c: Case) { + #expect(TranscriptionSteering.build(context: c.context) == c.expected) + } + + // MARK: - Selected text never becomes context + + @Test("selected text stays out of conversation context — the paste replaces it") + func selectedTextIsNotContext() { + // Selected text is about to be *overwritten* by the paste, so priming the + // model with it would condition the transcription on text that is on its way + // out. Only the text before the insertion point is real left-context. + let fields = TranscriptionSteering.build( + context: TranscriptionContext( + appName: nil, priorText: "keep this", selectedText: "REPLACED")) + #expect(fields.conversationContext == ["keep this"]) + } + + // MARK: - Documented caps + + @Test("an oversized key-terms list is fitted to the keyterms cap, keeping whole leading terms") + func keyTermsFittedToCap() { + // Key terms are the one unbounded input (a Settings list of any length), and + // the field's cap is the total across all terms — so a huge list must be cut + // to whole terms rather than pushing the request over the documented limit. + let terms = (0..<2000).map { "term\($0)" } + let fields = TranscriptionSteering.build( + context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: terms)) + #expect(fields.keyterms.first == "term0") + #expect(fields.keyterms.count < terms.count) + #expect(fields.keyterms.reduce(0) { $0 + $1.count } <= TranscriptionSteering.keytermsCharacterCap) + } + + @Test("a single key term larger than the cap drops the field entirely") + func keyTermsOmittedWhenNoneFit() { + let huge = String(repeating: "k", count: TranscriptionSteering.keytermsCharacterCap + 1) + let fields = TranscriptionSteering.build( + context: TranscriptionContext(appName: nil, priorText: nil, keyTerms: [huge])) + #expect(fields.keyterms.isEmpty) + } + + @Test("over-long prior text is clipped to the cap, keeping the text nearest the cursor") + func priorTextClippedToCapKeepingTail() { + // The words immediately before the insertion point are the ones that carry + // continuity, so a clip must drop the *head* — the opposite of how the key + // terms list is fitted. (FocusCapture already caps prior text far below this; + // the guard is here so a hand-built or future-widened context can't exceed + // the field's documented limit.) + let long = + String(repeating: "a", count: 100) + + String(repeating: "b", count: TranscriptionSteering.conversationContextCharacterCap) + let fields = TranscriptionSteering.build( + context: TranscriptionContext(appName: nil, priorText: long)) + let turn = fields.conversationContext.first ?? "" + #expect(turn.count == TranscriptionSteering.conversationContextCharacterCap) + #expect(turn.hasSuffix("b")) + #expect(!turn.contains("a")) + } +} From 61ce542ca0aa905193fbe033f39c9cf6066df124 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 29 Jul 2026 02:24:37 +0000 Subject: [PATCH 7/8] fix(stt): drop test-only Fields.isEmpty flagged by periphery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI's periphery scan fails on Fields.isEmpty — production encodes each steering field's presence individually, so the predicate was referenced only from tests. Fields is Equatable and production already uses .empty, so the tests compare against that instead of keeping a separate emptiness rule that could drift from the fields themselves. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01R28wj6vXZDW3iS7FWpHT1U --- Sources/BlurtEngine/STT/TranscriptionSteering.swift | 10 ++++------ Tests/BlurtEngineTests/TranscriptionContextTests.swift | 6 +++--- 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/Sources/BlurtEngine/STT/TranscriptionSteering.swift b/Sources/BlurtEngine/STT/TranscriptionSteering.swift index 486df5b..92fed52 100644 --- a/Sources/BlurtEngine/STT/TranscriptionSteering.swift +++ b/Sources/BlurtEngine/STT/TranscriptionSteering.swift @@ -52,13 +52,11 @@ enum TranscriptionSteering { let rewriteInstruction: String? /// Nothing to customize — every field omitted, so the service applies its - /// managed default prompt and its default cleanup rewrite. + /// managed default prompt and its default cleanup rewrite. Also the value + /// to compare against for "does this utterance customize anything?" — + /// `Fields` is `Equatable`, so no separate emptiness predicate exists to + /// drift from the fields themselves. static let empty = Fields(conversationContext: [], keyterms: [], rewriteInstruction: nil) - - /// True when this utterance customizes nothing. - var isEmpty: Bool { - conversationContext.isEmpty && keyterms.isEmpty && rewriteInstruction == nil - } } /// Cap the dictation API documents for `conversation_context`: 4096 characters diff --git a/Tests/BlurtEngineTests/TranscriptionContextTests.swift b/Tests/BlurtEngineTests/TranscriptionContextTests.swift index 76c6de0..0db647f 100644 --- a/Tests/BlurtEngineTests/TranscriptionContextTests.swift +++ b/Tests/BlurtEngineTests/TranscriptionContextTests.swift @@ -62,7 +62,7 @@ struct TranscriptionContextTests { ] for context in empties { #expect(context.isEmpty) - #expect(TranscriptionSteering.build(context: context).isEmpty) + #expect(TranscriptionSteering.build(context: context) == .empty) } // Renderable signals (a recognized bundle ID, key terms, prior text) each @@ -74,7 +74,7 @@ struct TranscriptionContextTests { ] for context in renderable { #expect(!context.isEmpty) - #expect(!TranscriptionSteering.build(context: context).isEmpty) + #expect(TranscriptionSteering.build(context: context) != .empty) } // …while carry-only signals make the context non-empty (worth carrying for @@ -82,7 +82,7 @@ struct TranscriptionContextTests { // paste replaces it, so it is never priming. let carryOnly = TranscriptionContext(appName: "Mail", priorText: nil, selectedText: "sel") #expect(!carryOnly.isEmpty) - #expect(TranscriptionSteering.build(context: carryOnly).isEmpty) + #expect(TranscriptionSteering.build(context: carryOnly) == .empty) } @Test("Equatable compares every field") From d0b49333e73c3c578f77993a4d8682c458a30d35 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 29 Jul 2026 02:35:00 +0000 Subject: [PATCH 8/8] style(docs): run prettier over AGENTS.md CI's prettier --check failed on the AGENTS.md rewrap from 4c9a7e5; everything else in the run was green. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01R28wj6vXZDW3iS7FWpHT1U --- AGENTS.md | 48 ++++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index db70738..83fd15c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -159,26 +159,26 @@ In Claude Code on the web, a `SessionStart` hook installs the portable linters a Each was tried the other way and reverted. If a task seems to require one, stop and ask first. (`.claude/skills/project-guardrails` is the compressed version of this list.) -| Don't | Because | -| --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | -| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. | -| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. | -| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — request steering belongs in `TranscriptionSteering`. | -| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | +| Don't | Because | +| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. | +| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. | +| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. | +| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — request steering belongs in `TranscriptionSteering`. | +| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. | | Send `config.prompt` at all | The field takes a _description of the audio_, not instructions, and a custom value replaces the service's managed default **including its language steering**. Formatting goes in `llm.instruction`, vocabulary in `keyterms_prompt`, preceding text in `conversation_context`. | -| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection — and a custom prompt is what drops the managed default's language steering in the first place. | -| Put formatting instructions in `config.prompt` | Reshaping output is not something the STT prompt acts on, so _"Transcribe speech into markdown."_ was a measured no-op. `llm.instruction` (_"Format the result as markdown."_) is the lever that works. | -| Pack key terms into the prompt as `Keywords: a, b, c.` | The API documents `keyterms_prompt` for exactly this, and warns against packing keyword lists into the prompt. | -| Add a "remove filler words (um, uh, like)" clause | Not something the STT prompt acts on — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | -| Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | -| Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | -| Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | -| Add a self-replacing install or background auto-updater | Updates are download-only; `mxcl/AppUpdater` and its in-place updater were removed. The once-a-day launch _check_ (`AutomaticUpdateCheck`) is fine; installing for the user, or polling, is not. Extend `UpdateCheckModel`. | -| Hand-edit `Blurt.xcodeproj/project.pbxproj` | Generated from `project.yml`; `check.sh`'s drift check fails on any manual edit (a Claude PreToolUse hook also blocks it). | -| Redirect the post-build install away from `/Applications` | TCC won't register apps in DerivedData/`/tmp`, so permission toggles never appear. | -| Touch the real Keychain in tests | `APIKeyStore` is the production item — a test that writes it triggers Keychain prompts and corrupts the real item's ACL. Use an isolated service (see `KeychainStoreTests`) or `InMemoryAPIKeyStore`. | -| Add backwards-compat shims for removed types | Deleted types stay deleted — no deprecated re-exports. | +| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection — and a custom prompt is what drops the managed default's language steering in the first place. | +| Put formatting instructions in `config.prompt` | Reshaping output is not something the STT prompt acts on, so _"Transcribe speech into markdown."_ was a measured no-op. `llm.instruction` (_"Format the result as markdown."_) is the lever that works. | +| Pack key terms into the prompt as `Keywords: a, b, c.` | The API documents `keyterms_prompt` for exactly this, and warns against packing keyword lists into the prompt. | +| Add a "remove filler words (um, uh, like)" clause | Not something the STT prompt acts on — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. | +| Add a keystroke-typing paste path or a length threshold | Injection is **always** clipboard paste (save → write → ⌘V → settle → restore), with the copied-to-clipboard degradation when the target is lost. | +| Add `LSUIElement` or a menu-bar-**only** mode | Blurt is a Dock app first. The `MenuBarExtra` status item is convenience layered on the Dock icon; the notch can hide a status item, so nothing may depend on it. A menu-bar-only variant was reverted twice. | +| Add a `KeyboardShortcuts` package or a key+modifier chord | The trigger is a single lone modifier, home-grown (`CGEventTap` + `DictationKeyGate`), and swallows nothing. | +| Add a self-replacing install or background auto-updater | Updates are download-only; `mxcl/AppUpdater` and its in-place updater were removed. The once-a-day launch _check_ (`AutomaticUpdateCheck`) is fine; installing for the user, or polling, is not. Extend `UpdateCheckModel`. | +| Hand-edit `Blurt.xcodeproj/project.pbxproj` | Generated from `project.yml`; `check.sh`'s drift check fails on any manual edit (a Claude PreToolUse hook also blocks it). | +| Redirect the post-build install away from `/Applications` | TCC won't register apps in DerivedData/`/tmp`, so permission toggles never appear. | +| Touch the real Keychain in tests | `APIKeyStore` is the production item — a test that writes it triggers Keychain prompts and corrupts the real item's ACL. Use an isolated service (see `KeychainStoreTests`) or `InMemoryAPIKeyStore`. | +| Add backwards-compat shims for removed types | Deleted types stay deleted — no deprecated re-exports. | Release-side invariants (hardened runtime and a secure timestamp on every nested mach-o and embedded framework, or notarization rejects the build; roll-forward-only for a bad release) live in @@ -384,11 +384,11 @@ to right ⌘), so views must not re-declare `TriggerKey.rightCommand.rawValue` t request-customization fields, each with one job. It's unit-tested in `Tests/BlurtEngineTests/TranscriptionSteeringTests.swift`. -| Field | Carries | Cap | -| ---------------------- | -------------------------------------------------------------- | ----------------------- | -| `conversation_context` | prior-cursor text, as a single turn | 4096 chars, clip head | -| `keyterms_prompt` | the user's key terms, verbatim | 2048 chars, whole terms | -| `llm.instruction` | the `AppKindPriming` formatting clause, or absent for the default | — | +| Field | Carries | Cap | +| ---------------------- | ----------------------------------------------------------------- | ----------------------- | +| `conversation_context` | prior-cursor text, as a single turn | 4096 chars, clip head | +| `keyterms_prompt` | the user's key terms, verbatim | 2048 chars, whole terms | +| `llm.instruction` | the `AppKindPriming` formatting clause, or absent for the default | — | **`config.prompt` is never sent, and that is the whole point of this design.** The field takes a _description of the audio_ ("Cardiology consultation about chest pain symptoms."), not instructions —