Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ DictationKeyTap (CGEventTap + DictationKeyGate) → AppCoordinator → Dictation
KeyInjector → focused app (clipboard-paste via a synthesized Cmd-V CGEvent)
```

**`MicCapture`** (`Sources/BlurtEngine/Audio/MicCapture.swift`) is an actor implementing `MicCaptureProtocol`. It captures with **`AVAudioRecorder`**, recording straight to a temp 16 kHz / mono / 16-bit PCM WAV — the exact geometry the Sync API wants, so `stop()` reads the file back as `[Float]` (via `AVAudioFile`) with no resampling pass. A **fresh recorder per session** resolves the current default input device at `record()` time; this is deliberate. A long-lived `AVAudioEngine` was used previously and its input graph bound to one device, going stale on a device switch (mic ↔ built-in) — raising `-10868` (`kAudioUnitErr_FormatNotSupported`) or capturing all-zero buffers. **Don't reintroduce an `AVAudioEngine`/`installTap` capture path.** The overlay meter (`levels`) comes from the recorder's dBFS power on a ~30 Hz timer, mapped to `0…1` by `linearLevel(fromPowerDB:)` (floored so room ambient reads as empty bars — the overlay's auto-gain inflates any steady nonzero level to full).
**`MicCapture`** (`Sources/BlurtEngine/Audio/MicCapture.swift`) is an actor implementing `MicCaptureProtocol`. It captures with **`AVAudioRecorder`**, recording straight to a temp 16 kHz / mono / 16-bit PCM WAV — the exact geometry the Sync API wants, so `stop()` reads the file back as `[Float]` (via `AVAudioFile`) with no resampling pass. A **fresh recorder per session** resolves the current default input device at `record()` time; this is deliberate. A long-lived `AVAudioEngine` was used previously and its input graph bound to one device, going stale on a device switch (mic ↔ built-in) — raising `-10868` (`kAudioUnitErr_FormatNotSupported`) or capturing all-zero buffers. **Don't reintroduce an `AVAudioEngine`/`installTap` capture path.** The overlay meter (`levels`) comes from the recorder's dBFS power on a ~30 Hz timer, mapped to `0…1` by `linearLevel(fromPowerDB:)` (floored so room ambient reads as empty bars rather than a meter that never rests).

**`AssemblyAITranscriber`** (`Sources/BlurtEngine/STT/AssemblyAITranscriber.swift`) implements `TranscriberProtocol` against AssemblyAI's **Sync** STT API (the endpoint `assembly dictate` uses): a single `POST https://sync.assemblyai.com/transcribe` with the captured audio as a raw S16LE PCM blob (`PCMEncoder`) in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, and a `prompt`), `X-AAI-Model: u3-sync-pro`. The `prompt` (built per utterance by `TranscriptionPrompt.build(context:)`) is prepended to the model's system prompt, so the transcript that comes back is **already polished** — there is no separate LLM post-processing step. The finished transcript comes back in the response body — no `/v2/upload`, no job submission, no polling. Truly synchronous: `TranscriberProtocol.transcribe(samples:sampleRate:)` is a single `async throws -> String` that returns the whole transcript at once (no streaming/deltas). The sync model handles audio from ~80 ms up to 120 s (server-side ~30 s inference deadline); those limits live in `SyncSTTLimits` and back `DictationSession`'s auto-release timeout (so a held hotkey stops before the cap).

Expand All @@ -87,7 +87,7 @@ The dictation trigger is a **single lone modifier key** (tap-to-toggle or hold-t

The app side, **`DictationKeyTap`** (`App/Blurt/Blurt/Hotkey/DictationKeyTap.swift`), bridges a `CGEventTap` (watching `flagsChanged` for the bound modifier and `keyDown` for any other key) into the gate. It **swallows nothing** — a lone modifier types nothing, and combos pass through so normal shortcuts keep working — unlike the old chord trigger.

The trigger is editable in the Shortcut section of the setup/settings window (`HotkeyStepView`), which is a `Picker` over `TriggerKey.allCases` that writes `TriggerKeyStore` (then `AppCoordinator.refreshBinding()` re-reads it into the tap). Use `DictateHotkey.label` / `DictateHotkey.keyCaps` (`App/Blurt/Blurt/Hotkey/DictateHotkey.swift`) for display strings.
The trigger is editable in the Shortcut section of the setup/settings window (`HotkeyStepView`), which is a `Picker` over `TriggerKey.allCases` that writes `TriggerKeyStore` (then `AppCoordinator.dictationBindingChanged()` re-reads it into the tap). For display strings, use `DictateHotkey.label` (`App/Blurt/Blurt/Hotkey/DictateHotkey.swift`) for one-shot reads, or `@AppStorage(TriggerKeyStore.defaultsKey)` + `TriggerKey.fromPersisted` in views that must re-render live on a Settings change.

## Transcription prompt

Expand Down
31 changes: 6 additions & 25 deletions App/Blurt/Blurt/AppCoordinator+UITesting.swift
Original file line number Diff line number Diff line change
Expand Up @@ -3,32 +3,13 @@
//
// XCUITest can't synthesize the lone right-modifier `flagsChanged` event the
// real trigger relies on — and the `CGEventTap` that reads it needs the
// process to be Accessibility-trusted, which the CI test host isn't. These
// mirror the key tap's `onStart`/`onStop`/`onCancel` closures so a UI test
// drives the *same* pipeline (`DictationSession`) the hotkey would, just via
// the test harness window instead of a key event. Compiled only in Debug, so
// they never ship in the notarized Release build.
// process to be Accessibility-trusted, which the CI test host isn't. The test
// harness window instead calls `beginDictation`/`endDictation`/
// `cancelDictation` directly — the same methods the key tap's closures
// dispatch to — so a UI test drives the *same* pipeline (`DictationSession`)
// and missing-key gate the hotkey would. Compiled only in Debug, so nothing
// here ships in the notarized Release build.
extension AppCoordinator {
/// Mirrors the key tap's `onStart`: honors the missing-key gate (so the
/// no-key path is testable), otherwise begins recording.
func uiTestBeginDictation() async {
guard hasAPIKey else {
onMissingAPIKey()
return
}
await session.press()
}

/// Mirrors the key tap's `onStop` — stop recording and run transcribe→inject.
func uiTestEndDictation() async {
await session.release()
}

/// Mirrors the key tap's `onCancel` — abandon the in-flight dictation.
func uiTestCancelDictation() async {
await session.cancel()
}

/// Fire a synthetic dictation-key press/release through the *real* key tap
/// (gate + callbacks), not the session directly. Lets the leak exercise
/// (`scripts/leaks.sh`) cover the DictationKeyTap object graph a real
Expand Down
206 changes: 115 additions & 91 deletions App/Blurt/Blurt/AppCoordinator.swift
Original file line number Diff line number Diff line change
Expand Up @@ -90,23 +90,13 @@ final class AppCoordinator {
// auto-repeat key events into the focused app while held.
let tap = DictationKeyTap(
onStart: { [weak self] in
Task { @MainActor in
guard let self else { return }
guard self.hasAPIKey else {
// No key yet — take the user straight to the fix instead of a
// transient message. The overlay pill isn't even visible in this
// state (it's only revealed once setup is complete).
self.onMissingAPIKey()
return
}
await self.session.press()
}
Task { @MainActor in await self?.beginDictation() }
},
onStop: { [weak self] in
Task { @MainActor in await self?.session.release() }
Task { @MainActor in await self?.endDictation() }
},
onCancel: { [weak self] in
Task { @MainActor in await self?.session.cancel() }
Task { @MainActor in await self?.cancelDictation() }
}
)
self.keyTap = tap
Expand Down Expand Up @@ -152,7 +142,7 @@ final class AppCoordinator {
if overlay == nil { overlay = OverlayWindowController() }
// Pre-roll the start/stop cues now that the app is ready, so the first
// chime's audio-queue setup never stalls the recording pill.
primeSounds()
cues.prime()
}

/// Hides the overlay pill. Called by the wizard when the app stops being fully
Expand All @@ -161,6 +151,35 @@ final class AppCoordinator {
overlay?.hide()
}

// MARK: - Dictation drivers
//
// The single begin/end/cancel path shared by the key tap's closures and the
// UI-test harness (`UITestSupport`), so the harness always exercises the same
// gate the hotkey does.

/// Begins a dictation as the hotkey would: the missing-key gate first, then
/// `session.press()`.
func beginDictation() async {
guard hasAPIKey else {
// No key yet — take the user straight to the fix instead of a
// transient message. The overlay pill isn't even visible in this
// state (it's only revealed once setup is complete).
onMissingAPIKey()
return
}
await session.press()
}

/// Stops recording and runs transcribe→inject.
func endDictation() async {
await session.release()
}

/// Abandons the in-flight dictation.
func cancelDictation() async {
await session.cancel()
}

/// Called when the user rebinds (or clears) the dictation shortcut in the
/// recorder, so the event tap starts matching the new chord. The shortcut no
/// longer gates readiness (it has a default and lives in Settings), so there's
Expand Down Expand Up @@ -238,94 +257,26 @@ final class AppCoordinator {

// MARK: - Dictation render

private var wasRecording = false
private var recordStartSound: AVAudioPlayer?
private var recordStopSound: AVAudioPlayer?

/// The cues are deliberate UI accents, not music — they are normalized to a
/// hot peak, so play them well below full scale so they read as a soft chime
/// rather than blasting at the system output level.
private static let cueVolume: Float = 0.35

/// (Re)loads the start/stop cue players for the currently selected sound pack.
/// `.none` (or a missing file) leaves a player nil, which `play(_:)` skips, so
/// "no sound" needs no extra branching on the hot path.
private func loadSounds() {
let pack = SoundPackStore().soundPack
recordStartSound = pack.startFileName.flatMap(Self.bundledSound(named:))
recordStopSound = pack.stopFileName.flatMap(Self.bundledSound(named:))
recordStartSound?.volume = Self.cueVolume
recordStopSound?.volume = Self.cueVolume
}

/// Loads a bundled chime (`Resources/Sounds/<name>.m4a`) fully into memory.
/// `AVAudioPlayer(contentsOf:)` decodes the AAC up front — unlike
/// `NSSound(…byReference: true)`, which deferred the disk read to the first
/// `play()` and so stalled the recording pill the first time you dictated. The
/// players are then pre-rolled in `primeSounds()` (called from the ready
/// transition) so even the audio-queue setup is paid before the hot path.
private static func bundledSound(named name: String) -> AVAudioPlayer? {
guard let url = Bundle.main.url(forResource: name, withExtension: "m4a") else { return nil }
return try? AVAudioPlayer(contentsOf: url)
}

/// Pre-rolls the cue players so the first start/stop chime adds no latency to
/// the pill. Called from `showOverlay()` — the "app is ready" transition — so
/// the audio queue is allocated and the buffers are filled well before the
/// first dictation. Idempotent: re-priming an already-prepared player is cheap.
private func primeSounds() {
loadSounds()
recordStartSound?.prepareToPlay()
recordStopSound?.prepareToPlay()
}
/// The record start/stop chimes (see `CueSoundPlayer` below).
private let cues = CueSoundPlayer()

/// Called when the user changes the sound pack in Settings: reload the cue
/// players (and re-prime them) so the next start/stop uses the new voice, then
/// preview the new voice so the choice is audible immediately.
/// players and preview the new voice so the choice is audible immediately.
func soundPackChanged() {
primeSounds()
previewCues()
}

/// Plays the selected pack's start, then stop cues a beat apart, as a preview
/// when switching voices in Settings. Silent for the `.none` pack (all players
/// are nil, which `play(_:)` skips).
private func previewCues() {
play(recordStartSound)
Task { @MainActor in
try? await Task.sleep(for: .milliseconds(380))
play(recordStopSound)
}
}

/// Plays a cue from the start, without ever blocking the caller. Rewinding
/// first means a cue replays cleanly even if the previous play hasn't been
/// reset, and keeping this off the visual path (callers `show` the pill first)
/// guarantees the sound never delays the overlay.
private func play(_ sound: AVAudioPlayer?) {
guard let sound else { return }
sound.currentTime = 0
sound.play()
cues.packChanged()
}

private func render(_ phase: PipelinePhase) {
let isRecording = phase == .recording

// Reveal the pill first, then fire the cue: `play()` must never sit in front
// of the visual state change. Pure phase→pill mapping lives in the engine
// (unit-tested there); .failed resolves to .error, which the pill flashes red
// then auto-reverts to idle.
// Reveal the pill first, then fire the cue: the sound must never sit in
// front of the visual state change. Pure phase→pill mapping lives in the
// engine (unit-tested there); .failed resolves to .error, which the pill
// flashes red then auto-reverts to idle.
overlay?.show(state: phase.overlayState)
// Mirror the phase onto the menu bar indicator (mapping lives in the engine,
// unit-tested alongside `overlayState`).
menuBarStatus = phase.menuBarStatus

if isRecording && !wasRecording {
play(recordStartSound)
} else if !isRecording && wasRecording {
play(recordStopSound)
}
wasRecording = isRecording
cues.transition(isRecording: phase == .recording)

// A .failed phase is a handled error (it doesn't crash the app), so the
// crash reporter never sees it — report the genuine faults here.
Expand Down Expand Up @@ -371,3 +322,76 @@ final class AppCoordinator {
}
}
}

/// Owns the record start/stop cue chimes: loading the selected pack, pre-rolling
/// so the first chime never stalls the recording pill, previewing on a pack
/// change, and firing on the recording edge. Kept out of `AppCoordinator`'s body
/// so chime behavior can change without churning the session↔UI wiring.
@MainActor
final class CueSoundPlayer {
private var startSound: AVAudioPlayer?
private var stopSound: AVAudioPlayer?
private var wasRecording = false

/// The cues are deliberate UI accents, not music — they are normalized to a
/// hot peak, so play them well below full scale so they read as a soft chime
/// rather than blasting at the system output level.
private static let cueVolume: Float = 0.35

/// (Re)loads and pre-rolls the cue players for the selected sound pack so
/// the first start/stop chime adds no latency to the pill; called from the
/// "app is ready" transition, well before the hot path. `.none` (or a
/// missing file) leaves a player nil, which `play(_:)` skips. Idempotent:
/// re-priming an already-prepared player is cheap.
func prime() {
let pack = SoundPackStore().soundPack
startSound = pack.startFileName.flatMap(Self.bundledSound(named:))
stopSound = pack.stopFileName.flatMap(Self.bundledSound(named:))
startSound?.volume = Self.cueVolume
stopSound?.volume = Self.cueVolume
startSound?.prepareToPlay()
stopSound?.prepareToPlay()
}

/// Reloads the players for a newly selected pack and previews the new voice
/// (start, then stop a beat apart) so the choice is audible immediately.
/// Silent for the `.none` pack (all players are nil, which `play(_:)` skips).
func packChanged() {
prime()
play(startSound)
Task { @MainActor in
try? await Task.sleep(for: .milliseconds(380))
play(stopSound)
}
}

/// Fires the start/stop cue on the recording edge. Call once per rendered
/// phase; only the idle↔recording transitions make a sound.
func transition(isRecording: Bool) {
if isRecording && !wasRecording {
play(startSound)
} else if !isRecording && wasRecording {
play(stopSound)
}
wasRecording = isRecording
}

/// Loads a bundled chime (`Resources/Sounds/<name>.m4a`) fully into memory:
/// `AVAudioPlayer(contentsOf:)` decodes the AAC up front, unlike
/// `NSSound(…byReference: true)`, whose deferred disk read stalled the pill
/// on the first dictation. `prime()` then pre-rolls the audio queue too.
private static func bundledSound(named name: String) -> AVAudioPlayer? {
guard let url = Bundle.main.url(forResource: name, withExtension: "m4a") else { return nil }
return try? AVAudioPlayer(contentsOf: url)
}

/// Plays a cue from the start, without ever blocking the caller. Rewinding
/// first means a cue replays cleanly even if the previous play hasn't been
/// reset, and keeping this off the visual path (callers reveal the pill
/// first) guarantees the sound never delays the overlay.
private func play(_ sound: AVAudioPlayer?) {
guard let sound else { return }
sound.currentTime = 0
sound.play()
}
}
Loading