diff --git a/AGENTS.md b/AGENTS.md index 7cffbc1..05e1f67 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,7 +51,7 @@ Full flow: Key files: -- `AudioEngine.swift` wraps `AVAudioEngine`. It follows System Default route changes with bounded recovery and binds explicit microphone selections directly to the capture input audio unit. +- `AudioEngine.swift` captures the microphone through `HALInputCapture` (a Core Audio input callback on the chosen device), not `AVAudioEngine`. `AVAudioEngine` couples input and output in one I/O unit, so a Bluetooth headset switching profile stopped the capture and a built-in microphone selected while a headset was the output delivered no samples. The capture follows the System Default at start and, with an explicit selection, uses that device; a sample-rate change does not interrupt it, and only a stopped capture is rebuilt, with bounded retries. - `GrammarEnhancer.swift` spawns the `copilot` CLI as a subprocess; `EnhancedModel` holds the evaluated cleanup models and loads model-specific prompt profiles. - `OverlayPanel.swift` is a borderless `NSPanel` with one waveform design shared by `.listening`, `.transcribing`, `.enhancing`, `.done`, and `.error`. - `UpdateManager.swift` wraps [AppUpdater](https://github.com/mxcl/AppUpdater) to check GitHub Releases, validate Developer ID signatures, and install and relaunch in place. diff --git a/CHANGELOG.md b/CHANGELOG.md index 35cc20f..95b71d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,19 @@ All notable changes to OpenWritr will be documented in this file. Each release's notes on GitHub are generated from its section here, and the broker publication handoff fails when the section for a tag is missing or empty. +## [1.7.4] — 2026-10-13 + +### Fixed +- Dictation works with Bluetooth headsets such as AirPods Pro and AirPods Max. The microphone is now captured through an input-only Core Audio callback instead of `AVAudioEngine`, which tied the capture to the output device: a headset switching to its microphone profile stopped the engine and lost the start of the recording, and selecting the built-in microphone while a headset was the output could deliver no audio at all +- Recording starts faster (about 70 ms instead of about 250 ms on the built-in microphone) +- The start tone and the Listening overlay now wait for the first audio, so with a Bluetooth headset they appear when speech is actually being captured. Before, the tone played as soon as the capture was requested, while the headset was still switching its output profile, and was not heard. The capture records the whole time, so nothing is lost while waiting + +### Added +- **Keep Last Dictation** in the menu bar menu. When it is on, OpenWritr keeps the text of the last dictation in memory only, never on disk or in the logs, and **Paste Last Dictation** pastes it again into the app in front. Use it when a long dictation could not be pasted because the target window was gone. It follows the same rules as a fresh dictation (typing in Windows App when that setting is on), pastes even with Auto-Paste off, and applies to clipboard-only recordings too. Turning the setting off or quitting discards the text. Off by default + +### Changed +- Microphone capture no longer depends on the output device + ## [1.7.3] — 2026-10-13 ### Fixed diff --git a/Info.plist b/Info.plist index 878953b..2617189 100644 --- a/Info.plist +++ b/Info.plist @@ -7,9 +7,9 @@ CFBundleIdentifier com.openwritr.app CFBundleVersion - 1.7.3 + 1.7.4 CFBundleShortVersionString - 1.7.3 + 1.7.4 NSHumanReadableCopyright Copyright © 2026 Torsten Mahr. Released under the MIT License. OpenWritrDescription diff --git a/README.md b/README.md index ff875ec..375fcf3 100644 --- a/README.md +++ b/README.md @@ -80,6 +80,10 @@ Clipboard-only works even with Auto-Paste off. Successful copying intentionally Normal Auto-Paste restores the original clipboard, including an empty clipboard, unless another application or the user has changed it. Output errors are shown explicitly; delayed restoration errors appear as a dismissible menu warning. With Auto-Paste off, standard recordings remain inside OpenWritr and are not copied automatically. +### Keep Last Dictation + +If a long dictation could not be pasted because the target window was gone, turn on **Keep Last Dictation** in the menu bar menu. OpenWritr then keeps the text of the last dictation in memory only: it is never written to disk or the logs, turning the setting off or quitting discards it, and it is kept even when pasting failed. **Paste Last Dictation** (shown while the setting is on) pastes it again into the app in front, with the same routing as a fresh dictation, including typing in Windows App when that setting is on. It pastes even with Auto-Paste off, and it also covers clipboard-only recordings. The setting is off by default. + ### Remote desktop apps (experimental) Windows App copies the Mac clipboard to the remote PC only after you leave its window, even with clipboard redirection set to Bidirectional. A paste into the remote session can therefore insert old text. Turn on **Type Text in Remote Desktop Apps (Experimental)** in Settings: while Windows App is frontmost, Auto-Paste then types the text key by key instead of pasting, and your clipboard is not touched. It applies only to Windows App (`com.microsoft.rdc.macos`), only with Auto-Paste on, and not to clipboard-only recordings (Option), which still copy. @@ -310,7 +314,8 @@ Sources/OpenWritr/ ├── OpenWritrApp.swift # App entry, presence modes, state machine ├── MenuBarView.swift # Menu bar dropdown UI ├── SettingsView.swift # Dedicated settings window -├── AudioEngine.swift # AVAudioEngine, 16kHz capture, realtime-safe +├── AudioEngine.swift # Capture lifecycle, 16kHz conversion, device selection +├── HALInputCapture.swift # Input-only Core Audio capture (no AVAudioEngine) ├── TranscriptionManager.swift # FluidAudio model loading + transcription ├── GrammarEnhancer.swift # Cleanup provider routing and remote providers ├── AppleIntelligenceEnhancer.swift # macOS 26+ on-device cleanup @@ -331,7 +336,7 @@ Sources/OpenWritr/ - **FluidAudio** — CoreML-optimized ASR framework - **NVIDIA Parakeet TDT 0.6B v3** — non-autoregressive transducer, 25 languages - **Apple Neural Engine** — hardware-accelerated inference via CoreML -- **AVAudioEngine** — low-latency microphone capture at 16kHz +- **Core Audio** — low-latency, input-only microphone capture, converted to 16kHz - **CGEventTap** — global Fn key detection (requires Accessibility permission) - **AppUpdater** — signed, in-app update checks against GitHub Releases diff --git a/Sources/OpenWritr/AudioDiagnostics.swift b/Sources/OpenWritr/AudioDiagnostics.swift index 812aee9..5af422c 100644 --- a/Sources/OpenWritr/AudioDiagnostics.swift +++ b/Sources/OpenWritr/AudioDiagnostics.swift @@ -79,6 +79,13 @@ struct AudioDiagnostics: Equatable, Sendable { ) } + /// Transport, channels and sample rate of one device, for logs. Never includes its name or UID. + static func deviceFormatSummary(for deviceID: AudioDeviceID) -> String { + let rate = (property(of: deviceID, selector: kAudioDevicePropertyNominalSampleRate) as Float64?) + .map { String(Int($0.rounded())) } ?? "unknown" + return "transport=\(transportName(of: deviceID)) channels=\(inputChannelCount(of: deviceID)) rate=\(rate)" + } + static func transportName(forCode code: UInt32) -> String { switch code { case kAudioDeviceTransportTypeBuiltIn: return "built-in" diff --git a/Sources/OpenWritr/AudioEngine.swift b/Sources/OpenWritr/AudioEngine.swift index d702c99..56c39bb 100644 --- a/Sources/OpenWritr/AudioEngine.swift +++ b/Sources/OpenWritr/AudioEngine.swift @@ -2,7 +2,6 @@ import AudioToolbox import CoreAudio import Foundation -import ObjCExceptionCatcher import os.log private let audioLog = Logger(subsystem: "com.openwritr.app", category: "AudioEngine") @@ -36,8 +35,6 @@ enum AudioEngineError: LocalizedError, Sendable { case inputDeviceAvailabilityCheckFailed(AudioDeviceID, OSStatus) case inputDeviceSelectionFailed(AudioDeviceID, OSStatus) case deviceMonitoringFailed(OSStatus) - case tapInstallationFailed(String) - case enginePreparationFailed(String) case engineStartFailed(String) case unexpectedFailure(String) @@ -57,10 +54,6 @@ enum AudioEngineError: LocalizedError, Sendable { return "OpenWritr could not select the requested microphone (\(Self.describe(status)))." case .deviceMonitoringFailed(let status): return "OpenWritr could not monitor microphone changes (\(Self.describe(status)))." - case .tapInstallationFailed(let reason): - return "OpenWritr could not access microphone audio: \(reason)" - case .enginePreparationFailed(let reason): - return "OpenWritr could not prepare microphone capture: \(reason)" case .engineStartFailed(let reason): return "OpenWritr could not start microphone capture: \(reason)" case .unexpectedFailure(let reason): @@ -72,7 +65,7 @@ enum AudioEngineError: LocalizedError, Sendable { switch self { case .captureCancelled: return nil - case .tapInstallationFailed, .enginePreparationFailed, .engineStartFailed: + case .engineStartFailed: return "Check Microphone access in System Settings > Privacy & Security, then retry." case .inputDeviceAvailabilityCheckFailed: return "Retry the microphone change." @@ -230,6 +223,10 @@ protocol AudioCapturing: AnyObject, Sendable { func setInputDevice(_ deviceID: AudioDeviceID?) -> Result func prepare() -> Result func startCapture() async -> Result + /// Waits until the capture has delivered its first audio, which a Bluetooth headset does only + /// after it has switched to its microphone profile. Returns false if none arrived in time or + /// the capture ended; the capture itself keeps recording the whole time. + func waitForFirstAudio(handle: CaptureHandle, timeout: Duration) async -> Bool func waitForCaptureToSettle( handle: CaptureHandle, idleWindow: Duration, @@ -240,6 +237,10 @@ protocol AudioCapturing: AnyObject, Sendable { func shutdown() -> Result } +extension AudioCapturing { + func waitForFirstAudio(handle: CaptureHandle, timeout: Duration) async -> Bool { true } +} + final class AudioEngine: AudioCapturing, @unchecked Sendable { private static let maximumRecoveryAttempts = 3 private let targetSampleRate: Double = 16_000 @@ -251,19 +252,18 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { private let callbackLock = NSLock() private var lifecycleState: AudioEngineLifecycleState = .idle - private var engine: AVAudioEngine? - private var isRunning = false - private var tapInstalled = false + private var capture: HALInputCapture? private var isShuttingDown = false private var shutdownRequested = false private var nextGeneration: UInt64 = 0 private var activeGeneration: UInt64? private var returnedGeneration: UInt64? private var activeDeviceID: AudioDeviceID? + /// True when the active capture follows the System Default input rather than an explicit selection. + private var activeDeviceFollowsDefault = false private var recoveryAttemptCount = 0 private var pendingConfigurationWork: DispatchWorkItem? private var selectedDeviceID: AudioDeviceID? - private var configObserver: Any? private var deviceListListenerBlock: AudioObjectPropertyListenerBlock? private var defaultInputListenerBlock: AudioObjectPropertyListenerBlock? private var devicesChangedCallback: (@Sendable () -> Void)? @@ -461,114 +461,41 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { audioLog.debug("lifecycle \(stateName, privacy: .public), generation \(generationText, privacy: .public)") } - private func constructFreshEngineOnQueue(generation: UInt64) throws { - guard case .starting = lifecycleState else { - throw AudioEngineError.captureCancelled - } - let newEngine = AVAudioEngine() - let tapFormat = try bindSelectedInputDeviceOnQueue(to: newEngine) - engine = newEngine - try installTapOnQueue(engine: newEngine, generation: generation, format: tapFormat) - } - - private func bindSelectedInputDeviceOnQueue(to engine: AVAudioEngine) throws -> AVAudioFormat? { - guard let selectedDeviceID else { return nil } - guard try Self.isInputDeviceAvailable(selectedDeviceID) else { - throw AudioEngineError.inputDeviceUnavailable(selectedDeviceID) - } - guard let audioUnit = engine.inputNode.audioUnit else { - throw AudioEngineError.inputDeviceSelectionFailed( - selectedDeviceID, - kAudio_ParamError - ) - } - - var deviceID = selectedDeviceID - let status = AudioUnitSetProperty( - audioUnit, - kAudioOutputUnitProperty_CurrentDevice, - kAudioUnitScope_Global, - 0, - &deviceID, - UInt32(MemoryLayout.size) - ) - guard status == noErr else { - throw AudioEngineError.inputDeviceSelectionFailed(selectedDeviceID, status) - } - - var streamDescription = AudioStreamBasicDescription() - var streamDescriptionSize = UInt32(MemoryLayout.size) - let hardwareFormatStatus = AudioUnitGetProperty( - audioUnit, - kAudioUnitProperty_StreamFormat, - kAudioUnitScope_Input, - 1, - &streamDescription, - &streamDescriptionSize + private func startInputCaptureOnQueue(deviceID: AudioDeviceID, generation: UInt64) throws { + let newCapture = HALInputCapture( + deviceID: deviceID, + handler: makeBufferHandler(generation: generation), + deviceLost: { [weak self] in + self?.lifecycleQueue.async { [weak self] in + self?.scheduleConfigurationDecisionOnQueue(generation: generation) + } + } ) - guard hardwareFormatStatus == noErr, - streamDescription.mSampleRate > 0, - streamDescription.mChannelsPerFrame > 0 - else { - throw AudioEngineError.inputDeviceAvailabilityCheckFailed( - selectedDeviceID, - hardwareFormatStatus - ) - } + try newCapture.start() + capture = newCapture + } - let clientFormatStatus = AudioUnitSetProperty( - audioUnit, - kAudioUnitProperty_StreamFormat, - kAudioUnitScope_Output, - 1, - &streamDescription, - streamDescriptionSize - ) - guard clientFormatStatus == noErr, - let format = AVAudioFormat(streamDescription: &streamDescription) - else { - throw AudioEngineError.inputDeviceSelectionFailed( - selectedDeviceID, - clientFormatStatus - ) - } - audioLog.notice( - "bound capture input directly to device \(selectedDeviceID) at \(format.sampleRate) Hz" - ) - return format + private func stopInputCaptureOnQueue() { + capture?.stop() + capture = nil } - private func recoverEngineOnQueue(generation: UInt64) throws { + /// Moves the running capture to the device the system now resolves, for example when the + /// System Default input changes while recording. + private func recoverCaptureOnQueue(generation: UInt64, deviceID: AudioDeviceID) throws { guard case .recovering = lifecycleState else { throw AudioEngineError.captureCancelled } - removeConfigObserverOnQueue() - stopEngineOnQueue() + stopInputCaptureOnQueue() guard !shutdownWasRequested() else { throw AudioEngineError.captureCancelled } - activeDeviceID = try resolveCaptureDeviceOnQueue() - guard !shutdownWasRequested() else { - throw AudioEngineError.captureCancelled - } - let recoveredEngine = AVAudioEngine() - let tapFormat = try bindSelectedInputDeviceOnQueue(to: recoveredEngine) - engine = recoveredEngine - try installTapOnQueue( - engine: recoveredEngine, - generation: generation, - format: tapFormat - ) - try prepareAndStartEngineOnQueue(recoveredEngine) - addConfigObserverOnQueue(engine: recoveredEngine, generation: generation) + activeDeviceID = deviceID + try startInputCaptureOnQueue(deviceID: deviceID, generation: generation) } - private func installTapOnQueue( - engine: AVAudioEngine, - generation: UInt64, - format: AVAudioFormat? = nil - ) throws { - let inputNode = engine.inputNode + /// Converts each buffer to mono 16 kHz, appends it to the capture, and reports its level. + private func makeBufferHandler(generation: UInt64) -> HALInputCapture.BufferHandler { let state = captureBuffer let target = desiredFormat let clock = captureClock @@ -578,7 +505,7 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { self.notifyAudioLevel(Self.rawRMSLevel(samples), generation: generation) } - let tapBlock: AVAudioNodeTapBlock = { buffer, _ in + return { buffer in let bufferFormat = buffer.format if bufferFormat.sampleRate == target.sampleRate && bufferFormat.channelCount == 1 { guard let channel = buffer.floatChannelData else { return } @@ -614,69 +541,6 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { state.append(samples, generation: generation, at: clock.now) levelHandler(samples) } - - var tapError: NSError? - guard ObjCTryCatch({ - inputNode.installTap(onBus: 0, bufferSize: 4096, format: format, block: tapBlock) - }, &tapError) else { - throw AudioEngineError.tapInstallationFailed( - tapError?.localizedDescription ?? "The audio input rejected the capture format." - ) - } - tapInstalled = true - } - - private func prepareAndStartEngineOnQueue(_ engine: AVAudioEngine) throws { - guard !shutdownWasRequested() else { - throw AudioEngineError.captureCancelled - } - var preparationError: NSError? - guard ObjCTryCatch({ engine.prepare() }, &preparationError) else { - throw AudioEngineError.enginePreparationFailed( - preparationError?.localizedDescription ?? "The audio engine rejected its configuration." - ) - } - - guard !shutdownWasRequested() else { - throw AudioEngineError.captureCancelled - } - do { - try engine.start() - isRunning = true - } catch { - throw AudioEngineError.engineStartFailed(error.localizedDescription) - } - } - - private func stopEngineOnQueue() { - guard let engine else { - isRunning = false - tapInstalled = false - return - } - if isRunning || engine.isRunning { - var stopError: NSError? - if !ObjCTryCatch({ engine.stop() }, &stopError) { - audioLog.error( - "failed to stop audio engine: \(stopError?.localizedDescription ?? "unknown error", privacy: .public)" - ) - } - } - isRunning = false - removeInstalledTapOnQueue() - self.engine = nil - } - - private func removeInstalledTapOnQueue() { - guard tapInstalled, let engine else { return } - tapInstalled = false - let inputNode = engine.inputNode - var removeError: NSError? - if !ObjCTryCatch({ inputNode.removeTap(onBus: 0) }, &removeError) { - audioLog.error( - "failed to remove audio tap: \(removeError?.localizedDescription ?? "unknown error", privacy: .public)" - ) - } } private func cleanupGenerationOnQueue( @@ -691,8 +555,7 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { audioLog.notice("discarded capture generation \(generation)") } } - removeConfigObserverOnQueue() - stopEngineOnQueue() + stopInputCaptureOnQueue() if activeGeneration == generation { activeGeneration = nil activeDeviceID = nil @@ -708,26 +571,6 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { // MARK: - Configuration changes - private func addConfigObserverOnQueue(engine: AVAudioEngine, generation: UInt64) { - removeConfigObserverOnQueue() - configObserver = NotificationCenter.default.addObserver( - forName: .AVAudioEngineConfigurationChange, - object: engine, - queue: nil - ) { [weak self] _ in - self?.lifecycleQueue.async { [weak self] in - self?.scheduleConfigurationDecisionOnQueue(generation: generation) - } - } - } - - private func removeConfigObserverOnQueue() { - if let configObserver { - NotificationCenter.default.removeObserver(configObserver) - self.configObserver = nil - } - } - private func scheduleConfigurationDecisionOnQueue(generation: UInt64) { guard !isShuttingDown, activeGeneration == generation else { return } switch lifecycleState { @@ -757,17 +600,28 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { } do { - guard let deviceID = activeDeviceID, - try Self.isInputDeviceAvailable(deviceID) - else { - failActiveGenerationOnQueue( - generation, - error: .inputDeviceUnavailable(activeDeviceID ?? kAudioObjectUnknown) - ) - return + // An explicit selection must stay available. A capture that follows the System Default + // moves to whatever the system resolves now, including when the old device went away. + let deviceID: AudioDeviceID + if activeDeviceFollowsDefault { + deviceID = try resolveCaptureDeviceOnQueue() + } else { + guard let active = activeDeviceID, + try Self.isInputDeviceAvailable(active) + else { + failActiveGenerationOnQueue( + generation, + error: .inputDeviceUnavailable(activeDeviceID ?? kAudioObjectUnknown) + ) + return + } + deviceID = active } - if let engine, isRunning, engine.isRunning { + // A change of sample rate (for example a Bluetooth headset switching to its microphone + // profile) does not interrupt the input callback, so only a stopped capture or a + // different resolved device needs work. + if deviceID == activeDeviceID, capture?.isRunning == true { audioLog.debug("benign configuration change for generation \(generation)") return } @@ -775,14 +629,14 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { guard recoveryAttemptCount < Self.maximumRecoveryAttempts else { failActiveGenerationOnQueue( generation, - error: .engineStartFailed("The audio engine stopped repeatedly during microphone recovery.") + error: .engineStartFailed("The microphone stopped repeatedly during recovery.") ) return } recoveryAttemptCount += 1 transitionOnQueue(to: .recovering, generation: generation) - try recoverEngineOnQueue(generation: generation) + try recoverCaptureOnQueue(generation: generation, deviceID: deviceID) guard !shutdownWasRequested(), activeGeneration == generation else { cleanupGenerationOnQueue(generation, discardSamples: true, finalState: .idle) return @@ -1168,16 +1022,15 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { throw AudioEngineError.captureCancelled } activeDeviceID = try resolveCaptureDeviceOnQueue() - try constructFreshEngineOnQueue(generation: generation) - guard !shutdownWasRequested(), !cancellation.isCancelled, let engine else { + activeDeviceFollowsDefault = selectedDeviceID == nil + guard !shutdownWasRequested(), !cancellation.isCancelled, let deviceID = activeDeviceID else { throw AudioEngineError.captureCancelled } captureBuffer.activate(generation: generation) - try prepareAndStartEngineOnQueue(engine) + try startInputCaptureOnQueue(deviceID: deviceID, generation: generation) guard !shutdownWasRequested(), !cancellation.isCancelled else { throw AudioEngineError.captureCancelled } - addConfigObserverOnQueue(engine: engine, generation: generation) transitionOnQueue(to: .capturing, generation: generation) audioLog.notice("capture generation \(generation) started on device \(self.activeDeviceID ?? 0)") return .success(CaptureHandle(generation: generation)) @@ -1195,6 +1048,28 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { } } + func waitForFirstAudio(handle: CaptureHandle, timeout: Duration = .seconds(2)) async -> Bool { + let startedAt = captureClock.now + while captureClock.now - startedAt < timeout { + if Task.isCancelled { return false } + let snapshot = captureBuffer.snapshot() + guard snapshot.isCapturing, snapshot.generation == handle.generation else { return false } + if snapshot.sampleCount > 0 { + let parts = (captureClock.now - startedAt).components + let milliseconds = Int(parts.seconds) * 1000 + Int(parts.attoseconds / 1_000_000_000_000_000) + audioLog.notice("first audio for generation \(handle.generation) after \(milliseconds) ms") + return true + } + do { + try await Task.sleep(for: .milliseconds(10)) + } catch { + return false + } + } + audioLog.notice("no audio arrived for generation \(handle.generation) within the timeout") + return false + } + func waitForCaptureToSettle( handle: CaptureHandle, idleWindow: Duration = .milliseconds(70), @@ -1245,8 +1120,7 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { transitionOnQueue(to: .stopping, generation: handle.generation) pendingConfigurationWork?.cancel() pendingConfigurationWork = nil - removeConfigObserverOnQueue() - stopEngineOnQueue() + stopInputCaptureOnQueue() activeGeneration = nil returnedGeneration = nil activeDeviceID = nil @@ -1277,8 +1151,7 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { returnedGeneration = nil activeDeviceID = nil recoveryAttemptCount = 0 - removeConfigObserverOnQueue() - stopEngineOnQueue() + stopInputCaptureOnQueue() removeDeviceListListenersOnQueue() selectedDeviceID = nil transitionOnQueue(to: .idle) @@ -1307,10 +1180,9 @@ final class AudioEngine: AudioCapturing, @unchecked Sendable { lifecycleQueue.async { [weak self] in guard let self, self.returnedGeneration == handle.generation, - let engine = self.engine + let capture = self.capture else { return } - engine.stop() - self.isRunning = false + capture.stop() self.scheduleConfigurationDecisionOnQueue(generation: handle.generation) } } diff --git a/Sources/OpenWritr/HALInputCapture.swift b/Sources/OpenWritr/HALInputCapture.swift new file mode 100644 index 0000000..1355e23 --- /dev/null +++ b/Sources/OpenWritr/HALInputCapture.swift @@ -0,0 +1,251 @@ +@preconcurrency import AVFoundation +import AudioToolbox +import CoreAudio +import Foundation +import os + +private let captureLog = Logger(subsystem: "com.openwritr.app", category: "AudioEngine") + +/// The sample types input devices deliver through a Core Audio I/O callback. +enum HALSampleKind: Equatable, Sendable { + case float32 + case int16 + case int32 + + /// Nil for formats OpenWritr cannot read: compressed, big-endian, other bit depths, or samples + /// padded or aligned inside a wider frame (the decoder assumes tightly packed samples). + init?(_ format: AudioStreamBasicDescription) { + guard format.mFormatID == kAudioFormatLinearPCM, + format.mFormatFlags & kAudioFormatFlagIsBigEndian == 0 + else { return nil } + let isFloat = format.mFormatFlags & kAudioFormatFlagIsFloat != 0 + let isSignedInteger = format.mFormatFlags & kAudioFormatFlagIsSignedInteger != 0 + let kind: HALSampleKind + switch (format.mBitsPerChannel, isFloat, isSignedInteger) { + case (32, true, _): kind = .float32 + case (16, false, true): kind = .int16 + case (32, false, true): kind = .int32 + default: return nil + } + let isNonInterleaved = format.mFormatFlags & kAudioFormatFlagIsNonInterleaved != 0 + let samplesPerFrame = isNonInterleaved ? 1 : Int(format.mChannelsPerFrame) + guard Int(format.mBytesPerFrame) == kind.bytesPerSample * samplesPerFrame else { return nil } + self = kind + } + + var bytesPerSample: Int { + switch self { + case .float32, .int32: return 4 + case .int16: return 2 + } + } +} + +enum HALSampleConverter { + /// The first channel of an interleaved (or single-channel) buffer as Float samples in -1...1. + static func firstChannel( + of raw: UnsafeRawPointer, + byteCount: Int, + channels: Int, + kind: HALSampleKind + ) -> [Float] { + guard channels > 0 else { return [] } + let frames = byteCount / (kind.bytesPerSample * channels) + guard frames > 0 else { return [] } + var samples = [Float](repeating: 0, count: frames) + switch kind { + case .float32: + let source = raw.assumingMemoryBound(to: Float.self) + for frame in 0.. Void + /// Called on the audio queue when the device stops being available. + typealias DeviceLostHandler = @Sendable () -> Void + + private let deviceID: AudioDeviceID + private let handler: BufferHandler + private let deviceLost: DeviceLostHandler + private let queue = DispatchQueue(label: "com.openwritr.audio-input", qos: .userInteractive) + private let lock = NSLock() + /// Separate from `lock` because the liveness callback runs on `queue` while `stop()` holds `lock`. + private let startedFlag = OSAllocatedUnfairLock(initialState: false) + + private var procID: AudioDeviceIOProcID? + private var aliveListener: AudioObjectPropertyListenerBlock? + + init(deviceID: AudioDeviceID, handler: @escaping BufferHandler, deviceLost: @escaping DeviceLostHandler) { + self.deviceID = deviceID + self.handler = handler + self.deviceLost = deviceLost + } + + deinit { + stop() + } + + var isRunning: Bool { + startedFlag.withLock { $0 } + } + + func start() throws { + lock.lock() + defer { lock.unlock() } + guard !isRunning else { return } + + let format = try Self.inputStreamFormat(of: deviceID) + guard HALSampleKind(format) != nil else { + throw AudioEngineError.engineStartFailed("The input device uses an audio format OpenWritr cannot read.") + } + + var newProcID: AudioDeviceIOProcID? + let createStatus = AudioDeviceCreateIOProcIDWithBlock(&newProcID, deviceID, queue) { [weak self] _, input, _, _, _ in + self?.handleInput(input) + } + guard createStatus == noErr, let newProcID else { + throw AudioEngineError.inputDeviceSelectionFailed(deviceID, createStatus) + } + procID = newProcID + + installAliveListener() + let startStatus = AudioDeviceStart(deviceID, newProcID) + guard startStatus == noErr else { + removeAliveListener() + AudioDeviceDestroyIOProcID(deviceID, newProcID) + procID = nil + throw AudioEngineError.inputDeviceSelectionFailed(deviceID, startStatus) + } + startedFlag.withLock { $0 = true } + captureLog.notice( + "started input capture on device \(self.deviceID) \(AudioDiagnostics.deviceFormatSummary(for: self.deviceID), privacy: .public)" + ) + } + + func stop() { + lock.lock() + defer { lock.unlock() } + guard let procID else { return } + startedFlag.withLock { $0 = false } + removeAliveListener() + AudioDeviceStop(deviceID, procID) + AudioDeviceDestroyIOProcID(deviceID, procID) + self.procID = nil + } + + // MARK: - Audio callback + + /// Runs on `queue`. Takes the first channel of the first buffer, which is the microphone for + /// the mono and interleaved layouts input devices use, and hands it on as a mono Float32 buffer + /// at the device's current sample rate. The device's format is read for every callback, so a + /// change while capturing is followed: a Bluetooth headset switching to its microphone profile + /// changes the rate, and a device may change its sample type. + private func handleInput(_ input: UnsafePointer) { + let list = UnsafeMutableAudioBufferListPointer(UnsafeMutablePointer(mutating: input)) + guard let first = list.first, + let data = first.mData, + first.mDataByteSize > 0, + first.mNumberChannels > 0, + let streamFormat = try? Self.inputStreamFormat(of: deviceID), + let kind = HALSampleKind(streamFormat) + else { return } + + let samples = HALSampleConverter.firstChannel( + of: UnsafeRawPointer(data), + byteCount: Int(first.mDataByteSize), + channels: Int(first.mNumberChannels), + kind: kind + ) + guard !samples.isEmpty, + let format = AVAudioFormat( + commonFormat: .pcmFormatFloat32, + sampleRate: streamFormat.mSampleRate, + channels: 1, + interleaved: false + ), + let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(samples.count)), + let destination = buffer.floatChannelData?[0] + else { return } + + destination.update(from: samples, count: samples.count) + buffer.frameLength = AVAudioFrameCount(samples.count) + handler(buffer) + } + + // MARK: - Device liveness + + private func installAliveListener() { + var address = AudioObjectPropertyAddress( + mSelector: kAudioDevicePropertyDeviceIsAlive, + mScope: kAudioObjectPropertyScopeGlobal, + mElement: kAudioObjectPropertyElementMain + ) + let block: AudioObjectPropertyListenerBlock = { [weak self] _, _ in + guard let self, !Self.isAlive(self.deviceID) else { return } + self.startedFlag.withLock { $0 = false } + self.deviceLost() + } + if AudioObjectAddPropertyListenerBlock(deviceID, &address, queue, block) == noErr { + aliveListener = block + } + } + + private func removeAliveListener() { + guard let aliveListener else { return } + var address = AudioObjectPropertyAddress( + mSelector: kAudioDevicePropertyDeviceIsAlive, + mScope: kAudioObjectPropertyScopeGlobal, + mElement: kAudioObjectPropertyElementMain + ) + AudioObjectRemovePropertyListenerBlock(deviceID, &address, queue, aliveListener) + self.aliveListener = nil + } + + // MARK: - Device properties + + private static func inputStreamFormat(of deviceID: AudioDeviceID) throws -> AudioStreamBasicDescription { + var address = AudioObjectPropertyAddress( + mSelector: kAudioDevicePropertyStreamFormat, + mScope: kAudioDevicePropertyScopeInput, + mElement: kAudioObjectPropertyElementMain + ) + var format = AudioStreamBasicDescription() + var size = UInt32(MemoryLayout.size) + let status = AudioObjectGetPropertyData(deviceID, &address, 0, nil, &size, &format) + guard status == noErr, format.mSampleRate > 0, format.mChannelsPerFrame > 0 else { + throw AudioEngineError.inputDeviceAvailabilityCheckFailed(deviceID, status) + } + return format + } + + private static func isAlive(_ deviceID: AudioDeviceID) -> Bool { + var address = AudioObjectPropertyAddress( + mSelector: kAudioDevicePropertyDeviceIsAlive, + mScope: kAudioObjectPropertyScopeGlobal, + mElement: kAudioObjectPropertyElementMain + ) + var alive: UInt32 = 1 + var size = UInt32(MemoryLayout.size) + guard AudioObjectGetPropertyData(deviceID, &address, 0, nil, &size, &alive) == noErr else { + return false + } + return alive != 0 + } +} diff --git a/Sources/OpenWritr/MenuBarView.swift b/Sources/OpenWritr/MenuBarView.swift index 10298cb..c12eb15 100644 --- a/Sources/OpenWritr/MenuBarView.swift +++ b/Sources/OpenWritr/MenuBarView.swift @@ -199,6 +199,20 @@ struct MenuBarView: View { } )) .disabled(!isReady) + + Toggle("Keep Last Dictation", isOn: Binding( + get: { viewModel.keepLastDictationEnabled }, + set: { viewModel.setKeepLastDictation($0) } + )) + .help("Keeps the last dictated text in memory only, so you can paste it again.") + .disabled(!isReady) + + if viewModel.keepLastDictationEnabled { + Button("Paste Last Dictation") { + Task { await viewModel.pasteLastDictation() } + } + .disabled(!viewModel.canPasteLastDictation) + } } @ViewBuilder diff --git a/Sources/OpenWritr/OpenWritrApp.swift b/Sources/OpenWritr/OpenWritrApp.swift index 7df1afd..b0a4926 100644 --- a/Sources/OpenWritr/OpenWritrApp.swift +++ b/Sources/OpenWritr/OpenWritrApp.swift @@ -113,6 +113,8 @@ final class AppViewModel { private static let customPromptsPreferenceKey = "enhancementCustomPromptsV1" private static let customPromptsStoreVersion = 1 static let defaultDoneDisplayDuration: Duration = .milliseconds(600) + /// Lets the menu bar menu close so the paste reaches the app that was frontmost. + static let defaultPasteAgainDelay: Duration = .milliseconds(350) static let defaultTransientErrorDisplayDuration: Duration = .seconds(2.5) var state: AppState = .idle var lastTranscription: String = "" @@ -130,6 +132,9 @@ final class AppViewModel { var autoPasteEnabled: Bool = true /// Experimental: type instead of paste when a remote desktop app is frontmost. var typeIntoRemoteDesktopsEnabled: Bool = false + /// Keeps the last dictated text in memory so it can be pasted again. Never persisted or logged. + var keepLastDictationEnabled: Bool = false + private(set) var lastDictation: String? var launchAtLogin: Bool = false var appPresence: AppPresence var hotkeyChoice: HotkeyChoice = .fn @@ -161,7 +166,9 @@ final class AppViewModel { private let captureDrainIdleDuration: Duration = .milliseconds(70) private let captureDrainTimeout: Duration = .milliseconds(350) + private let firstAudioTimeout: Duration = .seconds(2) private let doneDisplayDuration: Duration + private let pasteAgainDelay: Duration private let transientErrorDisplayDuration: Duration private let errorLogger: any ErrorLogging private let audioDiagnostics: (AudioDeviceID?) -> String @@ -199,6 +206,7 @@ final class AppViewModel { errorLogger: any ErrorLogging = UnifiedErrorLogger(category: "AppViewModel"), startsOperational: Bool = false, doneDisplayDuration: Duration = AppViewModel.defaultDoneDisplayDuration, + pasteAgainDelay: Duration = AppViewModel.defaultPasteAgainDelay, transientErrorDisplayDuration: Duration = AppViewModel.defaultTransientErrorDisplayDuration, applicationPresenceController: any ApplicationPresenceControlling = SystemApplicationPresenceController(), @@ -218,6 +226,7 @@ final class AppViewModel { self.errorLogger = errorLogger self.audioDiagnostics = audioDiagnostics self.doneDisplayDuration = doneDisplayDuration + self.pasteAgainDelay = pasteAgainDelay self.transientErrorDisplayDuration = transientErrorDisplayDuration self.applicationPresenceController = applicationPresenceController self.appPresenceDefaults = appPresenceDefaults @@ -489,6 +498,7 @@ final class AppViewModel { if defaults.object(forKey: "typeIntoRemoteDesktopsEnabled") != nil { typeIntoRemoteDesktopsEnabled = defaults.bool(forKey: "typeIntoRemoteDesktopsEnabled") } + keepLastDictationEnabled = defaults.bool(forKey: "keepLastDictationEnabled") if let raw = defaults.string(forKey: "hotkeyChoice"), let choice = HotkeyChoice(rawValue: raw) { hotkeyChoice = choice @@ -732,6 +742,47 @@ final class AppViewModel { clipboardWarning = nil } + // MARK: - Last dictation + + var canPasteLastDictation: Bool { + guard keepLastDictationEnabled, isOperational, !didShutdown, + let lastDictation, !lastDictation.isEmpty, + case .ready = state + else { return false } + return true + } + + /// Turning the setting off discards the kept text immediately. + func setKeepLastDictation(_ enabled: Bool) { + keepLastDictationEnabled = enabled + savePreference("keepLastDictationEnabled", value: enabled) + if !enabled { + lastDictation = nil + } + } + + /// Pastes the kept dictation into the frontmost app again, using the same routing as a + /// fresh dictation (typing for remote desktop apps, otherwise paste) regardless of Auto-Paste, + /// because the user asked for it explicitly. + func pasteLastDictation() async { + guard canPasteLastDictation else { return } + do { + try await Task.sleep(for: pasteAgainDelay) + } catch { + return + } + guard canPasteLastDictation, let text = lastDictation else { return } + + if pasteManager.outputText( + text, + destination: .standard, + autoPasteEnabled: true, + typeIntoRemoteDesktops: typeIntoRemoteDesktopsEnabled + ) == .cancelled { + presentPasteCancelledError() + } + } + private func handleClipboardRestoreFailure(_ error: PasteManagerError) { guard isOperational, !didShutdown else { return } clipboardWarning = error.localizedDescription @@ -1288,7 +1339,7 @@ final class AppViewModel { } } - private func startProcessingStoppedRecording(shortcut: RecordingShortcut) { + func startProcessingStoppedRecording(shortcut: RecordingShortcut) { guard isOperational, !didShutdown, let operationID = captureOperationID @@ -1419,15 +1470,7 @@ final class AppViewModel { !Task.isCancelled, case .preparingMicrophone = state else { return } - logAudioError("Microphone start failed: \(error.localizedDescription)") - if selectedInputDeviceID == nil { - scheduleSystemDefaultRecovery(showTransientError: true, initialError: error) - } else { - presentRuntimeError( - errorPresentation(kind: .audio, title: "Microphone Preparation Failed", error: error), - overlayMessage: "Microphone unavailable" - ) - } + presentMicrophoneStartFailure(error) case .success(let handle): guard captureOperationID == operationID, @@ -1455,6 +1498,31 @@ final class AppViewModel { return } + // A Bluetooth headset delivers audio only after it has switched to its microphone + // profile. The capture is already recording, so waiting loses nothing, and the tone and + // the Listening overlay then mean that speech is actually being captured. + let receivedAudio = await audioEngine.waitForFirstAudio(handle: handle, timeout: firstAudioTimeout) + guard captureOperationID == operationID, + captureHandle?.generation == handle.generation, + isOperational, + !didShutdown, + !Task.isCancelled, + case .preparingMicrophone = state + else { + clearPendingStartTask(ifCurrent: operationID) + _ = await stopCaptureOnce(handle) + return + } + + // A capture that stays silent must not announce itself as listening. + guard receivedAudio else { + invalidateCaptureOperation(ifCurrent: operationID) + _ = await stopCaptureOnce(handle) + guard isOperational, !didShutdown, case .preparingMicrophone = state else { return } + presentMicrophoneStartFailure(.engineStartFailed("No audio arrived from the microphone.")) + return + } + if releaseRequested { await stopListeningAndTranscribe( operationID: operationID, @@ -1480,6 +1548,18 @@ final class AppViewModel { } } + private func presentMicrophoneStartFailure(_ error: AudioEngineError) { + logAudioError("Microphone start failed: \(error.localizedDescription)") + if selectedInputDeviceID == nil { + scheduleSystemDefaultRecovery(showTransientError: true, initialError: error) + } else { + presentRuntimeError( + errorPresentation(kind: .audio, title: "Microphone Preparation Failed", error: error), + overlayMessage: "Microphone unavailable" + ) + } + } + private func stopListeningAndTranscribe( operationID: UUID, handle: CaptureHandle, @@ -1672,6 +1752,9 @@ final class AppViewModel { lastTranscription = text lastRawTranscription = rawText ?? "" lastWasEnhanced = wasEnhanced + if keepLastDictationEnabled { + lastDictation = text + } if pasteManager.outputText( text, @@ -1903,6 +1986,7 @@ final class AppViewModel { pasteManager.flushPendingRestore() recoverableRawTranscription = nil recoverableOutputDestination = .standard + lastDictation = nil overlayPanel.dismiss() updateManager.stopAutomaticChecks() diff --git a/Tests/OpenWritrTests/AppViewModelDictationFlowTests.swift b/Tests/OpenWritrTests/AppViewModelDictationFlowTests.swift index 9ea44cf..2fc478d 100644 --- a/Tests/OpenWritrTests/AppViewModelDictationFlowTests.swift +++ b/Tests/OpenWritrTests/AppViewModelDictationFlowTests.swift @@ -22,6 +22,111 @@ struct AppViewModelDictationFlowTests { #expect(viewModel.state.isReady) } + @Test func lastDictationIsNotKeptUnlessEnabled() async { + let dependencies = makeDependencies( + samples: [audibleSamples(count: 16_000)], + transcriptions: [.success("Synthetic transcript.")] + ) + let viewModel = dependencies.makeViewModel() + defer { viewModel.shutdown() } + + await recordAndStop(viewModel) + await viewModel.pasteLastDictation() + + #expect(viewModel.lastDictation == nil) + #expect(!viewModel.canPasteLastDictation) + #expect(dependencies.paster.pastedTexts == ["Synthetic transcript."]) + } + + @Test(arguments: [false, true]) + func keptDictationCanBePastedAgain(autoPasteEnabled: Bool) async { + let dependencies = makeDependencies( + samples: [audibleSamples(count: 16_000)], + transcriptions: [.success("Synthetic transcript.")] + ) + let viewModel = dependencies.makeViewModel() + defer { viewModel.shutdown() } + viewModel.keepLastDictationEnabled = true + viewModel.autoPasteEnabled = autoPasteEnabled + + await recordAndStop(viewModel) + #expect(viewModel.canPasteLastDictation) + await viewModel.pasteLastDictation() + + let expectedFirstPaste = autoPasteEnabled ? ["Synthetic transcript."] : [] + #expect(dependencies.paster.pastedTexts == expectedFirstPaste + ["Synthetic transcript."]) + #expect(viewModel.state.isReady) + } + + @Test func dictationIsKeptWhenThePasteFailed() async { + let dependencies = makeDependencies( + samples: [audibleSamples(count: 16_000)], + transcriptions: [.success("Synthetic transcript.")], + pasteOutcomes: [.cancelled] + ) + let viewModel = dependencies.makeViewModel() + defer { viewModel.shutdown() } + viewModel.keepLastDictationEnabled = true + + await recordAndStop(viewModel) + #expect(dependencies.paster.pastedTexts.isEmpty) + viewModel.dismissRuntimeError() + await waitUntil { viewModel.state.isReady } + await viewModel.pasteLastDictation() + + #expect(dependencies.paster.pastedTexts == ["Synthetic transcript."]) + } + + @Test func turningTheSettingOffDiscardsTheKeptDictation() async { + let dependencies = makeDependencies( + samples: [audibleSamples(count: 16_000)], + transcriptions: [.success("Synthetic transcript.")] + ) + let viewModel = dependencies.makeViewModel() + defer { viewModel.shutdown() } + viewModel.setKeepLastDictation(true) + defer { viewModel.setKeepLastDictation(false) } + + await recordAndStop(viewModel) + #expect(viewModel.lastDictation == "Synthetic transcript.") + viewModel.setKeepLastDictation(false) + + #expect(viewModel.lastDictation == nil) + #expect(!viewModel.canPasteLastDictation) + } + + @Test func shuttingDownDiscardsTheKeptDictation() async { + let dependencies = makeDependencies( + samples: [audibleSamples(count: 16_000)], + transcriptions: [.success("Synthetic transcript.")] + ) + let viewModel = dependencies.makeViewModel() + viewModel.keepLastDictationEnabled = true + + await recordAndStop(viewModel) + viewModel.shutdown() + + #expect(viewModel.lastDictation == nil) + } + + @Test(arguments: [false, true]) + func clipboardOnlyDictationIsKeptForPastingToo(autoPasteEnabled: Bool) async { + let dependencies = makeDependencies( + samples: [audibleSamples(count: 16_000)], + transcriptions: [.success("Synthetic copied transcript.")] + ) + let viewModel = dependencies.makeViewModel() + defer { viewModel.shutdown() } + viewModel.keepLastDictationEnabled = true + viewModel.autoPasteEnabled = autoPasteEnabled + + await recordAndStop(viewModel, outputDestination: .clipboard) + await viewModel.pasteLastDictation() + + #expect(dependencies.paster.copiedTexts == ["Synthetic copied transcript."]) + #expect(dependencies.paster.pastedTexts == ["Synthetic copied transcript."]) + } + @Test(arguments: [false, true]) func clipboardOnlyCopiesWithoutPasting(autoPasteEnabled: Bool) async { let dependencies = makeDependencies( @@ -375,6 +480,68 @@ struct AppViewModelDictationFlowTests { await waitUntil { viewModel.state.isReady } } + @Test func listeningWaitsForTheFirstAudioBeforeShowingTheOverlay() async { + let dependencies = makeDependencies( + samples: [audibleSamples(count: 16_000)], + transcriptions: [.success("Synthetic transcript.")] + ) + dependencies.audio.holdFirstAudio = true + let viewModel = dependencies.makeViewModel() + defer { viewModel.shutdown() } + viewModel.soundEnabled = false + + viewModel.startListening() + await waitUntil { dependencies.audio.firstAudioWaitCount == 1 } + + #expect(!isListening(viewModel.state), "A Bluetooth headset may not deliver audio yet") + #expect(!dependencies.overlay.didShowListening) + + dependencies.audio.deliverFirstAudio() + await waitUntil { isListening(viewModel.state) } + + #expect(dependencies.overlay.didShowListening) + } + + @Test func aSilentCaptureNeverAnnouncesListening() async { + let dependencies = makeDependencies( + samples: [audibleSamples(count: 16_000)], + transcriptions: [.success("Synthetic transcript.")] + ) + dependencies.audio.firstAudioArrives = false + let viewModel = dependencies.makeViewModel() + defer { viewModel.shutdown() } + viewModel.soundEnabled = true + + viewModel.startListening() + await waitUntil { + dependencies.errorLogger.messages.contains { $0.contains("No audio arrived") } + } + + #expect(!isListening(viewModel.state)) + #expect(!dependencies.overlay.didShowListening) + } + + @Test func releasingBeforeTheFirstAudioStillStopsAndTranscribes() async { + let dependencies = makeDependencies( + samples: [audibleSamples(count: 16_000)], + transcriptions: [.success("Synthetic transcript.")] + ) + dependencies.audio.holdFirstAudio = true + let viewModel = dependencies.makeViewModel() + defer { viewModel.shutdown() } + viewModel.soundEnabled = false + + viewModel.startListening() + await waitUntil { dependencies.audio.firstAudioWaitCount == 1 } + viewModel.startProcessingStoppedRecording(shortcut: RecordingShortcut()) + dependencies.audio.deliverFirstAudio() + await waitUntil { dependencies.paster.pastedTexts.count == 1 } + await waitUntil { viewModel.state.isReady } + + #expect(dependencies.paster.pastedTexts == ["Synthetic transcript."]) + #expect(viewModel.state.isReady) + } + private func recordAndStop( _ viewModel: AppViewModel, mode: RecordingShortcutMode = .normal, @@ -474,6 +641,7 @@ private struct DictationDependencies { errorLogger: errorLogger, startsOperational: true, doneDisplayDuration: .milliseconds(1), + pasteAgainDelay: .milliseconds(1), transientErrorDisplayDuration: transientErrorDisplayDuration, audioDiagnostics: { selected in "selection=\(selected.map { "explicit:\($0)" } ?? "system-default") synthetic" @@ -521,6 +689,23 @@ private final class FakeAudioCapture: AudioCapturing, @unchecked Sendable { return .success(CaptureHandle(generation: nextGeneration)) } + var holdFirstAudio = false + var firstAudioArrives = true + private(set) var firstAudioWaitCount = 0 + private var firstAudioContinuation: CheckedContinuation? + + func waitForFirstAudio(handle: CaptureHandle, timeout: Duration) async -> Bool { + firstAudioWaitCount += 1 + guard holdFirstAudio else { return firstAudioArrives } + await withCheckedContinuation { firstAudioContinuation = $0 } + return firstAudioArrives + } + + func deliverFirstAudio() { + firstAudioContinuation?.resume() + firstAudioContinuation = nil + } + func waitForCaptureToSettle( handle: CaptureHandle, idleWindow: Duration, @@ -668,6 +853,13 @@ private final class FakeOverlayPresenter: OverlayPresenting { } } + var didShowListening: Bool { + shownStates.contains { + if case .listening = $0 { return true } + return false + } + } + var doneCount: Int { shownStates.count { if case .done = $0 { diff --git a/Tests/OpenWritrTests/HALInputCaptureTests.swift b/Tests/OpenWritrTests/HALInputCaptureTests.swift new file mode 100644 index 0000000..5227f96 --- /dev/null +++ b/Tests/OpenWritrTests/HALInputCaptureTests.swift @@ -0,0 +1,135 @@ +import CoreAudio +import Foundation +import Testing +@testable import OpenWritr + +@Suite("Input sample decoding") +struct HALSampleDecodingTests { + private func format( + bits: UInt32, + flags: AudioFormatFlags, + id: AudioFormatID = kAudioFormatLinearPCM, + channels: UInt32 = 1, + bytesPerFrame: UInt32? = nil + ) -> AudioStreamBasicDescription { + AudioStreamBasicDescription( + mSampleRate: 48_000, mFormatID: id, mFormatFlags: flags, mBytesPerPacket: 0, mFramesPerPacket: 1, + mBytesPerFrame: bytesPerFrame ?? bits / 8 * channels, mChannelsPerFrame: channels, + mBitsPerChannel: bits, mReserved: 0 + ) + } + + @Test func recognisesSupportedSampleTypes() { + #expect(HALSampleKind(format(bits: 32, flags: kAudioFormatFlagIsFloat)) == .float32) + #expect(HALSampleKind(format(bits: 16, flags: kAudioFormatFlagIsSignedInteger)) == .int16) + #expect(HALSampleKind(format(bits: 32, flags: kAudioFormatFlagIsSignedInteger)) == .int32) + #expect(HALSampleKind(format(bits: 16, flags: kAudioFormatFlagIsSignedInteger, channels: 2)) == .int16) + let nonInterleaved = kAudioFormatFlagIsFloat | kAudioFormatFlagIsNonInterleaved + #expect(HALSampleKind(format(bits: 32, flags: nonInterleaved, channels: 2, bytesPerFrame: 4)) == .float32) + } + + @Test func rejectsSamplesPaddedInsideAWiderFrame() { + #expect(HALSampleKind(format(bits: 16, flags: kAudioFormatFlagIsSignedInteger, bytesPerFrame: 4)) == nil) + #expect(HALSampleKind(format(bits: 32, flags: kAudioFormatFlagIsSignedInteger, bytesPerFrame: 8)) == nil) + #expect(HALSampleKind(format(bits: 16, flags: kAudioFormatFlagIsSignedInteger, bytesPerFrame: 0)) == nil) + } + + @Test func rejectsUnreadableFormats() { + #expect(HALSampleKind(format(bits: 24, flags: kAudioFormatFlagIsSignedInteger)) == nil) + #expect(HALSampleKind(format(bits: 64, flags: kAudioFormatFlagIsFloat)) == nil) + #expect(HALSampleKind(format(bits: 16, flags: kAudioFormatFlagIsSignedInteger | kAudioFormatFlagIsBigEndian)) == nil) + #expect(HALSampleKind(format(bits: 0, flags: 0, id: kAudioFormatMPEG4AAC)) == nil) + } + + @Test func decodesMonoFloat() { + let input: [Float] = [0.25, -0.5, 1] + let output = input.withUnsafeBytes { + HALSampleConverter.firstChannel(of: $0.baseAddress!, byteCount: $0.count, channels: 1, kind: .float32) + } + #expect(output == input) + } + + @Test func decodesFirstChannelOfInterleavedInt16() { + let input: [Int16] = [16_384, 1, -16_384, 2, -32_768, 3] + let output = input.withUnsafeBytes { + HALSampleConverter.firstChannel(of: $0.baseAddress!, byteCount: $0.count, channels: 2, kind: .int16) + } + #expect(output == [0.5, -0.5, -1]) + } + + @Test func decodesFirstChannelOfInterleavedInt32() { + let input: [Int32] = [1_073_741_824, 7, -1_073_741_824, 8] + let output = input.withUnsafeBytes { + HALSampleConverter.firstChannel(of: $0.baseAddress!, byteCount: $0.count, channels: 2, kind: .int32) + } + #expect(output == [0.5, -0.5]) + } + + @Test func ignoresATrailingPartialFrame() { + let input: [Int16] = [100, 200, 300] + let output = input.withUnsafeBytes { + HALSampleConverter.firstChannel(of: $0.baseAddress!, byteCount: $0.count, channels: 2, kind: .int16) + } + #expect(output.count == 1) + } + + @Test func emptyInputYieldsNoSamples() { + let byte: UInt8 = 0 + let output = withUnsafeBytes(of: byte) { + HALSampleConverter.firstChannel(of: $0.baseAddress!, byteCount: 0, channels: 1, kind: .float32) + } + #expect(output.isEmpty) + } +} + +@Suite("Input-only capture") +struct HALInputCaptureTests { + @Test func startingOnAMissingDeviceFailsAndLeavesNothingRunning() { + let capture = HALInputCapture( + deviceID: 999_999, + handler: { _ in Issue.record("No audio may be delivered for a missing device") }, + deviceLost: {} + ) + + #expect(throws: AudioEngineError.self) { + try capture.start() + } + #expect(!capture.isRunning) + } + + @Test func stoppingACaptureThatNeverStartedIsSafe() { + let capture = HALInputCapture(deviceID: 999_999, handler: { _ in }, deviceLost: {}) + + capture.stop() + capture.stop() + + #expect(!capture.isRunning) + } + + @Test func selectingAMissingDeviceIsRejected() { + let engine = AudioEngine() + defer { _ = engine.shutdown() } + + let result = engine.setInputDevice(999_999) + + #expect(throws: AudioEngineError.self) { + try result.get() + } + } + + /// Needs a real microphone and permission, so it only runs when asked for: + /// `OPENWRITR_HARDWARE_TESTS=1 swift test --filter HALInputCaptureTests`. + @Test(.enabled(if: ProcessInfo.processInfo.environment["OPENWRITR_HARDWARE_TESTS"] == "1")) + func capturesRealAudioFromTheDefaultInput() async throws { + let engine = AudioEngine() + defer { _ = engine.shutdown() } + let result = await engine.startCapture() + let handle = try result.get() + + try await Task.sleep(for: .seconds(1.5)) + let samples = try #require(await engine.stopCapture(handle: handle)) + + #expect(samples.count > 16_000, "At least one second of 16 kHz audio is expected") + #expect(samples.contains { $0 != 0 }, "The capture must contain signal, not only zeros") + } +} diff --git a/docs/index.html b/docs/index.html index f4a7072..2853bfe 100644 --- a/docs/index.html +++ b/docs/index.html @@ -916,7 +916,7 @@

Apple CoreML & Neural Engine

Swift 6 & SwiftUI

-

Pure native implementation with strict concurrency safety. MenuBarExtra for the system tray, AVAudioEngine for low-latency capture, CGEventTap for global hotkeys.

+

Pure native implementation with strict concurrency safety. MenuBarExtra for the system tray, Core Audio for low-latency input-only capture, CGEventTap for global hotkeys.