Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ Each was tried the other way and reverted. If a task seems to require one, stop
| Add an external SPM dependency to the engine | Dependency-free by rule (biggest supply-chain risk); a `check.sh` guard fails on `.package(` in `Package.swift` or a `url:`/`github:` package in `project.yml`. Extend `BlurtEngine` instead. |
| Use `AVAudioEngine` / `installTap` for capture | A long-lived engine bound its input graph to one device and went stale on a mic↔built-in switch — `-10868` (`kAudioUnitErr_FormatNotSupported`) or all-zero buffers. `MicCapture` uses a fresh `AVAudioRecorder` per session. |
| Add streaming STT | The dictation API returns the full (already rewritten) text in one response; the overlay shows "Transcribing…" then the full text. |
| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `TranscriptionPrompt`. |
| Add a client-side LLM cleanup pass | Cleanup is the dictation API's server-side rewrite, requested by the `llm` block on the same `/v1/transcribe` call. No LLM Gateway client, no `StylerProtocol`, no styling stage, no second request — transcription steering belongs in `TranscriptionPrompt`. |
| Add local models or model downloads | Transcription is a remote AssemblyAI call: no on-device ASR/LLM, no model cache, no download UI. |
| Pin the prompt to English | Hurt non-English transcription; language is left to the model's own detection. |
| Add a "remove filler words (um, uh, like)" clause | Not in the STT model's trained instruction set — a no-op, deliberately dropped; disfluency removal is the server-side LLM rewrite's job. |
Expand Down Expand Up @@ -213,7 +213,7 @@ framework, or notarization rejects the build; roll-forward-only for a bad releas
```text
DictationKeyTap (CGEventTap + DictationKeyGate) → AppCoordinator → DictationSession (actor) → MicCapture
AssemblyAITranscriber (STT + LLM cleanup, AssemblyAI dictation API: one POST /transcribe)
AssemblyAITranscriber (STT + LLM cleanup, AssemblyAI dictation API: one POST /v1/transcribe)
KeyInjector → focused app (clipboard paste via a synthesized ⌘V CGEvent)
```
Expand Down Expand Up @@ -241,7 +241,7 @@ the seam they inject.
### `AssemblyAITranscriber` — `Sources/BlurtEngine/STT/AssemblyAITranscriber.swift`

Implements `TranscriberProtocol` against AssemblyAI's **dictation** API: a single
`POST https://dictation.assemblyai.com/transcribe` with the captured audio as a raw S16LE PCM blob
`POST https://dictation.assemblyai.com/v1/transcribe` with the captured audio as a raw S16LE PCM blob
in the `audio` multipart part plus a JSON `config` part (`sample_rate`, `channels`, `prompt`, and an
empty `llm` block). No model header — the service pins the STT model server-side. The `prompt`
(built per utterance by `TranscriptionPrompt`) steers _transcription_; the `llm` block asks the
Expand Down
4 changes: 2 additions & 2 deletions BLURTENGINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Before the first dictation can succeed the host must have:
```text
press() ──▶ MicCapture.start() release() ──▶ MicCapture.stop() → Data (raw S16LE PCM)
(16 kHz mono 16-bit PCM) AssemblyAITranscriber.transcribe(pcm:sampleRate:context:)
+ focus/context capture (one POST dictation.assemblyai.com/transcribe: STT + LLM rewrite)
+ focus/context capture (one POST dictation.assemblyai.com/v1/transcribe: STT + LLM rewrite)
+ connection warm-up KeyInjector.insert(text, after: priorText)
(clipboard paste via synthesized ⌘V)
```
Expand Down Expand Up @@ -133,7 +133,7 @@ func transcribe(pcm: Data, sampleRate: Int, context: TranscriptionContext?) asyn
func warmUp() async // optional; no-op default
```

`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt`, and — while enhanced transcripts are enabled, the default — an empty `llm` block requesting the service's default cleanup rewrite), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.current`), a `baseURL`, an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses — and an `enhancedTranscripts` closure deciding, per request, whether the `llm` block is sent (nil, the default, reads `EnhancedTranscriptsStore`). `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before.
`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/v1/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt`, and — while enhanced transcripts are enabled, the default — an empty `llm` block requesting the service's default cleanup rewrite), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.current`), a `baseURL`, an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses — and an `enhancedTranscripts` closure deciding, per request, whether the `llm` block is sent (nil, the default, reads `EnhancedTranscriptsStore`). `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before.

The model's limits live in `SyncSTTLimits` (16 kHz sample rate, ~0.1 s–120 s audio, and the auto-release math — the sync STT model behind the dictation service) — the single source shared by the mic, the session, and the request so recorded and declared geometry can't drift.

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ directories, so the app needs a stable install path to be usable at all.
Sources/BlurtEngine/ Swift 6 package owning the pipeline — no external dependencies
Audio/ MicCapture: fresh AVAudioRecorder per session, 16 kHz mono PCM,
live level meter; DX7/Juno-106 sound packs
STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/transcribe
STT/ AssemblyAITranscriber: one POST to dictation.assemblyai.com/v1/transcribe
(STT + LLM rewrite) + TranscriptionPrompt contextual priming
Pipeline/ DictationSession actor: press/release/cancel commands, phase
stream, auto-release before the API's recording cap
Expand Down
6 changes: 3 additions & 3 deletions Sources/BlurtEngine/STT/AssemblyAITranscriber.swift
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ private let transcriberLog = Logger(subsystem: BlurtIdentity.subsystem, category

/// `TranscriberProtocol` backed by AssemblyAI's **dictation** API.
///
/// A single `POST dictation.assemblyai.com/transcribe` carries the captured
/// A single `POST dictation.assemblyai.com/v1/transcribe` carries the captured
/// audio (raw S16LE PCM, exactly the bytes the mic recorded — there is no
/// re-encoding pass) plus a JSON `config` part, and the response body carries
/// both the verbatim transcript and — when the config requests one via its
Expand Down Expand Up @@ -65,7 +65,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol {
let config = try makeConfigData(sampleRate: sampleRate, prompt: prompt)
let boundary = "blurt-\(UUID().uuidString)"

var request = URLRequest(url: baseURL.appendingPathComponent("transcribe"))
var request = URLRequest(url: baseURL.appendingPathComponent("v1/transcribe"))
request.httpMethod = "POST"
// Bounds a stalled connection; see `requestTimeoutSeconds` for why an idle
// timeout is the right shape here.
Expand Down Expand Up @@ -98,7 +98,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol {
/// `transcribe` reuses it instead of paying DNS+TCP+TLS on the hot path
/// (~170 ms cold, more on mobile — measured). A throwaway GET to the host
/// root is enough to establish the HTTP/2 connection `URLSession` then reuses
/// for the POST to `/transcribe`; the response (an auth-less 4xx) is
/// for the POST to `/v1/transcribe`; the response (an auth-less 4xx) is
/// discarded. No key, so it never counts as a transcription. A short timeout
/// keeps a dead network from leaving the task hanging. Fire-and-forget: any
/// error is swallowed — a failed warm-up just means the next request pays
Expand Down
4 changes: 2 additions & 2 deletions Tests/BlurtEngineTests/AssemblyAITranscriberTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ struct HTTPClientTests {
let hits = Counter()
let transport = FakeHTTPTransport { request in
_ = hits.next()
guard request.url?.path.hasSuffix("/transcribe") == true,
guard request.url?.path == "/v1/transcribe",
request.httpMethod == "POST"
else { return (404, Data()) }
return (200, json(["text": "um hello world", "llm_response": "Hello world."]))
Expand Down Expand Up @@ -55,7 +55,7 @@ struct HTTPClientTests {
@Test("transcriber succeeds with a real context (builds and sends a prompt)")
func transcribeWithContext() async throws {
let transport = FakeHTTPTransport { request in
guard request.url?.path.hasSuffix("/transcribe") == true else { return (404, Data()) }
guard request.url?.path == "/v1/transcribe" else { return (404, Data()) }
return (200, json(["text": "hello world"]))
}

Expand Down
Loading