Skip to content

Commit f6eea86

Browse files
claude[bot]claude
andauthored
Custom style instructions for the cleanup rewrite (#130)
* feat: custom style instructions appended to the cleanup rewrite An Advanced-settings free-text field (e.g. "add some emojis where appropriate sparingly", "always write in lowercase") whose contents are appended to the cleanup instruction the dictation request's llm block carries, so the server-side rewrite also applies the user's formatting preferences. Empty or blank means the request is exactly what ships today. The dictation API rejects the whole request over its 2048-character instruction cap, so the appended text is capped at the real headroom the base instruction and a bridging preamble leave (CleanupInstruction.customStyleBudget, currently 352 characters) — enforced in the Settings field (with a visible counter) and again at send time, and derived from the actual lengths rather than restated. Storage follows KeyTermsStore (read-only store, @AppStorage is the sole writer, trimming on the read side); the transcriber reads it per request via an injected closure like enhancedTranscripts, so an edit applies to the very next dictation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwjo9LJw6z5PDyNeZjHbhc * fix: review findings on custom style instructions - Measure the instruction budget in UTF-8 bytes everywhere (cap arithmetic, engine trim, Settings counter and truncation), derived from the one budget definition. The API's 2048 cap was measured against the live endpoint but its unit was not; bytes are the largest plausible unit, so conservative against all of them. Truncation drops whole Characters so a multi-scalar emoji is never split — one shared String.prefix(maxUTF8Bytes:) used by both enforcement points, plus an emoji regression test. - Skip building the combined instruction when enhanced transcripts are off (the result was discarded); the over-cap log guard is unchanged. - Hide the Settings counter while the section is disabled, and give it an accessibility label ("N of M characters used") instead of the raw "N/M" digits. - Drop the transcriber-test assertions duplicating the equality check and CleanupInstructionTests' structure tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gwjo9LJw6z5PDyNeZjHbhc --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent a0b620b commit f6eea86

13 files changed

Lines changed: 339 additions & 44 deletions

AGENTS.md

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -410,7 +410,13 @@ response carries both `text` (verbatim) and
410410

411411
The instruction is `CleanupInstruction.text`, the winner of a GEPA run of
412412
`evals/dictation-prompt/` (see its README), compressed to fit the cap. Two later searches
413-
failed to beat it — the most recent scored 0.9043 against its 0.9101 on a held-out split.
413+
failed to beat it — the most recent scored 0.9043 against its 0.9101 on a held-out split. When the
414+
user has entered **custom style instructions** (`CustomStyleStore`, read per request via the
415+
transcriber's injected `customStyle` closure), `CleanupInstruction.sendable(appending:)` appends
416+
them after the base text with a bridging preamble, trimmed to the headroom the API's 2048
417+
instruction cap leaves (`customStyleBudget`, measured in UTF-8 bytes — the cap's own unit is
418+
unmeasured, and bytes are the conservative bound); empty or blank sends the base instruction
419+
unchanged.
414420

415421
Note what that does and does not establish. It is the best instruction the harness has
416422
produced, measured against other _text_ candidates on a _stand-in_ model; it has never been
@@ -617,6 +623,10 @@ Engine-side stores, all `UserDefaults`-backed value types with the same shape:
617623
**`DeveloperModeStore`** (`BlurtDeveloperMode`, off by default),
618624
**`EnhancedTranscriptsStore`** (`BlurtEnhancedTranscripts`, **on** by default — unset reads as
619625
enabled; gates the dictation request's `llm` cleanup-rewrite block, re-read at every request),
626+
**`CustomStyleStore`** (`BlurtCustomStyle`, the user's custom style instructions appended to the
627+
cleanup instruction via `CleanupInstruction.sendable(appending:)`, re-read at every request; empty
628+
sends the base instruction unchanged, and its `characterLimit` re-exports the headroom — in UTF-8
629+
bytes — the API's 2048 instruction cap leaves),
620630
**`OverlayOriginStore`** (the pill's dragged origin, x/y), **`LastUpdateCheckStore`**
621631
(`BlurtLastUpdateCheck`, the stamp throttling the automatic launch update check).
622632
- **`DefaultsKey`** (`Config/DefaultsKey.swift`) defines every key those stores write, one case each,

App/Blurt/Blurt/Wizard/SettingsWindowRoot.swift

Lines changed: 65 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -67,15 +67,16 @@ private struct GeneralSettingsTab: View {
6767
}
6868
}
6969

70-
/// The occasional stuff: the enhanced-transcripts switch, checking for an
71-
/// update, and the developer-mode log toggle. Kept out of General so the
72-
/// common pane stays short.
70+
/// The occasional stuff: the enhanced-transcripts switch, the custom style
71+
/// instructions, checking for an update, and the developer-mode log toggle.
72+
/// Kept out of General so the common pane stays short.
7373
private struct AdvancedSettingsTab: View {
7474
let updateModel: UpdateCheckModel
7575

7676
var body: some View {
7777
SettingsPane {
7878
TranscriptionSection()
79+
CustomStyleSection()
7980
UpdateSection(model: updateModel)
8081
DeveloperSection()
8182
}
@@ -113,6 +114,67 @@ private struct TranscriptionSection: View {
113114
}
114115
}
115116

117+
/// The Custom Style section of the Settings window: free-text style
118+
/// instructions appended to the cleanup instruction on every dictation request
119+
/// (see `CleanupInstruction.sendable(appending:)` / `CustomStyleStore`), so the
120+
/// enhanced-transcript polish also applies the user's formatting preferences.
121+
/// Optional — empty means the request is exactly what ships today. Disabled
122+
/// while enhanced transcripts are off, since the instruction it extends is not
123+
/// sent at all then.
124+
private struct CustomStyleSection: View {
125+
@AppStorage(EnhancedTranscriptsStore.defaultsKey)
126+
private var enhancedTranscripts = EnhancedTranscriptsStore.defaultValue
127+
128+
/// `@AppStorage` is the only writer of this slot (the store exposes no
129+
/// setter); trimming lives on the read side (`CustomStyleStore.instructions`)
130+
/// for the reasons on `KeyTermsStepView.text`. The length cap is enforced
131+
/// here, though: the dictation API rejects the whole request over its
132+
/// instruction limit, so text past `characterLimit` must never be storable.
133+
@AppStorage(CustomStyleStore.defaultsKey) private var text = ""
134+
135+
var body: some View {
136+
Section {
137+
TextField(
138+
text: $text,
139+
prompt: Text("e.g. add fitting emojis sparingly, or always write in lowercase"),
140+
axis: .vertical
141+
) {
142+
Text("Custom Style")
143+
}
144+
.labelsHidden()
145+
.lineLimit(2...6)
146+
.font(.body)
147+
.disableAutocorrection(true)
148+
.accessibilityIdentifier(UITestIdentifiers.customStyleField)
149+
.onChange(of: text) {
150+
if text.utf8.count > CustomStyleStore.characterLimit {
151+
text = text.prefix(maxUTF8Bytes: CustomStyleStore.characterLimit)
152+
}
153+
}
154+
.disabled(!enhancedTranscripts)
155+
} header: {
156+
Text("Custom Style")
157+
} footer: {
158+
HStack(alignment: .top) {
159+
Text(
160+
enhancedTranscripts
161+
? "Style preferences applied when polishing each dictation — casing, tone, emoji use."
162+
: "Style preferences need enhanced transcripts turned on.")
163+
if enhancedTranscripts {
164+
Spacer()
165+
// The API caps the combined instruction, so the room left is finite —
166+
// show it rather than truncating silently at the limit. Counted in
167+
// UTF-8 bytes, the unit the limit is enforced in.
168+
Text("\(text.utf8.count)/\(CustomStyleStore.characterLimit)")
169+
.monospacedDigit()
170+
.accessibilityLabel(
171+
"\(text.utf8.count) of \(CustomStyleStore.characterLimit) characters used")
172+
}
173+
}
174+
}
175+
}
176+
}
177+
116178
/// The Updates section of the Settings window: the running version and a
117179
/// "Check for Updates" button that runs the check and reports the result in a
118180
/// modal (see `UpdateCheckModel`). The same check is reachable from the

App/Blurt/Shared/UITestIdentifiers.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ enum UITestIdentifiers {
6464
static let soundPicker = "settings.sound.picker"
6565
static let developerToggle = "settings.developer.toggle"
6666
static let enhancedTranscriptsToggle = "settings.enhancedTranscripts.toggle"
67+
static let customStyleField = "settings.customStyle.field"
6768
static let updateCheck = "settings.update.check"
6869

6970
/// The dictation overlay pill (`OverlayView`).

BLURTENGINE.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ press() ──▶ MicCapture.start() release() ──▶ MicCapture.s
5959
Key properties of the design, which your integration can rely on:
6060

6161
- **One request per utterance, no streaming.** The dictation API returns the complete transcript — and its LLM-rewritten form — in the response body: no upload step, no job polling, no incremental deltas, no second request for the cleanup. `TranscriberProtocol.transcribe` is a single `async throws -> String`. UIs should show a "transcribing…" state and then the whole result; there is nothing to stream.
62-
- **Cleanup happens server-side, and it's optional.** The request's `llm` block asks the service to apply our own cleanup instruction (`CleanupInstruction.text` — delete disfluencies, change nothing else) to the verbatim transcript inside the same call. It is the only instruction on the request: the separate `config.prompt` field, which primes the _transcription_, is switched off at `TranscriptionPrompt.isEnabled` and omitted from every request. The block is gated by the **enhanced transcripts** setting (`EnhancedTranscriptsStore`, on by default): turned off, the config omits `llm` and the verbatim transcript is pasted as spoken. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one.
62+
- **Cleanup happens server-side, and it's optional.** The request's `llm` block asks the service to apply our own cleanup instruction (`CleanupInstruction.text` — delete disfluencies, change nothing else) to the verbatim transcript inside the same call. It is the only instruction on the request: the separate `config.prompt` field, which primes the _transcription_, is switched off at `TranscriptionPrompt.isEnabled` and omitted from every request. The block is gated by the **enhanced transcripts** setting (`EnhancedTranscriptsStore`, on by default): turned off, the config omits `llm` and the verbatim transcript is pasted as spoken. The user's **custom style instructions** (`CustomStyleStore`, empty by default) are appended to that instruction via `CleanupInstruction.sendable(appending:)`, trimmed to the headroom the API's 2048 instruction cap leaves (measured in UTF-8 bytes, the conservative bound — the cap's own unit is unmeasured); blank means the base instruction goes out unchanged. The engine pastes `llm_response`, falling back to the verbatim `text` when the best-effort rewrite failed (`llm_error`) — a degradation, never a user-facing error. There is no client-side LLM pass, no styling stage, and deliberately no hook for one.
6363
- **Latency is pre-paid where possible.** `press()` fires a detached `warmUp()` at the transcriber (pre-opening the HTTPS connection while the user speaks, ~170 ms saved cold) and kicks off the cross-process accessibility read of the focused field without awaiting it — the read is then consumed at transcribe time with a bounded wait (`DictationSession.contextWaitBudget`, 500 ms), so an unresponsive frontmost app costs the transcript its priming, never a multi-second stall — and never delays the recording indicator. On the way out, `release()` flips the phase to `.transcribing` _before_ reading the recorded audio back, so a host's stop cue fires at key-up rather than after the disk read.
6464
- **A held trigger auto-releases.** `DictationSession` stops recording after `maxRecordingSeconds` (default `SyncSTTLimits.autoReleaseSeconds`, 115 s) so audio never exceeds what the endpoint accepts, and transcribes what it has. Clips shorter than `SyncSTTLimits.minPCMBytes` (~100 ms of audio — an accidental tap) are dropped as a silent no-op rather than sent to earn a 400.
6565

@@ -133,7 +133,7 @@ func transcribe(pcm: Data, sampleRate: Int, context: TranscriptionContext?) asyn
133133
func warmUp() async // optional; no-op default
134134
```
135135

136-
`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt` (nil today, so the field is omitted), and — while enhanced transcripts are enabled, the default — an `llm` block whose one `instruction` field carries `CleanupInstruction.text`), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.current`), a `baseURL`, an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses — and an `enhancedTranscripts` closure deciding, per request, whether the `llm` block is sent (nil, the default, reads `EnhancedTranscriptsStore`). `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before.
136+
`AssemblyAITranscriber` is a stateless `Sendable` struct. One `POST https://dictation.assemblyai.com/transcribe` per utterance: the audio as raw S16LE PCM (the `pcm` blob, byte-for-byte) in the `audio` multipart part, plus a JSON `config` part (`sample_rate`, `channels`, the rendered `prompt` (nil today, so the field is omitted), and — while enhanced transcripts are enabled, the default — an `llm` block whose one `instruction` field carries `CleanupInstruction.text`, with any custom style instructions appended — `CleanupInstruction.sendable(appending:)`), with the API key in `Authorization` (no model header — the service pins the STT model server-side). The response carries the verbatim `text` and the rewritten `llm_response`; the transcriber returns the rewrite and falls back to `text` when it is null (the rewrite is best-effort — `llm_error` is logged, never surfaced as a failure). Its initializer takes an `apiKeyProvider` closure (defaults to `APIKeyStore.current`), a `baseURL`, an `HTTPTransport` — inject a fake transport (see `Tests/BlurtEngineTests/Stubs/FakeHTTPTransport.swift`) to test against canned responses — an `enhancedTranscripts` closure deciding, per request, whether the `llm` block is sent (nil, the default, reads `EnhancedTranscriptsStore`), and a `customStyle` closure supplying the style instructions appended to the cleanup instruction (nil, the default, reads `CustomStyleStore`). `warmUp()` fires a throwaway GET at the host root to pre-pool the connection; it never throws and any failure just means the real request pays connection setup as before.
137137

138138
The model's limits live in `SyncSTTLimits` (16 kHz sample rate, ~0.1 s–120 s audio, and the auto-release math — the sync STT model behind the dictation service) — the single source shared by the mic, the session, and the request so recorded and declared geometry can't drift.
139139

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
import Foundation
2+
3+
/// Storage for the user's custom style instructions — free text (e.g. "add some
4+
/// emojis where appropriate sparingly", "always write in lowercase") appended to
5+
/// the cleanup instruction so the dictation API's server-side rewrite also
6+
/// applies the user's formatting preferences (see
7+
/// `CleanupInstruction.sendable(appending:)`). Optional: unset or blank sends
8+
/// exactly the instruction that ships today. Inert while enhanced transcripts
9+
/// are off, since the `llm` block the instruction rides on is omitted entirely.
10+
/// `AssemblyAITranscriber` reads this at each request, so an edit applies to the
11+
/// very next dictation.
12+
///
13+
/// Read-only for the same reason as `KeyTermsStore`: the Settings field binds
14+
/// `@AppStorage` straight to `defaultsKey` and is the sole writer — a
15+
/// normalizing setter fights the text field — so trimming lives on the read
16+
/// side and a whitespace-only field still reads back as "no instructions".
17+
public struct CustomStyleStore {
18+
/// `UserDefaults` key for the raw text the user typed. Public so the Settings
19+
/// field can bind `@AppStorage` to it.
20+
public static let defaultsKey = DefaultsKey.customStyle.rawValue
21+
22+
/// The most UTF-8 bytes the Settings field accepts —
23+
/// `CleanupInstruction.customStyleBudget`, the real headroom the dictation
24+
/// API's 2048 instruction cap leaves after the base instruction (see
25+
/// `CleanupInstruction.characterCap` for why the unit is bytes).
26+
/// Re-exported here (the field's counter and the engine's trim have to agree)
27+
/// rather than restated, which is how the cap bug shipped once before.
28+
public static let characterLimit = CleanupInstruction.customStyleBudget
29+
30+
private let defaults: UserDefaults
31+
32+
init(defaults: UserDefaults = .standard) {
33+
self.defaults = defaults
34+
}
35+
36+
/// The instructions to append, trimmed, or `nil` when unset or blank — blank
37+
/// must mean "send the base instruction untouched", not an empty suffix.
38+
var instructions: String? {
39+
defaults.string(forKey: Self.defaultsKey).trimmedNonEmpty()
40+
}
41+
}

Sources/BlurtEngine/Config/DefaultsKey.swift

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ enum DefaultsKey: String, CaseIterable {
2525
case keyTerms = "BlurtKeyTerms"
2626
case developerMode = "BlurtDeveloperMode"
2727
case enhancedTranscripts = "BlurtEnhancedTranscripts"
28+
case customStyle = "BlurtCustomStyle"
2829
/// `OverlayOriginStore` persists a point, so it owns two keys rather than one.
2930
case overlayCustomOriginX = "BlurtOverlayCustomOriginX"
3031
case overlayCustomOriginY = "BlurtOverlayCustomOriginY"

Sources/BlurtEngine/STT/AssemblyAITranscriber.swift

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ public struct AssemblyAITranscriber: TranscriberProtocol {
2626
private let baseURL: URL
2727
private let transport: any HTTPTransport
2828
private let enhancedTranscriptsEnabled: @Sendable () -> Bool
29+
private let customStyle: @Sendable () -> String?
2930

3031
/// Idle timeout for the transcribe round trip — `URLRequest.timeoutInterval` is
3132
/// reset each time data moves, so this bounds *stalls*, not total elapsed time.
@@ -37,21 +38,25 @@ public struct AssemblyAITranscriber: TranscriberProtocol {
3738
private static let requestTimeoutSeconds: TimeInterval = 90
3839

3940
/// `enhancedTranscripts` decides, per request, whether the config carries
40-
/// the `llm` cleanup-rewrite block. Read at every `transcribe` so a settings
41-
/// change applies to the next dictation without rebuilding the transcriber.
42-
/// `nil` (the default) reads `EnhancedTranscriptsStore` — spelled as an
43-
/// optional rather than a default closure because a public default argument
44-
/// can't reference the store's internal `isEnabled`.
41+
/// the `llm` cleanup-rewrite block; `customStyle` supplies the user's custom
42+
/// style instructions appended to that block's cleanup instruction. Both are
43+
/// read at every `transcribe` so a settings change applies to the next
44+
/// dictation without rebuilding the transcriber. `nil` (the default) reads
45+
/// the corresponding store — spelled as optionals rather than default
46+
/// closures because a public default argument can't reference a store's
47+
/// internal member.
4548
public init(
4649
apiKeyProvider: @escaping @Sendable () -> String? = { APIKeyStore.current },
4750
baseURL: URL = URL(staticString: "https://dictation.assemblyai.com"),
4851
transport: any HTTPTransport = URLSession.shared,
49-
enhancedTranscripts: (@Sendable () -> Bool)? = nil
52+
enhancedTranscripts: (@Sendable () -> Bool)? = nil,
53+
customStyle: (@Sendable () -> String?)? = nil
5054
) {
5155
self.apiKeyProvider = apiKeyProvider
5256
self.baseURL = baseURL
5357
self.transport = transport
5458
self.enhancedTranscriptsEnabled = enhancedTranscripts ?? { EnhancedTranscriptsStore().isEnabled }
59+
self.customStyle = customStyle ?? { CustomStyleStore().instructions }
5560
}
5661

5762
// MARK: - Dictation request
@@ -131,15 +136,15 @@ public struct AssemblyAITranscriber: TranscriberProtocol {
131136
/// `URLProtocol` mocks can't observe reliably for `upload(from:)`).
132137
func makeConfigData(sampleRate: Int, prompt: String?) throws -> Data {
133138
let enhanced = enhancedTranscriptsEnabled()
134-
let instruction = CleanupInstruction.sendable
139+
let instruction = enhanced ? CleanupInstruction.sendable(appending: customStyle()) : nil
135140
if enhanced, instruction == nil {
136141
// Unreachable while the tests run: `CleanupInstructionTests` asserts the length.
137142
// Logged rather than trusted because the failure it guards against is silent —
138143
// the request would 400 and every dictation would error, so a line naming the
139144
// real cause is worth the one comparison per request it costs.
140145
transcriberLog.error(
141146
"""
142-
cleanup instruction is \(CleanupInstruction.text.count, privacy: .public) characters, \
147+
cleanup instruction is \(CleanupInstruction.text.utf8.count, privacy: .public) UTF-8 bytes, \
143148
over the \(CleanupInstruction.characterCap, privacy: .public) cap; \
144149
falling back to the service default
145150
""")

0 commit comments

Comments
 (0)