diff --git a/CHANGELOG.md b/CHANGELOG.md index f17d6be..81645b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,53 @@ All notable changes to bestASR are documented here. The format follows ## [Unreleased] +### Changed + +- **BREAKING — `Engine` now declares its prompt capability (#164)**: the protocol + gains a `promptCapability` requirement with **no default implementation**, so + every conformer — including any out-of-tree engine — must state whether it + consumes decoder conditioning text and up to how many tokens. A default was + considered and rejected: it would let a new backend inherit "no prompt support" + without its author ever considering the question, which is a quieter form of + the very problem this change fixes. The cost is a one-time source-breaking + change; the benefit is that the answer can no longer be assumed. + +- **Context injection reports the truth, and reaches more of your terms (#164)**: + the token budget now comes from the selected backend instead of a single global + constant. + + Two things change for you. First, a backend that ignores conditioning text no + longer prints `injected (N)` and a truncation list — it says plainly that it + does not support context biasing, and selection warns when your context cannot + take effect. Previously five of the seven backends ran the whole render-and- + truncate pipeline and discarded the result while still reporting a count; a + real run showed `injected (49) / truncated (53)` against a backend that used + none of them, and the two terms that mattered were both in the "injected" list + and both mis-transcribed. Acting on that number by trimming your term list + changed nothing. + + Second, on the Whisper backends the budget rises from 200 to their measured + 224-token ceiling, so **more of the same context directory now reaches the + model**. Expect slightly different transcripts there. This is the intended + improvement, not model drift. + + The engine-side clamp direction was reversed in the same change: overflow now + discards the lowest-priority phrases rather than the names at the front of the + prompt, which is what it had been dropping. Read that as a reversible bet, not + a settled correction. It trades against a different mechanism the previous + direction was built on — Whisper's reference decoder truncates an over-long + prompt by keeping its *tail*, and content nearest the transcription boundary + is widely reported to weigh more. Neither direction has been measured here. + If a WER regression shows up on long-context Whisper runs, this is the line to + suspect; the durable fix is to reorder the renderer so the highest-value items + land at the tail, not to flip the clamp back. See `PipelineWiringTests` for + the trade-off in full. + + Selection warns but does not re-rank. Whether a lower measured error rate is + worth losing context biasing has not been measured, so the trade-off is + surfaced rather than decided. + + ### Added - **Apple Speech backend (#121)**: `apple-speech` — the OS-native backend diff --git a/Sources/BestASRKit/Benchmark/BenchmarkRunner.swift b/Sources/BestASRKit/Benchmark/BenchmarkRunner.swift index 3a4821a..7e3e67e 100644 --- a/Sources/BestASRKit/Benchmark/BenchmarkRunner.swift +++ b/Sources/BestASRKit/Benchmark/BenchmarkRunner.swift @@ -174,6 +174,12 @@ public struct BenchmarkRunner { ) async -> BenchmarkOutcome { var measured: [MeasuredCandidate] = [] var failures: [BenchmarkFailure] = [] + // Candidates whose backend declares no prompt support, so the ±context + // pass was not run for them (#164 verify). Reported, never silent. + var contextSkipped: [String] = [] + // Candidates that CAN take a prompt but whose with-context pass threw. + // Kept apart from contextSkipped: same blank cell, different reason. + var contextFailed: [String] = [] guard let audioDuration = audio.duration, audioDuration > 0 else { return BenchmarkOutcome( @@ -252,10 +258,27 @@ public struct BenchmarkRunner { language: language) // Optional second pass with the context prompt (spec benchmark: - // Measure the context-biasing delta). Model is warm; failures - // here degrade to a note-worthy nil, not a candidate failure. + // Measure the context-biasing delta). Model is warm; a failure + // here is not a candidate failure — the baseline measurement + // still stands. + // + // Gated on the candidate's own declaration (#164 verify): a + // backend that takes no prompt must never receive one (spec + // asr-engine), and measuring a "with-context" pass there would + // publish a delta of ~0 that reads as "context does not help + // this backend" when the truth is that it cannot use context + // at all. + // + // Both ways of ending up without a delta are named in the + // notes, and named SEPARATELY: "declares no prompt support" + // and "the pass threw" are different facts about the backend, + // and reporting a transient decode failure as a capability + // limit would be false. var contextErrorRate: Double? - if let contextPrompt { + if contextPrompt != nil, !engine.promptCapability.supportsPrompt { + contextSkipped.append(candidate.backend.rawValue) + } + if let contextPrompt, engine.promptCapability.supportsPrompt { let contextOptions = TranscribeOptions( model: candidate.model, quantization: candidate.quantization, @@ -263,12 +286,16 @@ public struct BenchmarkRunner { prompt: contextPrompt, deterministicDecode: deterministicDecode ) - if let contextTranscript = try? await engine.transcribe( - audioPath: normalizedAudio.path, options: contextOptions) - { + do { + let contextTranscript = try await engine.transcribe( + audioPath: normalizedAudio.path, options: contextOptions) contextErrorRate = ErrorRate.compute( hypothesis: contextTranscript.text, reference: referenceText, kind: metricKind, language: language) + } catch { + let reason = (error as? TranscriptionError)?.errorDescription + ?? error.localizedDescription + contextFailed.append("\(candidate.backend.rawValue) (\(reason))") } } let record = BenchmarkRecord( @@ -297,10 +324,27 @@ public struct BenchmarkRunner { } } + // Naming the skipped backends keeps a mixed grid honest: without this + // line a report showing a DELTA column for some rows and blanks for + // others gives no reason for the blanks. + let skipNote = contextSkipped.isEmpty + ? [] + : [ + "context: no with-context pass for " + + Set(contextSkipped).sorted().joined(separator: ", ") + + " — these backends declare no prompt support" + ] + let failNote = contextFailed.isEmpty + ? [] + : [ + "context: with-context pass failed for " + + Set(contextFailed).sorted().joined(separator: "; ") + + " — the baseline measurement for these candidates still stands" + ] return BenchmarkOutcome( measured: measured, failures: failures, - notes: initialNotes, + notes: initialNotes + skipNote + failNote, metricKind: metricKind, language: language ) diff --git a/Sources/BestASRKit/CommandCore.swift b/Sources/BestASRKit/CommandCore.swift index d4997d4..90800e7 100644 --- a/Sources/BestASRKit/CommandCore.swift +++ b/Sources/BestASRKit/CommandCore.swift @@ -90,17 +90,36 @@ public struct CommandCore: Sendable { struct ContextBundle { let loaded: LoadedContext - let rendered: PromptRenderer.Rendered + /// `nil` when the selected backend takes no conditioning text, so no + /// prompt was rendered. Distinct from a bundle whose prompt happens to + /// be empty: nil means the question did not apply. + let rendered: PromptRenderer.Rendered? } /// Resolve + load + render. Returns nil when nothing resolves or the /// directory holds neither values nor ignorable files — zero impact. /// A directory that only holds unsupported files still returns a bundle /// (prompt nil) so the ignore list is disclosed loudly, never silently. - func loadContext(flag: String?) throws -> ContextBundle? { + /// + /// - Parameter capability: the selected backend's declaration. When it + /// reports no usable budget, rendering is **skipped entirely** rather than + /// performed and discarded (design D3): the truncation arithmetic would be + /// real work producing figures that describe nothing, and reporting an + /// injected count for a backend that consumes none is the misstatement + /// this whole change exists to remove. + /// + /// `nil` means "no single backend applies" — the benchmark's ±context + /// delta mode measures many candidates in one run — and keeps the + /// previous global default. + func loadContext(flag: String?, capability: PromptCapability? = nil) throws -> ContextBundle? { guard let loaded = try ContextLoader.load(flag: flag) else { return nil } if loaded.isEmpty && loaded.ignoredFiles.isEmpty { return nil } - return ContextBundle(loaded: loaded, rendered: PromptRenderer.render(loaded)) + if let capability, !capability.supportsPrompt { + return ContextBundle(loaded: loaded, rendered: nil) + } + let budget = capability?.effectiveBudget ?? PromptRenderer.defaultTokenBudget + return ContextBundle( + loaded: loaded, rendered: PromptRenderer.render(loaded, tokenBudget: budget)) } /// Grid-row lookup for a measured candidate (#16): keyed by the facts the @@ -114,26 +133,99 @@ public struct CommandCore: Sendable { } } + /// Wording used wherever a backend cannot consume conditioning text. Naming + /// the *predicate* once keeps the reason line, the selection warning and the + /// explain block from drifting apart while each caller supplies its own + /// subject — folding the subject in here produced "selected backend this + /// backend does not support …" (#164 verify). + static let contextUnsupportedNote = + "does not support context biasing — the context will not affect this transcription" + + /// The declaration of the engine registered for `backend`, or nil when none + /// is registered (the caller then keeps the previous global default). + func promptCapability(for backend: BackendID) -> PromptCapability? { + engines.first(where: { $0.id == backend })?.promptCapability + } + + /// Which capability the benchmark should render its single context prompt + /// against (#164 verify). The benchmark has no one "selected" engine, which + /// is why `loadContext` takes the capability as an optional — but the + /// prompt now only reaches candidates that declare support, so: + /// + /// - candidates agreeing on one budget → that budget, so this call site + /// obeys "the budget comes from the engine" like the other two; + /// - nothing in the grid can take a prompt → `.unsupported`, so no prompt + /// is rendered for a pass that cannot happen; + /// - candidates disagreeing → nil (keep the global default): one prompt + /// cannot honour two budgets, and the smaller backend's own clamp is + /// the remaining backstop. + /// + /// Note the deliberate asymmetry with `promptCapability(for:)`, whose nil + /// means "unknown — no engine registered" and tells its caller to keep the + /// global default. Here an unregistered candidate contributes nothing and + /// a grid of *only* unregistered candidates therefore yields `.unsupported` + /// rather than nil. That is intentional: this function's job is "can the + /// with-context pass happen at all", and a candidate with no engine cannot + /// run any pass — `BenchmarkRunner` will not measure it either. The case is + /// unreachable from `benchmark()` regardless, because `enumerateCandidates` + /// derives candidates from the same `engines` array, but the reasoning is + /// recorded here because two reviewers read the old wording in opposite + /// ways (#164 verify round 2). + func benchmarkPromptCapability(for candidates: [BenchmarkCandidate]) -> PromptCapability? { + var budgets = Set() + for candidate in candidates { + guard let capability = promptCapability(for: candidate.backend) else { continue } + if let budget = capability.effectiveBudget { budgets.insert(budget) } + } + if budgets.isEmpty { return .unsupported } + guard budgets.count == 1, let budget = budgets.first else { return nil } + return .supported(maxTokens: budget) + } + + /// Selection-time warning (design D5). Returns nil when there is nothing to + /// warn about — including when no engine is registered for the choice, since + /// inventing a warning from an unknown is worse than staying quiet. + static func contextCapabilityWarning(_ capability: PromptCapability?) -> String? { + guard let capability, !capability.supportsPrompt else { return nil } + return "the selected backend \(Self.contextUnsupportedNote)" + } + static func contextReasonLine(_ bundle: ContextBundle) -> String { - if bundle.rendered.injected.isEmpty { + guard let rendered = bundle.rendered else { + return "context: \(bundle.loaded.directory) — this backend \(Self.contextUnsupportedNote)" + } + if rendered.injected.isEmpty { return "context: \(bundle.loaded.directory) — 0 values injected; " + "\(bundle.loaded.ignoredFiles.count) file(s) ignored (run the context-ingest skill)" } - return "context: \(bundle.loaded.directory) — \(bundle.rendered.injected.count) value(s) injected" + return "context: \(bundle.loaded.directory) — \(rendered.injected.count) value(s) injected" } /// Explain-mode disclosure (design D9): resolved dir, injected values, /// truncated items, ignored files with ingestion guidance. + /// + /// When the backend takes no prompt there is no injected count and no + /// truncation list to report — printing `injected (N)` there was the + /// original complaint: it reads as "this worked", and a user acts on it by + /// trimming their term list, which changes nothing (spec `Explain discloses + /// context usage`). static func contextExplanation(_ bundle: ContextBundle) -> [String] { var lines = ["Context: \(bundle.loaded.directory)"] + guard let rendered = bundle.rendered else { + lines.append(" this backend \(Self.contextUnsupportedNote)") + for file in bundle.loaded.ignoredFiles { + lines.append(" ignored: \(file) — \(LoadedContext.ingestGuidance)") + } + return lines + } lines.append( - " injected (\(bundle.rendered.injected.count)): " - + (bundle.rendered.injected.isEmpty - ? "(none)" : bundle.rendered.injected.joined(separator: ", "))) - if !bundle.rendered.truncated.isEmpty { + " injected (\(rendered.injected.count)): " + + (rendered.injected.isEmpty + ? "(none)" : rendered.injected.joined(separator: ", "))) + if !rendered.truncated.isEmpty { lines.append( - " truncated (\(bundle.rendered.truncated.count)): " - + bundle.rendered.truncated.joined(separator: ", ")) + " truncated (\(rendered.truncated.count)): " + + rendered.truncated.joined(separator: ", ")) } for file in bundle.loaded.ignoredFiles { lines.append(" ignored: \(file) — \(LoadedContext.ingestGuidance)") @@ -291,13 +383,25 @@ public struct CommandCore: Sendable { let lang = await resolveAutoLanguage(audioPath: audio.path, resolved: audio.language) var rec = try await resolveRecommendation(selection: selection, language: lang.language) rec = rec.merging(reasons: lang.reasons, warnings: lang.warnings) - if let bundle = try loadContext(flag: selection.contextDir) { + let selectedCapability = promptCapability(for: rec.backend) + if let bundle = try loadContext( + flag: selection.contextDir, capability: selectedCapability) + { + // D5: surface the trade-off, do not decide it. The backend stays + // selected — whether a lower measured error rate is worth losing + // context biasing has not been measured, and quietly re-ranking on + // an unmeasured belief would be the same overreach in the other + // direction. + var warnings = rec.warnings + if let note = Self.contextCapabilityWarning(selectedCapability) { + warnings.append(note) + } rec = ASRRecommendation( backend: rec.backend, model: rec.model, quantization: rec.quantization, profile: rec.profile, language: rec.language, dataSource: rec.dataSource, measured: rec.measured, reason: rec.reason + [Self.contextReasonLine(bundle)], - warnings: rec.warnings + warnings: warnings ) } let document = RecommendationJSON( @@ -335,18 +439,34 @@ public struct CommandCore: Sendable { let audio = try AudioProber.probe( path: audioPath, requestedLanguage: selection.requestedLanguage) let lang = await resolveAutoLanguage(audioPath: audio.path, resolved: audio.language) - let rec = (try await resolveRecommendation(selection: selection, language: lang.language)) + let resolved = (try await resolveRecommendation( + selection: selection, language: lang.language)) .merging(reasons: lang.reasons, warnings: lang.warnings) - guard let engine = engines.first(where: { $0.id == rec.backend }) else { - throw BestASRError.runtime("no engine registered for backend \(rec.backend.rawValue)") + guard let engine = engines.first(where: { $0.id == resolved.backend }) else { + throw BestASRError.runtime( + "no engine registered for backend \(resolved.backend.rawValue)") } - let context = try loadContext(flag: selection.contextDir) + // The engine is resolved above, so the render budget can come from the + // backend that will actually receive the prompt (design D3). + let context = try loadContext( + flag: selection.contextDir, capability: engine.promptCapability) + + // The D5 warning belongs to *selection*, not to one subcommand — this + // command selects a backend too, so it carries the same warning + // `recommend` does when the choice cannot use the resolved context + // (#164 verify). Merged after loadContext so it fires only when a + // context directory actually resolved. + let rec = context == nil + ? resolved + : resolved.merging( + reasons: [], + warnings: [Self.contextCapabilityWarning(engine.promptCapability)].compactMap { $0 }) let transcript = try await engine.transcribe( audioPath: audio.path, options: TranscribeOptions( model: rec.model, quantization: rec.quantization, - language: lang.language, prompt: context?.rendered.prompt, + language: lang.language, prompt: context?.rendered?.prompt, noSpeechThreshold: noSpeechThreshold, compressionRatioThreshold: compressionRatioThreshold, logProbThreshold: logProbThreshold) @@ -518,22 +638,32 @@ public struct CommandCore: Sendable { ) } - // ±context delta mode (spec benchmark; design D6): context is loaded - // via the same three-layer resolution; the runner measures a second - // with-context pass per candidate while the cache stays baseline-only. - let contextBundle = try loadContext(flag: contextDir) + // ±context delta mode (spec benchmark): context is loaded via the same + // three-layer resolution; the runner measures a second with-context + // pass per candidate while the cache stays baseline-only. The render + // budget comes from the candidates that can actually consume it, and + // the runner skips the pass for those that cannot (#164 verify). + let contextBundle = try loadContext( + flag: contextDir, + capability: benchmarkPromptCapability(for: enumeration.candidates)) let outcome = await runner.run( candidates: enumeration.candidates, notes: enumeration.notes - + (contextBundle.map { - ["context: \($0.loaded.directory) — " - + "\($0.rendered.injected.count) value(s) in the with-context pass"] + + (contextBundle.map { bundle in + [ + bundle.rendered.map { + "context: \(bundle.loaded.directory) — " + + "\($0.injected.count) value(s) in the with-context pass" + } + ?? "context: \(bundle.loaded.directory) — " + + "no candidate supports a prompt; no with-context pass" + ] } ?? []), audio: audio, referenceText: referenceText, metricKind: metricKind, language: resolvedLanguage ?? "auto", - contextPrompt: contextBundle?.rendered.prompt, + contextPrompt: contextBundle?.rendered?.prompt, deterministicDecode: decodeDeterministic ) diff --git a/Sources/BestASRKit/Engines/AppleSpeechEngine.swift b/Sources/BestASRKit/Engines/AppleSpeechEngine.swift index 42fc3d9..44a45d5 100644 --- a/Sources/BestASRKit/Engines/AppleSpeechEngine.swift +++ b/Sources/BestASRKit/Engines/AppleSpeechEngine.swift @@ -52,6 +52,11 @@ import Speech public struct AppleSpeechEngine: Engine { public let id: BackendID = .appleSpeech + /// `SpeechAnalyzer` exposes no conditioning-text parameter. (It does have a + /// separate contextual-strings facility; that is a different mechanism and + /// is not wired here — declaring support would misdescribe what happens.) + public let promptCapability: PromptCapability = .unsupported + public init() {} // MARK: - Diagnostics diff --git a/Sources/BestASRKit/Engines/ChineseFamilyEngine.swift b/Sources/BestASRKit/Engines/ChineseFamilyEngine.swift index b0b5023..fd4f6c0 100644 --- a/Sources/BestASRKit/Engines/ChineseFamilyEngine.swift +++ b/Sources/BestASRKit/Engines/ChineseFamilyEngine.swift @@ -51,6 +51,10 @@ struct FluidAudioSenseVoicePipeline: TextTranscribing { public struct ChineseFamilyEngine: Engine { public let id: BackendID + /// Paraformer / SenseVoice take no conditioning text. The family shares one + /// engine type, so the declaration is shared too. + public let promptCapability: PromptCapability = .unsupported + let probeDuration: @Sendable (String) throws -> TimeInterval let pipelineFactory: @Sendable (String) async throws -> any TextTranscribing let pipelines = CreateOnceStore() diff --git a/Sources/BestASRKit/Engines/Engine.swift b/Sources/BestASRKit/Engines/Engine.swift index ca145bc..94fcf4e 100644 --- a/Sources/BestASRKit/Engines/Engine.swift +++ b/Sources/BestASRKit/Engines/Engine.swift @@ -68,6 +68,21 @@ public protocol Engine: Sendable { /// Backend-specific transcription returning raw segments. func transcribeRaw(audioPath: String, options: TranscribeOptions) async throws -> RawTranscription + + /// Whether this backend consumes a decoder conditioning prompt, and its + /// token ceiling. + /// + /// **Deliberately has no default implementation.** A default would let a new + /// engine inherit "no prompt support" without its author ever considering + /// the question — which is a quieter version of the exact failure this + /// declaration exists to remove: the system asserting something about a + /// backend that nobody checked. Requiring it is source-breaking for any + /// out-of-tree conformer, and that is the accepted cost: a bounded one-time + /// change, in exchange for every future engine being made to answer. + /// + /// Declaring this does not make it true. `EngineCapabilityTests` asserts + /// each backend's declaration against what it actually forwards. + var promptCapability: PromptCapability { get } } extension Engine { diff --git a/Sources/BestASRKit/Engines/ExternalProcessEngine.swift b/Sources/BestASRKit/Engines/ExternalProcessEngine.swift index db7051b..f71ee13 100644 --- a/Sources/BestASRKit/Engines/ExternalProcessEngine.swift +++ b/Sources/BestASRKit/Engines/ExternalProcessEngine.swift @@ -20,6 +20,11 @@ public struct ExternalProcessEngine: Engine { static let supportedProtocols: Set = [1] public let id: BackendID + + /// Protocol v1 has no prompt field, so there is nowhere to put conditioning + /// text even if an adapter wanted it. Adding one would be a protocol + /// version bump, not a declaration change. + public let promptCapability: PromptCapability = .unsupported let command: [String] /// Test seam — production timeout is `max(120s, 4x audio duration)` (D3); /// duration is unknown before probing, so the floor applies to short files. diff --git a/Sources/BestASRKit/Engines/ParakeetEngine.swift b/Sources/BestASRKit/Engines/ParakeetEngine.swift index bce5ed1..60a6c33 100644 --- a/Sources/BestASRKit/Engines/ParakeetEngine.swift +++ b/Sources/BestASRKit/Engines/ParakeetEngine.swift @@ -95,6 +95,13 @@ struct FluidAudioParakeetPipeline: ParakeetTranscribing { public struct ParakeetEngine: Engine { public let id: BackendID = .fluidParakeet + /// Parakeet is a transducer/CTC model — there is no conditioning-text input + /// to give it. Measured directly: with a 49-term context injected, it hit + /// none of the target terms, while the same audio through a Whisper backend + /// picked them up. Declaring this is what stops the CLI reporting an + /// injection that cannot happen. + public let promptCapability: PromptCapability = .unsupported + /// Inter-token silence that starts a new raw segment. static let segmentGapSeconds: TimeInterval = 0.8 diff --git a/Sources/BestASRKit/Engines/PromptCapability.swift b/Sources/BestASRKit/Engines/PromptCapability.swift new file mode 100644 index 0000000..280761e --- /dev/null +++ b/Sources/BestASRKit/Engines/PromptCapability.swift @@ -0,0 +1,60 @@ +import Foundation + +/// Whether a backend consumes a decoder conditioning prompt, and how much of +/// one it can take. +/// +/// "Prompt" here means the ASR decoder's conditioning text — Whisper's +/// `initial_prompt` / `promptTokens` — not a prompt in the LLM sense. The +/// distinction matters because the two live at different layers and only one of +/// them exists in this codebase. +/// +/// ## Why this type exists +/// +/// The token budget used to be a single global constant applied to every +/// backend. Only two of the engines in this package actually read the prompt; +/// for the rest the whole render-and-truncate pipeline ran and the result was +/// discarded — while the CLI still reported `injected (N)`. The system claimed +/// to have done something it had not. Measured on a real run: 102 context items +/// produced "injected (49) / truncated (53)" against a backend that consumed +/// none of them, and the two terms the user cared about were both in the +/// "injected" list and both mis-transcribed. +/// +/// ## Two cases, deliberately +/// +/// An earlier proposal had a third case — supported, but with an unknown limit, +/// leaving truncation to the backend. It is not here because nothing in the tree +/// needs it: both Whisper-family backends cap at the same measured 224. Adding a +/// case with no instance would oblige every conformer, and every future engine +/// author, to handle a branch whose semantics could not be validated against +/// anything real. If a backend that genuinely cannot state a limit shows up, +/// the case can be added then — with a concrete instance to define it against. +public enum PromptCapability: Sendable, Equatable { + /// The backend ignores conditioning text. Nothing should be rendered for it. + case unsupported + + /// The backend accepts conditioning text up to `maxTokens` tokens. + /// + /// Tokens, not characters and not items — the renderer measures in tokens + /// because that is what the decoder window is denominated in. + case supported(maxTokens: Int) + + /// The budget actually usable, or `nil` when there is none. + /// + /// A declared maximum of zero — or, defensively, a negative one — collapses + /// to `nil` rather than being passed downstream. The spec requires zero to + /// behave exactly like `unsupported`, and putting that rule in the type + /// means no call site can forget it or implement it slightly differently. + /// Rendering into a zero budget would otherwise produce an empty prompt and + /// hand the backend a meaningless empty string. + public var effectiveBudget: Int? { + switch self { + case .unsupported: + return nil + case .supported(let maxTokens): + return maxTokens > 0 ? maxTokens : nil + } + } + + /// Whether context biasing can do anything at all for this backend. + public var supportsPrompt: Bool { effectiveBudget != nil } +} diff --git a/Sources/BestASRKit/Engines/WhisperCppEngine.swift b/Sources/BestASRKit/Engines/WhisperCppEngine.swift index af56194..6b97993 100644 --- a/Sources/BestASRKit/Engines/WhisperCppEngine.swift +++ b/Sources/BestASRKit/Engines/WhisperCppEngine.swift @@ -9,6 +9,11 @@ import Foundation public struct WhisperCppEngine: Engine { public let id: BackendID = .whisperCpp + /// Same ceiling as the WhisperKit path, from the horse's mouth: + /// `whisper-cli --help` documents `--prompt PROMPT (max n_text_ctx/2 + /// tokens)`, and Whisper's `n_text_ctx` is 448. + public let promptCapability: PromptCapability = .supported(maxTokens: 224) + /// Where GGML model files live, e.g. ggml-small-q5_0.bin. public let modelDirectory: URL /// Override for tests; nil means "search PATH". diff --git a/Sources/BestASRKit/Engines/WhisperKitEngine.swift b/Sources/BestASRKit/Engines/WhisperKitEngine.swift index befa4ac..05c4fe7 100644 --- a/Sources/BestASRKit/Engines/WhisperKitEngine.swift +++ b/Sources/BestASRKit/Engines/WhisperKitEngine.swift @@ -31,6 +31,10 @@ extension WhisperKit: TranscribingPipeline { public struct WhisperKitEngine: Engine { public let id: BackendID = .whisperKit + /// Whisper conditions on `initial_prompt`; the decoder window caps it at + /// `n_text_ctx / 2` = 224 tokens, which `clampedPromptTokens` also enforces. + public let promptCapability: PromptCapability = .supported(maxTokens: 224) + public init() { self.init(pipelineFactory: { model in try await WhisperKit(WhisperKitConfig(model: model, download: true)) @@ -60,10 +64,20 @@ public struct WhisperKitEngine: Engine { } } - /// Whisper's practical prompt window is ~224 tokens; the renderer budgets - /// ~200 with a heuristic, this clamp is the tokenizer-measured net. + /// Keeps the FRONT of the prompt, because that is where the value is. + /// + /// This clamp is the tokenizer-measured net under Whisper's ~224-token + /// window; the renderer's budget now comes from `promptCapability` rather + /// than a global constant, so the two numbers finally agree. + /// + /// `PromptRenderer` orders names, then terms, then phrases — highest-value + /// first. This used to take the suffix, so overflow discarded exactly the + /// names the context directory exists to inject. No observable difference + /// while the budget (200) stayed below this limit (224); raising the budget + /// per-engine is what would have made the two mechanisms disagree, so the + /// direction is corrected first and on its own (design D4). static func clampedPromptTokens(_ tokens: [Int], limit: Int = 224) -> [Int] { - tokens.count <= limit ? tokens : Array(tokens.suffix(limit)) + tokens.count <= limit ? tokens : Array(tokens.prefix(limit)) } /// Decode options for one run. skipSpecialTokens MUST stay true: without it diff --git a/Tests/BestASRKitTests/BackendEngineTests.swift b/Tests/BestASRKitTests/BackendEngineTests.swift index 9fa010a..749f080 100644 --- a/Tests/BestASRKitTests/BackendEngineTests.swift +++ b/Tests/BestASRKitTests/BackendEngineTests.swift @@ -89,10 +89,33 @@ struct PromptForwardingTests { let big = Array(0..<500) let clamped = WhisperKitEngine.clampedPromptTokens(big) #expect(clamped.count == 224) - #expect(clamped.last == 499) // suffix keeps the most recent tokens #expect(WhisperKitEngine.clampedPromptTokens([1, 2, 3]) == [1, 2, 3]) } + /// Design D4 — the clamp must keep the same end of the list the renderer + /// considers most important. + /// + /// `PromptRenderer` emits names, then terms, then phrases, so the front of + /// the list is the highest-value content. The clamp took the *suffix*, + /// which discards exactly those names. There is no observable difference + /// while the budget (200) sits below the clamp (224) — which is precisely + /// why this is fixed first and separately: once the budget is raised to 224 + /// the two mechanisms disagree, and a combined change would make the cause + /// unattributable. + /// + /// Feeds more than 224 tokens directly rather than going through the + /// budget, so the test cannot silently pass just because the budget is + /// currently lower than the clamp. + @Test func `The clamp keeps the highest-priority tokens, not the most recent`() { + let tokens = Array(0..<500) + let clamped = WhisperKitEngine.clampedPromptTokens(tokens) + #expect(clamped.count == 224) + #expect( + clamped.first == 0, + "the clamp dropped the front of the prompt — that is where the names are") + #expect(clamped.last == 223, "expected the first 224 tokens, got a different window") + } + @Test func `Options prompt flows through the engine seam`() async throws { // MockEngine's raw closure sees the same options the caller passed — // the transcribe template method forwards prompt untouched. diff --git a/Tests/BestASRKitTests/BenchmarkTests.swift b/Tests/BestASRKitTests/BenchmarkTests.swift index e7aaff6..2137225 100644 --- a/Tests/BestASRKitTests/BenchmarkTests.swift +++ b/Tests/BestASRKitTests/BenchmarkTests.swift @@ -222,14 +222,61 @@ struct BenchmarkCacheTests { struct ContextDeltaBenchmarkTests { /// Baseline mishears ("hello"), the context prompt fixes it ("hello world") /// against reference "hello world": WER 0.5 → 0.0, delta -0.5. + /// + /// Declares `.supported` because it reads `options.prompt` — a double that + /// consumes the prompt while declaring `.unsupported` is exactly the + /// mismatch `EngineCapabilityTests` exists to forbid in production code. static func biasedEngine() -> MockEngine { - MockEngine(id: .whisperKit, available: true) { _, options in + MockEngine( + id: .whisperKit, available: true, promptCapability: .supported(maxTokens: 224) + ) { _, options in let text = options.prompt == nil ? "hello" : "hello world" return RawTranscription( segments: [.init(start: 0, end: 2, text: text)], language: "en", duration: 2) } } + /// Records the prompt seen on every pass, so a test can assert that a + /// backend declaring no support was never handed one. + final class PromptWitness: @unchecked Sendable { + private let lock = NSLock() + private var seen: [String?] = [] + func record(_ prompt: String?) { + lock.lock() + defer { lock.unlock() } + seen.append(prompt) + } + var all: [String?] { + lock.lock() + defer { lock.unlock() } + return seen + } + } + + /// Baseline succeeds; the with-context pass throws. A real backend can hit + /// this transiently (model eviction, a decode error on the second pass). + static func contextFailingEngine() -> MockEngine { + MockEngine( + id: .whisperKit, available: true, promptCapability: .supported(maxTokens: 224) + ) { _, options in + if options.prompt != nil { + throw NSError( + domain: "MockEngine", code: 2, + userInfo: [NSLocalizedDescriptionKey: "context pass exploded"]) + } + return RawTranscription( + segments: [.init(start: 0, end: 2, text: "hello")], language: "en", duration: 2) + } + } + + static func witnessEngine(_ id: BackendID, _ witness: PromptWitness) -> MockEngine { + MockEngine(id: id, available: true) { _, options in + witness.record(options.prompt) + return RawTranscription( + segments: [.init(start: 0, end: 2, text: "hello")], language: "en", duration: 2) + } + } + private let candidate = BenchmarkCandidate( backend: .whisperKit, model: "tiny", quantization: "default") private let audio = AudioInfo( @@ -298,6 +345,167 @@ struct ContextDeltaBenchmarkTests { #expect(results[0]["delta"] as? Double == -0.5) } + /// #164 verify round 1 — found independently by four review lenses. + /// + /// The ±context pass used to hand the rendered prompt to EVERY candidate, + /// including engines declaring `.unsupported`, which never read it. Two + /// consequences: a wasted full decode per such candidate, and a persisted + /// `contextErrorRate` ≈ baseline that the report prints as a `%(CTX)` + /// delta — which reads as "context does not help this backend" when the + /// truth is "this backend cannot use context at all". That is the same + /// misstatement class #164 exists to remove, relocated to the benchmark + /// exit, and it contradicts this change's own spec sentence: "an engine + /// that declares no support never receives a rendered prompt". + @Test func `A candidate declaring no prompt support never gets the with-context pass`() async { + let witness = PromptWitness() + let runner = BenchmarkRunner( + engines: [Self.witnessEngine(.fluidParakeet, witness)], host: Fixtures.m5Max, + probe: FakeClock(step: 1).probe()) + let outcome = await runner.run( + candidates: [ + BenchmarkCandidate( + backend: .fluidParakeet, model: "parakeet", quantization: "default") + ], + notes: [], audio: audio, referenceText: "hello world", metricKind: .wer, + language: "en", contextPrompt: "鄭澈, world" + ) + #expect(witness.all.allSatisfy { $0 == nil }, "an unsupported backend was handed a prompt") + #expect(witness.all.count == 2, "the wasted third pass must not run") + #expect(outcome.measured.first?.contextErrorRate == nil, "no delta may be invented") + #expect( + outcome.notes.contains { $0.contains("no prompt support") }, + "the skip must be disclosed, not silent") + } + + /// A mixed grid is the reachable case: `ModelGrid.rows` carries parakeet, + /// chinese-family and apple-speech rows, so an unfiltered `benchmark + /// --context` run measures supporting and non-supporting backends together. + @Test func `A mixed grid measures the delta only where the prompt can land`() async { + let witness = PromptWitness() + let runner = BenchmarkRunner( + engines: [Self.biasedEngine(), Self.witnessEngine(.fluidParakeet, witness)], + host: Fixtures.m5Max, probe: FakeClock(step: 1).probe()) + let outcome = await runner.run( + candidates: [ + candidate, + BenchmarkCandidate( + backend: .fluidParakeet, model: "parakeet", quantization: "default"), + ], + notes: [], audio: audio, referenceText: "hello world", metricKind: .wer, + language: "en", contextPrompt: "鄭澈, world" + ) + let byBackend = Dictionary( + uniqueKeysWithValues: outcome.measured.map { ($0.record.backend, $0) }) + #expect(byBackend["whisperkit"]?.contextErrorRate == 0.0) + #expect(byBackend["fluid-parakeet"]?.contextErrorRate == nil) + #expect(witness.all.allSatisfy { $0 == nil }) + #expect(outcome.notes.contains { $0.contains("fluid-parakeet") }) + } + + /// #164 verify round 2 (logic lens). The capability gate closed one route to + /// an unexplained blank in the DELTA column; a second route was left open. + /// + /// A backend that *does* declare support, whose baseline pass succeeds but + /// whose with-context pass throws, gets `contextErrorRate == nil` and does + /// not enter `contextSkipped` — so the report shows the same blank cell with + /// no reason given. The `try?` predates this change, but round 2 added a + /// completeness claim above it that this path did not honour. + /// + /// Reported separately from the capability skip on purpose: telling a user + /// their whisper.cpp run "declares no prompt support" would be false. + @Test func `A failed with-context pass is reported, not left as a blank cell`() async { + let runner = BenchmarkRunner( + engines: [Self.contextFailingEngine()], host: Fixtures.m5Max, + probe: FakeClock(step: 1).probe()) + let outcome = await runner.run( + candidates: [candidate], notes: [], audio: audio, + referenceText: "hello world", metricKind: .wer, language: "en", + contextPrompt: "鄭澈, world" + ) + // The candidate is still measured — a context-pass failure is not a + // candidate failure (spec benchmark: warn-continue). + #expect(outcome.measured.count == 1) + #expect(outcome.failures.isEmpty) + #expect(outcome.measured.first?.contextErrorRate == nil) + #expect( + outcome.notes.contains { $0.contains("with-context pass failed") }, + "a blank DELTA cell must carry its reason; got: \(outcome.notes)") + #expect( + !outcome.notes.contains { $0.contains("no prompt support") }, + "a failed pass must not be reported as a capability skip") + } + + /// The benchmark has no one "selected" engine, which is why `loadContext` + /// takes an optional capability. But the prompt now only reaches candidates + /// that declare support, so when those agree there IS a single applicable + /// budget — and the spec's "the budget comes from the selected engine + /// rather than a single global constant" should hold on this call site too. + @Test func `The benchmark renders against the budget its candidates agree on`() { + let core = CommandCore( + engines: [ + MockEngine.fixed(.whisperKit, promptCapability: .supported(maxTokens: 224)), + MockEngine.fixed(.whisperCpp, promptCapability: .supported(maxTokens: 224)), + MockEngine.fixed(.fluidParakeet), + ], + detect: { Fixtures.m5Max }) + func grid(_ backends: [BackendID]) -> [BenchmarkCandidate] { + backends.map { BenchmarkCandidate(backend: $0, model: "m", quantization: "default") } + } + // Whisper-only, and mixed with a backend that cannot consume it: the + // agreeing budget wins — the non-consumer constrains nothing. + #expect( + core.benchmarkPromptCapability(for: grid([.whisperKit, .whisperCpp])) + == .supported(maxTokens: 224)) + #expect( + core.benchmarkPromptCapability(for: grid([.whisperKit, .fluidParakeet])) + == .supported(maxTokens: 224)) + // Nothing in the grid can take a prompt → do not render one at all. + #expect(core.benchmarkPromptCapability(for: grid([.fluidParakeet])) == .unsupported) + } + + @Test func `Disagreeing budgets fall back to the global default`() { + let core = CommandCore( + engines: [ + MockEngine.fixed(.whisperKit, promptCapability: .supported(maxTokens: 224)), + MockEngine.fixed(.whisperCpp, promptCapability: .supported(maxTokens: 64)), + ], + detect: { Fixtures.m5Max }) + let grid = [BackendID.whisperKit, .whisperCpp].map { + BenchmarkCandidate(backend: $0, model: "m", quantization: "default") + } + // One prompt cannot honour two budgets; nil keeps the previous global + // default and leaves the smaller backend's own clamp as the backstop. + #expect(core.benchmarkPromptCapability(for: grid) == nil) + } + + @Test func `A grid that cannot take a prompt says so instead of counting values`() async throws { + let dir = try makeTempDir() + defer { try? FileManager.default.removeItem(at: dir) } + let ctxDir = dir.appendingPathComponent("ctx") + try FileManager.default.createDirectory(at: ctxDir, withIntermediateDirectories: true) + try #"{"version":1,"terms":["world"]}"#.write( + to: ctxDir.appendingPathComponent("context.json"), atomically: true, encoding: .utf8) + let audioPath = try makeWavFile(in: dir, seconds: 2.0) + let srt = dir.appendingPathComponent("truth.srt").path + try "1\n00:00:00,000 --> 00:00:02,000\nhello world\n".write( + toFile: srt, atomically: true, encoding: .utf8) + let core = CommandCore( + engines: [MockEngine.fixed(.fluidParakeet)], + detect: { Fixtures.m5Max }, + store: BenchmarkStore(directory: dir.appendingPathComponent("store")), + probe: FakeClock(step: 1).probe() + ) + let report = try await core.benchmark( + audioPath: audioPath, referencePath: srt, language: "en", + backendFilter: ["fluid-parakeet"], modelFilter: nil, profileName: "medium", + asJSON: false, contextDir: ctxDir.path + ) + // "0 value(s) in the with-context pass" would be the old misstatement: + // technically true, and read as "your context is empty". + #expect(!report.contains("value(s) in the with-context pass")) + #expect(report.contains("no candidate supports a prompt")) + } + @Test func `No context directory means single-pass runs and an unchanged report shape`() async { let runner = BenchmarkRunner( engines: [Self.biasedEngine()], host: Fixtures.m5Max, @@ -342,6 +550,11 @@ struct BenchmarkNormalizationTests { let collector = PathCollector() let engine = MockEngine( id: .whisperKit, available: true, + // Stands in for WhisperKit, so it must declare WhisperKit's + // capability — the ±context pass is now gated on the declaration + // (#164 verify), and an undeclared double would skip that pass and + // make this a two-pass assertion for the wrong reason. + promptCapability: .supported(maxTokens: 224), raw: { path, _ in collector.record(path) return RawTranscription( diff --git a/Tests/BestASRKitTests/CLITests.swift b/Tests/BestASRKitTests/CLITests.swift index df72523..19aad0c 100644 --- a/Tests/BestASRKitTests/CLITests.swift +++ b/Tests/BestASRKitTests/CLITests.swift @@ -327,8 +327,15 @@ final class OptionsBox: @unchecked Sendable { } } +/// Stands in for the WhisperKit backend, so it declares what that backend +/// declares. A mock claiming `.whisperKit` while reporting no prompt support +/// would be the exact declaration/behavior mismatch #164 removes — and the +/// context here would be skipped, which is what caught this. private func capturingEngine(_ box: OptionsBox) -> MockEngine { - MockEngine(id: .whisperKit, available: true) { _, options in + MockEngine( + id: .whisperKit, available: true, + promptCapability: .supported(maxTokens: 224) + ) { _, options in box.append(options) return RawTranscription( segments: [.init(start: 0.0, end: 2.5, text: "hello world")], @@ -400,14 +407,19 @@ struct ContextCommandTests { defer { try? FileManager.default.removeItem(at: dir) } let audio = try makeWavFile(in: dir) let ctxDir = try makeContextFixture(in: dir) + // Stands in for whisperKit, so it declares whisperKit's capability — + // otherwise the context is correctly skipped and there is no injected + // count to assert (#164). let core = CommandCore( - engines: [MockEngine.fixed(.whisperKit)], + engines: [ + MockEngine.fixed(.whisperKit, promptCapability: .supported(maxTokens: 224)) + ], detect: { Fixtures.m5Max }, store: BenchmarkStore(directory: dir.appendingPathComponent("store")), probe: FakeClockProbe.probe() ) let selection = SelectionRequest( - profileName: "medium", backendOverride: nil, modelOverride: nil, + profileName: "medium", backendOverride: "whisperkit", modelOverride: nil, requestedLanguage: "auto", contextDir: ctxDir) let output = try await core.recommendJSON(audioPath: audio, selection: selection) let json = try #require( diff --git a/Tests/BestASRKitTests/ContextBudgetTests.swift b/Tests/BestASRKitTests/ContextBudgetTests.swift new file mode 100644 index 0000000..ebe1c0c --- /dev/null +++ b/Tests/BestASRKitTests/ContextBudgetTests.swift @@ -0,0 +1,247 @@ +import Foundation +import Testing + +@testable import BestASRKit + +/// Where the render budget comes from, and when rendering happens at all +/// (design D3, spec `Render context into a natural-language prompt with +/// priority and budget`). +struct ContextBudgetTests { + + /// A context directory holding enough terms that a 200-token budget + /// truncates and a 224-token one truncates less — so "which budget was + /// used" is observable rather than asserted against a number. + private func makeContextDir() throws -> URL { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("ctx-budget-\(UUID().uuidString)", isDirectory: true) + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + let terms = (0..<120).map { "\"term-number-\($0)\"" }.joined(separator: ", ") + let json = "{ \"version\": 1, \"terms\": [\(terms)] }" + try json.write( + to: dir.appendingPathComponent("context.json"), atomically: true, encoding: .utf8) + return dir + } + + /// Same injection shape the other CommandCore tests use: mock engines, + /// fixed host, temp store, deterministic clock. + private func core(engines: [any Engine]) -> CommandCore { + let scratch = FileManager.default.temporaryDirectory + .appendingPathComponent("core-\(UUID().uuidString)", isDirectory: true) + return CommandCore( + engines: engines, + detect: { Fixtures.m5Max }, + store: BenchmarkStore(directory: scratch.appendingPathComponent("store")), + probe: FakeClock(step: 1.0).probe()) + } + + /// The two budgets must actually differ on this fixture, or every other + /// assertion here would be vacuous. + @Test func `The fixture distinguishes a 200-token budget from a 224-token one`() throws { + let dir = try makeContextDir() + defer { try? FileManager.default.removeItem(at: dir) } + let loaded = try #require(try ContextLoader.load(flag: dir.path)) + let at200 = PromptRenderer.render(loaded, tokenBudget: 200) + let at224 = PromptRenderer.render(loaded, tokenBudget: 224) + #expect( + at200.injected.count < at224.injected.count, + "fixture is too small to tell the budgets apart — the budget tests would be vacuous") + } + + /// 4.1 — the budget comes from the selected engine, not the global constant. + @Test func `Rendering uses the budget the engine declared`() throws { + let dir = try makeContextDir() + defer { try? FileManager.default.removeItem(at: dir) } + let subject = core(engines: []) + + let bundle = try #require( + try subject.loadContext(flag: dir.path, capability: .supported(maxTokens: 224))) + let rendered = try #require(bundle.rendered) + + let loaded = try #require(try ContextLoader.load(flag: dir.path)) + #expect(rendered.injected.count == PromptRenderer.render(loaded, tokenBudget: 224).injected.count) + #expect(rendered.injected.count != PromptRenderer.render(loaded, tokenBudget: 200).injected.count) + } + + /// 4.2 — D3: an engine that ignores conditioning text gets no render at all. + /// Not "render and discard": the truncation figures would be real work + /// producing a number that describes nothing. + @Test func `An unsupported engine skips rendering entirely`() throws { + let dir = try makeContextDir() + defer { try? FileManager.default.removeItem(at: dir) } + let subject = core(engines: []) + + let bundle = try #require(try subject.loadContext(flag: dir.path, capability: .unsupported)) + #expect(bundle.rendered == nil, "an unsupported engine must not get a rendered prompt") + #expect(bundle.loaded.directory == dir.path, "the directory is still disclosed") + } + + /// 4.3 — a declared maximum of zero takes the same path as no support, so a + /// backend can never be handed an empty-string prompt. + @Test func `A zero declared budget takes the unsupported path`() throws { + let dir = try makeContextDir() + defer { try? FileManager.default.removeItem(at: dir) } + let subject = core(engines: []) + + let bundle = try #require( + try subject.loadContext(flag: dir.path, capability: .supported(maxTokens: 0))) + #expect(bundle.rendered == nil, "a zero budget must not produce a prompt") + } + + // MARK: - 5.1 honest disclosure (spec `Explain discloses context usage`) + + /// The original complaint in one assertion: a backend that consumes no + /// prompt must not be described as having had values injected. `injected + /// (49)` reads as success, and the user acts on it by trimming their term + /// list — which changes nothing, because nothing was ever injected. + @Test func `Explain does not report an injected count for an unsupported backend`() throws { + let dir = try makeContextDir() + defer { try? FileManager.default.removeItem(at: dir) } + let subject = core(engines: []) + let bundle = try #require(try subject.loadContext(flag: dir.path, capability: .unsupported)) + + let explanation = CommandCore.contextExplanation(bundle).joined(separator: "\n") + #expect(!explanation.contains("injected"), "got: \(explanation)") + #expect(!explanation.contains("truncated"), "got: \(explanation)") + #expect(explanation.contains("does not support context biasing"), "got: \(explanation)") + #expect(explanation.contains(dir.path), "the directory is still disclosed") + + let reason = CommandCore.contextReasonLine(bundle) + #expect(!reason.contains("value(s) injected"), "got: \(reason)") + #expect(reason.contains("does not support context biasing"), "got: \(reason)") + } + + /// The supported path must be untouched — this change is about removing a + /// false claim, not about changing what a working run reports. + @Test func `Explain still reports injected values for a supporting backend`() throws { + let dir = try makeContextDir() + defer { try? FileManager.default.removeItem(at: dir) } + let subject = core(engines: []) + let bundle = try #require( + try subject.loadContext(flag: dir.path, capability: .supported(maxTokens: 224))) + + let explanation = CommandCore.contextExplanation(bundle).joined(separator: "\n") + #expect(explanation.contains("injected ("), "got: \(explanation)") + #expect(!explanation.contains("does not support context biasing")) + #expect(CommandCore.contextReasonLine(bundle).contains("value(s) injected")) + } + + // MARK: - 5.2 selection surfaces the trade-off (spec `Selection accounts + // for prompt support when context is present`, design D5) + + /// `backendOverride` is used so the scenario under test is the one the spec + /// describes. Without it the router picks by cold-start prior — it chose + /// whisper.cpp, a backend with no registered engine here, so the assertion + /// would have been about routing rather than about capability. + private static func selection( + contextDir: String? = nil, backend: String? = nil + ) -> SelectionRequest { + SelectionRequest( + profileName: "medium", backendOverride: backend, modelOverride: nil, + requestedLanguage: "en", contextDir: contextDir) + } + + private func audioFixture() throws -> (URL, String) { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("ctx-route-\(UUID().uuidString)", isDirectory: true) + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + return (dir, try makeWavFile(in: dir)) + } + + /// Warned, not excluded. Whether a lower error rate is worth losing context + /// biasing has not been measured here, so the system surfaces the trade-off + /// instead of deciding it silently (D5). + @Test func `Selecting a backend without prompt support warns but still uses it`() async throws { + let (dir, audio) = try audioFixture() + defer { try? FileManager.default.removeItem(at: dir) } + let ctx = try makeContextDir() + defer { try? FileManager.default.removeItem(at: ctx) } + + let subject = core(engines: [MockEngine.fixed(.fluidParakeet)]) + let json = try await subject.recommendJSON( + audioPath: audio, + selection: Self.selection(contextDir: ctx.path, backend: "fluid-parakeet")) + + #expect( + json.contains("does not support context biasing"), + "selection must surface that the context cannot take effect; got: \(json)") + #expect(json.contains("fluid-parakeet"), "the backend must still be the one selected") + } + + @Test func `A supporting backend produces no prompt-capability warning`() async throws { + let (dir, audio) = try audioFixture() + defer { try? FileManager.default.removeItem(at: dir) } + let ctx = try makeContextDir() + defer { try? FileManager.default.removeItem(at: ctx) } + + let subject = core(engines: [ + MockEngine( + id: .whisperKit, available: true, + promptCapability: .supported(maxTokens: 224) + ) { _, _ in + RawTranscription(segments: [], language: "en", duration: 1) + } + ]) + let json = try await subject.recommendJSON( + audioPath: audio, + selection: Self.selection(contextDir: ctx.path, backend: "whisperkit")) + #expect(!json.contains("does not support context biasing"), "got: \(json)") + } + + /// #164 verify round 1: the spec sentence is about *selection* — "backend + /// selection SHALL … warn when the selected backend declares no prompt + /// support" — and is not qualified by subcommand. `transcribe` selects a + /// backend too, so it must carry the warning `recommend` carries; it used + /// to fill `warnings` with language-detection notes only. + /// + /// Deliberately not fixed here: `transcribe` prints nothing without + /// `--explain` — pre-existing for *every* warning it produces, so it is a + /// separate issue rather than something #164 introduced. This asserts the + /// warning reaches the surface `transcribe` actually has. + @Test func `Transcribe carries the same prompt-capability warning as recommend`() async throws { + let (dir, audio) = try audioFixture() + defer { try? FileManager.default.removeItem(at: dir) } + let ctx = try makeContextDir() + defer { try? FileManager.default.removeItem(at: ctx) } + + let subject = core(engines: [MockEngine.fixed(.fluidParakeet)]) + let outcome = try await subject.transcribe( + audioPath: audio, + selection: Self.selection(contextDir: ctx.path, backend: "fluid-parakeet"), + formatName: "txt", + outputPath: dir.appendingPathComponent("out.txt").path) + + #expect( + outcome.explanation.contains("! the selected backend does not support context biasing"), + "transcribe must warn like recommend does; got: \(outcome.explanation)") + } + + @Test func `A supporting backend transcribes without a capability warning`() async throws { + let (dir, audio) = try audioFixture() + defer { try? FileManager.default.removeItem(at: dir) } + let ctx = try makeContextDir() + defer { try? FileManager.default.removeItem(at: ctx) } + + let subject = core(engines: [ + MockEngine.fixed(.whisperKit, promptCapability: .supported(maxTokens: 224)) + ]) + let outcome = try await subject.transcribe( + audioPath: audio, + selection: Self.selection(contextDir: ctx.path, backend: "whisperkit"), + formatName: "txt", + outputPath: dir.appendingPathComponent("out.txt").path) + + #expect(!outcome.explanation.contains("! the selected backend does not support")) + } + + /// With no context resolved, capability must not influence selection at all + /// — no warning, no change of criteria. + @Test func `No context means no prompt-capability warning`() async throws { + let (dir, audio) = try audioFixture() + defer { try? FileManager.default.removeItem(at: dir) } + + let subject = core(engines: [MockEngine.fixed(.fluidParakeet)]) + let json = try await subject.recommendJSON( + audioPath: audio, selection: Self.selection(backend: "fluid-parakeet")) + #expect(!json.contains("does not support context biasing"), "got: \(json)") + } +} diff --git a/Tests/BestASRKitTests/EngineCapabilityTests.swift b/Tests/BestASRKitTests/EngineCapabilityTests.swift new file mode 100644 index 0000000..9f232b1 --- /dev/null +++ b/Tests/BestASRKitTests/EngineCapabilityTests.swift @@ -0,0 +1,90 @@ +import Foundation +import Testing + +@testable import BestASRKit + +/// Minimal stand-in so the Chinese-family engine can be constructed; this test +/// only reads a declaration and never transcribes. +private struct InertTranscriber: TextTranscribing { + func transcribe(audioPath: String, language: String?) async throws -> String { "" } +} + +/// Every backend's prompt declaration, checked against what it actually does +/// (spec `Common engine interface`). +/// +/// Declaring a capability does not make it true, and a declaration nobody +/// verifies is the same class of claim as the `injected (N)` line this whole +/// change exists to remove. So this asserts both halves: the value each engine +/// declares, and — structurally — that only the engines declaring support are +/// the ones whose source actually forwards `options.prompt` to a backend. +struct EngineCapabilityTests { + static let engineSources = URL(fileURLWithPath: #filePath) + .deletingLastPathComponent() + .deletingLastPathComponent() + .deletingLastPathComponent() + .appendingPathComponent("Sources/BestASRKit/Engines") + + /// The measured ceiling shared by both Whisper backends. + /// + /// `whisper-cli --help` states `--prompt PROMPT (max n_text_ctx/2 tokens)`, + /// and Whisper's `n_text_ctx` is 448 — so 224. WhisperKit's own clamp uses + /// the same number, independently arrived at, which is why a single third + /// "limit unknown" case was not needed (design D1). + static let whisperPromptLimit = 224 + + @Test func `The Whisper-family backends declare support at the measured 224 ceiling`() { + #expect( + WhisperKitEngine().promptCapability == .supported(maxTokens: Self.whisperPromptLimit)) + #expect( + WhisperCppEngine().promptCapability == .supported(maxTokens: Self.whisperPromptLimit)) + } + + @Test func `Backends that ignore conditioning text declare no support`() { + #expect(ParakeetEngine().promptCapability == .unsupported) + #expect(AppleSpeechEngine().promptCapability == .unsupported) + #expect( + ChineseFamilyEngine( + id: .fluidParaformer, + probeDuration: { _ in 1.0 }, + pipelineFactory: { _ in InertTranscriber() } + ).promptCapability == .unsupported) + #expect( + ExternalProcessEngine(id: .mlxAudio, command: ["/bin/echo"]).promptCapability + == .unsupported) + } + + /// The declaration must match the code, not the author's intention. + /// + /// An engine that forwards `options.prompt` to its backend is claiming to + /// use it; one that never mentions it cannot be. This compares the two sets + /// and fails on either kind of drift — a backend that quietly starts using + /// the prompt without declaring support, or one that declares support it + /// does not act on. + @Test func `Each declaration matches whether the source actually forwards the prompt`() throws { + let declaredSupported = ["WhisperKitEngine", "WhisperCppEngine"] + var actuallyForwards: [String] = [] + for name in [ + "WhisperKitEngine", "WhisperCppEngine", "ParakeetEngine", + "AppleSpeechEngine", "ChineseFamilyEngine", "ExternalProcessEngine", + ] { + let source = try String( + contentsOf: Self.engineSources.appendingPathComponent("\(name).swift"), + encoding: .utf8) + // Strip comments so prose about prompts is not mistaken for use. + let code = source + .replacingOccurrences(of: #"/\*[\s\S]*?\*/"#, with: " ", options: .regularExpression) + .replacingOccurrences(of: #"(?m)//.*$"#, with: " ", options: .regularExpression) + if code.contains("options.prompt") { actuallyForwards.append(name) } + } + #expect( + actuallyForwards.sorted() == declaredSupported.sorted(), + """ + Prompt declaration and behavior disagree. + Declares support: \(declaredSupported.sorted()) + Forwards options.prompt: \(actuallyForwards.sorted()) + An engine that started using the prompt must declare it, and one that + declares support must act on it — that mismatch is the bug this + change removes. + """) + } +} diff --git a/Tests/BestASRKitTests/PipelineWiringTests.swift b/Tests/BestASRKitTests/PipelineWiringTests.swift index ce98066..42b1f3b 100644 --- a/Tests/BestASRKitTests/PipelineWiringTests.swift +++ b/Tests/BestASRKitTests/PipelineWiringTests.swift @@ -139,14 +139,34 @@ struct PipelineWiringTests { // stayed green before this line (#12 verify F3) } - /// #12: the 224-token clamp must act on the ENCODED prompt at the seam — - /// keeping the suffix (nearest context wins under Whisper's left-context - /// window), not the prefix. - @Test func `Overlong prompt is clamped to the trailing 224 tokens at the seam`() async throws { + /// #12: the 224-token clamp must act on the ENCODED prompt at the seam. + /// + /// **The direction was reversed by #164 (design D4), and the two rationales + /// are on different axes — this is a real trade-off, not a bug fix.** + /// + /// The original (#12) kept the SUFFIX, reasoning that "nearest context wins + /// under Whisper's left-context window" — a claim about decoder influence: + /// tokens closest to where transcription begins weigh most. + /// + /// D4 keeps the PREFIX, reasoning that `PromptRenderer` emits names, then + /// terms, then phrases, so the front holds the highest-value content — a + /// claim about what is worth keeping. Dropping the suffix drops phrases; + /// dropping the prefix dropped the proper nouns the context directory exists + /// to inject. + /// + /// Both can be true at once: the front may carry the better content while + /// the tail carries more decoder weight. Which wins has NOT been measured + /// here. D4 chose content priority because the observed failure was proper + /// nouns being mis-transcribed while sitting in the "injected" list. If + /// measurement later shows position dominates, this is the line to revisit — + /// and the honest fix would be to reorder the renderer so the highest-value + /// items land at the tail, not to re-flip the clamp against its own + /// priority ordering. + @Test func `Overlong prompt is clamped to the leading 224 tokens at the seam`() async throws { let spy = SpyPipeline(tokenizer: FakeTokenizer()) let engine = WhisperKitEngine(pipelineFactory: { _ in spy }) - // 150 a's + 150 b's: every 224-window is now DISTINCT, so a - // prefix-keeping clamp cannot masquerade as suffix-keeping + // 150 a's + 150 b's: every 224-window is DISTINCT, so a suffix-keeping + // clamp cannot masquerade as prefix-keeping // (#12 verify F1 — homogeneous data made the direction vacuous). let longPrompt = String(repeating: "a", count: 150) + String(repeating: "b", count: 150) _ = try await engine.transcribe( @@ -158,7 +178,7 @@ struct PipelineWiringTests { let sent = try #require(spy.lastOptions) let full = FakeTokenizer().encode(text: " " + longPrompt) // 301 tokens #expect(sent.promptTokens?.count == 224) - #expect(sent.promptTokens == Array(full.suffix(224))) - #expect(sent.promptTokens != Array(full.prefix(224))) // direction really is decidable now + #expect(sent.promptTokens == Array(full.prefix(224))) + #expect(sent.promptTokens != Array(full.suffix(224))) // direction really is decidable now } } diff --git a/Tests/BestASRKitTests/PromptCapabilityTests.swift b/Tests/BestASRKitTests/PromptCapabilityTests.swift new file mode 100644 index 0000000..64af88c --- /dev/null +++ b/Tests/BestASRKitTests/PromptCapabilityTests.swift @@ -0,0 +1,54 @@ +import Foundation +import Testing + +@testable import BestASRKit + +/// The capability declaration itself (design D1/D2, spec `Common engine +/// interface`). +/// +/// The type carries the zero-budget rule rather than leaving it to callers: +/// "declared supported with a maximum of zero" and "declared unsupported" must +/// be indistinguishable downstream, and an invariant every call site has to +/// remember is an invariant that eventually gets forgotten. +struct PromptCapabilityTests { + + @Test func `An unsupported engine has no usable budget`() { + #expect(PromptCapability.unsupported.effectiveBudget == nil) + #expect(PromptCapability.unsupported.supportsPrompt == false) + } + + @Test func `A supported engine reports the budget it declared`() { + #expect(PromptCapability.supported(maxTokens: 224).effectiveBudget == 224) + #expect(PromptCapability.supported(maxTokens: 224).supportsPrompt) + } + + /// Spec `Common engine interface`: "A backend that declares support with a + /// maximum token count of zero SHALL be treated identically to a backend + /// that declares no support." Negative is folded in for the same reason — + /// there is no sensible reading of it, and silently rendering into a + /// negative budget is worse than declining. + @Test func `A zero or negative declared maximum is treated as no support`() { + for bogus in [0, -1, Int.min] { + let capability = PromptCapability.supported(maxTokens: bogus) + #expect( + capability.effectiveBudget == nil, + "maxTokens \(bogus) must not yield a usable budget") + #expect(capability.supportsPrompt == false, "maxTokens \(bogus) must not claim support") + } + } + + /// D1 rejected a third "supported, limit unknown" case because nothing in + /// the tree needs it. This pins that decision: the type is exhaustively + /// handled by two branches, so adding a third later is a deliberate, + /// visible act rather than a silent widening. + @Test func `The capability has exactly two shapes`() { + let cases: [PromptCapability] = [.unsupported, .supported(maxTokens: 1)] + for capability in cases { + switch capability { + case .unsupported, .supported: + continue // exhaustive without a default — if a case is added, this stops compiling + } + } + #expect(cases.count == 2) + } +} diff --git a/Tests/BestASRKitTests/TestSupport.swift b/Tests/BestASRKitTests/TestSupport.swift index e99dc66..c9c10d5 100644 --- a/Tests/BestASRKitTests/TestSupport.swift +++ b/Tests/BestASRKitTests/TestSupport.swift @@ -8,6 +8,11 @@ import Foundation struct MockEngine: Engine { let id: BackendID let available: Bool + /// Declared like any other engine — the protocol requirement has no default + /// (design D2), and a test double is exactly where a silent default would + /// hide a wrong assumption. `var` with a default keeps every existing + /// call site working while letting a test drive the supported branch. + var promptCapability: PromptCapability = .unsupported let raw: @Sendable (String, TranscribeOptions) throws -> RawTranscription func isAvailable() async -> Bool { available } @@ -20,13 +25,14 @@ struct MockEngine: Engine { static func fixed( _ id: BackendID, available: Bool = true, + promptCapability: PromptCapability = .unsupported, segments: [RawTranscription.RawSegment] = [ .init(start: 0.0, end: 2.5, text: "hello world") ], language: String? = "en", duration: Double? = 2.5 ) -> MockEngine { - MockEngine(id: id, available: available) { _, _ in + MockEngine(id: id, available: available, promptCapability: promptCapability) { _, _ in RawTranscription(segments: segments, language: language, duration: duration) } } diff --git a/openspec/changes/per-engine-prompt-capability/.openspec.yaml b/openspec/changes/per-engine-prompt-capability/.openspec.yaml new file mode 100644 index 0000000..f2d586c --- /dev/null +++ b/openspec/changes/per-engine-prompt-capability/.openspec.yaml @@ -0,0 +1,4 @@ +schema: spec-driven +created: 2026-08-07 +created_by: che cheng +created_with: claude diff --git a/openspec/changes/per-engine-prompt-capability/design.md b/openspec/changes/per-engine-prompt-capability/design.md new file mode 100644 index 0000000..cdc9861 --- /dev/null +++ b/openspec/changes/per-engine-prompt-capability/design.md @@ -0,0 +1,113 @@ +## Context + +context biasing 目前的資料流是:載入 context 目錄 → `PromptRenderer.render` 依 names → terms → phrases 優先序渲染成逗號分隔詞表、受約 200 token 預算截斷 → 交給引擎。 + +問題在最後一步:`Engine` 是 public protocol,目前有三個 requirement(識別碼、可用性偵測、原始轉錄),**沒有任何地方描述該引擎是否消費 prompt**。實際上 6 個 conformer 只有兩個 whisper backend 會讀,其餘 4 個把它丟掉。渲染與截斷卻對全部引擎照跑,CLI 的 explain 也對全部引擎印出注入計數。 + +約束: + +- `Engine` 是 **public** protocol,契約同時對 repo 外與未來的引擎作者生效。增加 requirement 是 source-breaking。 +- 兩個 whisper backend 的上限已實證收斂為 **224**(`whisper-cli --help` 明載 `--prompt PROMPT (max n_text_ctx/2 tokens)`,Whisper `n_text_ctx = 448`;WhisperKit 端的 clamp 亦為 224)。 +- 現行 200 的註解本身即寫明「為 Whisper 而調」——設計沒錯,錯在被無條件套用到不是 Whisper 的東西。 +- `PromptRenderer.render` 全 codebase 僅一個呼叫點,位於命令核心組裝 context bundle 之處,故管線改動面小;成本集中在 protocol 與 conformer。 + +## Goals / Non-Goals + +**Goals:** + +- 讓「此引擎是否支援 prompt、上限多少」成為引擎自己宣告的、型別層可檢查的事實。 +- 預算由所選引擎決定,而非全域常數。 +- 不支援 prompt 時**不執行** render、**不宣稱**注入——消滅「宣稱與實際不符」。 +- 選型在有 context 時把 prompt 支援度納入考量。 + +**Non-Goals:** + +- 不評比哪個引擎在特定音訊類型上較佳,也不調整 benchmark 語料。 +- 不處理長 prompt 誘發的解碼重複問題。 +- 不更動 context.json schema、三層目錄解析、文件擷取。 +- 不引入 prompt 內容的語意最佳化(例如依音訊自動挑詞)。 + +## Decisions + +### D1:能力宣告採二態,而非三態 + +採 `unsupported` 與 `supported(maxTokens:)` 兩態。 + +原始 issue 提議三態,第三態為「支援但上限未知,交由後端自行截斷」。**否決理由:該態目前沒有任何 instance。** 兩個 whisper backend 上限相同且已知。為不存在的情況新增 case,會要求 6 個 conformer 與所有未來作者處理一個無法驗證語意的分支。日後若真出現無法宣告上限的後端,屆時有真實 instance 可據以定義行為,再擴充。 + +代價:未來新增第三態是 source-breaking 的第二次。接受——比現在憑想像定義語意好。 + +### D2:protocol requirement 不給預設實作 + +不提供 `extension Engine { var promptCapability: ... { .unsupported } }`。 + +給預設會讓新引擎**預設靜默不支援 context**,這正是本變更要消滅的失敗模式(系統宣稱與實際不符)的變體:作者沒想過這件事,系統替他選了一個看起來安全的答案。本變更的核心命題是「能力必須被明確宣告」,給預設與初衷相反。 + +代價:6 個既有 conformer 全部要改,且對 repo 外的實作是 source-breaking。這是**明知的**取捨——一次有界的成本,換取往後每個新引擎都被迫回答這個問題。 + +### D3:不支援時跳過整段 render,而非渲染後丟棄 + +引擎宣告 `unsupported` 時,命令核心不呼叫 render。理由有二:一是截斷計算與 `exhausted` 判定對該引擎完全無意義,跑了只是浪費並產生誤導性的統計;二是 explain 的輸出必須據此改變措辭,兩者需在同一個判斷點決定。 + +### D4:截斷方向與優先序必須一致 + +渲染的優先序是 names → terms → phrases(最重要在前),而 WhisperKit 端的 token clamp 取的是**後綴**,砍掉的正是最前面的人名。目前預算 200 < 上限 224 撞不到,故無可觀測差異;但預算一旦依引擎提高到 224,兩個機制的方向相反會產生實際錯誤。本變更把 clamp 改為取前綴,**必須在提高預算之前完成**,否則兩個變因混在一起無法歸因。 + +### D5:routing 在有 context 時納入支援度,但不硬性過濾 + +`recommend` 偵測到 context 目錄存在時,把 prompt 支援度納入選型考量;選中 `unsupported` 引擎時發出警告。 + +不採「硬性排除不支援的引擎」:WER 與 context 支援何者較重要並未量測,硬排可能把明顯較準的引擎擋掉。警告把判斷交還使用者,且不宣稱系統知道它其實不知道的事。 + +## Implementation Contract + +**Behavior(使用者可觀察到的變化)** + +- 以支援 prompt 的後端轉錄且存在 context 時:行為與現況相同,惟預算改由該後端宣告(兩個 whisper backend 為 224,高於現行 200,故可注入的項目數會增加)。 +- 以不支援 prompt 的後端轉錄且存在 context 時:**不再**出現注入計數與截斷清單;改為一則明確訊息,指出該後端不支援 context biasing。 +- 存在 context 目錄而選型選中不支援的後端時:發出警告,指出 context 將不生效。 + +**Interface / data shape** + +- `Engine` protocol 新增一個唯讀屬性,回傳 prompt 能力,型別為二態列舉:不支援;支援並帶一個非負的最大 token 數。 +- 該列舉為 public,與 `Engine` 同一模組公開。 +- `PromptRenderer` 的渲染入口接受 token 預算參數(既有),呼叫端改為傳入所選引擎宣告的上限;`defaultTokenBudget` 不再作為隱含全域預設被單一呼叫點依賴。 +- explain 的 context 區段在不支援時輸出「不支援」訊息,而非注入/截斷計數。 + +**Failure modes** + +- 引擎宣告 `supported(maxTokens:)` 但 `maxTokens` 為 0:視同不支援,走同一條「不執行 render」路徑,不得產生空 prompt 傳給後端。 +- context 目錄不存在:維持現況零影響,不因本變更產生任何新輸出。 +- 外部程序引擎(協定型後端)宣告 `unsupported`:其協定目前無 prompt 欄位,不得因本變更在協定上新增欄位。 + +**Acceptance criteria** + +- 對每個 conformer 存在測試斷言其宣告值,且宣告與該引擎是否真的把 prompt 傳給後端一致。 +- 存在測試:以 `unsupported` 引擎搭配非空 context 執行,斷言 explain 輸出不含注入計數、且渲染未被執行。 +- 存在測試:以 `supported(224)` 引擎執行,斷言實際採用的預算為 224 而非 200。 +- 存在測試:截斷方向與渲染優先序一致——超出上限時被丟棄的是低優先項,而非最前面的人名。 +- 全套件測試通過。 + +**Scope boundaries** + +- 在範圍內:`Engine` protocol 與其 6 個 conformer、prompt 渲染的預算來源、explain 的 context 區段措辭、選型對 context 存在的反應、WhisperKit 端 clamp 的方向。 +- 在範圍外:context.json schema、目錄解析、文件擷取、benchmark 語料與排名、解碼參數(除 clamp 方向外)、外部引擎協定的欄位定義。 + +## Risks / Trade-offs + +- **改 public protocol 是廣播式改動,對 repo 外實作 source-breaking** → 這是 D2 的明知取捨。以 release note 標示 breaking,並在 6 個 in-repo conformer 一次改完,讓編譯器成為完整性檢查。 +- **預算由 200 提高到 224,改變既有使用者的實際 prompt 內容** → 同一份 context 在同一後端上會注入更多項目,逐字稿可能改變。屬預期改善,但應在 release note 標明「context 注入量會增加」,避免被誤認為模型行為漂移。 +- **D4 的 clamp 方向改動在現行預算下無可觀測差異,容易被誤判為無效改動而被省略** → 必須在提高預算之前落地並附測試,測試需以超過上限的輸入直接驗證方向,而非依賴當前預算值。 +- **D5 的警告可能被使用者忽略,效果有限** → 接受。硬性過濾需要「WER 與 context 支援孰重」的量測依據,目前沒有;寧可警告而不假裝知道。 +- **`maxTokens` 為 0 的宣告是型別上合法但語意可疑的狀態** → 於 Failure modes 明訂等同不支援,並以測試鎖住,避免產生空 prompt。 + +## Migration Plan + +1. 先落地 D4(clamp 方向),與預算改動分離,確保方向修正可獨立歸因。 +2. 新增能力列舉與 protocol requirement;此時編譯失敗會精確列出所有未宣告的 conformer。 +3. 逐一為 6 個 conformer 補宣告:兩個 whisper backend 為支援並帶上限 224,其餘 4 個為不支援。 +4. 改渲染呼叫點:由所選引擎取得預算;不支援時跳過 render。 +5. 改 explain 措辭與選型警告。 +6. 補齊 Acceptance criteria 所列測試,跑全套件。 + +無資料遷移、無持久化狀態變更,故不需回填或相容層。回退方式為整體回退本變更。 diff --git a/openspec/changes/per-engine-prompt-capability/proposal.md b/openspec/changes/per-engine-prompt-capability/proposal.md new file mode 100644 index 0000000..9d1212c --- /dev/null +++ b/openspec/changes/per-engine-prompt-capability/proposal.md @@ -0,0 +1,73 @@ +## Summary + +把 context 的 prompt token 預算從全域寫死的 200 改為**由所選引擎宣告的能力**,並讓不支援 prompt 的後端停止宣稱自己注入了 context。 + +## Motivation + +`PromptRenderer.defaultTokenBudget = 200` 是單一全域常數,套用到所有 backend。但 prompt 能力是**逐引擎**的性質:`Engine` 是 public protocol,目前有 6 個 conformer,其中只有 `WhisperKitEngine` 與 `WhisperCppEngine` 真的消費 prompt;`ParakeetEngine`、`AppleSpeechEngine`、`ChineseFamilyEngine`、`ExternalProcessEngine` 完全不讀它。 + +對那 4 個引擎,整套 render 照跑、截斷照發生、`injected (N)` 照印——然後產物被丟棄。**系統宣稱做了它沒做的事。** + +這不是設計潔癖。2026-08-07 轉錄一批學術會議錄音時,`recommend` 在 `high` profile 選出完全不吃 prompt 的 fluid-parakeet,CLI 仍印出 `injected (49)` / `truncated (53)`: + +| 目標詞 | parakeet + context | whisper + context | +|---|---|---| +| Joint Thurstonian | 「during Sonya」 | 「Joint Thornia Models」 | +| Likert | 「the scale or L F」 | — | + +兩個詞都在「已注入」的 49 個裡,parakeet 命中 0。使用者據 `injected (N)` 去精簡詞表,是白工。截斷也是真的發生的:102 項只注入 49、截斷 53,其中 8 個 phrase 因預算耗盡被整類丟棄——對不吃 prompt 的引擎而言,這整段取捨毫無意義。 + +## Proposed Solution + +1. `Engine` protocol 增加 prompt 能力宣告,**二態**:`unsupported` 或 `supported(maxTokens:)`。 +2. 6 個 conformer 各自宣告:兩個 whisper backend 為 `supported(224)`,其餘 4 個為 `unsupported`。 +3. `PromptRenderer.render` 的預算由所選引擎提供,`defaultTokenBudget` 不再擔任全域預設。 +4. 引擎宣告 `unsupported` 時**整段跳過 render**,explain 明講「此後端不支援 context biasing」,不印 `injected (N)`。 +5. `recommend` / routing 在偵測到 context 目錄存在時,把 prompt 支援度納入選型;至少在選中 `unsupported` 引擎時警告。 + +上限 224 已由實證收斂:`whisper-cli --help` 明載 `--prompt PROMPT (max n_text_ctx/2 tokens)`,Whisper 的 `n_text_ctx = 448` → 224,與 `WhisperKitEngine.clampedPromptTokens(limit: 224)` 一致。 + +## Alternatives Considered + +**三態能力宣告**(`unsupported` / `supported(N)` / `supportedUnknownLimit`)。原始 issue 這樣提,但**第三態目前沒有任何 instance**——兩個 whisper backend 上限相同。加一個沒人用的 case,等於要求 6 個 conformer 與所有未來作者處理一個不存在的情況。日後真出現無法宣告上限的後端再加,屆時有真實 instance 可驗證語意。 + +**給 protocol requirement 預設實作 `.unsupported`**。改動較小、非破壞性,但新引擎會**預設靜默不支援 context**——正是本變更要消滅的失敗模式(宣稱與實際不符)的變體。不採用:本變更的核心命題就是能力必須被明確宣告。 + +## Non-Goals + +- **不**決定哪個引擎在會議廳音訊上最好。本變更只處理能力宣告的正確性。benchmark 語料代表性另案處理。 +- **不**處理 whisper 長 prompt 誘發的重複迴圈。那是獨立缺陷,修好本變更後那條路徑仍需另行處理。 +- **不**改變 context.json 的 schema、三層目錄解析、或文件擷取流程。 +- **不**調整 `clampedPromptTokens` 之外的 WhisperKit 解碼設定。 + +## Impact + +- Affected specs: `asr-engine`(新增能力宣告要求)、`context-calibration`(預算來源與 explain 誠實性)、`asr-routing`(選型納入 prompt 支援度) +- Affected code: + - Modified: + - Sources/BestASRKit/Engines/Engine.swift + - Sources/BestASRKit/Engines/WhisperKitEngine.swift + - Sources/BestASRKit/Engines/WhisperCppEngine.swift + - Sources/BestASRKit/Engines/ParakeetEngine.swift + - Sources/BestASRKit/Engines/AppleSpeechEngine.swift + - Sources/BestASRKit/Engines/ChineseFamilyEngine.swift + - Sources/BestASRKit/Engines/ExternalProcessEngine.swift + - Sources/BestASRKit/CommandCore.swift + - Sources/BestASRKit/Benchmark/BenchmarkRunner.swift + - Tests/BestASRKitTests/BackendEngineTests.swift + - Tests/BestASRKitTests/BenchmarkTests.swift + - Tests/BestASRKitTests/CLITests.swift + - Tests/BestASRKitTests/PipelineWiringTests.swift + - Tests/BestASRKitTests/TestSupport.swift + - CHANGELOG.md + - New: + - Sources/BestASRKit/Engines/PromptCapability.swift + - Tests/BestASRKitTests/PromptCapabilityTests.swift + - Tests/BestASRKitTests/EngineCapabilityTests.swift + - Tests/BestASRKitTests/ContextBudgetTests.swift + - Removed: (none) + +> 本清單於 #164 verify round 1 對齊實作:原先列了 `Context/PromptRenderer.swift` +> 與 `ContextTests.swift`(兩者實際未動——`ContextTests` 全部直接呼叫 +> `PromptRenderer.render`,不經 `ContextBundle.rendered`),且漏列全部新檔與 +> `BenchmarkRunner.swift`。 diff --git a/openspec/changes/per-engine-prompt-capability/specs/asr-engine/spec.md b/openspec/changes/per-engine-prompt-capability/specs/asr-engine/spec.md new file mode 100644 index 0000000..2610e89 --- /dev/null +++ b/openspec/changes/per-engine-prompt-capability/specs/asr-engine/spec.md @@ -0,0 +1,37 @@ +## MODIFIED Requirements + +### Requirement: Common engine interface + +Every ASR backend SHALL implement the common `Engine` interface (`id`, `isAvailable`, `transcribeRaw`, `promptCapability`), and `BackendID` SHALL enumerate exactly the backends with a bundled runtime: `whisperkit`, `whisper.cpp`, and `fluid-parakeet`. + +`promptCapability` SHALL declare whether the backend consumes a decoder conditioning prompt, as one of exactly two states: unsupported, or supported with a maximum token count. The interface SHALL NOT provide a default implementation of `promptCapability`, so that every current and future backend is required to declare it explicitly rather than inherit a silent default. + +A backend that declares support with a maximum token count of zero SHALL be treated identically to a backend that declares no support. + +#### Scenario: Three backends enumerate + +- **WHEN** `BackendID.allCases` is consulted (e.g. by `list-backends`) +- **THEN** it yields `whisperkit`, `whisper.cpp`, and `fluid-parakeet`, each constructible as an engine + +#### Scenario: Non-Whisper engine inherits the normalization seam + +- **WHEN** any input that is not 16 kHz mono is transcribed through `Engine.transcribe` with the fluid-parakeet backend +- **THEN** the engine's `transcribeRaw` receives the normalized 16 kHz mono path (AudioNormalizer, #36), identical to the Whisper backends + +#### Scenario: Every backend declares its prompt capability + +- **WHEN** each engine conforming to `Engine` is consulted for `promptCapability` +- **THEN** the two Whisper-family backends declare support with a maximum of 224 tokens +- **AND** every other backend declares no support + +#### Scenario: A declared capability matches what the backend actually does + +- **WHEN** an engine declares support for a prompt +- **THEN** that engine forwards the rendered prompt to its underlying runtime +- **AND** an engine that declares no support never receives a rendered prompt + +#### Scenario: A zero-token budget is treated as no support + +- **WHEN** an engine declares support with a maximum token count of zero +- **THEN** the system takes the same path as for an unsupported engine +- **AND** no empty prompt is passed to the backend diff --git a/openspec/changes/per-engine-prompt-capability/specs/asr-routing/spec.md b/openspec/changes/per-engine-prompt-capability/specs/asr-routing/spec.md new file mode 100644 index 0000000..0a55745 --- /dev/null +++ b/openspec/changes/per-engine-prompt-capability/specs/asr-routing/spec.md @@ -0,0 +1,26 @@ +## ADDED Requirements + +### Requirement: Selection accounts for prompt support when context is present + +When a context directory is resolved for the run, backend selection SHALL take each candidate's declared prompt capability into account, and SHALL warn when the selected backend declares no prompt support, stating that the supplied context will have no effect on the transcription. + +Selection SHALL NOT exclude a backend solely because it declares no prompt support. Whether a lower measured error rate outweighs the loss of context biasing has not been measured, so the system SHALL surface the trade-off rather than decide it silently. + +When no context directory is resolved, prompt capability SHALL NOT influence selection. + +#### Scenario: Selecting a backend without prompt support warns the user + +- **WHEN** a context directory is resolved and selection picks a backend declaring no prompt support +- **THEN** the system emits a warning that the context will not affect this transcription +- **AND** the selected backend is still used + +#### Scenario: Prompt capability is ignored when no context is present + +- **WHEN** no context directory is resolved +- **THEN** selection proceeds on its existing criteria alone +- **AND** no prompt-capability warning is emitted + +#### Scenario: A supporting backend produces no warning + +- **WHEN** a context directory is resolved and selection picks a backend declaring prompt support +- **THEN** no prompt-capability warning is emitted diff --git a/openspec/changes/per-engine-prompt-capability/specs/context-calibration/spec.md b/openspec/changes/per-engine-prompt-capability/specs/context-calibration/spec.md new file mode 100644 index 0000000..b3bc1ee --- /dev/null +++ b/openspec/changes/per-engine-prompt-capability/specs/context-calibration/spec.md @@ -0,0 +1,63 @@ +## MODIFIED Requirements + +### Requirement: Render context into a natural-language prompt with priority and budget + +The system SHALL render context values into a comma-separated natural-language vocabulary list — never JSON — in the priority order names (with aliases) first, then terms, then phrases, subject to the token budget declared by the selected engine's `promptCapability` (tokenizer-measured on the WhisperKit path; a conservative character heuristic on the whisper-cli path). Items that do not fit SHALL be skipped whole and recorded as truncated. + +The budget SHALL be obtained from the selected engine rather than from a single global constant, so that a backend whose practical limit differs is not held to another backend's limit. + +When the selected engine declares no prompt support, the system SHALL NOT render a prompt at all, and SHALL NOT compute or report truncation. + +Where the rendering priority order and any engine-side token clamping interact, the clamp SHALL preserve the highest-priority items, so that names are never dropped in favour of phrases. + +#### Scenario: Rendering follows the priority order + +- **WHEN** context has names, terms, and phrases within budget +- **THEN** the prompt lists all names and aliases first, then terms, then phrases + +##### Example: worked example from the design discussion + +- **GIVEN** names [{"name": "鄭澈", "aliases": ["Che"], "role": "主持人"}] and terms ["benchmark-driven", "CoreML"] +- **WHEN** the prompt is rendered +- **THEN** the prompt is exactly "鄭澈, Che, benchmark-driven, CoreML" + +#### Scenario: Budget overflow drops lowest-priority items first and records them + +- **WHEN** the combined values exceed the budget +- **THEN** phrases are dropped before terms and terms before names +- **AND** every dropped item is recorded in the truncation list + +#### Scenario: The budget comes from the selected engine + +- **WHEN** context is rendered for a backend declaring a maximum of 224 tokens +- **THEN** the rendering budget used is 224 +- **AND** it is not the previously hardcoded global value of 200 + +#### Scenario: No rendering happens for a backend without prompt support + +- **WHEN** a context directory holds values and the selected backend declares no prompt support +- **THEN** no prompt is rendered +- **AND** no truncation list is produced + +#### Scenario: Engine-side clamping keeps the highest-priority items + +- **WHEN** a rendered prompt exceeds the engine's own token clamp +- **THEN** the retained tokens are those of the highest-priority items +- **AND** names are not discarded while lower-priority phrases are retained + +### Requirement: Explain discloses context usage + +When context was loaded, the explain output SHALL disclose: the resolved directory, the injected values (count and items), the truncated items (when any), and the ignored files (when any). + +When the selected engine declares no prompt support, the explain output SHALL state that the backend does not support context biasing, and SHALL NOT report injected or truncated values — reporting an injection count for a backend that discards the prompt misrepresents what the system did. + +#### Scenario: Explain shows what was injected and what was skipped + +- **WHEN** transcription runs with a context directory containing values, an over-budget phrase, and a pdf +- **THEN** explain lists the injected values, the truncated phrase, and the ignored pdf with conversion guidance + +#### Scenario: Explain does not claim injection for an unsupported backend + +- **WHEN** transcription runs with a populated context directory on a backend declaring no prompt support +- **THEN** explain states that the backend does not support context biasing +- **AND** explain reports no injected count and no truncated list diff --git a/openspec/changes/per-engine-prompt-capability/tasks.md b/openspec/changes/per-engine-prompt-capability/tasks.md new file mode 100644 index 0000000..28f8b47 --- /dev/null +++ b/openspec/changes/per-engine-prompt-capability/tasks.md @@ -0,0 +1,37 @@ +## 1. 先修截斷方向(實作 design 決策 D4:截斷方向與優先序必須一致) + +與預算改動分離,確保方向修正可獨立歸因。對應 spec 需求 `Render context into a natural-language prompt with priority and budget` 中「clamp SHALL preserve the highest-priority items」一句。 + +- [x] 1.1 將 WhisperKit 端的 prompt token clamp 由保留後綴改為保留前綴,使其與渲染的 names → terms → phrases 優先序一致。驗收:新增測試以超過上限的 token 序列為輸入,斷言保留的是序列開頭而非結尾;該測試不得依賴當前預算值(必須直接餵超過 224 的輸入),否則預算仍為 200 時測試會空轉。 + +## 2. 建立能力宣告(實作 design 決策 D1:能力宣告採二態,而非三態;D2:protocol requirement 不給預設實作) + +對應 spec 需求 `Common engine interface`。 + +- [x] 2.1 依 D1(二態,不含「上限未知」第三態)在引擎模組新增 public 列舉表示 prompt 能力:不支援;支援並帶最大 token 數。驗收:型別可由模組外部取用;列舉恰有兩個 case。 +- [x] 2.2 依 D2 在 `Engine` protocol 新增唯讀屬性 `promptCapability` 回傳該列舉,**不提供預設實作**。驗收:此時建置必然失敗,且錯誤訊息精確列出所有尚未宣告的 conformer——這份清單即為 3.1/3.2 的完整性依據。 + +## 3. 逐一宣告,滿足 `Common engine interface`(依賴 2.2;彼此檔案不重疊) + +- [x] 3.1 [P] 兩個 Whisper 家族後端(WhisperKit 與 whisper.cpp)宣告為支援、上限 224。驗收:測試斷言兩者的宣告值皆為支援且上限 224;224 的依據為 `whisper-cli --help` 所載 `--prompt PROMPT (max n_text_ctx/2 tokens)` 與 Whisper 的 `n_text_ctx = 448`。 +- [x] 3.2 [P] 四個不消費 prompt 的後端(Parakeet、Apple Speech、Chinese family、外部程序)宣告為不支援。驗收:測試斷言四者皆宣告不支援;並斷言其轉錄路徑不接收已渲染的 prompt(對應 `Common engine interface` 的「declared capability matches what the backend actually does」情境)。 +- [x] 3.3 確認建置回復成功且無 conformer 遺漏。驗收:全專案建置通過,2.2 所列的錯誤清單已全數消除。 + +## 4. 改預算來源與跳過路徑(實作 design 決策 D3:不支援時跳過整段 render,而非渲染後丟棄) + +對應 spec 需求 `Render context into a natural-language prompt with priority and budget`。 + +- [x] 4.1 渲染呼叫點改為向所選引擎取得預算,取代原本依賴全域預設常數的行為。驗收:測試以宣告上限 224 的引擎執行,斷言實際採用的預算為 224 而非 200。 +- [x] 4.2 依 D3,所選引擎宣告不支援時整段跳過渲染,不計算截斷。驗收:測試以不支援的引擎搭配非空 context 執行,斷言未產生截斷清單、且渲染未被執行。 +- [x] 4.3 將「宣告支援但最大 token 數為 0」導向與不支援相同的路徑,滿足 `Common engine interface` 的零預算情境。驗收:測試斷言此情況不產生任何 prompt(尤其不得產生空字串 prompt 傳給後端)。 + +## 5. 誠實化輸出與選型 + +- [x] 5.1 滿足 spec 需求 `Explain discloses context usage`:explain 的 context 區段在引擎不支援時改為明講該後端不支援 context biasing,且不輸出注入計數與截斷清單。驗收:測試斷言不支援情境下的輸出不含注入計數、且含不支援訊息;支援情境的既有輸出不變。 +- [x] 5.2 滿足 spec 需求 `Selection accounts for prompt support when context is present`,依 design 決策 D5(routing 在有 context 時納入支援度,但不硬性過濾):已解析到 context 目錄且選中不支援 prompt 的後端時發出警告,說明所提供的 context 不會影響本次轉錄;不因此排除該後端。驗收:測試涵蓋三種情形——有 context 且選中不支援者(發警告且仍使用該後端)、有 context 且選中支援者(無警告)、無 context(無警告且選型準則不受影響)。 + +## 6. 收尾驗證 + +- [x] 6.1 對照 design 的 Implementation Contract 之 Acceptance criteria 逐條確認測試存在且通過。驗收:每一條 criteria 都能指向一個具名測試。 +- [x] 6.2 執行全套件測試並確認全綠。驗收:測試指令回傳成功,且失敗數為 0。 +- [x] 6.3 於 CHANGELOG 記錄兩項對使用者可見的變化:`Engine` protocol 新增 requirement 屬 breaking change(D2 的取捨);同一份 context 在 Whisper 後端上的注入量會因預算由 200 提高到 224 而增加。驗收:CHANGELOG 含這兩點,措辭指明後者屬預期改善而非模型行為漂移。