feat: prompt capability is a per-engine declaration, not a global constant (#164)

CHANGELOG.md

@@ -5,6 +5,53 @@ All notable changes to bestASR are documented here. The format follows
 
 ## [Unreleased]
 
+### Changed
+
+- **BREAKING — `Engine` now declares its prompt capability (#164)**: the protocol
+  gains a `promptCapability` requirement with **no default implementation**, so
+  every conformer — including any out-of-tree engine — must state whether it
+  consumes decoder conditioning text and up to how many tokens. A default was
+  considered and rejected: it would let a new backend inherit "no prompt support"
+  without its author ever considering the question, which is a quieter form of
+  the very problem this change fixes. The cost is a one-time source-breaking
+  change; the benefit is that the answer can no longer be assumed.
+
+- **Context injection reports the truth, and reaches more of your terms (#164)**:
+  the token budget now comes from the selected backend instead of a single global
+  constant.
+
+  Two things change for you. First, a backend that ignores conditioning text no
+  longer prints `injected (N)` and a truncation list — it says plainly that it
+  does not support context biasing, and selection warns when your context cannot
+  take effect. Previously five of the seven backends ran the whole render-and-
+  truncate pipeline and discarded the result while still reporting a count; a
+  real run showed `injected (49) / truncated (53)` against a backend that used
+  none of them, and the two terms that mattered were both in the "injected" list
+  and both mis-transcribed. Acting on that number by trimming your term list
+  changed nothing.
+
+  Second, on the Whisper backends the budget rises from 200 to their measured
+  224-token ceiling, so **more of the same context directory now reaches the
+  model**. Expect slightly different transcripts there. This is the intended
+  improvement, not model drift.
+
+  The engine-side clamp direction was reversed in the same change: overflow now
+  discards the lowest-priority phrases rather than the names at the front of the
+  prompt, which is what it had been dropping. Read that as a reversible bet, not
+  a settled correction. It trades against a different mechanism the previous
+  direction was built on — Whisper's reference decoder truncates an over-long
+  prompt by keeping its *tail*, and content nearest the transcription boundary
+  is widely reported to weigh more. Neither direction has been measured here.
+  If a WER regression shows up on long-context Whisper runs, this is the line to
+  suspect; the durable fix is to reorder the renderer so the highest-value items
+  land at the tail, not to flip the clamp back. See `PipelineWiringTests` for
+  the trade-off in full.
+
+  Selection warns but does not re-rank. Whether a lower measured error rate is
+  worth losing context biasing has not been measured, so the trade-off is
+  surfaced rather than decided.
+
+
 ### Added
 
 - **Apple Speech backend (#121)**: `apple-speech` — the OS-native backend

Sources/BestASRKit/Benchmark/BenchmarkRunner.swift

@@ -174,6 +174,12 @@ public struct BenchmarkRunner {
     ) async -> BenchmarkOutcome {
         var measured: [MeasuredCandidate] = []
         var failures: [BenchmarkFailure] = []
+        // Candidates whose backend declares no prompt support, so the ±context
+        // pass was not run for them (#164 verify). Reported, never silent.
+        var contextSkipped: [String] = []
+        // Candidates that CAN take a prompt but whose with-context pass threw.
+        // Kept apart from contextSkipped: same blank cell, different reason.
+        var contextFailed: [String] = []
 
         guard let audioDuration = audio.duration, audioDuration > 0 else {
             return BenchmarkOutcome(
@@ -252,23 +258,44 @@ public struct BenchmarkRunner {
                     language: language)
 
                 // Optional second pass with the context prompt (spec benchmark:
-                // Measure the context-biasing delta). Model is warm; failures
-                // here degrade to a note-worthy nil, not a candidate failure.
+                // Measure the context-biasing delta). Model is warm; a failure
+                // here is not a candidate failure — the baseline measurement
+                // still stands.
+                //
+                // Gated on the candidate's own declaration (#164 verify): a
+                // backend that takes no prompt must never receive one (spec
+                // asr-engine), and measuring a "with-context" pass there would
+                // publish a delta of ~0 that reads as "context does not help
+                // this backend" when the truth is that it cannot use context
+                // at all.
+                //
+                // Both ways of ending up without a delta are named in the
+                // notes, and named SEPARATELY: "declares no prompt support"
+                // and "the pass threw" are different facts about the backend,
+                // and reporting a transient decode failure as a capability
+                // limit would be false.
                 var contextErrorRate: Double?
-                if let contextPrompt {
+                if contextPrompt != nil, !engine.promptCapability.supportsPrompt {
+                    contextSkipped.append(candidate.backend.rawValue)
+                }
+                if let contextPrompt, engine.promptCapability.supportsPrompt {
                     let contextOptions = TranscribeOptions(
                         model: candidate.model,
                         quantization: candidate.quantization,
                         language: effectiveLanguage,
                         prompt: contextPrompt,
                         deterministicDecode: deterministicDecode
                     )
-                    if let contextTranscript = try? await engine.transcribe(
-                        audioPath: normalizedAudio.path, options: contextOptions)
-                    {
+                    do {
+                        let contextTranscript = try await engine.transcribe(
+                            audioPath: normalizedAudio.path, options: contextOptions)
                         contextErrorRate = ErrorRate.compute(
                             hypothesis: contextTranscript.text,
                             reference: referenceText, kind: metricKind, language: language)
+                    } catch {
+                        let reason = (error as? TranscriptionError)?.errorDescription
+                            ?? error.localizedDescription
+                        contextFailed.append("\(candidate.backend.rawValue) (\(reason))")
                     }
                 }
                 let record = BenchmarkRecord(
@@ -297,10 +324,27 @@ public struct BenchmarkRunner {
             }
         }
 
+        // Naming the skipped backends keeps a mixed grid honest: without this
+        // line a report showing a DELTA column for some rows and blanks for
+        // others gives no reason for the blanks.
+        let skipNote = contextSkipped.isEmpty
+            ? []
+            : [
+                "context: no with-context pass for "
+                    + Set(contextSkipped).sorted().joined(separator: ", ")
+                    + " — these backends declare no prompt support"
+            ]
+        let failNote = contextFailed.isEmpty
+            ? []
+            : [
+                "context: with-context pass failed for "
+                    + Set(contextFailed).sorted().joined(separator: "; ")
+                    + " — the baseline measurement for these candidates still stands"
+            ]
         return BenchmarkOutcome(
             measured: measured,
             failures: failures,
-            notes: initialNotes,
+            notes: initialNotes + skipNote + failNote,
             metricKind: metricKind,
             language: language
         )