Don't let a half-read batch transcript bury a whole live one - #471
Don't let a half-read batch transcript bury a whole live one#471Optic00 wants to merge 1 commit into
Conversation
…ript The onnx backend skips a window whose recognize() raises, on purpose: one bad window shouldn't fail a meeting. But the transcript then covers less audio than the recording holds, and nothing said so. The live-transcript rescue (stenolabs#207) only fires when the batch FAILED or came back as the silence sentinel, and a half-read batch is neither -- so it passed the gate and silently replaced a complete live transcript with a full-of-holes one. No error, just less text. Report how much audio the surviving windows actually covered, roll it up across channels, and let the rescue fire when it falls under half. Measured in seconds of audio, not in windows: windows overlap and the last one is short, so on 61 s the two windows are [0,60) and [45,61) and losing the first leaves 16 seconds while 1-of-2 would report half. A window also only counts once it has contributed -- counting at recognize() time called a window discarded for unusable timing a success. Channels emptied by bleed handling don't vote, since what their transcription missed says nothing about the transcript built from the other one. Keyed on coverage, not length: the length threshold this gate used to have was removed because it replaced correct-but-short transcripts. A backend that does no windowing reports None, and None stays "unknown" rather than counting as complete -- only the onnx path can lose a window quietly, since parakeet-mlx has no per-chunk except and fails loudly.
|
We've triggered an ultrareview automatically — This PR modifies the core decision logic for transcript selection and introduces a new window-coverage metric involving complex overlap calculations across channels, posing a risk to data integrity.. I'll post findings when complete. An ultrareview is cubic's deepest review, catching hard-to-find bugs in the most critical PRs. It runs a longer, multi-pass analysis using cubic's most capable review models, and typically takes around 30 minutes. It consumes your team's reviewed-lines allowance at 3× the standard rate. Automated ultrareviews are disabled by default. We triggered this run as part of your trial. Want cubic to do this for every high-risk PR? Enable auto-ultrareview in your settings. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
Ultrareview completed in 10m 41s
1 issue found across 6 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/_parakeet_onnx.py">
<violation number="1" location="src/_parakeet_onnx.py:585">
P3: The new coverage warning can report an inflated “windows usable” count. The code logs `windows_recognized`, but that counter is incremented before the token/timestamp length check that can discard a window, so discarded windows still appear as “usable” in the message. Using wording that matches the counter (or tracking a true usable counter) would keep diagnostics aligned with actual behavior.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
| covered_seconds = covered_samples / _SAMPLE_RATE | ||
| if covered_seconds < total_seconds: | ||
| logger.warning( | ||
| "ONNX transcription read %.0fs of %.0fs (%d of %d windows usable) — " |
There was a problem hiding this comment.
P3: The new coverage warning can report an inflated “windows usable” count. The code logs windows_recognized, but that counter is incremented before the token/timestamp length check that can discard a window, so discarded windows still appear as “usable” in the message. Using wording that matches the counter (or tracking a true usable counter) would keep diagnostics aligned with actual behavior.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/_parakeet_onnx.py, line 585:
<comment>The new coverage warning can report an inflated “windows usable” count. The code logs `windows_recognized`, but that counter is incremented before the token/timestamp length check that can discard a window, so discarded windows still appear as “usable” in the message. Using wording that matches the counter (or tracking a true usable counter) would keep diagnostics aligned with actual behavior.</comment>
<file context>
@@ -539,4 +578,20 @@ def _transcribe_windows(ts_model: Any, samples) -> _SimpleResult:
+ covered_seconds = covered_samples / _SAMPLE_RATE
+ if covered_seconds < total_seconds:
+ logger.warning(
+ "ONNX transcription read %.0fs of %.0fs (%d of %d windows usable) — "
+ "the transcript is missing roughly %.0fs of audio",
+ covered_seconds, total_seconds,
</file context>
| "ONNX transcription read %.0fs of %.0fs (%d of %d windows usable) — " | |
| "ONNX transcription read %.0fs of %.0fs (%d of %d windows recognized) — " |
1 participant
Description
The onnx backend skips a window whose
recognize()raises, on purpose: one bad window shouldn't fail a whole meeting. But the transcript then covers less audio than the recording holds, and nothing said so.The live-transcript rescue (#207) only fires when the batch failed or came back as the exact silence sentinel. A half-read batch is neither, so it passed the gate and silently replaced a complete live transcript with a full-of-holes one. No error, just less text.
This reports how much audio the surviving windows covered, rolls it up across channels, and lets the rescue fire when it falls under half.
Measured in seconds of audio, not windows. Windows overlap and the last one is short: on 61 s they are
[0,60)and[45,61), so losing the first leaves 16 seconds while a 1-of-2 count would report half. A window counts only once it has contributed — counting atrecognize()time called a window later discarded for unusable timing a success. Channels emptied by bleed handling don't vote, since what their transcription missed says nothing about the transcript built from the other one.Keyed on coverage, not length: the length threshold this gate used to have was removed because it replaced correct-but-short transcripts.
Platform scope: only the onnx path (Windows/Linux) can lose a window quietly. parakeet-mlx has no per-chunk
except, so on macOS a bad window fails the whole call loudly and is already handled. A backend that does no windowing reportsNone, andNonestays "unknown" rather than counting as complete.Type of Change
Testing
python -m unittest discover tests), 12 of them new: the duration-based metric (losing the first window is not the same loss as losing the last), a window dropped for mismatched token/timestamp lengths not counting as read, the worst-channel roll-up ignoring channels that report nothing, and the gate from both sides.ruff checkclean on every file touched, no new findings againstmain.process_streamingchanged underneath. The decision now lives in_unusable_batch_reasonand the test calls it.Additional Notes
The 0.5 threshold is a judgment value, not a measured one, and deliberately low — a meeting that lost a window or two is still better than the streaming text. Happy to move it.
Not covered here: when coverage lands between the threshold and 1.0, the batch still wins silently. Filling only the gaps from the live transcript would be better, but needs timestamps in the live snapshot.
Summary by cubic
Prevents a partially read ONNX batch transcript from overwriting a complete live transcript. We now compute window coverage and fall back to the live transcript when coverage is below 50% (measured in seconds of audio).
window_coveragein ONNX windowed runs using overlap-aware covered seconds; include in result dicts and roll up across channels by the worst reported value. Channels with no surviving segments don’t vote._unusable_batch_reasonto gate the live fallback on: batch failure, exact silence sentinel, or low coverage;Nonecoverage is treated as unknown (e.g.,whisper.cpp,parakeet-mlx) and doesn’t penalize.transcriber.pyandsimple_recorder.py; remove length-based fallback to avoid replacing correct-but-short transcripts.None, worst-channel roll-up, and the new fallback predicate.Written for commit 3cb0070. Summary will update on new commits.