Bound streaming fallback latency

[a692570]

What changed

• accept stable streaming transcripts without blocking on redundant batch verification
• cap suspicious streaming verification at two seconds and avoid retrying it
• reduce the post-insert overlay hold from 900 ms to 300 ms
• add regression coverage for the fallback policy and overlay timing

Why

Recent local logs showed Bolo remaining in Thinking for 16 seconds even though usable streaming text was already available. The synchronous batch verification request timed out and retried before allowing insertion.

Verification

• cargo fmt --check
• cargo test, 55 passed
• cargo build --release
• restarted and verified the local release binary and helper processes

[entelligence-ai-pr-reviews]

EntelligenceAI PR Summary

Refactors streaming transcription logic in src/main.rs to improve clarity, correctness, and configurability.

• Added STREAMING_BATCH_VERIFY_TIMEOUT, STT_REQUEST_TIMEOUT, and POST_INSERT_OVERLAY_HOLD named constants to replace inline magic values
• Introduced verify_streaming_transcript method using a dedicated 2-second deadline for batch verification calls
• Extended transcribe_with_model with a request_timeout parameter, applied via .timeout() on the HTTP request
• Fixed streaming_batch_fallback_reason to evaluate the word-rate condition before the source condition
• Added stable_best_available as a recognized non-fallback source in fallback reason logic
• Updated existing tests and added two new tests covering constant boundary conditions

---

Confidence Score: 5/5 - Safe to Merge

Safe to merge — this PR refactors streaming transcription logic in src/main.rs by replacing inline magic values with named constants (STREAMING_BATCH_VERIFY_TIMEOUT, STT_REQUEST_TIMEOUT, POST_INSERT_OVERLAY_HOLD), introducing a dedicated verify_streaming_transcript method with a bounded 2-second deadline, and adding a request_timeout parameter to transcribe_with_model. The changes improve correctness by bounding previously unbounded fallback latency and enhance maintainability through explicit configuration. No review comments were generated and no pre-existing unresolved issues are present.

Key Findings:

• Named constants replace magic numbers throughout src/main.rs, eliminating ambiguity around timeout values and making future tuning straightforward without risk of inconsistent edits.
• The new request_timeout parameter on transcribe_with_model applied via .timeout() correctly bounds HTTP request duration, preventing indefinite hangs that could have blocked the streaming pipeline.
• The dedicated verify_streaming_transcript method encapsulates batch verification with its own deadline, improving separation of concerns and making timeout behavior explicit and auditable.
• No automated review comments were raised and heuristic analysis found zero critical, significant, or medium issues, indicating the refactor is mechanically clean with no introduced regressions.

Files requiring special attention

• src/main.rs

[entelligence-ai-pr-reviews]

Walkthrough

This PR refactors streaming transcription handling in src/main.rs by replacing magic values with named constants, introducing a dedicated verify_streaming_transcript method with a shorter 2-second deadline, adding per-call request_timeout support to transcribe_with_model, and fixing fallback reason evaluation logic to check word-rate before source while recognizing stable_best_available as a non-fallback source.

Changes

File(s)

Summary

src/main.rs

Introduced named constants (STREAMING_BATCH_VERIFY_TIMEOUT, STT_REQUEST_TIMEOUT, POST_INSERT_OVERLAY_HOLD) to replace magic values; added verify_streaming_transcript method with a 2-second batch verification deadline; updated transcribe_with_model to accept a request_timeout parameter applied via .timeout(); fixed streaming_batch_fallback_reason to evaluate word-rate check before source check and accept stable_best_available as non-fallback; updated existing tests and added two new constant-boundary tests.

Sequence Diagram

This diagram shows the interactions between components:

sequenceDiagram
    participant App as Application
    participant StreamHandler as Streaming Handler
    participant BatchVerifier as verify_streaming_transcript
    participant Transcriber as transcribe_with_model
    participant STT as STT API (Telnyx)
    participant Overlay as Overlay UI

    App->>StreamHandler: streaming transcript received
    activate StreamHandler

    Note over StreamHandler: Check streaming_batch_fallback_reason()
    Note over StreamHandler: source "final" or "stable_best_available" -> no fallback reason
    Note over StreamHandler: other sources -> "non_final_streaming_result"
    alt source is "final" or "stable_best_available"
        StreamHandler->>BatchVerifier: verify_streaming_transcript(wav, warmup)
        activate BatchVerifier
        Note over BatchVerifier: timeout = STREAMING_BATCH_VERIFY_TIMEOUT (2s)
        BatchVerifier->>Transcriber: transcribe_with_model(..., timeout=2s)
        activate Transcriber
        Transcriber->>STT: POST /stt with per-request timeout
        STT-->>Transcriber: batch transcript
        deactivate Transcriber
        Transcriber-->>BatchVerifier: batch result
        BatchVerifier-->>StreamHandler: batch transcript
        deactivate BatchVerifier

        alt batch word count > streaming word count
            StreamHandler->>App: use batch transcript
        else streaming word count >= batch
            StreamHandler->>App: use streaming transcript
        end

    else non-final source
        StreamHandler->>App: skip batch verification (non_final_streaming_result)
    end

    deactivate StreamHandler

    App->>App: insert text & record metrics

    App->>Overlay: send Overlay(Copied)
    App->>Overlay: HideOverlayAfter(POST_INSERT_OVERLAY_HOLD = 300ms)

    Note over App, STT: Full transcription path uses STT_REQUEST_TIMEOUT (12s)
    Note over BatchVerifier, STT: Batch verification uses STREAMING_BATCH_VERIFY_TIMEOUT (2s)

Loading