From 36e079d7417f9a13eb6d71a9a7da6a54aee86ca5 Mon Sep 17 00:00:00 2001 From: Tadas Petra <60107328+tadaspetra@users.noreply.github.com> Date: Thu, 2 Jul 2026 09:07:06 -0500 Subject: [PATCH] Replace realtime Scribe streaming with post-recording batch transcription The realtime WebSocket session died during long silences (idle/session limits), killing the transcript for the rest of the take. Transcription is now a batch pass over the finalized recording, ordered so footage durability never depends on the network: - Stop flow saves the recovery checkpoint before any transcription call; failure or timeout falls back to full-duration sections with a visible warning instead of blocking or losing the take. - New main-process transcription-service extracts audio from the finalized file via ffmpeg stream copy and calls batch Scribe (scribe_v2, word timestamps); the API key never reaches the renderer. - Word timestamps map into recording time via per-file recorder start offsets instead of the old wall-clock estimate. - Record is blocked while the previous take is still post-processing so overlapping take flows cannot race on the shared recovery checkpoint. - Removed the WebSocket/AudioWorklet/PCM pipeline, live segment editing UI, scribe token IPC, and scribe-status module (net -215 lines). Tests: segment builder + timeout helper (15), transcription service temp-file cleanup and failure paths (10), IPC handler validation (3); media-cleanup and register-handlers suites updated. pnpm run check passes. Co-Authored-By: Claude Fable 5 --- .env.example | 2 +- AGENTS.md | 2 +- docs/production/feature-inventory.md | 19 +- docs/production/runbook.md | 2 +- docs/production/target-architecture.md | 4 +- src/audio-processor.ts | 15 - src/main.ts | 4 +- src/main/ipc/register-handlers.ts | 23 +- src/main/services/scribe-service.ts | 10 +- src/main/services/transcription-service.ts | 150 ++++++ src/preload.ts | 2 +- src/renderer/app.ts | 467 +++++------------- src/renderer/features/media-cleanup.ts | 25 - .../features/transcript/batch-transcript.ts | 100 ++++ .../features/transcript/scribe-status.ts | 119 ----- src/shared/electron-api.ts | 17 +- tests/unit/batch-transcript.test.ts | 143 ++++++ tests/unit/media-cleanup.test.ts | 37 -- tests/unit/register-handlers.test.ts | 58 ++- tests/unit/scribe-status.test.ts | 68 --- tests/unit/transcription-service.test.ts | 205 ++++++++ tsconfig.build.json | 1 - 22 files changed, 827 insertions(+), 646 deletions(-) delete mode 100644 src/audio-processor.ts create mode 100644 src/main/services/transcription-service.ts create mode 100644 src/renderer/features/transcript/batch-transcript.ts delete mode 100644 src/renderer/features/transcript/scribe-status.ts create mode 100644 tests/unit/batch-transcript.test.ts delete mode 100644 tests/unit/scribe-status.test.ts create mode 100644 tests/unit/transcription-service.test.ts diff --git a/.env.example b/.env.example index 0b20037..a8df4b3 100644 --- a/.env.example +++ b/.env.example @@ -1,2 +1,2 @@ -# Required for realtime transcript token generation. +# Required for post-recording batch transcription (main process only). ELEVENLABS_API_KEY= diff --git a/AGENTS.md b/AGENTS.md index 8029222..6bf03df 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -241,7 +241,7 @@ Default to: - Very long timelines with dense camera-overlay keyframes can produce extremely nested ffmpeg filter expressions that exceed parser limits and fail render; the overlay graph needs to stay bounded for long sessions. - Heavy H.264 screen exports (high resolution, high quality, dense UI detail) can stress QuickTime playback more than NLEs; explicit yuv420p and a compatible profile/level—and avoiding unnecessary output resolution—improves default macOS playback. - `@electron/packager` dependency pruning can miss transitive packages under `pnpm`’s symlinked `node_modules` layout (fresh CI installs surface this); the packaging smoke harness avoids relying on that prune path. -- The Scribe live-transcription `AudioWorklet` (`audio-processor`) must load as a plain browser worklet script; compiling it with the main-process TypeScript target (e.g. CommonJS/NodeNext) breaks loading (`exports` undefined) and silently disables transcription-driven features. +- Transcription is a batch pass over the finalized recording (main-process `transcription-service` extracts audio via ffmpeg stream copy, then calls ElevenLabs batch Scribe with word timestamps); it must never run before the finalized files and the recovery checkpoint are on disk, and its failure/timeout must fall back to full-duration sections instead of blocking or losing the take. The realtime Scribe WebSocket/AudioWorklet pipeline was removed because long silences and session limits killed the live socket mid-recording; take flows are serialized (`stopRecordingInFlight` guards Record) so overlapping post-processing cannot race on the shared recovery checkpoint. - `src/renderer/styles/main.css` is Tailwind build output (`build:styles`, minified); full `check` rebuilds styles via nested steps, so tracking a non-matching formatted copy causes perpetual diffs—prefer generating locally and not treating it as hand-edited source. - Recording durability depends on streaming `MediaRecorder` chunks directly to disk via IPC temp `.part` files (renamed on finalize) instead of buffering in the renderer; screen and camera are independent recorder pipelines, so stop/save/finalize must tolerate one failing without blocking the other, and recovery normalization must require the screen file but tolerate a missing camera file so a partial failure does not discard an otherwise-recoverable take. Partial WebM files report `duration=Infinity`; resolve real duration by seeking a probe `