Run commands off the main thread and drop locks around ffmpeg work#3
Merged
Conversation
Playback was silent: the playhead was a wall-clock rAF loop and only video frames were fetched. Now every audio-bearing clip is audible during playback. - engine: audio_pcm() decodes a source window to mono s16le PCM via the ffmpeg binary (input-side -ss fast-seek, 32 kHz default) - app: get_audio command returns the raw bytes via tauri::ipc::Response (a JSON number array would balloon ~5x), decoding off-lock like get_frame - frontend: new AudioEngine schedules each clip on an AudioContext with volume, fades (transitions approximated as fade-in), speed and reverse applied; buffers are cached with an LRU duration cap; the playhead follows the audio clock while it runs so picture chases sound - edits mid-playback re-anchor the audio so the new cut is heard live
Trimming previously meant typing source points into the Inspector. Clips now have 6px ew-resize handles on both edges (pointer tool): dragging retrims with the usual snapping (playhead / clip edges / 0), clamped to the available source handle, neighboring clips, and a 0.05s minimum; still images extend freely since they loop. Reversed clips map edges to the opposite source ends. Project::trim gains an optional timeline_start applied in the same edit, so a left-edge trim keeps the right edge put and stays a single undo step; the Tauri command, MCP tool, and browser fallback pass it through.
J/K/L drive the transport the standard way: L plays forward, J reverse, K pauses, and repeat taps double the rate up to 8x (audio follows forward shuttle, pitch-shifted; reverse is silent). I/O set in/out marks at the playhead — kept ordered, cleared with Shift+I/O — rendered as brackets plus a shaded span on the ruler. The marks will drive range export next.
ExportOptions.range renders only a span of the timeline. The graph is built from Timeline::slice(start, end): a copy holding just the window, shifted to t=0 — boundary clips get their source windows cut (honoring speed and reverse), edge fades and orphaned transitions dropped, and an animated clip cut at the front keeps its pose via a resampled keyframe at the new start; overlays are clipped and shifted the same way. The progress total follows the span, and two-pass exports slice both passes. The export dialog offers "In → out" as a range choice whenever both marks are set; the MCP export tool documents the new field.
Track.duck flags a track (the DUCK toggle on audio track headers, or the set_track_duck command/MCP tool) so its audio dips under the rest of the mix on export: the graph mixes ducked and non-ducked clips into two buses and sidechain-compresses the ducked bus against the other (threshold .05, 8:1, 20ms/400ms) before the final sum — a music bed falls automatically under dialogue. With nothing flagged, or nothing to key from, the flat mix is emitted byte-identical to before. ExportOptions.loudnorm (a toggle in the export dialog's Audio section) appends single-pass loudnorm to -14 LUFS / -1.5 dBTP on the final mix, resampled back to the output rate.
The MediaBin transcript tab is now an editing surface: each line resolves to the timeline clip carrying it, clicking a line seeks the playhead there, the line under the playhead is highlighted, and an x cuts that sentence out of the cut — lines no longer on the timeline show struck through. The primitive is Project::cut_clip_range(clip_id, from, to): remove a source-time span from a clip (split around the intersection, honoring speed/reverse, edge fades and transitions kept on the outer boundaries) and ripple the track closed, all one undo step. Exposed as a Tauri command, an MCP tool (an agent can now delete sentences by transcript timestamps), and in the browser fallback.
Every Tauri command was a sync fn, which Tauri v2 executes on the main thread — so exports, analysis, frame decodes, waveforms and imports froze the whole window for their duration. Quick commands are now #[tauri::command(async)] (async runtime), and heavy ones are async fns that run their ffmpeg/disk work on the blocking pool via a blocking() helper. Lock hygiene: get_waveform/get_energy (GUI + MCP) and the MCP get_frame/skim_asset/preview_timeline tools decoded whole files while holding the shared project mutex, stalling every other op. New lock-free Project::decode_waveform/decode_energy/decode_contact_sheet/probe_asset statics let both adapters resolve inputs under the lock and release it before the slow part; heavy MCP tools are async + spawn_blocking so a minutes-long export no longer pins a tokio worker. Both lock helpers now recover from a poisoned mutex instead of bricking the session. Frontend: multi-file imports probe concurrently, the waveform cache is single-flight, and project-changed bursts from agent edits coalesce into one refresh in flight plus one queued instead of a re-fetch per event.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every Tauri command was a sync
fn, which Tauri v2 executes on the main thread. Exporting, analyzing, decoding preview frames (fetched continuously during playback), bucketing waveforms, and probing imports all froze the entire window for their duration. On top of that, several paths ran whole-file ffmpeg work while holding the shared project mutex, stalling every other GUI command and the MCP agent.Changes
Tauri commands (
kerf-app/src/lib.rs)#[tauri::command(async)]so they run on the async runtime instead of the main thread.export_timeline,analyze_asset,get_frame,get_timeline_frame,get_waveform,get_energy,get_audio,import_asset,open_project,save_project_as,export_srt) →async fn+ ablocking()helper on the blocking pool, resolving inputs under the lock and releasing it before the slow part.get_waveform/get_energyno longer hold the project mutex during the whole-file decode.MCP server (
kerf-app/src/mcp.rs)analyze_asset,get_frame,skim_asset,preview_timeline,get_waveform,get_energy,export→ async +spawn_blocking;get_frame/skim_asset/preview_timeline/waveforms previously decoded under the project lock, so an agent skimming footage froze the user's edits.kerf-core (
project.rs)decode_preview_frame:decode_waveform,decode_energy,decode_contact_sheet, plusprobe_asset(probe without&selfso parallel imports really probe in parallel).Frontend
project-changedbursts from agent edits coalesce into at most one refresh in flight + one queued, instead of a full re-fetch per event.Verification
Ran the desktop app (WSLg) and drove both surfaces:
preview_timelinereturned a real ffmpeg-encoded composite through async →spawn_blocking; 4 concurrent composites + aget_timeline_stateread all returned (read in ~12 ms, no lock starvation).cargo test -p kerf-core --no-default-features(117 passed),cargo clippy(no new warnings),bun run check(0 errors),bun run buildall green.Not exercised: heavy decode against long real sources (no MCP import tool and the native file picker isn't automatable headless) — the code paths are identical in shape to the pre-existing lock-free
get_frame/get_timeline_framepattern.🤖 Generated with Claude Code