Studio can't represent or render agent-built short-video edits (transitions, beat-cuts, subtitles) — timeline stays out of sync with the deliverable

[nanxingw]

Summary

I just produced a full 3-minute short-video (39 generated shots, 42 native-speed beats, varied transitions, ducked BGM, burned Chinese subtitles). The entire final video had to be built with raw ffmpeg, completely outside Studio — so the Studio timeline (the user's direct visual entry point) never reflected the actual deliverable. This issue collects the concrete reasons the composition/render pipeline couldn't carry the work, so Studio can become usable for agent-built edits.

Related: #93 (transition endpoints all 500).

Concrete blockers

1. Transitions are unusable. All four cinematic transition endpoints (/api/transitions/{light-leak,glitch,domain-warp,grav-lens}) return 500 (see Cinematic transition endpoints all return 500 (broken geq filter expr) — blocks Studio-native transitions #93). The only other transition the render pipeline supports is opacity-keyframe crossfade (dissolve) — no glitch / flash / whip / match-cut. A short-video needs varied transitions; there's no way to express them.

2. composition.yaml can't represent a real fast-cut edit. The schema models clips-on-tracks with basic crossfades, but a polished short-video needs, per beat: a transition-in effect (flash/glitch/dip/whip), a reframe/punch-in crop, a sub-clip trim (use seconds 4–8 of an 8s source), and sometimes a speed change. None of these have a representation on the track, so a 42-beat cut has nowhere to live in Studio.

3. No way to import a rendered result back onto the timeline. After building the master mp4, there's no "place this mp4 as a clip" / "flatten generated beats to the timeline" verb, so I can't even make Studio show the deliverable as a single clip without hand-editing yaml.

4. Subtitles have no first-class, styleable path. captionStrategy: overlay renders through the composition, but (a) I built the master outside the composition, and (b) the machine's default ffmpeg (Homebrew) is compiled without libass, so the subtitles filter doesn't exist — only the managed ~/.autoviral/bin/ffmpeg has it. There's no documented "burn styled captions (font/size/outline/position) into the exported file" control.

5. Render is a black box. POST /api/works/:id/render returns a jobId, but GET /api/works/:id/render/:jobId → 404 (no status endpoint), and there's no intermediate preview. By contrast, raw ffmpeg lets me snapshot any frame and probe loudness instantly, which I need to verify honestly and iterate fast.

Net effect

Studio remained stuck on the earliest (wrong, slow-motion) version while the shipped video lived only as output/final_v3_sub.mp4. The user explicitly said "Studio is the user's direct visual entry point, this is very important" — and right now agents that do anything beyond trivial clip-adds are forced to abandon it.

Suggested fixes (priority order)

1. Fix the 4 transition endpoints (Cinematic transition endpoints all return 500 (broken geq filter expr) — blocks Studio-native transitions #93).
2. Extend the track/clip schema to carry per-clip transition-in type+duration, sub-clip in/out trims, reframe (scale/position), and a speed field the render pipeline honors — so a beat-cut is representable.
3. Add an "import mp4 as clip" and/or "flatten beats to timeline" verb so a rendered result can round-trip into Studio.
4. First-class caption/subtitle track with styling (font/size/outline/position), burned via a libass-enabled ffmpeg; ship a libass-enabled bundled ffmpeg so this never depends on the host's Homebrew build.
5. Add a render job status endpoint and an intermediate/proxy preview.