refactor: captions/audio pipeline — consolidate the ep5 fix series into clean architecture #227

Context

The ep5 incident produced a series of working but surgical fixes (PR #223, PR #226, issue #225). They ship correct behavior — verified frame-by-frame on formula-care-ep5 — but each one patched a symptom inside an architecture that made those bugs possible. This issue tracks the proper refactor.

What the fix series bolted on (and where it creaks)

1. Caption timing is corrected in 4 different places instead of one:

   • transformCue (shift/drop/clamp vs clip trim window) in merge-ass.ts
   • monotonizeEntries (non-monotonic whisper words) in ass.ts
   • refineWordTimings (leading/trailing silence vs first/last word) in primitives/silence.ts
   • per-node analysis memoization + refine wiring inside audio-element.ts
     A caption cue is now transformed by up to 3 sequential passes owned by 3 different modules. There should be ONE timing model: words → (refine) → cues → (window remap) → timeline, as a single pure pipeline with one test surface.

2. Two resolve paths still exist (the root cause of the 429 storm): standalone resolve.ts (~1400 lines, now with memoizedElementResolve bolted on) vs renderers with withDedup/pendingFiles. Stage 3/4 of the graph plan (render = compile + executePlan) was landed in PR refactor: computation-graph IR, render lifecycle, renderers split #223, but async components still generate through the standalone path during resolveLazy, outside the executor and its concurrency limit (semaphore follow-up in async components bypass the plan executor concurrency limit — 429 storm (~36-42 parallel API calls) #225 still open). The executor should be the only thing that generates; standalone resolvers should be thin wrappers over it.

3. Caption pipeline state lives in /tmp with hand-rolled path management. uniqueTempPath fixed the Date.now() collision, but the deeper problem is that renderCaptions/mergeAssFiles communicate through temp files at all. In-memory ASS structures (parse once, transform, serialize once at burn time) would delete the entire class of file-collision/ordering bugs — and most of merge-ass.ts.

   • Related: remaining Date.now()-only temp paths in packshot.ts, slider.ts, swipe.ts, text-measure.ts, render.ts carry the same latent race, just outside the hot path today.

4. renderCaptions / resolveSrtContent is a 4-way branch stack (srt path / string src / AudioNode src / speech src / native words / whisper fallback) that grew a 5th branch for memoized AudioNode transcripts. Captions should consume ONE input shape (words + text), with a small resolver in front mapping every src type to it.

5. AudioNode is doing too much (audio-element.ts): element identity, thenable semantics, meta write-back, per-option memoization maps, refine policy, gateway-model fallback. The analysis layer (transcribe/silence/ranges + memo) wants to be its own module with AudioNode as a thin facade.

6. HoistedCaption/window plumbing (flatten.ts → render.ts → merge-ass.ts) passes trim windows positionally through 3 hops with ! indexing (flatten.hoistedCaptions[i]!). Fold window + offset + emoji shift into one per-clip caption descriptor computed in one place.

7. Idempotency-Key is client-side only. The API stores the key but (to verify) does not dedupe on it server-side — see async components bypass the plan executor concurrency limit — 429 storm (~36-42 parallel API calls) #225. API-side enforcement is the real guarantee; the SDK header is best-effort.

Suggested shape (rough)

• captions/ module: words → cues → timeline pure pipeline (monotonize + refine + group + window-remap as composable steps), in-memory ASS model, single serializer; renderers only call it
• generation: executor-only (executePlan), standalone resolvers delegate; resolveLazy-phase generation goes through the same semaphore/dedup (async components bypass the plan executor concurrency limit — 429 storm (~36-42 parallel API calls) #225)
• audio analysis: primitives/analysis.ts owning transcribe/silence/refine/memo; AudioNode delegates
• kill temp-file IPC between caption stages; uniqueTempPath everywhere a real file is still needed (packshot/slider/swipe/text-measure/render)

References

• PR refactor: computation-graph IR, render lifecycle, renderers split #223 (compile/IR/executor + audio nodes), PR fix: caption timing in trimmed clips, in-flight resolve dedup, idempotency keys #226 (ep5 fix series: transformCue, monotonize, dedup, idempotency, refine, uniqueTempPath)
• async components bypass the plan executor concurrency limit — 429 storm (~36-42 parallel API calls) #225 (rate-limit root cause; semaphore + API-side idempotency follow-ups)
• Verified artifacts: templates/output/formula-care-ep5/ (before/after frames, ASS snapshots per fix)

Non-goal: changing public syntax (vid.audio, speechRange, <Captions src>, audio: "native") — all of it stays; this is internal consolidation.