Repository navigation
Plan the music with the animation and score the BGM to its timing #42
Description
Goal
motion-render-pipeline-ja.mp4
Claude plans how the music and the animation play together, and generates a BGM that follows the animation's timing at second-level or finer precision.
The music and the animation only line up by chance today. In the v1 promo, the Enter press was aligned to the BGM lift by hand: measure the track, then move the animation.
Principle: no approval gates
Claude runs every step below on its own, in the background, and never stops to ask for approval.
- Theme only: Claude sets the directing intent itself, deciding where to build tension and where to release it.
- Intent given (for example, "break just before the door opens, drop the moment it opens"): Claude follows it and works out the details.
- The user asks to refine the intent in the session: only then does Claude develop the storyboard together with them.
Every intermediate artifact is kept as a file in the work directory, so Claude can show it when asked ("show me the storyboard").
Pipeline
- Storyboard: what each scene says and shows.
- Animatic: when each motion happens within a scene, in seconds.
- Animation: the HTML film, built from the animatic.
- Cue sheet: built from the film's actual
film.attimes, not from the plan. Each cue has a time, a type (hit, break, drop, build, sting, ...) and the intent in words. - BGM: a Lyria 3.5 prompt built from the cue sheet, with
[mm:ss - mm:ss]sections, a BPM and a per-section intensity. - Alignment: Claude detects the beats and section changes in the generated track, measures each cue's offset, and corrects it. A correction is a shift, a trim, a small time-stretch, or a mix-side edit such as a stop-and-resume break.
Acceptance criteria
- From a theme, or a theme plus one intent sentence, the run goes to the MP4 without stopping to ask.
- No new user-facing options. Claude decides, and the reference docs hold the toolbox and the criteria for choosing.
- A check reports, for every cue, the measured offset between the cue time and the musical event.
- After alignment, every cue lands within 100 ms of its musical event, verified on the built MP4.
- The narration stays intelligible: breaks and drops sit at scene boundaries or between sentences.
Open questions
- How precisely Lyria 3.5 follows timestamps is untested. Its timestamps are in whole seconds, so sub-second precision depends on two things:
- a bar-grid design: fix the BPM first and put the key moments on downbeats;
- alignment after the track is generated.
- Beat detection needs a tool. motion-render depends only on ffmpeg today, and a new dependency adds an install step for every user.
- If timestamp prompting falls short for breaks, Lyria RealTime is an alternative: it has
mute_drums, a fixedbpmand live prompt steering. It is an experimental model and instrumental only.
References
- Lyria 3.5 timing and structure: https://ai.google.dev/gemini-api/docs/music-generation#timing
- Lyria prompt guide (detailed structure, intensity): https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/music/music-gen-prompt-guide
- Lyria RealTime controls: https://ai.google.dev/gemini-api/docs/realtime-music-generation#controls
Activity
umiyosh commentedon Nov 5, 2026
Handoff from #44 (faster runs)
The work for #44 (PR #45) changes how an html film is made. The steps this issue adds should plug into that shape, so they get the same speed-ups instead of adding a long serial step.
What is there now
<work>/plan.md— the storyboard and timed plan, written once before any scene. Per scene: what it shows, the parts it uses, every motion with the seconds passed tofilm.at, the sound effects, and the state of shared elements at the scene's start and end. Scene writers treat these seconds as fixed.film/scenes/NN-name.html, written at once by subagents, one per two scenes (at most 7), fromplan.mdand the sharedindex.html, put in at<!-- scenes -->when the page opens.motion-render check <film> <file>checks one file's scenes alone.prompt+durationonly, so they are made fromplan.mdand moving anatlater costs nothing.motion-render times <work>— a per-step table from<work>/steps.jsonl(each command's span, and Claude's time between them).references/pipeline.md— which step reads and writes what and who waits for it, plus the rules for adding a step.How this issue's steps map onto it
plan.md. Do not add a separate file.plan.mdalready are the animatic. Add the musical intent per scene there (for example acues:line: time, type, intent).film.cue(scene, sec, type, intent)that returnsfilm.at(scene, sec)and records the cue, so a command can dump the cue sheet from the page.plan.md, start Lyria right afterplan.md, and let step 6 absorb the drift between planned and actual seconds.scripts/steps.ts, and add it toBACKGROUNDonly if nothing waits for it.Measuring
Run the same film before and after, and put the
motion-render times <work>table in the PR. Also report the number of Lyria generations, since regenerating the BGM is the step most likely to grow the run.