Skip to content

Commit 9ef68ee

Browse files
committed
merge: origin/staging (kernel cutover + analysis redesign + BPM) into local line (audio-intelligence + Suno + drop placement)
# Conflicts: # docs/Features/Kernel-Client.md # src/architecture/getStateAccessPolicy.ts # src/components/panels/properties/AnalysisTab.tsx # src/components/panels/properties/analysisWorkspace/AnalysisSceneBlob.tsx # src/components/panels/properties/analysisWorkspace/AnalysisSceneList.tsx # src/components/panels/properties/analysisWorkspace/AnalysisWorkspace.css # src/components/panels/properties/analysisWorkspace/AnalysisWorkspace.tsx # src/components/panels/properties/analysisWorkspace/analysisSceneListModel.ts # src/components/panels/properties/analysisWorkspace/analysisTranscriptChunks.ts # src/services/aiTools/handlers/analysis.ts # src/services/kernelClient/transcriptMoments.ts # src/services/kernelClient/types.ts # src/services/transcription/clipTranscriptResolver.ts # tests/unit/AnalysisSceneList.test.tsx # tests/unit/AnalysisWorkspace.test.tsx # tests/unit/analysisTranscriptChunks.test.ts
2 parents 5f77ac8 + f1591fa commit 9ef68ee

72 files changed

Lines changed: 6344 additions & 171 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/Features/AI-Integration.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,13 @@ Model-powered editing with the shared editor tool catalog, Kie.ai Cloud or local
2525

2626
## FlashBoard Chat
2727

28+
> **Kernel-first routing:** when an agent-kernel service is configured
29+
> (`ms.kernel.url` + `ms.kernel.token`), mechanical editing requests are
30+
> compiled, simulated, and verified by the kernel and executed locally in
31+
> one undo group before any provider call happens; everything else falls
32+
> back to the chat loop described below. See
33+
> `docs/Features/Kernel-Client.md` for the full flow and fallback matrix.
34+
2835
### Location
2936
- Floating FlashBoard composer chat mode
3037

@@ -298,7 +305,14 @@ See [MuScriptor Music-to-MIDI](./MuScriptor.md) for the complete runtime, mappin
298305

299306
## AI Editor Tools
300307

301-
### 86 Tools across 16 Exported Definition Groups
308+
### Tool Registry (parity-gated)
309+
310+
The registry currently holds 133 tool definitions, of which 97 are exposed
311+
to the chat model (policy-gated; the dev bridge additionally sees
312+
diagnostics-only tools). Exact counts are locked by
313+
`tests/unit/aiToolRegistryParity.test.ts` and mirrored 1:1 by the agent
314+
kernel's manifest registry — a tool that is only partially registered
315+
(definition, policy, or handler missing) fails the build.
302316

303317
> **Note:** Kie.ai requests are capped at 128 model-exposed tools. Bridge-only diagnostics can exist as handler/policy entries without being exposed to the chat model. Gaussian Splat tool definitions also exist in `src/services/aiTools/definitions/gaussian.ts`, but that file is not currently exported through `AI_TOOLS`.
304318

‎docs/Features/FlashBoard.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ FlashBoard is not a separate model backend. It is a composer/runtime layer on to
1717
- `cloud` for hosted Kling 3.0, hosted Seedance 2.0 / Fast, hosted Nano Banana 2, hosted ElevenLabs speech, and hosted Suno music
1818
- `elevenlabs` for lower-level development compatibility; the Media generator tray uses hosted speech
1919
- `suno` for lower-level development compatibility; the Media generator tray uses hosted Kie.ai-backed Suno generation
20-
- compact chat for prompt discussion and editor actions through hosted Kie.ai in production, with a personal Kie.ai key or local Lemonade available in non-production development; requests include the Media-chat system prompt, current timeline summary, and callable AI tools routed through the shared chat dispatcher
20+
- compact chat for prompt discussion and editor actions through hosted Kie.ai in production, with a personal Kie.ai key or local Lemonade available in non-production development; requests include the Media-chat system prompt, current timeline summary, and callable AI tools routed through the shared chat dispatcher. When an agent-kernel service is configured, mechanical editing requests are handled kernel-first before any provider call (see `docs/Features/Kernel-Client.md`)
2121

2222
The Media Panel generator tray offers two collapsed launch actions: `Generate` opens the normal generation prompt, while `Chat` opens a separate chat prompt window with provider/model/temperature controls and reasoning effort for Kie.ai GPT models. The chat model menu includes GPT 5.6 Luna/Terra/Sol, GPT 5.5/5.4, Claude Opus 4.8, Claude Sonnet 5, and chat-only Claude Fable 5. The generator tray uses hosted Cloud entries for Kie.ai-backed media; local/BYO Kie.ai keys do not replace that generation path. `Generate` remains the only action that queues media generation.
2323

‎docs/Features/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,8 @@ The docs in this folder were re-audited against the current codebase and now tra
4242
|---|---|
4343
| [Timeline](./Timeline.md) | Tracks, clips, nested comps, markers, selection, and editing flow |
4444
| [Transition Compositions](./Transition-Compositions.md) | Mapped-v3 transition source layout, templates, parity, and explicit legacy upgrades |
45-
| [Timeline Rulers](./Timeline-Rulers.md) | Stacked multi-ruler lanes (time/timecode/frames/bars) via the **Rulers** checklist, TempoMap-driven bars+beats, per-composition persistence, and the click-to-select active-lane seam (#257) |
45+
| [Timeline Rulers](./Timeline-Rulers.md) | Stacked multi-ruler lanes (time/timecode/frames/bars/tempo) via the **Rulers** checklist, TempoMap-driven bars+beats, and per-composition persistence (#257) |
46+
| [Tempo And Metronome](./Tempo-And-Metronome.md) | Editable tempo track with BPM/meter flags and ramps, tempo-driven grid and snapping, MIDI content that follows tempo, and a metronome click (#299) |
4647
| [Slot Grid](./Slot-Grid.md) | 12x4 live grid overlay, slot clip trimming, layer triggering, and deck warmup behavior |
4748
| [Keyframes](./Keyframes.md) | Animated properties, effect params, masks, fades, easing, and visibility rules |
4849
| [Preview](./Preview.md) | Main preview, source monitor, output windows, RAM preview, and target routing |
Lines changed: 224 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,224 @@
1+
[Back to Docs](./README.md)
2+
3+
# Tempo, Time Signature And Metronome
4+
5+
**Status:** Implemented (issue #299). Builds on the read-only TempoMap shipped by
6+
[Timeline Rulers](./Timeline-Rulers.md) (#257) and makes it **editable**, makes
7+
tempo drive the **timeline grid and snapping**, makes **MIDI content follow tempo
8+
changes**, and adds a **metronome click**.
9+
10+
---
11+
12+
## What you can do
13+
14+
| Action | Where |
15+
|---|---|
16+
| Show the tempo track | **Rulers → Tempo** — a lane under Bars + Beats |
17+
| Add a tempo or meter change | Right-click the Tempo lane → **Add tempo change** / **Add time signature change** |
18+
| Change an existing one | Right-click its flag → **Change tempo** / **Change time signature** |
19+
| Ramp instead of jump | Right-click its flag → **Ramp from previous tempo** |
20+
| Move a change | Drag its flag (snaps to bars; **Alt** for free placement) |
21+
| Delete a change | Right-click its flag → **Delete** |
22+
| Bar/beat grid behind clips | Enable **Rulers → Bars + Beats** |
23+
| Grid resolution | **Rulers → Bars + Beats → `+`** — Bar / Beat / 1/8 / 1/16 / triplets |
24+
| Metronome | The metronome button right of **Rulers**; its caret opens volume and Every beat / Bars only |
25+
26+
The **project tempo** is the first flag, pinned at the start of the timeline. Its
27+
BPM and meter are editable; it cannot be moved, deleted, or made a ramp — there
28+
is no earlier tempo to glide from, and the map must never be empty.
29+
30+
The tempo lane also appears in the **piano roll**, above its Bars ruler, as a
31+
read-only mirror. It follows the same Rulers → Tempo toggle.
32+
33+
---
34+
35+
## The model
36+
37+
Three ideas carry the whole feature. Each one is a decision that was wrong at
38+
least once during implementation, so the reasoning matters more than the code.
39+
40+
### 1. BPM sets duration; the time signature only groups
41+
42+
A quarter note's duration comes from the BPM. The time signature says how beats
43+
are grouped into bars and which note value is counted — it changes no duration.
44+
45+
So **content is anchored to its quarter-note position**, not to its (bar, beat)
46+
address. Anchoring to bar/beat meant that switching 4/4 → 3/4 at the same tempo
47+
re-addressed every note and physically moved it (a note at 6 s slid to 5 s),
48+
leaving the bars ruler out of sync with the content under it. With quarters:
49+
50+
- a meter change moves **nothing**, for any meter, including 6/8 where the
51+
counted beat changes from a quarter to an eighth;
52+
- a tempo change moves content by the tempo ratio alone;
53+
- both at once move content by the tempo ratio only.
54+
55+
`remapAcrossMaps` in `src/timeline/tempo/tempoRemap.ts`.
56+
57+
### 2. MIDI is musical; media is linear
58+
59+
Changing the tempo rewrites MIDI clip windows, note starts/durations and the
60+
four CC automation lanes, so a melody written at 120 stays on its bars at 100.
61+
Video, audio and image clips keep their seconds — a film edit must not reflow
62+
because someone set a tempo. Both land in **one undo entry** with the tempo edit.
63+
64+
The test lives in one predicate, `trackFollowsTempo(track)`, which is the seam
65+
for a future per-track `timebase: 'musical' | 'linear'` flag.
66+
67+
Durations are never scaled by a factor: an end is remapped and the remapped
68+
start subtracted, which is the only correct answer across a tempo boundary.
69+
70+
### 3. A tempo mark is a musical object
71+
72+
A flag means "90 BPM **at bar 11**", not "at 40 seconds". Editing an earlier
73+
tempo — or turning this one into a ramp — changes how long the preceding interval
74+
takes, so the flag's **seconds** must move to keep its **bar**. Without that, a
75+
ramp flag placed on bar 11 drifts to bar 11.5.
76+
77+
Every edit therefore re-anchors: each event's quarter position is read in the map
78+
it was placed against, then the seconds are rebuilt under the new tempo profile.
79+
One exception — an **explicitly set position wins**: a flag you drag or insert
80+
lands exactly where you dropped it, or it would slide out from under the cursor.
81+
82+
`reanchorTempoEvents` in `src/timeline/tempo/tempoEdits.ts`.
83+
84+
---
85+
86+
## Ramps
87+
88+
A `ramp` event is *reached* by interpolation: the tempo glides linearly, in time,
89+
from the previous event's BPM to this one across the interval leading into it.
90+
A `jump` (the default) is an instant step.
91+
92+
This is not cosmetic. Beats stop accumulating at a constant rate, so
93+
`TempoMap.ts` integrates a linearly varying tempo: elapsed beats are **quadratic**
94+
in time, and the inverse (beat → seconds) solves that quadratic in closed form,
95+
so it stays exact rather than iterating.
96+
97+
60 → 120 BPM over 8 s covers **12 beats**, not 8 — the average tempo — which is
98+
why a ramp reaches bar 4 where a jump reaches bar 3. Beat lines get progressively
99+
closer through an accelerando and further apart through a ritardando.
100+
101+
Everything downstream inherits it for free: the grid, snapping, the MIDI remap
102+
and the metronome all read the same projection.
103+
104+
The lane shows a dashed sloped line across the ramped interval — rising for a
105+
speed-up, falling for a slow-down — plus a ↗/↘ arrow on the flag.
106+
107+
---
108+
109+
## Grid and snapping
110+
111+
**An enabled Bars + Beats ruler wins the grid.** Enabling the lane is already the
112+
user saying "I am working in bars"; requiring a second, invisible "active lane"
113+
selection on top of that was the seam #257 stored but never used, and it is
114+
retired. `activeRulerLaneId` now means only the lane highlight.
115+
116+
- Bars **replace** the time/frame grid rather than overlaying it.
117+
- Lines thin by pixel spacing using the **same thresholds as the ruler ticks**, so
118+
the grid and the ruler above it can never disagree about what exists at a
119+
given zoom: subdivisions drop first, then beats, then bars go to every 2nd/4th.
120+
- Snapping follows the identical rule, so you can only snap to a line you can see.
121+
122+
Grid snapping uses a **pixel-derived threshold** (10 px), not the fixed
123+
`SNAP_THRESHOLD_SECONDS` used for clip edges and the playhead. A 1/16 at 120 BPM
124+
is 0.125 s apart — narrower than that fixed window — so a seconds threshold would
125+
leave grid snapping permanently engaged with overlapping capture zones. Alt still
126+
bypasses snapping entirely.
127+
128+
Both snap paths are covered: `getSnappedPosition` (clip drags) and
129+
`resolveTimelineClipPointerTime` (tool-driven interactions).
130+
131+
---
132+
133+
## Metronome
134+
135+
A look-ahead scheduler (25 ms timer, 0.12 s window) built on the same pattern as
136+
`midiPlaybackScheduler`: a timeline↔AudioContext anchor, re-anchor on a >0.25 s
137+
seek, silence at non-1x speed, and a dedup set so a beat inside two consecutive
138+
windows fires once.
139+
140+
Beat times come from `iterateBarBeatLines`, so the click tracks meter, mid-window
141+
tempo changes and ramps with no extra math.
142+
143+
The voice is one oscillator plus a gain with a ~40 ms exponential decay —
144+
1000 Hz at full level on the downbeat, 800 Hz at 0.7 otherwise. No assets.
145+
146+
**The click never touches the master bus.** It owns a `GainNode` wired directly to
147+
`AudioContext.destination`; it shares the context via `ensureSharedContext()` but
148+
never registers a node route, so it cannot enter master metering, the master
149+
FX/limiter chain, or any master-bus tap. Export renders through a separate
150+
offline path, so a live-only node is excluded structurally; an `isExporting`
151+
guard sits on top of that.
152+
153+
Metronome settings (on/off, volume, beats vs bars) and the grid resolution are
154+
**per-user localStorage view state**, never project content. Lane visibility, the
155+
tempo map itself and the active lane are **project content** and persist per
156+
composition.
157+
158+
---
159+
160+
## Data model
161+
162+
```ts
163+
TempoEvent {
164+
id: string // stable identity for editing, dragging and React keys
165+
time: number // seconds; sorted ascending; the first event is pinned at 0
166+
bpm: number // clamped to [20, 999]
167+
numerator: number
168+
denominator: number // 1 | 2 | 4 | 8 | 16 | 32
169+
curve?: 'jump' | 'ramp' // absent reads as 'jump'
170+
}
171+
```
172+
173+
`id` and `curve` are optional in the durable project tier and backfilled on load
174+
by `normalizeRulerLaneState`, so projects saved before this feature open
175+
unchanged with no version bump.
176+
177+
The editable BPM range is **deliberately not** `MIN_TEMPO_BPM` / `MAX_TEMPO_BPM`
178+
from `services/audio/beatOnset/beatGridEstimation.ts` (60 / 200). Those are
179+
octave-folding bins for autocorrelation *detection*; reusing them would reject a
180+
40 BPM largo and a 240 BPM drum'n'bass track.
181+
182+
Invariants live in one pure module and are enforced on every path — at least one
183+
event, the first pinned at 0 and never a ramp, sorted and unique by time, unique
184+
ids, clamped values. Writing onto an occupied bar replaces the event there.
185+
186+
---
187+
188+
## Where the code lives
189+
190+
| Piece | File |
191+
|---|---|
192+
| Tempo projection (bars, beats, quarters, ramps) | `src/timeline/tempo/TempoMap.ts` |
193+
| Editing invariants + musical re-anchoring | `src/timeline/tempo/tempoEdits.ts` |
194+
| Content remap (MIDI follows tempo) | `src/timeline/tempo/tempoRemap.ts` |
195+
| Grid geometry + snap candidates | `src/timeline/tempo/barsGrid.ts` |
196+
| Store actions (history-aware) | `src/stores/timeline/tempoSlice.ts` |
197+
| Tempo lane UI | `src/components/timeline/components/TempoRulerLane.tsx` |
198+
| Body grid canvas | `src/components/timeline/components/TimelineTrackGridCanvas.tsx` |
199+
| Metronome scheduler / voice | `src/services/audio/metronomeScheduler.ts`, `src/engine/audio/metronomeVoice.ts` |
200+
| Toolbar control | `src/components/timeline/MetronomeButton.tsx` |
201+
202+
Tempo edits are **content**, so every action captures a history snapshot — after
203+
the mutation, matching the store's post-state model. Ruler lane toggles remain
204+
view state and stay out of undo.
205+
206+
---
207+
208+
## Not included
209+
210+
- **Count-in.** A real pre-count sounds the click for N bars while content stays
211+
silent, which needs a new transport phase rather than a playhead roll-back;
212+
`play()` has nowhere for "running but not advancing". Deferred rather than
213+
bending playback for it.
214+
- **A separate time-signature track.** Tempo and meter stay one event type; each
215+
event sets both.
216+
- **Per-track timebase flag.** The predicate seam exists, the schema field
217+
does not.
218+
- **Tap tempo** and **detect tempo from an audio clip** —
219+
`beatGridEstimation.ts` already estimates BPM from audio, so wiring "set the
220+
project tempo from this clip" is a cheap follow-up now that editing exists.
221+
- **Ingesting a tempo map from an imported MIDI file.**
222+
- **Audio time-stretching to follow tempo** — media stays linear by design.
223+
- **Piano-roll note snapping.** Notes are free-placed by decision (#182); an
224+
optional snap-to-subdivision is an open question.

‎docs/Features/Timeline-Rulers.md‎

Lines changed: 24 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,25 @@
33
# Multi-Ruler Infrastructure (Integration Plan)
44

55
**Status:** Implemented (Packets 1–7). The single timeline ruler is now a stack of
6-
coexisting ruler lanes (Time / Timecode / Frames / Bars+Beats), chosen from the
7-
**Rulers** checklist (left of the View dropdown) and persisted per composition.
8-
Bars+Beats is driven by a TempoMap (constant 4/4 @ 60 BPM today, list-of-events
9-
ready). Clicking a lane sets the active lane (highlighted) — the seam a future grid
10-
will snap to. This phase ships **no grid/snap behavior** yet. Issue: #257.
6+
coexisting ruler lanes (Time / Timecode / Frames / Bars+Beats / Tempo), chosen from
7+
the **Rulers** checklist (left of the View dropdown) and persisted per composition.
8+
Bars+Beats is driven by a TempoMap. Issue: #257.
9+
10+
> **Superseded in part by issue #299** — see
11+
> [Tempo And Metronome](./Tempo-And-Metronome.md). That work made the TempoMap
12+
> **editable** (a Tempo lane with draggable BPM/meter flags and tempo ramps), gave
13+
> the body grid and snapping real bar/beat behaviour, and added a metronome. Two
14+
> statements below are no longer true:
15+
>
16+
> - *"ships no grid/snap behavior"* — an enabled **Bars + Beats** lane now drives
17+
> the grid behind the clips and clip snapping.
18+
> - *"clicking a lane sets the active lane — the seam a future grid will snap to"*
19+
> — that seam was **retired**. The grid keys off the bars lane being *enabled*,
20+
> not off `activeRulerLaneId`, which now means only the lane highlight. Requiring
21+
> a second, invisible selection to get a matching grid was the wrong UX.
22+
>
23+
> The lane stack, the TempoMap type, the pure tick generators and the
24+
> per-composition persistence described below are unchanged and still current.
1125
1226
Each `### Packet N` section below is annotated with a ✅ "Implemented" summary of
1327
what actually shipped, followed by the original spec for reference.
@@ -38,9 +52,11 @@ per-composition. Bars+Beats is driven by a **TempoMap** that is a constant
3852
4/4 @ 60 BPM today but list-of-events-ready for future tempo / time-signature
3953
changes at arbitrary points.
4054

41-
This phase ships **no grid/snap behavior** — it only stores an
42-
`activeRulerLaneId` so a future grid knows which lane is authoritative ("snap to
43-
the one clicked").
55+
This phase shipped **no grid/snap behavior** — it only stored an
56+
`activeRulerLaneId` so a future grid would know which lane was authoritative
57+
("snap to the one clicked"). Issue #299 delivered the grid and snapping but
58+
**dropped that seam**: an enabled Bars+Beats lane wins the grid, with no separate
59+
active-lane selection. See [Tempo And Metronome](./Tempo-And-Metronome.md).
4460

4561
## Background: how professional DAWs do it
4662

0 commit comments

Comments
 (0)