Skip to content

Integrate RES audio updates on the audio sigma schedule - #7

Open
morluto wants to merge 1 commit into
antirez:mainfrom
morluto:agent/audio-res-native-schedule
Open

Integrate RES audio updates on the audio sigma schedule#7
morluto wants to merge 1 commit into
antirez:mainfrom
morluto:agent/audio-res-native-schedule

Conversation

@morluto

@morluto morluto commented Aug 11, 2026

Copy link
Copy Markdown

Fixes #6.

Problem

The RES denoiser currently expresses the audio update in the video-sigma
coordinate:

D_v = audio_latent +
    sigma_video * time_shift_slope(sigma_video) * audio_velocity;

It then passes that estimate to h3_res_step with sigmas.video.

This is a valid continuous-time reparameterization, but it makes the denoised
forcing nonlinear even for constant audio velocity. At sparse step counts,
the RES discretization consequently has substantially more endpoint error
than the native audio-sigma formulation.

Change

This PR keeps each velocity conversion paired with the schedule used by RES:

flowchart LR
    VV["Video velocity"] --> VD["x + sigma_video * v"]
    VD --> VR["RES on sigmas.video"]

    AV["Audio velocity"] --> AD["x + sigma_audio * v"]
    AD --> AR["RES on sigmas.audio"]
Loading

The shared velocity-step helper constructs the denoised estimate using the
same sigma grid passed to RES. The audio wrapper owns the schedule.audio
invariant, preventing the coefficient and integration grid from drifting
apart again.

h3_dit_denoise now applies video velocity on sigmas.video and audio
velocity on sigmas.audio. The Euler paths, schedule construction, and DiT
forward pass are unchanged.

Regression evidence

For constant velocity, $D_a=x+\sigma_a v_a$ is constant, giving an exact
solver oracle. A scalar sample starting at 1.0 with velocity 0.5 has the
exact endpoint 1.5 at sigma zero.

Steps Existing formulation Native audio formulation Expected Existing endpoint error
4 1.843430758 1.500000000 1.5 22.8954%
7 1.620174885 1.500000000 1.5 8.0117%
20 1.494925022 1.500000238 1.5 0.3383%

The regression covers all three step counts and accepts an absolute endpoint
error below 1e-6.

A mutation check restoring the previous coefficient and video grid fails at
four steps:

result=1.843430758
absolute error=0.343430758

Restoring the native audio formulation passes all three cases. These numbers
are deterministic solver-state measurements; they do not represent decoded
audio quality percentages.

Validation

  • Constant-velocity solver regression passes at 4, 7, and 20 steps.
  • Previous-behavior mutation fails the regression.
  • Strict Clang syntax checks pass.
  • git diff --check passes.

@morluto
morluto marked this pull request as ready for review August 11, 2026 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RES audio sampling accumulates low-step error on the video sigma grid

1 participant