Skip to content

Audit fixes, offline HyperFrames runtimes, ComfyUI open-weight workflows - #942

Closed
sungmen999 wants to merge 31 commits into
OpenMontageInc:mainfrom
sungmen999:deep-audit-2026-11-11
Closed

sungmen999 wants to merge 31 commits into
OpenMontageInc:mainfrom
sungmen999:deep-audit-2026-11-11

Conversation

@sungmen999

Copy link
Copy Markdown

Summary

This PR has 31 commits from a deep audit of the pipeline and two feature branches. Grouped by area:

engine         lib/checkpoint.py enforces the decision-log, review, gate and budget contracts
tools          tool schemas, estimates and checks match their code
               audio_mixer tracks take offset_seconds and loop
               composition_validator also checks images[] and backgroundVideo
hyperframes    GSAP and three.js come from pinned npm installs, so renders work offline
               music fades are written as volume tweens
               an asset is restaged whenever its content changes (the size check missed this)
threejs_world  index.html has an import map for `three`, so the 3D world now renders
character      the scene is sized to the profile's canvas; figures are centred and on the ground
remotion       Explainer music ducks under narration
               titles split by grapheme
               the last cut holds to the end
               CalloutBox renders lines and bullets
comfyui / tts  bundled MiniMax H3 (local) and Qwen-Image-2.1 workflows
               comfyui_tts speaks through VoxCPM2
backlot        playback and scroll survive re-renders; project state carries an ETag
skills / docs  pipeline skills and manifests match the schemas and code; AGENT_GUIDE.md is split by section
ci / qa        CI runs Python 3.10 and 3.11 with Playwright and type-checks remotion
               the QA scripts exit 1 when a step fails

remotion-composer/package.json gains two dependencies, both installed by make setup:

  • gsap 3.14.2, under GSAP's Standard "no charge" license
  • three 0.181.2, under MIT

tools/gsap_runtime.py and tools/three_runtime.py copy these into each workspace. When they are not installed, both fall back to the pinned CDN URLs.

Evidence

Full suite after: 4154 passed, 26 skipped, 0 failed, and make lint passes. The suite ran against an ffmpeg build that has libass, drawtext and libvorbis.

Each fix below was checked with a real render:

  • threejs_world
    • Before: the page logged Failed to resolve module specifier "three". The canvas stayed black on "BUILDING WORLD GRAPH…", yet the render exited 0.
    • After: terrain, landmarks and the camera path render. Two runs produced the same frames: one with all outbound traffic except localhost denied, one online through the CDN fallback.
  • GSAP offline
    • Before: with network denied, the page logged Failed to download CDN script … gsap.min.js and Cannot read properties of null (reading 'timeline'), and the render exited 1.
    • After: the render completes and the tweens run.
  • Character scene
    • Before: the 1280x720 scene sat in the top-left of the 1920x1080 canvas, and the figures' feet were cut off. A vertical re-render then reused the stale landscape copy, which had the same byte size, so the lower half was black.
    • After: video_compose renders at the default 1920x1080 and at youtube_shorts 1080x1920 both fill the frame.
  • Explainer ducking: the music band drops 9.1 dB under speech. With ducking off it stays flat.
  • HyperFrames music fade: the music measures −49.3 dB at the start, −31.5 dB in the middle and −49.1 dB at the end.

Merge Danger

Door: two-way

This PR changes only code, tests, skills and docs, with no data migration. Reverting it restores the old behaviour.

Blast Radius: wide

  • It touches the checkpoint engine. Stricter gates can block runs that used to pass without notice.
  • It changes the Remotion, HyperFrames and Three.js render paths.
  • It changes the skill text agents follow.
  • make setup installs two more npm packages.

🤖 Generated with Claude Code

sungmen999 and others added 30 commits November 10, 2026 17:50
comfyui_video runs model_family "minimax_h3_local" from bundled T2V/I2V
API graphs (base mode, 20 steps, native audio), and comfyui_image gains
model_family "qwen_image_2.1" (40 steps, CFG 1) next to the flux2 default.
The model stacks live in tools/_comfyui/metadata.py and contract tests
check the graphs against them. docs/vast-ai-open-weight-stack.md is the
install guide for one rented GPU.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…fault

A ComfyUI server set up for MiniMax H3 and Qwen-Image-2.1 has no WAN 2.2
or FLUX 2 files. comfyui_video judged readiness on WAN alone, so preflight
listed video generation as unavailable and video_selector dropped the tool,
and a call without model_family ran the missing default and failed.

- comfyui_video: readiness per bundled family; an operation is ready when
  WAN or H3 can run it. get_info() reports model_family_statuses and the
  default_model_family a call without model_family will run.
- comfyui_video / comfyui_image: a call that names no model_family runs
  the first installed family (wan2.2 / flux2 first, as before). An explicit
  family is never swapped. Results carry model_family and
  model_family_auto_selected.
- ComfyUIClient.list_models(): a complete listing is reused for 10 s, so a
  preflight costs five /object_info requests per tool instead of fifteen
  or more (12 s -> 3 s for comfyui_video.get_info() over an SSH tunnel).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A machine set up for local image, video and music had no narration: every
TTS provider needed an API key or a separate install. comfyui_tts runs the
VoxCPM_TTS node of wildminder/ComfyUI-VoxCPM (model and nodes Apache-2.0)
on the same server, so tts_selector finds a provider there.

- Voice by local reference clip (uploaded and cloned), by a voice file in
  the server's input folder (with <voice>.reference.txt as transcript), or
  by a plain-words description. A call with only text gets a default
  narrator and a fixed seed, so sections of one script match.
- The server saves FLAC; any other output extension is converted locally
  with ffmpeg, so a .wav path never holds FLAC bytes.
- Status is DEGRADED when the server lacks the node pack, and execute()
  then names the pack to install.
- ComfyUIClient gains upload_audio(), input_file_text() and
  list_input_audio() (both COMBO formats of /object_info).

The node inputs in the two bundled graphs were read from /object_info of
ComfyUI 0.44.0 with the node pack installed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`make setup` installs piper-tts into the project venv, but the tool looked
for `piper` on PATH only, so a preflight run with `.venv/bin/python` (the
venv not activated) reported no offline TTS. It also reported AVAILABLE
with no voice model downloaded, and the documented download command
(`piper --download-dir`) does not exist in piper-tts 1.8.

- The command is taken from PATH or from the interpreter's folder.
- Voices live in ~/.piper/models (PIPER_DATA_DIR overrides) and are passed
  with --data-dir. Status is DEGRADED when the command is there but no
  voice is; get_info() lists installed_voices.
- With no model given, the default voice is used when installed, else the
  first installed one. An unknown voice fails with the download command
  (`python -m piper.download_voices <voice> --data-dir <dir>`).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
tests/tools/test_comfyui_live_gpu.py generates one small piece of media per
ComfyUI-backed capability against the server behind COMFYUI_SERVER_URL:
image and video through the selectors without naming a model, image to
video from that image, music, speech (designed voice, cloned server voice,
cloned uploaded clip), and the transcriber reading the speech back. It is
marked live_api, so it only runs with OPENMONTAGE_ALLOW_NETWORK=1.

First run, 2026-11-10, rented RTX 5090, ComfyUI 0.44.0: 12 passed in 8 min
40 s. The MiniMax H3 image-to-video graph ran for the first time and was
accepted as bundled. The docs, the comfyui skill and the two runtime
estimates now carry these measured numbers with their settings, in place
of "not yet run" and a third-party extrapolation.

The guide gains the VoxCPM2 install steps for comfyui_tts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The session network guard lets loopback through, and a ComfyUI server is
loopback: on the developer's machine, or a rented GPU behind an SSH tunnel.
tests/contracts/test_phase2_contracts.py::test_image_selector_no_provider
calls the real image selector with no mock, so with such a server up, and
a model installed that a call without model_family can run, every suite
run generated a real image there (seen on 2026-11-10: four images on a
rented GPU, two of them in the middle of someone else's job).

The guard now also refuses loopback connections to a ComfyUI: port 8188,
and any loopback port a COMFYUI_*_SERVER_URL variable names when the
session starts. ComfyUI tools then see "no server". OPENMONTAGE_ALLOW_NETWORK=1
lifts it, as for live_api tests.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The autoharness plugin writes per-session locks, intent queues and run
records to .claude/autoharness/, and a ledger and sidecar next to each
skill it lands. They are local runtime state. The landed skills under
.claude/skills/ stay tracked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HeroTitle rendered each character as its own flex item, so a title wider
than the frame (common on 9:16 shorts) could wrap in the middle of a word,
e.g. "painstak / ingly". Characters are now grouped into one nowrap span per
word, and the space between words comes from the flex column gap.

Each character keeps its index in the full title, so the stagger timing and
the accent on the first eight characters are unchanged.

Checked with Remotion stills of the HeroTitle composition at 1920x1080,
frame 150: the default single-line title renders byte-identical to before;
a title with a long word on the line edge now wraps between words.
`tsc --noEmit` passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit df658bce892de5c2ab6d59f3f6230c39d1a01fd1)
A calm cinematic playbook (identity category: cinematic) for slow,
quiet content: calm palette and music mood, clear conversational voice.
It loads through styles/playbook_loader and passes the playbook tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A prompt for a later session: list what must be installed on top of the
vastai-comfyui toolkit so one rented GPU runs the whole open-weight
stack. It stops for the user's decisions before any edit and never
rents a GPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Decision log has one source, projects/<id>/decision_log.json. A stray
  artifacts/decision_log.json is folded in. Re-using a decision_id with a
  different `selected` raises. Writes are atomic.
- A completed or awaiting_human checkpoint without `review` is logged and
  listed by get_unreviewed_stages(). get_stale_stages() is advisory.
- Gated non-canonical stages need their produced artifact. checkpoint.policy
  takes effect (manual_all gates every stage). An unknown pipeline_type fails
  loudly instead of falling back to the canonical stage list.
- Tool spend is synced into the project cost ledger at each checkpoint.
  resolve_budget_usd() is the one budget rule (approved amount, else the
  manifest default capped by config.yaml). cap mode refuses a paid call that
  does not fit. Rented GPU time can be recorded with
  `python -m tools.cost_tracker record`.
- One tool module that fails to import no longer takes discovery down.
  provider_menu_summary(compact=True) groups setup offers.
- A style_playbook written with a checkpoint is recorded on project.json.
- One .env parser. Dead config keys (paths.*, checkpoint.storage_dir) removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Input schemas match what execute() reads, across the registry (new
  contract test).
- jimeng_video prices a `duration` request (a 10 s clip was estimated at the
  5 s price). music_gen returns its error instead of raising ValueError.
- composition_validator resolves project-relative narration paths.
- video_compose final review reports music_present only when the
  composition declares a music bed.
- The HyperFrames scaffold text card stacks the subtitle under the title.
- FFmpeg feature detection (drawtext, libass, libvorbis) for tools and tests;
  showcase_card reports itself unavailable without drawtext.
- agent_skills pointers no longer name other vendors' skills (piper_tts,
  transcriber, audio_mixer).
- Hunyuan tests no longer sleep for real.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Hero and scene videos keep their time and mute state, and drawer scroll
  survives a board re-render.
- Project ids containing #, %, & or Thai resolve. The stage rail is keyboard
  operable, the modal is a dialog, and tap targets fit at 390 px.
- The state endpoint sends an ETag and answers 304.
- The board flags stages with no review and decision-log drift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- HeroTitle, ProductReveal and TitledVideo split text with splitGraphemes()
  (Intl.Segmenter with a fallback). split("") drew emoji as U+FFFD boxes and
  Thai vowels and tone marks on dotted circles.
- HeroTitle accents the whole first word, not the first 8 characters.
- Explainer holds the last cut through tailPaddingSeconds instead of ending
  on a second of bare background.
- Add an `npm run typecheck` script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AGENT_GUIDE.md is read before every reply. It now states every binding rule
and routes to docs/agent-guide/{preflight,decisions,planning,
workspace-and-contracts,layers}.md for the procedures, which moved over
unchanged. 49,611 B -> 24,778 B. A router test keeps headings that skills
cite present.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Director skills in all 12 pipelines: example artifacts validate; fields a
  closed schema forbids move under metadata; selected_concept is read
  through concept_options[]; edit sets render_runtime; Remotion-only cut
  fields go to artifacts/remotion_composition.json; the HyperFrames gate is
  described as the tool runs it.
- Executive-producer limits name the manifest orchestration fields instead
  of numbers; the budget row follows cost_tracker; talking-head gains
  max_wall_time_minutes.
- Superseded explainer, animation and cinematic idea-directors are pointer
  stubs to proposal-director.
- reviewer.md records the review on the checkpoint, at most two rounds.
- export_bundle in the animation, cinematic and character-animation publish
  stages.
- Contract tests for the EP limit source, manifest/skill tool consistency
  and the idea-director stubs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Python 3.10 and 3.11 matrix, with Chromium so tests/backlot UI tests run.
- A Node job runs the remotion-composer typecheck.
- `make lint` compiles every first-party module, not four files.
- playwright added to requirements-dev.txt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`audio.music.ducking` was accepted by the pipeline skills and ignored by the
Explainer, so music played at one flat level under the voice. lib/ducking.ts
derives speech intervals from the word captions (merged across pauses up to
0.6 s; without captions the whole video counts as speech) and lowers the
music gain to 0.35 with a 0.15 s attack and 0.4 s release. All four values
can be overridden.

Measured on a test render (narration 0-2 s, 220 Hz music bed): the music
band sits 9.1 dB lower while speech plays and returns to its level after;
with ducking off it is flat.

The explainer and animation edit, compose and executive-producer skills gain
`audio_mix_plan: "composition"` for this path; `premix` stays the choice
when there are no captions or the render is not Remotion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…roundVideo

video_compose stages `images[]` and `backgroundVideo` into the Remotion
public dir, but the validator only looked at `source` and
`backgroundImage`. A missing anime_scene frame passed validation and showed
up as a black frame in the render. Both are now checked; the animation
compose-director no longer tells agents to check images by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SCENE_TYPES.md describes `callout` as a boxed message with bullets, but
CalloutBox printed `text` as one run: newlines collapsed and "- item"
lines merged into one line on the audit render. Each line now renders on its
own; lines starting with "- ", "* " or a bullet glyph get an accent-coloured
bullet. A single-line callout renders as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scaffold_workspace read the music bed's fade_in_seconds / fade_out_seconds
and dropped them, so the bed started and stopped at full level. They are now
GSAP volume tweens on the root timeline (data-volume stays the baseline),
the form hyperframes-core documents.

Measured on a 6 s render (fade in 1.5 s, out 2 s, volume 0.6): -49.3 dB in
the first 0.3 s, -31.5 dB mid-clip, -49.1 dB in the last 0.3 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A music bed with a quiet intro or shorter than the video needed a separate
FFmpeg call before full_mix: audio_mixer could only delay a track and pad or
trim the whole mix. Tracks now take `offset_seconds` (input seek) and
`loop` (-stream_loop, cut to target_duration - start_seconds, which is
required). A track's fade-out lands at the end it actually plays to. An
in-point past the end of the file is an error.

The explainer and animation compose-directors drop the FFmpeg pre-step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The music plan offers royalty-free stock (music_search tools such as
pixabay_music), but music_source.source_type had no value for it, so the
explainer proposal-director recorded it through a metadata workaround. Add
`stock_search`; `provider` names the search tool.

Also correct two video_compose docstrings: a missing renderer_family blocks
the render (the docstring said warn), and render routes by the locked
render_runtime with no silent FFmpeg fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/qa/test_04..08 run their whole workload at import time and report by
printing. pytest imported them while collecting the main suite (about 17 s
on every run) and their failures could never fail anything; run as scripts
they always exited 0. A conftest now keeps them out of collection (0.02 s),
and each script exits 1 when a tool step or check fails.

QA_PLAN.md listed scripts that do not exist (test_01-03), a Windows path,
and no test_09; it now matches the folder and says how to run each script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ARCHITECTURE.md: the ledger holds spend only; $0 calls stay in
  events.jsonl and a run with no paid call and no budget writes no
  cost_log.json (the end-to-end audit run read the missing file as a gap).
- AGENT_GUIDE.md: framework-smoke is `beta` in its manifest (the schema has
  no `test` value) and has no director skills; the table now says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fallback path compared against a literal U+200D, invisible in review
and easy to lose in an edit. Use "\u200d". Grapheme contract tests (both
the Intl.Segmenter and the fallback path) still pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HyperFrames scaffolds, the character preview and the Three.js world loaded
GSAP from cdn.jsdelivr.net (3.14.2, and 3.12.5 in the character preview), so
a render without network failed: the HyperFrames compiler could not fetch
the script and every timeline registration threw.

remotion-composer/package.json now pins gsap 3.14.2, which `make setup`
already installs. tools/gsap_runtime.stage_gsap copies that build next to
the generated HTML and returns its src; without a local install it returns
the pinned CDN URL as before. The scaffold result reports it as gsap_src.

The character sub-composition no longer loads GSAP itself: the runtime
clones only its <template> into a host index.html that already loads it.

Verified with renders under sandbox-exec denying all outbound traffic except
localhost: the CDN workspace fails, the local one renders with its tweens,
and the character scene and its Playwright preview animate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The vendored .agents/skills/hyperframes-* examples load GSAP from a CDN.
skills/core/hyperframes.md and skills/meta/bespoke-composition.md now say
to use the gsap.min.js that scaffold_workspace copies into the workspace,
and that sub-compositions use the host's GSAP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… install

world-runtime.js imported GLTFLoader.js straight from the CDN. That module
imports 'three' by bare specifier and the page had no import map, so the
browser threw "Failed to resolve module specifier 'three'", the runtime
module never ran, and every render was the HTML overlay over a black canvas
stuck on "BUILDING WORLD GRAPH…" while the render still exited 0.

The runtime now imports "three" and "three/addons/…", and index.html gets
an import map for both. remotion-composer/package.json pins three 0.181.2,
which `make setup` installs; tools/three_runtime.stage_three copies the
modules the runtime needs, plus every module they import by relative path,
into vendor/three/ in the workspace. Without a local install the map points
at the pinned CDN.

Verified by rendering a 12 s world twice. The first ran with local three
under sandbox-exec, which denied all outbound traffic except localhost. The
second ran online through the CDN map. Both showed terrain, landmarks and
the camera path, and neither logged a page error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scaffold skipped copying an asset when the staged file already had the
same size as the source. video_compose renders into one workspace per
project (<project>/hyperframes), so re-rendering after a change that kept
the size reused the stale copy. A character composition re-rendered for
youtube_shorts differs from the landscape one only by swapping "1920" and
"1080". The vertical render therefore mounted the 1920x1080 scene: it
covered the top 1080 px, with black below.

Cuts, narration and music now go through _stage_copy. It uses filecmp's
shallow check: an unchanged file, whose (size, mtime) copy2 preserved, is
not read again, and any other file is compared byte by byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
character_rig_renderer declared its composition as a fixed 1280x720 box
with a 920px SVG. The scaffold that mounts it sizes the canvas from the
media profile video_compose forwards, which is 1920x1080 by default, and no
profile is 1280x720. The scene therefore sat in the top-left corner, and
because the SVG was taller than the box, the figures' feet were cut off.

The renderer now takes the same `profile` as video_compose and resolves it
with the scaffold's own _resolve_dimensions. The composition root declares
that size and fills its slot, and the SVG fits its square viewBox into 90%
of the slot. The result reports the size as `canvas`. The character
compose skill says to pass one profile to both tools.

Several figures were also scaled about the SVG origin, so they rose off
the ground, and they were spaced from x=110, so the group sat left of
centre and the first figure's arm left the viewBox. They are now spread
evenly about the centre and scaled about their feet. The placement sits on
an outer group, because GSAP tweens the .character group's own transform.

Verified with video_compose renders of two figures at the default
1920x1080 and at youtube_shorts 1080x1920. Both fill the frame, the
figures stand on the ground, and both are fully in view.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant