Repository navigation
Audit fixes, offline HyperFrames runtimes, ComfyUI open-weight workflows - #942
Closed
sungmen999 wants to merge 31 commits into
Closed
sungmen999 wants to merge 31 commits into
sungmen999 wants to merge 31 commits into
Conversation
comfyui_video runs model_family "minimax_h3_local" from bundled T2V/I2V API graphs (base mode, 20 steps, native audio), and comfyui_image gains model_family "qwen_image_2.1" (40 steps, CFG 1) next to the flux2 default. The model stacks live in tools/_comfyui/metadata.py and contract tests check the graphs against them. docs/vast-ai-open-weight-stack.md is the install guide for one rented GPU. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…fault A ComfyUI server set up for MiniMax H3 and Qwen-Image-2.1 has no WAN 2.2 or FLUX 2 files. comfyui_video judged readiness on WAN alone, so preflight listed video generation as unavailable and video_selector dropped the tool, and a call without model_family ran the missing default and failed. - comfyui_video: readiness per bundled family; an operation is ready when WAN or H3 can run it. get_info() reports model_family_statuses and the default_model_family a call without model_family will run. - comfyui_video / comfyui_image: a call that names no model_family runs the first installed family (wan2.2 / flux2 first, as before). An explicit family is never swapped. Results carry model_family and model_family_auto_selected. - ComfyUIClient.list_models(): a complete listing is reused for 10 s, so a preflight costs five /object_info requests per tool instead of fifteen or more (12 s -> 3 s for comfyui_video.get_info() over an SSH tunnel). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A machine set up for local image, video and music had no narration: every TTS provider needed an API key or a separate install. comfyui_tts runs the VoxCPM_TTS node of wildminder/ComfyUI-VoxCPM (model and nodes Apache-2.0) on the same server, so tts_selector finds a provider there. - Voice by local reference clip (uploaded and cloned), by a voice file in the server's input folder (with <voice>.reference.txt as transcript), or by a plain-words description. A call with only text gets a default narrator and a fixed seed, so sections of one script match. - The server saves FLAC; any other output extension is converted locally with ffmpeg, so a .wav path never holds FLAC bytes. - Status is DEGRADED when the server lacks the node pack, and execute() then names the pack to install. - ComfyUIClient gains upload_audio(), input_file_text() and list_input_audio() (both COMBO formats of /object_info). The node inputs in the two bundled graphs were read from /object_info of ComfyUI 0.44.0 with the node pack installed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
`make setup` installs piper-tts into the project venv, but the tool looked for `piper` on PATH only, so a preflight run with `.venv/bin/python` (the venv not activated) reported no offline TTS. It also reported AVAILABLE with no voice model downloaded, and the documented download command (`piper --download-dir`) does not exist in piper-tts 1.8. - The command is taken from PATH or from the interpreter's folder. - Voices live in ~/.piper/models (PIPER_DATA_DIR overrides) and are passed with --data-dir. Status is DEGRADED when the command is there but no voice is; get_info() lists installed_voices. - With no model given, the default voice is used when installed, else the first installed one. An unknown voice fails with the download command (`python -m piper.download_voices <voice> --data-dir <dir>`). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
tests/tools/test_comfyui_live_gpu.py generates one small piece of media per ComfyUI-backed capability against the server behind COMFYUI_SERVER_URL: image and video through the selectors without naming a model, image to video from that image, music, speech (designed voice, cloned server voice, cloned uploaded clip), and the transcriber reading the speech back. It is marked live_api, so it only runs with OPENMONTAGE_ALLOW_NETWORK=1. First run, 2026-11-10, rented RTX 5090, ComfyUI 0.44.0: 12 passed in 8 min 40 s. The MiniMax H3 image-to-video graph ran for the first time and was accepted as bundled. The docs, the comfyui skill and the two runtime estimates now carry these measured numbers with their settings, in place of "not yet run" and a third-party extrapolation. The guide gains the VoxCPM2 install steps for comfyui_tts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The session network guard lets loopback through, and a ComfyUI server is loopback: on the developer's machine, or a rented GPU behind an SSH tunnel. tests/contracts/test_phase2_contracts.py::test_image_selector_no_provider calls the real image selector with no mock, so with such a server up, and a model installed that a call without model_family can run, every suite run generated a real image there (seen on 2026-11-10: four images on a rented GPU, two of them in the middle of someone else's job). The guard now also refuses loopback connections to a ComfyUI: port 8188, and any loopback port a COMFYUI_*_SERVER_URL variable names when the session starts. ComfyUI tools then see "no server". OPENMONTAGE_ALLOW_NETWORK=1 lifts it, as for live_api tests. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The autoharness plugin writes per-session locks, intent queues and run records to .claude/autoharness/, and a ledger and sidecar next to each skill it lands. They are local runtime state. The landed skills under .claude/skills/ stay tracked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HeroTitle rendered each character as its own flex item, so a title wider than the frame (common on 9:16 shorts) could wrap in the middle of a word, e.g. "painstak / ingly". Characters are now grouped into one nowrap span per word, and the space between words comes from the flex column gap. Each character keeps its index in the full title, so the stagger timing and the accent on the first eight characters are unchanged. Checked with Remotion stills of the HeroTitle composition at 1920x1080, frame 150: the default single-line title renders byte-identical to before; a title with a long word on the line edge now wraps between words. `tsc --noEmit` passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit df658bce892de5c2ab6d59f3f6230c39d1a01fd1)
A calm cinematic playbook (identity category: cinematic) for slow, quiet content: calm palette and music mood, clear conversational voice. It loads through styles/playbook_loader and passes the playbook tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A prompt for a later session: list what must be installed on top of the vastai-comfyui toolkit so one rented GPU runs the whole open-weight stack. It stops for the user's decisions before any edit and never rents a GPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Decision log has one source, projects/<id>/decision_log.json. A stray artifacts/decision_log.json is folded in. Re-using a decision_id with a different `selected` raises. Writes are atomic. - A completed or awaiting_human checkpoint without `review` is logged and listed by get_unreviewed_stages(). get_stale_stages() is advisory. - Gated non-canonical stages need their produced artifact. checkpoint.policy takes effect (manual_all gates every stage). An unknown pipeline_type fails loudly instead of falling back to the canonical stage list. - Tool spend is synced into the project cost ledger at each checkpoint. resolve_budget_usd() is the one budget rule (approved amount, else the manifest default capped by config.yaml). cap mode refuses a paid call that does not fit. Rented GPU time can be recorded with `python -m tools.cost_tracker record`. - One tool module that fails to import no longer takes discovery down. provider_menu_summary(compact=True) groups setup offers. - A style_playbook written with a checkpoint is recorded on project.json. - One .env parser. Dead config keys (paths.*, checkpoint.storage_dir) removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Input schemas match what execute() reads, across the registry (new contract test). - jimeng_video prices a `duration` request (a 10 s clip was estimated at the 5 s price). music_gen returns its error instead of raising ValueError. - composition_validator resolves project-relative narration paths. - video_compose final review reports music_present only when the composition declares a music bed. - The HyperFrames scaffold text card stacks the subtitle under the title. - FFmpeg feature detection (drawtext, libass, libvorbis) for tools and tests; showcase_card reports itself unavailable without drawtext. - agent_skills pointers no longer name other vendors' skills (piper_tts, transcriber, audio_mixer). - Hunyuan tests no longer sleep for real. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Hero and scene videos keep their time and mute state, and drawer scroll survives a board re-render. - Project ids containing #, %, & or Thai resolve. The stage rail is keyboard operable, the modal is a dialog, and tap targets fit at 390 px. - The state endpoint sends an ETag and answers 304. - The board flags stages with no review and decision-log drift. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- HeroTitle, ProductReveal and TitledVideo split text with splitGraphemes()
(Intl.Segmenter with a fallback). split("") drew emoji as U+FFFD boxes and
Thai vowels and tone marks on dotted circles.
- HeroTitle accents the whole first word, not the first 8 characters.
- Explainer holds the last cut through tailPaddingSeconds instead of ending
on a second of bare background.
- Add an `npm run typecheck` script.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AGENT_GUIDE.md is read before every reply. It now states every binding rule
and routes to docs/agent-guide/{preflight,decisions,planning,
workspace-and-contracts,layers}.md for the procedures, which moved over
unchanged. 49,611 B -> 24,778 B. A router test keeps headings that skills
cite present.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Director skills in all 12 pipelines: example artifacts validate; fields a closed schema forbids move under metadata; selected_concept is read through concept_options[]; edit sets render_runtime; Remotion-only cut fields go to artifacts/remotion_composition.json; the HyperFrames gate is described as the tool runs it. - Executive-producer limits name the manifest orchestration fields instead of numbers; the budget row follows cost_tracker; talking-head gains max_wall_time_minutes. - Superseded explainer, animation and cinematic idea-directors are pointer stubs to proposal-director. - reviewer.md records the review on the checkpoint, at most two rounds. - export_bundle in the animation, cinematic and character-animation publish stages. - Contract tests for the EP limit source, manifest/skill tool consistency and the idea-director stubs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Python 3.10 and 3.11 matrix, with Chromium so tests/backlot UI tests run. - A Node job runs the remotion-composer typecheck. - `make lint` compiles every first-party module, not four files. - playwright added to requirements-dev.txt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`audio.music.ducking` was accepted by the pipeline skills and ignored by the Explainer, so music played at one flat level under the voice. lib/ducking.ts derives speech intervals from the word captions (merged across pauses up to 0.6 s; without captions the whole video counts as speech) and lowers the music gain to 0.35 with a 0.15 s attack and 0.4 s release. All four values can be overridden. Measured on a test render (narration 0-2 s, 220 Hz music bed): the music band sits 9.1 dB lower while speech plays and returns to its level after; with ducking off it is flat. The explainer and animation edit, compose and executive-producer skills gain `audio_mix_plan: "composition"` for this path; `premix` stays the choice when there are no captions or the render is not Remotion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…roundVideo video_compose stages `images[]` and `backgroundVideo` into the Remotion public dir, but the validator only looked at `source` and `backgroundImage`. A missing anime_scene frame passed validation and showed up as a black frame in the render. Both are now checked; the animation compose-director no longer tells agents to check images by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SCENE_TYPES.md describes `callout` as a boxed message with bullets, but CalloutBox printed `text` as one run: newlines collapsed and "- item" lines merged into one line on the audit render. Each line now renders on its own; lines starting with "- ", "* " or a bullet glyph get an accent-coloured bullet. A single-line callout renders as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scaffold_workspace read the music bed's fade_in_seconds / fade_out_seconds and dropped them, so the bed started and stopped at full level. They are now GSAP volume tweens on the root timeline (data-volume stays the baseline), the form hyperframes-core documents. Measured on a 6 s render (fade in 1.5 s, out 2 s, volume 0.6): -49.3 dB in the first 0.3 s, -31.5 dB mid-clip, -49.1 dB in the last 0.3 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A music bed with a quiet intro or shorter than the video needed a separate FFmpeg call before full_mix: audio_mixer could only delay a track and pad or trim the whole mix. Tracks now take `offset_seconds` (input seek) and `loop` (-stream_loop, cut to target_duration - start_seconds, which is required). A track's fade-out lands at the end it actually plays to. An in-point past the end of the file is an error. The explainer and animation compose-directors drop the FFmpeg pre-step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The music plan offers royalty-free stock (music_search tools such as pixabay_music), but music_source.source_type had no value for it, so the explainer proposal-director recorded it through a metadata workaround. Add `stock_search`; `provider` names the search tool. Also correct two video_compose docstrings: a missing renderer_family blocks the render (the docstring said warn), and render routes by the locked render_runtime with no silent FFmpeg fallback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/qa/test_04..08 run their whole workload at import time and report by printing. pytest imported them while collecting the main suite (about 17 s on every run) and their failures could never fail anything; run as scripts they always exited 0. A conftest now keeps them out of collection (0.02 s), and each script exits 1 when a tool step or check fails. QA_PLAN.md listed scripts that do not exist (test_01-03), a Windows path, and no test_09; it now matches the folder and says how to run each script. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- ARCHITECTURE.md: the ledger holds spend only; $0 calls stay in events.jsonl and a run with no paid call and no budget writes no cost_log.json (the end-to-end audit run read the missing file as a gap). - AGENT_GUIDE.md: framework-smoke is `beta` in its manifest (the schema has no `test` value) and has no director skills; the table now says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fallback path compared against a literal U+200D, invisible in review and easy to lose in an edit. Use "\u200d". Grapheme contract tests (both the Intl.Segmenter and the fallback path) still pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HyperFrames scaffolds, the character preview and the Three.js world loaded GSAP from cdn.jsdelivr.net (3.14.2, and 3.12.5 in the character preview), so a render without network failed: the HyperFrames compiler could not fetch the script and every timeline registration threw. remotion-composer/package.json now pins gsap 3.14.2, which `make setup` already installs. tools/gsap_runtime.stage_gsap copies that build next to the generated HTML and returns its src; without a local install it returns the pinned CDN URL as before. The scaffold result reports it as gsap_src. The character sub-composition no longer loads GSAP itself: the runtime clones only its <template> into a host index.html that already loads it. Verified with renders under sandbox-exec denying all outbound traffic except localhost: the CDN workspace fails, the local one renders with its tweens, and the character scene and its Playwright preview animate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The vendored .agents/skills/hyperframes-* examples load GSAP from a CDN. skills/core/hyperframes.md and skills/meta/bespoke-composition.md now say to use the gsap.min.js that scaffold_workspace copies into the workspace, and that sub-compositions use the host's GSAP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… install world-runtime.js imported GLTFLoader.js straight from the CDN. That module imports 'three' by bare specifier and the page had no import map, so the browser threw "Failed to resolve module specifier 'three'", the runtime module never ran, and every render was the HTML overlay over a black canvas stuck on "BUILDING WORLD GRAPH…" while the render still exited 0. The runtime now imports "three" and "three/addons/…", and index.html gets an import map for both. remotion-composer/package.json pins three 0.181.2, which `make setup` installs; tools/three_runtime.stage_three copies the modules the runtime needs, plus every module they import by relative path, into vendor/three/ in the workspace. Without a local install the map points at the pinned CDN. Verified by rendering a 12 s world twice. The first ran with local three under sandbox-exec, which denied all outbound traffic except localhost. The second ran online through the CDN map. Both showed terrain, landmarks and the camera path, and neither logged a page error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scaffold skipped copying an asset when the staged file already had the same size as the source. video_compose renders into one workspace per project (<project>/hyperframes), so re-rendering after a change that kept the size reused the stale copy. A character composition re-rendered for youtube_shorts differs from the landscape one only by swapping "1920" and "1080". The vertical render therefore mounted the 1920x1080 scene: it covered the top 1080 px, with black below. Cuts, narration and music now go through _stage_copy. It uses filecmp's shallow check: an unchanged file, whose (size, mtime) copy2 preserved, is not read again, and any other file is compared byte by byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
character_rig_renderer declared its composition as a fixed 1280x720 box with a 920px SVG. The scaffold that mounts it sizes the canvas from the media profile video_compose forwards, which is 1920x1080 by default, and no profile is 1280x720. The scene therefore sat in the top-left corner, and because the SVG was taller than the box, the figures' feet were cut off. The renderer now takes the same `profile` as video_compose and resolves it with the scaffold's own _resolve_dimensions. The composition root declares that size and fills its slot, and the SVG fits its square viewBox into 90% of the slot. The result reports the size as `canvas`. The character compose skill says to pass one profile to both tools. Several figures were also scaled about the SVG origin, so they rose off the ground, and they were spaced from x=110, so the group sat left of centre and the first figure's arm left the viewBox. They are now spread evenly about the centre and scaled about their feet. The placement sits on an outer group, because GSAP tweens the .character group's own transform. Verified with video_compose renders of two figures at the default 1920x1080 and at youtube_shorts 1080x1920. Both fill the frame, the figures stand on the ground, and both are fully in view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR has 31 commits from a deep audit of the pipeline and two feature branches. Grouped by area:
remotion-composer/package.jsongains two dependencies, both installed bymake setup:gsap3.14.2, under GSAP's Standard "no charge" licensethree0.181.2, under MITtools/gsap_runtime.pyandtools/three_runtime.pycopy these into each workspace. When they are not installed, both fall back to the pinned CDN URLs.Evidence
Full suite after: 4154 passed, 26 skipped, 0 failed, and
make lintpasses. The suite ran against an ffmpeg build that has libass, drawtext and libvorbis.Each fix below was checked with a real render:
Failed to resolve module specifier "three". The canvas stayed black on "BUILDING WORLD GRAPH…", yet the render exited 0.Failed to download CDN script … gsap.min.jsandCannot read properties of null (reading 'timeline'), and the render exited 1.video_composerenders at the default 1920x1080 and atyoutube_shorts1080x1920 both fill the frame.Merge Danger
Door: two-way
This PR changes only code, tests, skills and docs, with no data migration. Reverting it restores the old behaviour.
Blast Radius: wide
make setupinstalls two more npm packages.🤖 Generated with Claude Code