feat(agent-runtime): autonomous OpenMontage agent runtime + LLM prompt director (Gap B)#7
Draft
FluffyAIcode wants to merge 28 commits into
Draft
feat(agent-runtime): autonomous OpenMontage agent runtime + LLM prompt director (Gap B)#7FluffyAIcode wants to merge 28 commits into
FluffyAIcode wants to merge 28 commits into
Conversation
… mode=agent (Gap B Phase 1) OpenMontage has no Python pipeline runner (host LLM is the orchestrator), so mode=agent needs a new unattended runtime. Adds services/agent_runtime/: - llm.py: urllib-only cloud LLM client (anthropic|openai|kakeya|stub, auto-detect from keys). - run.py: LLM -> brief/script/scene_plan (schema-validated + checkpointed) -> assets via video_selector (auto best provider) + tts_selector + music_gen -> edit -> audio_mixer + video_compose (ffmpeg) -> final mp4. AGENT_RUNTIME_FAKE_ASSETS=1 for offline CI. - tests: stub-LLM + ffmpeg end-to-end (valid h264 + checkpoints) + JSON extraction. 23 pass. ADR 0016 + loop-log iteration 41. Needs a cloud LLM key to run live; quality is provider-bounded. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…ave no system ffmpeg) _prepare_ffmpeg symlinks the bundled imageio-ffmpeg binary as 'ffmpeg' on PATH so video_compose/ audio_mixer subprocesses work without a brew install. Verified end-to-end on the head Mac (stub LLM + fake assets -> real mp4). Offline tests still pass. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…y provider option diffusers HunyuanVideoPipeline / HunyuanVideoImageToVideoPipeline; 13B, native 720p ~5s clips. --backend hunyuan, ops framework,i2v; HUNYUAN_OFFLOAD (default on) for VRAM. For an A/B vs WAN to decide whether Hunyuan becomes the default quality layer. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
… as quality layer Real same-prompt 720p A/B on vast: WAN quality=high = unusable abstract blobs (bad 1.3B seed); HunyuanVideo = coherent photorealistic cinematic clip. Verdict: Hunyuan = default quality provider. Caveats: ~8min/2s@720p, VRAM-heavy, prompt-adherence/long-form/license noted. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…kills) Per ADR 0016, enrich each scene into a rich native-T2V prompt before generation, using the provider's advertised agent_skills (ai-video-gen/ltx2) as guidance, and persist directed prompts to assets/director_prompts.json. Extends offline test. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
cursor
Bot
changed the title
One tiny LLM round-trip to verify the reasoner (head Kakeya/Gemma gRPC runtime via OpenAI HTTP shim) before committing to a full pipeline run. Exit 0/1/2 = ok/error/ no-endpoint. Docs updated with the live mode=agent verification recipe. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…rvice) Fold the head's working transport into the repo: AGENT_LLM=kakeya_grpc talks token-id level to the user-run Kakeya engine (Gemma 26B @ :51051) via its gRPC SDK. External dependency contract (KAKEYA_REPO required; kakeya/transformers imported lazily) so grpc/ transformers never become core deps. Clear RuntimeError when KAKEYA_REPO unset; offline guard test. Verified live by owner (--check-llm -> OK). Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…/gateway/key) The public gateway enforces X-API-Key which the owner cannot retrieve, blocking mode=agent smoke tests. The runtime is a plain module; --plan-only runs LLM planning + prompt director, writes director_prompts.json, and exits before any render. Refactor run() to compute directed prompts up front. Offline plan-only test. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Gemma's 4-bit script stage emitted enhancement_cues.type='visual' (out of the script schema enum), failing validation. Tighten plan_script prompt to the exact enum + minimal sections, and add defensive _sanitize_script() that drops unknown section keys and out-of-enum/malformed cues before validation. Folds owner's local plan_script patch into the repo. Offline test reproduces the drift. Live --plan-only on the head (real Gemma via kakeya_grpc) confirmed exit 0. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
The X-API-Key requirement repeatedly blocked the web UI because the key had to be re-typed each time. Prefill the key field from localStorage on load and persist it on submit, so it is entered once per browser. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Web UI can now trigger the full agent pipeline (mode=agent) and pick quality/length for direct video, without hand-crafting API calls. Quality/length auto-disable in agent mode (the LLM plans its own scenes/pacing). Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
First framework/refine RPC was hitting INTERNAL BrokenPipe because the cold ~30s model download/load happened mid-RPC over the held-SSH pipe. Add --preload (load at boot; server binds only after, so supervisor keeps it out of the pool until ready) and HF_HUB_DISABLE_PROGRESS_BARS/TQDM_DISABLE to avoid flooding the SSH stderr pipe. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…unnel) Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…x BrokenPipe) The worker's stdout/stderr was piped over the held SSH; diffusion progress bars on stderr both raised BrokenPipeError when the channel hiccuped and the heavy traffic triggered 'Connection reset by peer' that killed the worker mid-refine. Redirect worker output to a local file so the held SSH carries ~no data and only keeps the process alive. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
… -> tmux mode Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…ne hang) The high-quality path upscales the proposer to 1280x720 before refine, so the Wan VAE must encode+decode a full-frame 720p clip in fp32. Untiled this allocates a huge tensor and HANGS after the denoise loop (worker stuck in VAE decode, GPU 0%, final message never sent -> orchestrator wedged at 58%). enable_tiling()+enable_slicing() processes spatial tiles / temporal slices, keeping memory bounded. Applied to T2V/v2v and I2V pipelines. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…upscale Full-frame v2v at 1280x720 hangs in the Wan VAE (denoise completes, worker stuck at GPU 0%, never returns -> job wedges at 58%). Refine at a capped longest-side (default 512, proven stable ~6s) and let the existing post-refine LANCZOS upscale reach the 720p target. Env REFINE_MAX_DIM tunable; 0 = original full-res behavior. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…rker thread) The distributed v2v refine op wedges in the worker's gRPC thread after denoise (VAE decode never returns; confirmed framework/native-T2V @720p completes in-thread and the 1MB result traverses the tunnel fine). Repoint 'high' to native 1280x720 generation on the strongest worker (Blackwell CUDA), no refine -- reliable + aligns with ADR 0015's 'native T2V beats low-res seed + refine'. Verified end-to-end: public download = h264 1280x720. (Also keeps REFINE_MAX_DIM cap as defense for any single-refine use.) Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
… worker thread) All three tiers now use the reliable direct native-T2V path (draft 384x224 / standard 832x480 / high 1280x720), differing by resolution+frames. The v2v refine op (single AND tiled) wedges in the worker's gRPC thread, so it's removed from the default tiers. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…ad hang; tiers -> native T2V direct Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…ium layer) Add orchestrator --prefer-backend (pin the generator to a backend substring, e.g. hunyuan; falls back to fastest). Gateway: prefer_backend param + high preset prefer_backend=hunyuan, native 1280x720 DIRECT, 30 steps / 45 frames. draft/standard stay on fast WAN; high routes to the Hunyuan worker when present. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…ecode hang) Hunyuan native 720p produces断崖式 better quality standalone, but its heavy VAE decode HANGS when run inside the worker's gRPC daemon thread (identical post-denoise GPU-0% stall as WAN v2v; both work standalone). Needs the worker to run the heavy op off the daemon thread (main thread/subprocess). Until then high stays on reliable WAN native; --prefer-backend=hunyuan routing is wired and ready to flip on once that's fixed. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…thread VAE-decode hang Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
Fixes the in-worker-thread VAE-decode hang (Hunyuan, WAN v2v): the gRPC servicer thread deadlocks on heavy CUDA decode, but the same op runs fine in a standalone process's main thread. SubprocBackend spawns a persistent child (spawn ctx) that loads the model and runs ops in its MAIN thread, streaming per-step progress back over a Queue and handing frames via a temp .npy; the gRPC side does only IPC + a file read (no CUDA on the gRPC threads). Gated by --subproc for cuda/hunyuan. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…durably SubprocBackend fixed the in-worker decode hang, so flip 'high' to prefer_backend=hunyuan (native 1280x720, 30 steps/45 frames; falls back to WAN native if no hunyuan worker). Supervisor now (VAST_HUNYUAN=1) launches the hunyuan --subproc worker in tmux, maintains its tunnel (VAST_HUNYUAN_LOCAL->REMOTE), and appends it to the worker list. Verified public render: h264 1280x720x45. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…720p live as high tier Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
…ance reviewer
(1) generate_clip renders via the gateway /v1/videos when AGENT_GATEWAY_URL is set
(quality=AGENT_VIDEO_QUALITY; high -> Hunyuan native 720p), else video_selector with a
resolution hint. Director/editor (runtime) + render farm (gateway) split.
(2) plan_style_bible: one fixed subject+style 'continuity bible' embedded into every directed
prompt for cross-scene consistency (assets/style_bible.json; AGENT_CONSISTENCY).
(3) review_plan: LLM reviewer over brief+prompts (consistency/coherence/quality) with optional
auto-revise (AGENT_AUTO_REVISE) and human-approval gate (AGENT_REQUIRE_APPROVAL);
assets/review.json. All shown under --plan-only. Tests updated (5 passed).
Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Autonomous agent runtime for the gateway's
mode=agent(ADR 0016):brief → script → scene_plan → prompt-director → per-scene assets → compose, LLM-driven, schema-validated + checkpointed. Plus the distributed render path (ADR 0017) and three production-grade upgrades.LLM providers (
llm.py)AGENT_LLM = anthropic | openai | kakeya | kakeya_grpc | stub.kakeya_grpctalks directly to the head's Kakeya Gemma 26B gRPCRuntimeService(external dep contract viaKAKEYA_REPO). Owner-verified live.Prompt director + 3 upgrades
generate_cliprenders via the gateway/v1/videoswhenAGENT_GATEWAY_URLis set (quality=AGENT_VIDEO_QUALITY;high→ Hunyuan native 720p), elsevideo_selector+ resolution hint.plan_style_bible→ one fixed subject+style embedded into every directed prompt (assets/style_bible.json;AGENT_CONSISTENCY).review_planLLM pass (consistency/coherence/quality) → optional auto-revise (AGENT_AUTO_REVISE) + human-approval gate (AGENT_REQUIRE_APPROVAL);assets/review.json.Distributed render farm (ADR 0017)
SubprocBackend(--subproc): runs heavy CUDA generation in a spawned child's MAIN thread (fixes the in-gRPC-thread VAE-decode hang).--prefer-backend. Supervisor manages the Hunyuan worker + tunnel durably.CLI
--check-llm(verify LLM endpoint),--plan-only(planning + bible + director + review, no GPU/gateway/key).Validation
pytest tests/tools/test_agent_runtime.py -q→ 5 passed (E2E, plan-only w/ bible+review, directed-prompt, kakeya_grpc guard, script-sanitizer).Docs
ADR 0016 (runtime + 3 upgrades), ADR 0017 (Hunyuan); loop log iterations 43–47.