diff --git a/docs/adr/0001-kakeya-integration-loop-log.md b/docs/adr/0001-kakeya-integration-loop-log.md index 9b5128f76..b3e01c7f7 100644 --- a/docs/adr/0001-kakeya-integration-loop-log.md +++ b/docs/adr/0001-kakeya-integration-loop-log.md @@ -1122,6 +1122,223 @@ auto-recovery. (Caveat: the ~84 G I2V model re-downloads unless `HF_HOME` is on --- +## Iteration 41 — Gap B Phase 1: autonomous agent runtime (mode=agent) — ADR 0016 + +**Finding:** OpenMontage has **no Python pipeline runner** — the host LLM is normally the +orchestrator. So `mode=agent` integration = building a new **autonomous agent runtime** (a cloud-LLM +reasoner + tool-calling loop). In-repo LLM is only `kakeya_llm` (mechanical). + +**Built:** `services/agent_runtime/` (`AGENT_RUNTIME_CMD`): +- `llm.py` — urllib-only cloud LLM client (`AGENT_LLM=anthropic|openai|kakeya|stub`, auto-detect). +- `run.py` — unattended orchestrator: LLM → brief→script→scene_plan (schema-validated + checkpointed + via `lib/checkpoint`), assets via `video_selector` (auto-routes to best provider; premium if + configured, else WAN draft) + `tts_selector` + `music_gen`, deterministic edit → `audio_mixer` + + `video_compose` (ffmpeg) → final mp4. `AGENT_RUNTIME_FAKE_ASSETS=1` for CI. + +**Topology:** agent runtime on the head Mac CPU (cloud LLM, doesn't steal video GPU); headless = video +worker; vast = heavy gen — the cost/stability recommendation. + +**Validated offline (no keys/GPU):** stub LLM + ffmpeg fixtures → full plan→validate→checkpoint→ +assets→mix→compose → valid h264 mp4 + checkpoints. 23 tests pass (runtime+gateway+pipeline). + +**Honest scope:** MVP "auto" path (no research/proposal/approval/reviewer governance). **Needs a +cloud LLM key** to actually run; quality is provider-bounded (WAN draft unless a premium video +provider key is set — then `video_selector` auto-upgrades). Not yet "all skills". Live `mode=agent` +test on agent.kakeya.ai pending the key. + +--- + +## Iteration 42 — A/B: HunyuanVideo vs WAN (real frames) → adopt Hunyuan as quality layer (ADR 0017) + +Same prompt (border collie, snowy forest, cinematic), both 1280×720 ~2s on vast Blackwell. +- **WAN** (`quality=high`: 1.3B T2V seed → I2V-14B): frame = chaotic abstract blobs, **no recognizable + dog/forest** — unusable. Root cause: garbage low-res 1.3B seed amplified by I2V. +- **Hunyuan** (`hunyuanvideo-community/HunyuanVideo`, 45f/30steps, no offload): frame = **coherent, + photorealistic** dog in a snowy forest, cinematic DoF — production-looking. + +**Verdict (frame-level visual judgment): adopt HunyuanVideo as the default quality layer.** Night-and- +day. Caveats: ~8 min / 2s clip @720p (~13–16 s/step), VRAM-heavy (evicted the 85 G WAN worker to avoid +OOM), prompt adherence imperfect (rendered a Shiba-type dog), long-form still needs Hunyuan-I2V, +license stricter (Tencent community). Added `HunyuanBackend` to `grpc_worker.py` (`--backend hunyuan`, +ops framework/i2v). Next: register `hunyuan_video` as a provider for `video_selector`/agent; resolve +WAN-14B vs Hunyuan VRAM (load-on-demand); wire Hunyuan-I2V for continuity. + +--- + +## Iteration 43 — agent runtime: LLM "prompt director" step (ADR 0016, per user) + +The user clarified the "LLM prompt director" is **already in OpenMontage's architecture** (the +Layer-3 video skills) and asked to keep executing the agent-runtime integration per ADR 0016 — +i.e. *activate* that existing knowledge, not invent a new step. Implemented in +`services/agent_runtime/run.py`: + +- **`_resolve_video_skill(registry)`** — reads the `agent_skills` advertised by the video provider + `video_selector` routes to (e.g. `ai-video-gen`/`ltx2`) from `.agents/skills` or `.claude/skills`, + and feeds that prompting guidance to the LLM. So per-scene prompts are written the way the chosen + model wants, sourced from the skills OpenMontage already ships. +- **`direct_video_prompt(llm, scene, brief, skill)`** — rewrites each scene description into ONE + rich native-T2V prompt (subject+action+setting+lighting+camera+style, ~40–70 words, positive + phrasing). This is the cross-attention insight from ADR 0015: **enrich the text condition and + hand a strong prompt to native T2V** instead of refining a weak low-res seed via I2V. Falls back + to the raw scene on any LLM error. Needs **no video GPU** — runs as soon as the LLM is up. +- Directed prompts are persisted to `assets/director_prompts.json` for transparency; the assets + loop now generates from the **directed** prompt (not the bare scene description). + +**Validation (offline, no keys/GPU):** `tests/tools/test_agent_runtime.py` extended — the stub LLM +handles the "video prompt director" system prompt and the test asserts each scene received a +directed prompt (`startswith "directed cinematic shot, "`, `!= raw`) via `director_prompts.json`. +`2 passed`. + +**LLM endpoint smoke test (`--check-llm`).** The head now runs the Kakeya **Gemma 26B** runtime +(gRPC :51051, kept alive via `caffeinate`) next to the gateway (:8088, `/healthz agent_runtime:true`). +Added `python -m services.agent_runtime.run --check-llm`: one tiny LLM round-trip prints +provider/model + latency, exits 0 (OK) / 1 (endpoint error) / 2 (no endpoint, provider=stub) — so the +reasoner is verified in ~seconds before committing to a 20–30 min pipeline run. Cloud agent cannot +submit `mode=agent` directly (gateway enforces `X-API-Key`); the live planning+director run is driven +by the key holder. + +**Live transport = `kakeya_grpc` (direct gRPC, not the HTTP shim).** Folded the head's working +provider into the repo: `AGENT_LLM=kakeya_grpc` calls the Kakeya engine's native gRPC `RuntimeService` +at token-id level (the stable surface — the OpenAI HTTP shim is deprecated/single-session). It is an +**external dependency contract** — `KAKEYA_REPO` must point at a Kakeya checkout (top-level `kakeya` +pkg + `sdks/python`); `kakeya`/`transformers` import **lazily** only when selected, so grpc/transformers +never enter OpenMontage's core deps (D5/D7). Flow: `apply_chat_template(enable_thinking=False)` → +`Client.create_session(eos_token_ids)` → `append` → `generate(max_tokens)` → `decode`. Env: +`KAKEYA_GRPC_ADDRESS=127.0.0.1:51051`, `KAKEYA_TOKENIZER_ID` (→ `KAKEYA_VERIFIER_ID` fallback). Head +deps in `~/.venv-distwan`: grpcio 1.81, transformers 5.12, mlx-lm 0.31. **Owner confirmed live:** +`--check-llm` → `provider=kakeya_grpc model=kakeya-local` → `OK`. Offline guard test asserts a missing +`KAKEYA_REPO` yields a clear `RuntimeError` (no obscure ImportError, no core-dep creep). + +**`--plan-only` (unblock testing without the gateway key).** The public gateway enforces +`X-API-Key` (`AGENT_GATEWAY_API_KEY`), which the owner couldn't retrieve — repeatedly blocking +`mode=agent` smoke tests via `agent.kakeya.ai`. Key insight: the agent runtime is a plain module; +the gateway is just a wrapper. Added `--plan-only` — runs LLM planning (brief→script→scene_plan) + +the prompt director, writes `director_prompts.json`, and **exits before any video render**. So the +owner validates Gemma + director on the head in minutes with **no GPU, no gateway, no key**: +`python -m services.agent_runtime.run --prompt "…" --plan-only`. Refactored `run()` to compute all +directed prompts up front (cleaner separation of director vs. generation). Offline test asserts +plan-only emits 2 directed prompts and renders no clips. + +**LIVE validation on the head (real Gemma, no GPU/gateway/key).** Owner ran `--plan-only` against the +`kakeya_grpc` runtime → exit 0. Log: `provider=kakeya_grpc model=kakeya-local fake_assets=False` → +`planning brief…` (`brief: The Dawn Hunter`) → `planning script/scenes…` → `3 scenes` → +`prompt-director: loaded provider guidance (8000 chars)` → 3× `scene i: directed prompt — ` → `director_prompts.json`. **Gap B core loop (Gemma planning + Layer-3 prompt +director) is proven live.** + +**Schema-drift fix (`enhancement_cues.type`).** Gemma's 4-bit script stage emitted an out-of-enum +`enhancement_cues.type="visual"`, failing the strict `script.schema.json` (enum: overlay/broll/diagram/ +stat_card/code_snippet/animation; sections `additionalProperties:false`). Hardened `plan_script`: the +prompt now lists the exact enum + "keep sections minimal, no unknown fields", AND a defensive +`_sanitize_script()` strips unknown section keys and drops out-of-enum/malformed cues before +validation (prompt steering alone is fragile for a 4-bit model). Offline test reproduces the +`type="visual"` drift and asserts the sanitized script passes the schema. `5 passed`. (Folds the +owner's local plan_script patch into the repo so head==repo, no divergence.) + +--- + +## Iteration 44 — new Blackwell vast node + BrokenPipe root-cause (held-SSH) → tmux mode + +Wired a new vast box (RTX PRO 6000 **Blackwell**, 96 GB) as the 3rd node and chased an +`INTERNAL: BrokenPipeError` that killed real jobs. + +- **Onboard:** authorized the head key (fixed a recurring *merged authorized_keys line* — the existing + key lacked a trailing newline, so an appended key concatenated and broke both), repointed + `~/.kakeya/vast.env`, verified Blackwell torch (2.12.1+cu130, sm_120 matmul OK), deps preinstalled. +- **BrokenPipe root cause:** the **held-worker** mode runs the worker as a child of a persistent SSH + with stdout/stderr piped over that SSH. Diffusion writes progress bars to stderr; the fragile SSH + channel got `Connection reset by peer`, so the worker's write raised `BrokenPipeError` (surfaced as + the op error) AND the reset killed the worker mid-op (no Python traceback → killed by signal). The + heavy stderr traffic likely *caused* the resets. Verified the op itself is fine: framework AND v2v + refine run to completion **directly on the box** (`REFINE_OK`, ~6 s). +- **Fix:** this box's image (unlike the earlier one) **keeps tmux/processes alive after SSH logout** + (tested). So switched it to non-held **tmux** mode (`VAST_HOLD_WORKER=0`): the worker is detached + from SSH, output goes to a tmux pane/file, and SSH resets no longer kill it. Verified: the worker + now survives and the 720p refine runs to 14/14 on the worker (previously it was killed → BrokenPipe). + Draft `mode=video` jobs complete end-to-end over the public gateway. Held-mode path was also + hardened (--preload, worker output → file, HF/tqdm bars off) for boxes that need it. +- **Remaining (separate) issue:** for `quality=high` (full-frame 720p single-refine) the worker + computes the result but the gRPC progress/result **stream wedges over the SSH `-L` tunnel after + ~8 messages** — a transport flow-control deadlock, NOT the original crash. Needs a tunnel/transport + fix (gRPC window/keepalive tuning on the orchestrator channel, chunked result, or a sturdier + forward). Draft works; standard (tiled) likely avoids the single large stream. + +--- + +## Iteration 45 — "tunnel stall" was a v2v-in-worker-thread hang → all tiers use native T2V DIRECT + +Chased the `quality=high` wedge (job stuck at refine 58%). It was **not** the tunnel: + +- The progress reaching 58% is expected (14 effective steps / 24 `total` for strength 0.6). The + **final result never arrives** because the worker is stuck **after** the denoise loop. +- Bisected it: `framework`/native-T2V at **1280x720** completes in the worker (final 1 MB mp4 sent + over the tunnel OK, public download = h264 1280x720); **standalone v2v** at 512x288 completes in a + plain process; but the **`refine` (v2v) op hangs in the worker's gRPC servicer thread** after + denoise (VAE decode/return never completes, GPU 0%) — at 720p AND at the capped 512x288. So it is a + v2v-op × gRPC-thread interaction, independent of resolution and of the SSH tunnel. VAE + tiling/slicing (added) did not help. +- **Fix:** drop v2v from the default tiers and use the **reliable native-T2V DIRECT** path on the + strongest worker (the Blackwell CUDA box) for all tiers — which also matches ADR 0015's finding + that native T2V beats a low-res seed + I2V/refine. Presets: draft 384x224, **standard 832x480**, + **high 1280x720**, all `refine_mode=direct`, 25 frames (4n+1) + interpolation. +- **Verified end-to-end via `agent.kakeya.ai`** (public download + ffprobe): standard = h264 + 832x480x25; high = h264 1280x720, 47 frames (~2 s). Draft also fine. +- Left in place as defense/diagnostics: `REFINE_MAX_DIM` cap on the single-refine path, VAE + tiling on the CUDA backend, and worker op-error tracebacks + final-message logging. + +**Open:** the v2v generative refine (extra detail over native T2V) remains disabled pending a +root-cause of the in-thread VAE-decode hang (suspect CUDA × gRPC C-core threading; py-spy blocked by +no SYS_PTRACE in the vast container). Native-T2V direct is the reliable quality path meanwhile. + +--- + +## Iteration 46 — Hunyuan 720p routing wired; blocked by the same in-worker-thread decode hang + +Tried to make `high` = HunyuanVideo (13B) native 720p (ADR 0017 premium layer). Built the routing +and confirmed the blocker is the worker concurrency model, not Hunyuan: + +- **Wired end-to-end:** orchestrator `--prefer-backend ` (pin the generator to a backend, + e.g. `hunyuan`; falls back to fastest) + gateway `prefer_backend` param + `high` preset. Downloaded + the 40 GB model on the Blackwell, ran a dedicated `--backend hunyuan` worker on :50053 (no offload), + tunneled it to the head, added it to the worker list. A public `high` job correctly routed: + `DIRECT on 127.0.0.1:50054 (hunyuan-diffusers) @ 1280x720x45`, and Hunyuan **generated all 30 + steps** at 100% GPU (~14 s/step) — generation works. +- **Blocker:** after denoise, the worker **hangs at GPU 0%** in the Hunyuan VAE decode — the + *identical* post-denoise stall as WAN v2v (iter 45). Standalone Hunyuan (the iter-42 A/B) AND + standalone WAN v2v both complete; they only hang when the **heavy CUDA decode runs inside the gRPC + servicer's daemon thread** (`_stream`'s `threading.Thread`). WAN native-T2V's tiny VAE is the only + heavy decode that slips through — which is why all reliable tiers use it. (Can't py-spy: vast + container lacks SYS_PTRACE.) +- **Decision:** reverted `high` to reliable **WAN native 720p** (verified: public h264 1280x720). + The `--prefer-backend=hunyuan` routing is wired and one line away from flipping on. **Real fix: + run the heavy generation op OFF the daemon thread** — a CUDA subprocess worker (spawn) or a + main-thread work-queue in the gРПС worker — so the decode doesn't deadlock with the gRPC C-core + threads. That worker-concurrency change is the next focused task to unlock Hunyuan (and v2v). + +--- + +## Iteration 47 — subprocess worker fixes the decode hang; Hunyuan 720p LIVE as the `high` tier + +Implemented the worker-concurrency fix and shipped Hunyuan as the premium tier. + +- **`SubprocBackend` (`--subproc`):** runs the real backend in a persistent **spawned child process** + whose MAIN thread does all CUDA work (load + denoise + the heavy VAE decode). The gRPC servicer + thread only does IPC: it streams per-step progress back over a `Queue` and receives the result + frames via a temp `.npy` handoff — **no CUDA on the gRPC threads**, so the post-denoise decode no + longer hangs. (Root finding: the decode also *eventually* completes non-subproc but unreliably/very + slowly with GPU idle; the child-main-thread path is the robust fix.) +- **Hunyuan 720p productized:** `high` → `prefer_backend=hunyuan` (native 1280x720, 30 steps, 45 + frames; WAN-native fallback via `_pick_fw` if no hunyuan worker). The supervisor (`VAST_HUNYUAN=1`) + launches the `--backend hunyuan --subproc` worker in tmux on the box, maintains a 2nd tunnel + (`VAST_HUNYUAN_LOCAL`→`REMOTE`), and appends it to the dynamic worker list — durable + auto-recover. +- **Verified end-to-end on `agent.kakeya.ai`** (public download + ffprobe): a plain `{"quality":"high"}` + job routed to `hunyuan-subproc @ 1280x720x45` and produced **h264 1280x720, 45 frames** (also an + explicit-param run). Generation ~30×14 s + decode; the worker no longer wedges. +- Model (40 GB `hunyuanvideo-community/HunyuanVideo`) cached on the box; HUNYUAN_OFFLOAD=0 (Blackwell + 96 GB). Tiers: draft 384x224 / standard 832x480 (WAN native) / **high 1280x720 (Hunyuan native)**. + +--- + ## Open follow-ups (next iterations) - **Phase 2b — native gRPC transport.** Add an optional `kakeya` Python SDK transport for the bounded-memory long-context path (W3), behind the same tool, once the proto diff --git a/docs/adr/0016-autonomous-agent-runtime.md b/docs/adr/0016-autonomous-agent-runtime.md new file mode 100644 index 000000000..335cc0066 --- /dev/null +++ b/docs/adr/0016-autonomous-agent-runtime.md @@ -0,0 +1,123 @@ +# ADR 0016 — Autonomous agent runtime for the public gateway (Gap B, Phase 1) + +**Status:** Accepted; MVP implemented + offline-validated. Live run pending a cloud LLM key. +**Date:** 2026-06-23 +**Related:** ADR 0011/0013 (agent gateway, re-anchor), AGENT_GUIDE.md (instruction-driven model). + +## 1. Problem (Gap B) + +`agent.kakeya.ai` only did **bare text→video** through the WAN cluster. The gateway's `mode=agent` +requires an `AGENT_RUNTIME_CMD`, which was unattached (`agent_runtime=false`) — so none of +OpenMontage's pipeline/skills (script → scene_plan → assets via selectors → compose) ran. The user +asked to **integrate the OpenMontage agent** so the public service runs a real multi-stage pipeline. + +## 2. Key finding + +OpenMontage is **instruction-driven**: there is **no Python pipeline runner** — the *host LLM* +(Cursor/Claude) is normally the orchestrator, reading manifests + director skills and driving tools. +To run **unattended** behind the gateway, we must supply our own LLM reasoner. In-repo LLM access is +only `kakeya_llm`/`llm_selector` (mechanical text). So this ADR introduces a new **autonomous agent +runtime**. + +## 3. Decision + +New module `services/agent_runtime/` exposed as `AGENT_RUNTIME_CMD` +(`python -m services.agent_runtime.run --prompt … --out …`): + +- **`llm.py`** — LLM client: `AGENT_LLM = anthropic|openai|kakeya|kakeya_grpc|stub` (auto-detected + from `ANTHROPIC_API_KEY`/`OPENAI_API_KEY`/`KAKEYA_GRPC_ADDRESS`/`KAKEYA_ENDPOINT`). `complete_json` + parses + retries. Creative reasoning runs here; the host agent is no longer required at runtime. + - The cloud + `kakeya` (HTTP shim) providers are **urllib-only** (no heavy deps). + - **`kakeya_grpc`** (the head's live path) talks **directly** to the user-run Kakeya engine over + its native gRPC `RuntimeService` (token-id level — the real stable surface; the HTTP shim is + deprecated/single-session). It is an **external dependency contract**: OpenMontage does **not** + vendor the Kakeya SDK/proto — point `KAKEYA_REPO` at a checkout (must expose a top-level `kakeya` + package + `sdks/python`) and the `kakeya`/`transformers` imports happen **lazily** only when this + provider is selected, so grpc/transformers never become core deps (ADR 0001 D5/D7). Env: + `KAKEYA_GRPC_ADDRESS` (e.g. `127.0.0.1:51051`), `KAKEYA_TOKENIZER_ID` (HF/MLX tokenizer dir; + falls back to `KAKEYA_VERIFIER_ID`). Flow: tokenizer `apply_chat_template(enable_thinking=False)` + → `Client.create_session(eos_token_ids=…)` → `append` → `generate(max_tokens)` → `decode`. + Host deps (head `~/.venv-distwan`): `grpcio`, `transformers`, `mlx`/`mlx-lm`. Missing + `KAKEYA_REPO` → clear actionable `RuntimeError`. +- **`run.py`** — minimal unattended orchestrator that reuses the real machinery: + - LLM produces **brief → script → scene_plan** (each schema-validated via `schemas.artifacts` + and checkpointed via `lib/checkpoint`, exactly like the governed pipeline; director-skill text + is loaded as LLM context — "intelligence in the skills"). + - **prompt director** (`direct_video_prompt`): before each clip, the LLM rewrites the scene + description into ONE rich, model-appropriate **native-T2V** prompt (subject+action+setting+ + lighting+camera+style, ~40–70 words, positive phrasing). Guidance is the **Layer-3 skill the + video provider advertises** (`_resolve_video_skill` reads the provider's `agent_skills`, e.g. + `ai-video-gen`/`ltx2`, from `.agents/skills` or `.claude/skills`) — i.e. it activates the prompt- + director knowledge OpenMontage already ships rather than inventing a new one. This is the + cross-attention insight from ADR 0015: enrich the text condition and hand a strong prompt to T2V, + instead of refining a weak low-res seed via I2V. Directed prompts are persisted to + `assets/director_prompts.json` for transparency. Needs no video GPU — runs as soon as the LLM is up. + - assets: per-scene video via **`video_selector`** (auto-routes to the BEST available provider — + premium Seedance/Kling/Veo if configured, else the local WAN draft cluster) using the **directed** + prompt, narration via **`tts_selector`**, one music track via **`music_gen`** (skipped when unconfigured). + - deterministic **edit_decisions** → **`audio_mixer`** + **`video_compose`** (ffmpeg) → final mp4. +- `AGENT_RUNTIME_FAKE_ASSETS=1` swaps providers for ffmpeg fixtures so the harness is CI-testable + with no keys/GPU. + +**Topology** (per the cost/stability recommendation): agent runtime runs on the **head Mac CPU** +(cloud LLM) next to the gateway; headless Mac stays a video worker; vast does heavy generation. The +agent uses a cloud LLM (light CPU/network) — it does **not** steal GPU from video. + +## 3b. Quality, consistency, governance (added) + +Three upgrades toward production-grade, all in `run.py` (LLM-driven, schema-light sidecars): + +- **(1) Render quality passthrough.** `generate_clip` renders each scene through the **gateway** + `/v1/videos` when `AGENT_GATEWAY_URL` is set (`quality=AGENT_VIDEO_QUALITY`, default `standard`; + `high` → Hunyuan native 720p — the tier from ADR 0017), reusing the whole distributed pipeline. + Without a gateway URL it falls back to `video_selector` with a `resolution` hint. The runtime is the + director/editor; the gateway is the render farm. (`AGENT_GATEWAY_API_KEY` or `~/.kakeya/...` key.) +- **(2) Cross-scene consistency.** `plan_style_bible` produces ONE fixed `subject`+`style` (a textual + "continuity bible") that `direct_video_prompt` embeds verbatim into EVERY shot's prompt, so the + recurring subject/style stays consistent across scenes (mitigates the "different dog each shot" + gap). Written to `assets/style_bible.json`. Toggle `AGENT_CONSISTENCY` (default on). +- **(3) Governance reviewer.** `review_plan` is an LLM reviewer over the brief + directed prompts + (subject consistency / coherence / prompt quality) → `{approved, issues, revised_prompts}`. With + `AGENT_AUTO_REVISE` (default on) flagged prompts are auto-fixed; `AGENT_REQUIRE_APPROVAL=1` gates + (halts with the plan written for human approval). Written to `assets/review.json`. Toggle + `AGENT_REVIEW` (default on). All visible under `--plan-only` (no GPU). + +## 4. Honest scope / limits + +- This is an **MVP "auto" path**, not the full governed `animated-explainer` pipeline (no + research/proposal, human-approval gates, reviewer governance, or per-stage director nuance). +- **Needs a cloud LLM key** to reason (none configured). Without it, `mode=agent` stays honestly + "not attached". +- **Output quality is provider-bounded.** With only the WAN cluster, clips are draft quality. + Configure a **premium video provider** (e.g. fal.ai/Seedance) and `video_selector` auto-upgrades — + that is the real path to production-grade. TTS/music/images need their own keys. +- Not "all skills integrated": skills light up as their providers are configured + as more pipelines + are wired. This is the first integration, not the finish line. + +## 5. Validation + +`tests/tools/test_agent_runtime.py` (offline, no keys/GPU): stub LLM emits valid brief/script/ +scene_plan; `AGENT_RUNTIME_FAKE_ASSETS=1` → full plumbing (plan → validate → **prompt director** → +checkpoint → per-scene assets → audio mix → ffmpeg compose) → **valid h264 mp4** + checkpoints for +idea/script/scene_plan/assets/edit. The test asserts each scene received a **directed** prompt +(enriched, `!= raw` description) via `director_prompts.json`. JSON-extraction unit test. 2 pass. + +## 6. Next + +- Deploy: set `AGENT_RUNTIME_CMD=" -m services.agent_runtime.run"` + an LLM endpoint on the + head; live-test `mode=agent` via `agent.kakeya.ai`. **LLM endpoint smoke test:** run + `python -m services.agent_runtime.run --check-llm` first — one tiny round-trip prints + provider/model + latency and exits 0/1/2 (ok/error/no-endpoint), so the reasoner is verified + before a full (slow) pipeline run. The head's live config uses **`AGENT_LLM=kakeya_grpc`** against + the Gemma 26B gRPC runtime (`KAKEYA_GRPC_ADDRESS=127.0.0.1:51051`, `KAKEYA_REPO`, `KAKEYA_TOKENIZER_ID`) + — `--check-llm` prints `OK`, confirming end-to-end reachability. +- **`--plan-only`**: runs the LLM planning (brief→script→scene_plan) + the prompt director and writes + `director_prompts.json`, then exits **before any video render**. This is the fast, + **no-GPU / no-gateway / no-API-key** way to validate that the LLM + director actually work on the + head: `python -m services.agent_runtime.run --prompt "…" --plan-only`. The gateway's `X-API-Key` + (set as `AGENT_GATEWAY_API_KEY`) is only needed for the *public* path; the runtime itself is a plain + module and needs no key — run it directly to verify the feature. +- Configure premium video provider for production-grade clips (the quality answer from the + draft-vs-premium discussion). +- Grow toward the governed pipeline (proposal/approval/reviewer, more director skills) and more + skills (avatar, images, music) as keys are added. diff --git a/docs/adr/0017-hunyuanvideo-quality-provider.md b/docs/adr/0017-hunyuanvideo-quality-provider.md new file mode 100644 index 000000000..0555a40a5 --- /dev/null +++ b/docs/adr/0017-hunyuanvideo-quality-provider.md @@ -0,0 +1,48 @@ +# ADR 0017 — HunyuanVideo as the default quality video provider (A/B vs WAN) + +**Status:** Accepted (decision: adopt Hunyuan as the quality layer). Productization pending. +**Date:** 2026-06-23 +**Related:** ADR 0015/0016 (quality/duration, agent runtime), the "four open-source models as +providers" design. + +## 1. Question + +Does replacing/augmenting WAN 2.1 with **HunyuanVideo** (13B) improve video quality enough to make it +the default quality layer? Decided by a **real A/B** on the same prompt, judged on actual frames. + +## 2. Setup + +- vast RTX PRO 6000 Blackwell (97 G). Same prompt: *"a border collie dog running through a snowy + forest, cinematic, soft winter light."* Both 1280×720, ~2 s. +- **WAN** = current `quality=high` path (1.3B T2V seed → I2V-14B @720p), via the public gateway. +- **Hunyuan** = `hunyuanvideo-community/HunyuanVideo` (diffusers `HunyuanVideoPipeline`), 45 frames, + 30 steps, no offload. New `HunyuanBackend` added to `grpc_worker.py` (`--backend hunyuan`, ops + framework/i2v). + +## 3. Result (frame-level visual judgment) + +- **WAN**: a chaotic, abstract blob field — **no recognizable dog or forest**, heavy SR artifacts. + Effectively unusable. (Root cause: the 1.3B low-res T2V *seed* is garbage; I2V amplifies it.) +- **Hunyuan**: a **coherent, photorealistic** dog walking through a snowy forest, cinematic shallow + depth of field, realistic snow + blurred trunks. **Clearly production-looking** for a single clip. + +**Verdict: adopt HunyuanVideo as the default quality layer.** The gap is night-and-day. + +## 4. Honest caveats + +- **Speed/VRAM:** Hunyuan ~13–16 s/step × 30 + VAE decode ≈ **~8 min for a ~2 s 720p clip**, and it + needs the GPU largely to itself (~49 GB active; the resident 85 GB WAN worker had to be evicted to + avoid OOM). Slower + heavier than WAN. +- **Prompt adherence:** it rendered a Shiba/Akita-type dog, not specifically a *border collie* — scene + quality is excellent but fine-grained subject control needs prompt work (Layer-3 skill). +- **Long-form:** multi-shot >5 s still needs `HunyuanVideo-I2V` chunking (drift risk remains). +- **License:** Tencent community license (region/MAU limits) — stricter than WAN's Apache-2.0. + +## 5. Decision / next + +1. **Hunyuan = default quality provider** (single-clip T2V hero path); WAN demoted to fast/draft + + fallback. Both stay (the four-model provider design). +2. **Productize:** register `hunyuan_video` as a `video_generation` provider so the agent/ + `video_selector` routes to it for quality; resolve the VRAM tradeoff (can't keep WAN-14B (85 G) + + Hunyuan resident — load the quality model on demand or dedicate the box to Hunyuan). +3. Wire HunyuanVideo-I2V for long-form continuity; improve prompting for subject fidelity. diff --git a/services/agent_gateway/deploy/vast_refiner_supervisor.sh b/services/agent_gateway/deploy/vast_refiner_supervisor.sh index 3bf2af603..18babc339 100755 --- a/services/agent_gateway/deploy/vast_refiner_supervisor.sh +++ b/services/agent_gateway/deploy/vast_refiner_supervisor.sh @@ -87,9 +87,18 @@ once() { if ! pgrep -f "KAKEYA_HELD=$VAST_SSH_HOST" >/dev/null 2>&1; then pkill -f "KAKEYA_HELD=" 2>/dev/null # clean stale holds (e.g. old host) OPS="framework,refine,i2v"; [ -z "${VAST_I2V_MODEL:-}" ] && OPS="framework,refine" + # Two hardening fixes for INTERNAL BrokenPipe on real jobs: + # 1) --preload: load the T2V model at BOOT (server binds only after), so the first RPC is + # instant instead of triggering a ~30s cold load mid-RPC. + # 2) Redirect the worker's stdout/stderr to a FILE ON THE BOX (not through the held-SSH pipe). + # The diffusion writes progress bars to stderr; sending that over the SSH channel both + # (a) raised BrokenPipeError inside the op when the channel hiccuped, and (b) the heavy + # traffic itself triggered "Connection reset by peer" that killed the worker mid-job. + # With output on a local file the held SSH carries ~no data and just keeps the process alive. + # HF/tqdm bars are also disabled as defense in depth. Read the worker log at $VAST_WORKER_LOG. nohup ssh $SSH_OPTS -o ServerAliveInterval=20 -o ServerAliveCountMax=1000 \ -p "$VAST_SSH_PORT" "$VAST_SSH_USER@$VAST_SSH_HOST" \ - "cd /workspace/distwan && KAKEYA_HELD=$VAST_SSH_HOST HF_HOME=${VAST_HF_HOME:-/root/.hf_home} CUDA_I2V_MODEL='${VAST_I2V_MODEL:-}' CUDA_I2V_OFFLOAD='${VAST_I2V_OFFLOAD:-0}' exec ${VAST_VENV_PY:-/venv/main/bin/python} grpc_worker.py --backend cuda --host 0.0.0.0 --port $VAST_REMOTE_PORT --ops $OPS" \ + "cd /workspace/distwan && KAKEYA_HELD=$VAST_SSH_HOST HF_HOME=${VAST_HF_HOME:-/root/.hf_home} HF_HUB_DISABLE_PROGRESS_BARS=1 TQDM_DISABLE=1 CUDA_I2V_MODEL='${VAST_I2V_MODEL:-}' CUDA_I2V_OFFLOAD='${VAST_I2V_OFFLOAD:-0}' exec ${VAST_VENV_PY:-/venv/main/bin/python} grpc_worker.py --backend cuda --host 0.0.0.0 --port $VAST_REMOTE_PORT --ops $OPS --preload > ${VAST_WORKER_LOG:-/workspace/distwan/worker.log} 2>&1" \ >> "$HOME/.openmontage-logs/vast_held_worker.log" 2>&1 & log "spawned held worker ssh -> $VAST_SSH_HOST (ops=$OPS); model load may take minutes" sleep 3 @@ -114,10 +123,29 @@ once() { sleep 2 fi + # Optional Hunyuan premium worker (subproc) for the 'high' tier (ADR 0017). Runs as a SECOND tmux + # worker on the same box (port VAST_HUNYUAN_REMOTE), tunneled to VAST_HUNYUAN_LOCAL on the head, and + # appended to the worker list so the gateway's --prefer-backend=hunyuan routing can reach it. + HY="" + if [ "${VAST_HUNYUAN:-0}" = "1" ]; then + HR="${VAST_HUNYUAN_REMOTE:-50053}"; HL="${VAST_HUNYUAN_LOCAL:-50054}" + ssh $SSH_OPTS -p "$VAST_SSH_PORT" "$VAST_SSH_USER@$VAST_SSH_HOST" \ + "tmux has-session -t hunyuan 2>/dev/null || tmux new-session -d -s hunyuan \"cd /workspace/distwan && HF_HOME=${VAST_HF_HOME:-/root/.hf_home} HUNYUAN_OFFLOAD=${VAST_HUNYUAN_OFFLOAD:-0} DISTWAN_TMP=/workspace HF_HUB_DISABLE_PROGRESS_BARS=1 TQDM_DISABLE=1 ${VAST_VENV_PY:-/venv/main/bin/python} grpc_worker.py --backend hunyuan --subproc --ops framework --host 0.0.0.0 --port $HR > /workspace/distwan/hunyuan_sp.log 2>&1\"" 2>/dev/null + if ssh $SSH_OPTS -p "$VAST_SSH_PORT" "$VAST_SSH_USER@$VAST_SSH_HOST" \ + "bash -c 'exec 3<>/dev/tcp/127.0.0.1/$HR' 2>/dev/null && echo up" 2>/dev/null | grep -q up; then + nc -z 127.0.0.1 "$HL" >/dev/null 2>&1 || { + pkill -f "L $HL:localhost:$HR" 2>/dev/null; sleep 1 + ssh $SSH_OPTS -o ExitOnForwardFailure=yes -fN -L "$HL:localhost:$HR" \ + -p "$VAST_SSH_PORT" "$VAST_SSH_USER@$VAST_SSH_HOST" 2>/dev/null; sleep 2; } + nc -z 127.0.0.1 "$HL" >/dev/null 2>&1 && HY=",127.0.0.1:$HL" + fi + fi + if reachable_local; then - write_workers "$BASE_WORKERS,127.0.0.1:$VAST_LOCAL_PORT"; log "vast refiner ONLINE -> 3-node (added 127.0.0.1:$VAST_LOCAL_PORT)" + write_workers "$BASE_WORKERS,127.0.0.1:$VAST_LOCAL_PORT$HY" + log "vast refiner ONLINE -> $BASE_WORKERS,127.0.0.1:$VAST_LOCAL_PORT$HY" else - write_workers "$BASE_WORKERS"; log "tunnel failed -> 2-Mac fallback" + write_workers "$BASE_WORKERS$HY"; log "wan tunnel failed -> $BASE_WORKERS$HY" fi } diff --git a/services/agent_gateway/server.py b/services/agent_gateway/server.py index 6745fa2ef..56632fb59 100644 --- a/services/agent_gateway/server.py +++ b/services/agent_gateway/server.py @@ -157,12 +157,21 @@ def _auth(x_api_key: Optional[str] = Header(default=None)): "draft": {"fw_width": 384, "fw_height": 224, "fw_frames": 13, "proposer_steps": 5, "refine_steps": 12, "out_width": 640, "out_height": 384, "frames": 25, "fps": 12, "interpolate": 1, "interp_method": "linear", "refine_mode": "direct"}, - "standard": {"fw_width": 480, "fw_height": 256, "fw_frames": 13, "proposer_steps": 6, + # standard = native T2V @ 832x480 DIRECT (same reliable path as 'high'; the tiled v2v refine + # hangs in the worker thread, so all tiers use direct native generation, differing by res/frames). + "standard": {"fw_width": 832, "fw_height": 480, "fw_frames": 25, "proposer_steps": 6, "refine_steps": 16, "out_width": 832, "out_height": 480, "frames": 25, - "fps": 12, "interpolate": 1, "interp_method": "linear", "refine_mode": ""}, - "high": {"fw_width": 512, "fw_height": 288, "fw_frames": 17, "proposer_steps": 8, - "refine_steps": 24, "out_width": 1280, "out_height": 720, "frames": 25, - "fps": 24, "interpolate": 2, "interp_method": "mci", "refine_mode": "single"}, + "fps": 16, "interpolate": 1, "interp_method": "linear", "refine_mode": "direct"}, + # 'high' = HunyuanVideo (13B) NATIVE 1280x720 — the premium quality layer (ADR 0017), routed via + # prefer_backend='hunyuan' to the subprocess Hunyuan worker (--backend hunyuan --subproc, which + # runs the heavy VAE decode off the gRPC threads — fixes the in-worker-thread hang). Hunyuan is + # not distilled, so ~30 steps; 45 frames (4n+1) native, no interpolation. If no hunyuan worker is + # up, _pick_fw falls back to the fastest worker = WAN native 1280x720 (still reliable). Verified + # end-to-end on agent.kakeya.ai: public h264 1280x720x45. + "high": {"fw_width": 1280, "fw_height": 720, "fw_frames": 45, "proposer_steps": 30, + "refine_steps": 24, "out_width": 1280, "out_height": 720, "frames": 45, + "fps": 24, "interpolate": 1, "interp_method": "linear", "refine_mode": "direct", + "prefer_backend": "hunyuan"}, } @@ -185,6 +194,7 @@ class VideoRequest(BaseModel): out_width: Optional[int] = None out_height: Optional[int] = None refine_mode: Optional[str] = Field(None, pattern="^(auto|direct|single|tiled)$") + prefer_backend: Optional[str] = Field(None, pattern="^[a-z0-9_-]{0,24}$") # Long-form (ADR 0015 Phase 2): chunks>1 (or longform=true + seconds) = autoregressive I2V gen. longform: bool = False chunks: Optional[int] = Field(None, ge=1, le=20) @@ -213,6 +223,8 @@ def resolved(self) -> dict: p["chunks"] = max(1, math.ceil((want - p["chunk_overlap"]) / net)) if p["chunks"] == 1 and p["seconds"] > 0: # single-pass: seconds drives frame count p["frames"] = max(2, round(p["seconds"] * p["fps"])) + if self.prefer_backend is not None: + p["prefer_backend"] = self.prefer_backend p["quality"] = self.quality return p @@ -252,6 +264,8 @@ def _run_video_job(job: Job): cmd += ["--seconds", str(p["seconds"])] if p.get("refine_mode"): # quality preset / explicit override of refine topology cmd += ["--refine-mode", p["refine_mode"]] + if p.get("prefer_backend"): # pin a tier to a model worker (e.g. 'high' -> hunyuan) + cmd += ["--prefer-backend", p["prefer_backend"]] if p.get("chunks", 1) > 1: # long-form (autoregressive I2V continuity) cmd += ["--chunks", str(p["chunks"]), "--chunk-frames", str(p["chunk_frames"]), "--chunk-overlap", str(p["chunk_overlap"])] @@ -412,15 +426,42 @@ def index(): border:1px solid #1c2942;border-radius:10px;padding:12px;font-size:12.5px;color:#a9c2e6} video{width:100%;border-radius:12px;margin-top:12px;background:#000} input[type=password]{padding:10px;border-radius:8px;border:1px solid #2a3a59;background:#0e1830;color:#e7eefc} +select{padding:9px 10px;border-radius:8px;border:1px solid #2a3a59;background:#0e1830;color:#e7eefc;font:inherit} +label.fld{display:flex;flex-direction:column;gap:4px;font-size:12px;color:#8fa6c8} +label.fld select:disabled{opacity:.45}

OpenMontage — Agent Video

Describe a clip. The distributed cluster (MLX proposer + CUDA refiner) renders it.

+
+ + + +
- +
-

Direct text→video via /v1/videos. First render can take a few minutes.

+

Direct text→video via /v1/videos. First render can take a few minutes.