Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# OneCue · 文本到短视频工作流

OneCue 是一套面向 Apple Silicon Mac 的本地短视频 MVP:Codex 负责理解内容、文案与分镜,Image 2 生成逐镜图片,包 B 生成逐镜人声,包 A/FFmpeg 校验 Schema v1 Job Bundle 并合成竖屏视频。当前已能从一段自然语言需求推进到候选成片,并保留校验、缓存、执行报告和镜头级返修能力。
OneCue 是一套面向 Apple Silicon Mac 的本地短视频 MVP:Codex 负责理解内容、文案与分镜,Image 2 生成逐镜静态图片,包 B 生成逐镜人声,包 A/FFmpeg 校验 Schema v1 Job Bundle 并合成竖屏视频。当前已能从一段自然语言需求推进到候选成片,并保留校验、缓存、执行报告和镜头级返修能力。

当前不使用 BGM;音频只来自包 B 的解说或角色对话。基础动态仍是 FFmpeg 虚拟摄影机推拉、平移与轻漂移,并非物体级自然动画;雨落、流水、人物动作等尚未实现。这是当前最明显的质量边界,不在此处夸大
当前第一版采用“多张静态分镜图随叙事硬切”的路线:不使用 BGM,不对图片做推拉、平移、漂移、拆层微动态或 I2V;音频只来自包 B 的解说或角色对话。每张图片仍会由包 A 编码为与该镜头人声等长的视频片段,再与字幕和其他镜头合成为成片。动态研究与兼容代码暂留,但不进入默认工作流

## 系统构成

- `【包A】视频引擎包`:Job Bundle 契约、TTS 编排、ASS 字幕、基础运镜、转场、缓存、报告与最终合成。
- `【包A】视频引擎包`:Job Bundle 契约、TTS 编排、静态镜头编码、ASS 字幕、硬切、缓存、报告与最终合成。
- `【包B】语音引擎包`:基于 dots.tts 的本地 TTS 服务,默认监听 `127.0.0.1:7860`。
- `skills/short-video-director`:可选但强烈推荐的 Codex 薄导演层,识别策划、新建、续接、检查、渲染和返修模式;它不实现另一套视频能力。
- `【包A】视频引擎包/docs/short_video_v2`:V2 的权威工作流、Schema 说明、图片约定、核心管线、动态边界与验收文档
- `【包A】视频引擎包/docs/short_video_v2`:V2 的权威工作流、Schema 说明、图片约定、核心管线与验收文档;动态文档仅作历史研究记录

## 仓库不包含的内容

Expand Down Expand Up @@ -41,7 +41,7 @@ python3.12 scripts/setup_macos_source.py --model mf

该脚本会在包 B 内创建 `runtime/python`,安装锁定依赖,以 editable 方式安装包 B,下载 `rednote-hilab/dots.tts-mf`,并按仓库中的 manifest 校验大小与 SHA-256。校验不通过时不应启动服务。

可选质量版模型速度更慢、占用更大:
可选质量版模型速度更慢、占用更大,当前第一版并不需要;仅在另行比较音质时下载

```bash
python3.12 scripts/download_macos_models.py --model soar
Expand Down Expand Up @@ -118,11 +118,15 @@ Job Bundle 必须是自包含的 Schema v1 目录。请先阅读 `【包A】视

```bash
"【包B】语音引擎包/runtime/python/bin/python3.12" -m unittest \
"【包A】视频引擎包/tests/test_director_workflow_docs.py" -v
"【包B】语音引擎包/runtime/python/bin/python3.12" -m unittest discover \
-s "【包A】视频引擎包/tests" -p 'test_v2_*.py' -v
"【包A】视频引擎包/tests/test_director_workflow_docs.py" \
"【包A】视频引擎包/tests/test_v2_job_bundle_contract.py" \
"【包A】视频引擎包/tests/test_v2_core_runtime.py" \
"【包A】视频引擎包/tests/test_v2_core_pipeline.py" \
"【包A】视频引擎包/tests/test_v2_mvp_acceptance.py" -v
```

计划 01、04、07 等旧阶段实验测试仍保留在 `【包A】视频引擎包/tests`,但不属于第一版日常门。需要审计历史兼容性时,才使用 `test_v2_*.py` 全量发现。

包 B 的运行策略与 API 契约测试:

```bash
Expand Down
37 changes: 23 additions & 14 deletions skills/short-video-director/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,45 +1,54 @@
---
name: short-video-director
description: Direct OneCue's short-video V2 workflow from natural-language content or an existing Schema v1 Job Bundle. Use when Codex is working in a OneCue checkout and needs to plan, create, resume, inspect, render, or revise a short video, including Brief and storyboard design, Image 2 keyframe planning, Job Bundle validation, package A/B rendering, evidence review, user review, and shot-scoped repair.
description: Direct OneCue short-video work from natural-language content, an existing Schema v1 Job Bundle, or user-supplied footage. Use when Codex needs to plan, create, resume, inspect, render, or revise either the established static-storyboard route or a Codex-owned footage edit built with general-video, media-use, and HyperFrames.
---

# Short Video Director

Act only as a thin director and navigation layer over the repository's established workflow. The core Codex → Image 2 → package B → package A/FFmpeg chain does not depend on this skill. Do not implement image, TTS, video, FFmpeg, cache, schema, or state-machine capabilities here.
Act only as a thin director and navigation layer. Route established static-storyboard work to the repository workflow, and route selected user-supplied footage to Codex's media workflow. Do not implement image, TTS, video, FFmpeg, cache, schema, or state-machine capabilities here.

## Route the request

Choose exactly one primary mode, then record it:

- **Plan**: Produce the Brief, script, and shot draft. Do not generate images, synthesize speech, or render.
- **Create**: Advance from content to a candidate video through the existing project workflow. Stop for user review after the candidate is ready.
- **Resume**: Read the task record and existing artifacts first. Continue from the first unmet hard gate.
- **Inspect**: Perform read-only checks of a Job Bundle, report, cache, final, services, or current state.
- **Render**: Validate an existing Job Bundle before calling the existing `video_v2 render` CLI.
- **Revise**: Identify the shot and affected layer. Make the smallest change and scope rebuilding with `--shot` when applicable.
- **Plan**: Produce the brief, script, source decisions, and shot draft. Do not generate media, synthesize speech, edit, or render.
- **Create**: Advance from content and selected sources to a candidate through the chosen material route. Stop for user review after the candidate is ready.
- **Resume**: Read the chosen route's task record or composition and existing artifacts first. Continue from the first unmet hard gate.
- **Inspect**: Perform read-only checks of supplied media or the chosen route's bundle, composition, reports, cache, final, services, and current state.
- **Render**: In Static V1, validate the Job Bundle before `video_v2 render`; in Codex footage, check the HyperFrames composition before its established render loop.
- **Revise**: Identify the timestamp, source or shot, and affected layer. Make the smallest change; use `--shot` only in Static V1 when applicable.

Infer safe defaults for low-impact omissions. Ask only when missing information changes facts, brand/content direction, external cost or authorization, or the candidate result materially.

## Choose the material route

After choosing the primary mode, choose one orthogonal material route:

- **Static V1 route**: Use when the result is built from independent Image 2 stills. Keep the established Codex → Image 2 → package B → package A/FFmpeg contract unchanged.
- **Codex footage route**: Use when the user supplies video for assessment or possible use. Mixed video-and-image work also uses this route. Codex owns the footage analysis, edit plan, media operations, composition, audio treatment, rendering, and review through general-video, media-use, and HyperFrames; package A and package B do not process this route.
- If supplied footage is irrelevant, redundant, unreadable, or weaker than another truthful visual, record why it was omitted. Supplying footage is permission to assess it, not an obligation to force it into the edit. After an `omit` decision, switch to the Static V1 route only if the remaining requested result actually fits that contract.

## Read only what the mode needs

- Read [references/project-map.md](references/project-map.md) to locate the checkout, authoritative documents, CLI, services, records, and protected baselines.
- Read [references/workflow-and-gates.md](references/workflow-and-gates.md) for Plan, Create, Resume, Render, mode transitions, automatic continuation, and pause conditions.
- Read [references/quality-and-revision.md](references/quality-and-revision.md) for Inspect, review, failure diagnosis, cache interpretation, media evidence, and minimal revision.
- Read the repository's `director_workflow_v1.md` before executing any plan, create, render, or revision path. Follow its linked contracts instead of copying them into this skill.
- For the Static V1 route, read [references/project-map.md](references/project-map.md), [references/workflow-and-gates.md](references/workflow-and-gates.md), and [references/quality-and-revision.md](references/quality-and-revision.md). Read the repository's `director_workflow_v1.md` before executing any plan, create, render, or revision path.
- For the Codex footage route, read [references/video-material-workflow.md](references/video-material-workflow.md). Then load the installed hyperframes, general-video, and media-use skills as that reference directs. Do not load the static Job Bundle contracts unless the request also contains a separate Static V1 task.

## Preserve the project contract

- The following Schema, CLI, cache, and shot rules apply only to the Static V1 route.
- Locate the repository from the current workspace or the path supplied by the user. Do not assume one person's absolute checkout path.
- Treat natural language as an upstream Codex input. Feed package A only a valid, self-contained Schema v1 Job Bundle.
- For every newly created V1 shot, set `motion.preset` to `static`, `motion.strength` to `low`, and `transition_out` to `cut` with `duration_sec: 0`. Keep `visual.focus` for crop placement. Do not choose motion or crossfade presets merely because Schema v1 still accepts them.
- Use the existing `python3 -m video_v2 validate|render` CLI. Do not import the pipeline to create another orchestrator.
- Preserve the user's dirty worktree, existing Job Bundles, reports, cache, and final files. Never reset, checkout, clean, or overwrite unrelated work.
- Use explicit argv, `shell=False`, and finite timeouts for external commands. Never turn user text into shell syntax.
- Do not add a wrapper script by default. Consider one only if two independent real cases prove the existing CLI cannot express the same deterministic step, with tests and a minimal design first. Never add a second Schema, cache, timeline, caption system, media checker, Web UI, or provider routing layer.
- The Codex footage route is deliberately outside Schema v1. Never place supplied video in a V1 Job Bundle, claim package A or package B supports it, or create a parallel OneCue schema or runner. Keep originals intact and store only task-local plans, derivatives, compositions, renders, and review evidence.

## Continue and stop safely

Continue through authorized, local, reversible work: read-only checks, task-local records, content design, validation, targeted tests, report reading, and bounded contract repair.

Pause for material ambiguity; new third-party/paid/cloud APIs, credentials, downloads, payment, or publishing; destructive deletion or cache cleanup; a need to change Schema/core pipeline/tool scope; and final user review of a candidate video. When the user has explicitly requested creation, the current session's built-in Image 2 capability is the established keyframe route, not a new external API expansion.
Pause for material ambiguity; new third-party/paid/cloud APIs, credentials, downloads, payment, or publishing; destructive deletion or cache cleanup; a need to change Schema/core pipeline/tool scope; and final user review of a candidate video. In the Static V1 route, also pause for any request to expand the active route into image animation. When the user has explicitly requested creation, the current session's built-in Image 2 capability is the established keyframe route, not a new external API expansion.

Never describe FFmpeg push, pull, pan, tilt, or drift as natural semantic motion. Rain, water flow, vehicle travel, breathing, body motion, and similar object-level movement remain unimplemented; the formal advanced-provider count is zero.
The active V1 result is a sequence of distinct static storyboard images, not one image for the whole narration and not animated stills. In that static route, do not invoke FFmpeg push/pull/pan/tilt/drift, crossfades, HyperFrames, DepthFlow, I2V, BGM, environmental audio, or SFX. The Codex footage route is the separate scope in which HyperFrames may be used, under its own review gates.
1 change: 1 addition & 0 deletions skills/short-video-director/VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1.1.0
4 changes: 2 additions & 2 deletions skills/short-video-director/agents/openai.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
interface:
display_name: "短视频导演"
short_description: "以自然语言策划、制作、续接、检查、渲染与返修 OneCue 短视频"
default_prompt: "Use $short-video-director to plan or continue a short-video V2 task from natural-language input."
short_description: "以自然语言统筹静态分镜或用户视频素材的短视频制作"
default_prompt: "Use $short-video-director to plan, create, inspect, or revise a OneCue short video from static storyboard assets or user-supplied footage."
3 changes: 2 additions & 1 deletion skills/short-video-director/references/project-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,15 @@ Read these from `<package-a>/docs/short_video_v2/` as needed:
- `job_bundle_v1.md`: machine input contract
- `image_workflow_v1.md`: Image 2 keyframe method
- `core_pipeline_v1.md`: rendering, cache, failure, and CLI behavior
- `motion_feasibility_v1.md`: honest motion boundary
- `mvp_acceptance_v1.md`: accepted MVP baseline
- `workflow_acceptance_v1.md`: accepted director workflow and skill boundary
- `templates/brief_v1.md`: optional content-design aid
- `templates/review_v1.md`: optional review and revision aid

Do not substitute older root-level vision documents for these frozen v1 facts.

`motion_feasibility_v1.md`, `motion_experiment_playbook_v1.md`, plans 04/07, and their experiments are historical motion research. They are not active-route documents and should be opened only when the user explicitly asks to revisit image animation.

## Evidence and records

Task-local Job Bundles, reports, cache, final videos, and acceptance evidence normally live under ignored `成片/` directories. Treat any existing formal acceptance bundle and final as protected. Validate it read-only and use an isolated copy for failure injection or revision rehearsal.
Expand Down
12 changes: 7 additions & 5 deletions skills/short-video-director/references/quality-and-revision.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Quality and revision

This reference governs only the **Static V1 route** and its Job Bundle artifacts. Review or revise a Codex-owned footage edit with [video-material-workflow.md](video-material-workflow.md) instead.

## Inspect without mutation

For an existing Job Bundle, read the task record if it already exists and run `validate --json`. Inspect the render report, cache manifest, final hash, ffprobe facts, and complete decode as requested. Do not create or update a task record, render, modify the formal bundle, or regenerate media merely to answer whether it can continue; report facts in the reply.
Expand All @@ -16,7 +18,7 @@ Use the repository review template to separate:

- contract and service facts;
- codec, size, frame rate, pixel format, audio, duration, and complete decode;
- per-shot content, image, voice, caption, and basic-motion observations;
- per-shot content, static image, voice, caption, and hard-cut observations;
- cache hits/rebuilds and protected hashes;
- user feedback and the first unmet gate.

Expand All @@ -28,14 +30,14 @@ Map the request to a `shot_id`, time point, and affected layer:

- text or caption: change only the necessary shot and adjacent continuity;
- voice: rebuild only affected audio/shot dependencies;
- focus or motion: change only legal fields for that shot;
- focus or crop: change only `visual.focus` for that shot and keep motion `static/low`;
- keyframe composition: edit or regenerate only that keyframe, then update its SHA-256;
- transition or cross-shot pacing: allow final recomposition while reusing valid audio/shot caches.
- hard-cut or cross-shot pacing: keep `cut/0`; allow final recomposition while reusing valid audio/shot caches.

Snapshot the old final/report hashes, validate again, then use `--shot` to limit allowed rebuilding. Add `--force` only when the selected cache must be invalidated. Verify affected cache counts, report, final hash, ffprobe, and complete decode.

Never regenerate every image, every voice, or the whole video for a bounded shot problem. Never switch providers as a revision shortcut.

## Motion wording
## Active visual wording

Call FFmpeg behavior virtual-camera motion: push, pull, pan, tilt, or drift. State that rain, flowing water, vehicle travel, breathing, gestures, clothing/hair response, and comparable natural semantic motion are not implemented. Advanced natural-scene providers are not part of the formal route.
Describe the active result as multiple static storyboard images that change with the narration. Do not promise or invoke image animation, virtual-camera motion, crossfades, HyperFrames, DepthFlow, or I2V. Schema support for older motion presets is compatibility, not an invitation to select them in a new task.
Loading