feat(pipeline): add the story.video product

docs/cli.md

@@ -195,7 +195,7 @@ an ADV episode -> JSON
 
 ## nnnotes export story
 
-an ADV episode -> the story product and the products it plays with (scene, assets, Live2D models, cue sheets, chat, CRI Lips)
+an ADV episode -> the story product and the products it plays with (scene, assets, Live2D models, cue sheets, videos, chat, CRI Lips)
 
 | Argument | Description |
 | --- | --- |

docs/commands.md

@@ -188,6 +188,7 @@ OUT/storyassets/<name>.json    each stage, frame, effect, post effect, still, ti
                                chat icon and stamp the episode loads (`story.asset`; <name>: its key, "/" as "~")
 OUT/models/<id>.json           each Live2D model of the episode (`live2d.model`)
 OUT/cuesheets/<sheet>.json     each cue sheet of the episode, decoded (`story.sound`)
+OUT/videos/<name>.json         each video of the episode, as WebM (`story.video`; <name>: its asset, "/" as "~")
 OUT/storychat/default.json     the shared chat rows, when the episode shows chat (`story.chat`)
 OUT/crilips/default.json       the CRI Lips analysis data, when a voice can reach it (`story.crilips`)
 OUT/assets/<sha256>.<ext>      the files the manifests name
@@ -197,7 +198,9 @@ OUT/assets/<sha256>.<ext>      the files the manifests name
 `story.json` names the episode's chat rows, its effect instances and, in `parts`, the manifest of each product above
 it plays with. The command runs twice: it asks for the story, reads its `story.json`, then asks for the story and
 every product `parts` names and places them. It stops with a usage error before the second run when a key of such a
-product is not in the catalog. The episode's talk windows and videos are listed in `episode.json` `resources`.
+product is not in the catalog. The episode's talk windows are listed in `episode.json` `resources`. The videos are
+the `movie.webm` of `cri.movie` with `format` `webm`: VP9 as stored, re-encoded when the video is in another codec or
+has an alpha stream.
 
 A product several stories play with (the scene, a stage, a model, a cue sheet) is one product: stories exported into
 one directory share its manifest and files, and the depot keeps every result for the next export.

docs/export.md

@@ -17,8 +17,8 @@ nnnotes export story 10462 -o out/stories/10462            # everything the epis
 
 `export story` writes the story product of an episode and the products it plays with, as a product layout below `-o`:
 the story index in `stories/<id>.json`, the stage scene in `storyscenes/default.json`, each stage, frame, effect and
-other asset in `storyassets/`, each Live2D model in `models/`, each cue sheet in `cuesheets/`, and, when the episode
-uses them, the shared chat rows in `storychat/default.json` and the CRI Lips data in `crilips/default.json`. The files
+other asset in `storyassets/`, each Live2D model in `models/`, each cue sheet in `cuesheets/`, each video in
+`videos/` and, when the episode uses them, the shared chat rows in `storychat/default.json` and the CRI Lips data in `crilips/default.json`. The files
 themselves are under `assets/`.
 
 Products that several stories share (the scene, assets, models and cue sheets) are placed once, so exporting several

docs/products.md

@@ -23,7 +23,10 @@ A product stage may also read cue sheets: the catalog keys `cue_sheets` gives (e
 sheet is the ACB its key loads, held in a bundle (the `acb` file of an object of the key) or a raw file of the
 catalog that `cri.audio` decoded. The task's inputs then hold each ACB (`acb:<key>`) and, as `sheets`, the files
 `cri.audio` made of it (`{key: {acb, status, reason, files: {name: {sha256, size, ext}}}}`); it depends on those
-`cri.audio` tasks, and a product file may be one of those files as stored, without copying its bytes. A stage that
+`cri.audio` tasks, and a product file may be one of those files as stored, without copying its bytes. Movies work the
+same way: the catalog keys `movies` gives, each the USM its key loads (the `usm` file of an object of the key, or a raw
+file of the catalog that `cri.movie` decoded), as `usm:<key>` and, in `movies`, the files `cri.movie` made of it with
+their facts (`{key: {usm, status, reason, files: {name: {sha256, size, ext, facts}}}}`). A stage that
 reads files of the APK names them in `APK`: inputs `apk:<path in the APK>`, from the fact `apk`.
 
 All are read from the planning facts alone, so a command can ask for what its products need before anything runs:
@@ -360,10 +363,12 @@ A story episode plays with products that several episodes share, made by `nnnote
 | `story.crilips` | `default` | `crilips/default.json` | none (it reads the APK) | `crilips.json` and `crilips.bin` |
 | `story.scene` | `default` | `storyscenes/default.json` | the ADV scene `EmbScene/Adv`, the prefabs `EmbCommon/Prefab/CameraManager`, `EmbCommon/Adv/Prefab/AdvCharacterField`, `AdvBackgroundField` and `AdvGlobalVolume`, the settings `EmbCommon/Adv/Settings/AdvPlayerSettings` and `AdvMasterIdSettings`, and the built-in `GraphicsSettings`, `QualitySettings` and `PlayerSettings` | `scene.json` and the object table |
 | `story.asset` | a catalog key a story loads: a stage (`Adv/Stage/<name>`), frame, effect, post effect, still, timeline, chat window, chat icon or stamp, transition | `storyassets/<name>.json`, `<name>` the key with `/` as `~` | the key | `asset.json` and the object table |
+| `story.video` | a video key `Cri/Video/<asset>` (the video of a Movie / Clip command) | `videos/<name>.json`, `<name>` the asset with `/` as `~` | the key | `video.json` and `movie.webm` |
 | `story.chat` | `default` | `storychat/default.json` | the shared chat shards `Adv/Chat/AdvChat-Text`, `Adv/Chat/AdvChat-Sound` and `Adv/Chat/AdvChat-SoundCueSheet` | `chat.json` |
 | `story` | a `MasterAdv` id | `stories/<id>.json` | the episode key `Adv/Episode/<asset>/<asset>` (`<asset>`: the row's `_advEpisodeAsset`), its shard keys (the key with `-Text`, `-Sound`, `-SoundCueSheet`, `-Video`) and `EmbCommon/Adv/Settings/AdvPlayerSettings` | `episode.json` and `story.json` |
 
-`story.sound`, `story.crilips`, `story.chat` and `story` keep no objects: their files are all they hold.
+`story.sound`, `story.crilips`, `story.video`, `story.chat` and `story` keep no objects: their files are all they
+hold.
 `story.sound` reads its cue sheet's `cri.audio` files; `cues.json` maps each cue name to the
 first stream of that name (in stream order): `file` (the sound, next to `cues.json`), `sampleRate`, `channels`,
 `samples` (frames at `sampleRate`, without encoder padding) and, for a looping stream, `loopStart` and `loopEnd`
@@ -391,8 +396,18 @@ the product `unsupported` (`unsupported.crilips.version`). Only a request makes
 The renderers' shaders and post-processing textures are reached through the graphics settings' references (the
 render pipeline asset, its renderer data and their features). `story.asset` holds the objects of one catalog key
 and what they reach; an asset several stories load is one product. `asset.json` has `format` 1, `key`, `objects`
-(the key's object ids), `bindings` and `shaders` as above. The CRI Lips data and the story assets are made only when
-a request names them; the `product` layout makes `story.scene` when the selection holds every key of it.
+(the key's object ids), `bindings` and `shaders` as above. The CRI Lips data, the story assets and the videos are
+made only when a request names them; the `product` layout makes `story.scene` when the selection holds every key of
+it.
+
+`story.video` is one movie of a story. Its key loads a CriWare.Assets MonoBehaviour describing the movie and the movie
+itself, a USM; `movie.webm` is the `movie.webm` that `cri.movie` makes of the USM with its parameter `format` `webm`
+([stages](stages.md#cri-stages)): the first video stream, VP9 as stored or re-encoded as VP9 when it is in another
+codec or has an alpha stream (the alpha becoming the WebM's alpha channel), and the first audio stream as Opus.
+`video.json` has `format` 1, `file` (`movie.webm`), `codec` (the stored video codec: `vp9`, `h264` or
+`sofdec.prime`), `width` and `height` (the coded size), `displayWidth` and `displayHeight`, `frameRate`
+(`[numerator, denominator]`), `frames` and `tracks`, the tracks of `movie.webm` (`kind`, `channel`, `codec` and, for a
+converted stream, its `source` codec).
 
 `story.chat` holds the rows of the chat shards every chat of a story reads. `chat.json` has `format` 1, `text` (the
 chat message texts), `sounds` (their sound rows) and `cueSheets` (the cue sheet rows those name), each the `_allData`
@@ -412,11 +427,11 @@ language, the voices, sounds, cue sheets and videos the commands name, and the r
 | `episode` | `"episode.json"`, the file next to it |
 | `chats` | `{chat id: MasterAdvChat row}`: each chat id the commands name (`TargetChatID`) |
 | `effects` | `{TargetName: "Adv/Effect/<asset>"}`: the effect instances, the first asset an Effect command loads under a name |
-| `parts` | the manifests of the products the story plays with: `scene` (`storyscenes/default.json`), `assets` (`{address: manifest}` of each `story.asset`), `models` (`{address: manifest}` of each `live2d.model`), `sounds` (`{cue sheet: manifest}` of each `story.sound`), `chat` (`storychat/default.json` when a command names a chat, else null) and `crilips` (`crilips/default.json` when a voice command has a target and voice cues and sets neither `IgnoreData` nor `IgnoreLipSync`, else null) |
+| `parts` | the manifests of the products the story plays with: `scene` (`storyscenes/default.json`), `assets` (`{address: manifest}` of each `story.asset`), `models` (`{address: manifest}` of each `live2d.model`), `sounds` (`{cue sheet: manifest}` of each `story.sound`), `videos` (`{address: manifest}` of each `story.video`), `chat` (`storychat/default.json` when a command names a chat, else null) and `crilips` (`crilips/default.json` when a voice command has a target and voice cues and sets neither `IgnoreData` nor `IgnoreLipSync`, else null) |
 
 A model's MotionSync (model.json `motionSync`) decides whether a lip-synced voice runs the CRI Lips analysis, so
-`crilips` names the data whenever a voice can reach it. The episode's talk windows and videos are listed in
-`episode.json` `resources`. `parts` depend on the episode's commands, which only the story's result holds: a command
+`crilips` names the data whenever a voice can reach it. The episode's talk windows are listed in `episode.json`
+`resources`. `parts` depend on the episode's commands, which only the story's result holds: a command
 asks for the `story` product first, reads its `story.json`, then asks for the story with every product `parts` names
 (`nnnotes.pipeline.story.part_products`); `export story` works this way. The `product` layout makes the story of
 every episode whose key the selection holds.

docs/quickstart.md

@@ -68,8 +68,8 @@ nnnotes export story 10462 -o out/story_10462
 `export episode` writes the episode's command list with its lines in five languages and the index of its voices,
 sounds and videos. `export story` writes the story product of the episode and the products it plays with below
 `out/story_10462`: the story index, the stage scene, the assets, the Live2D models, the cue sheets decoded to audio
-(AAC by default, `--format` chooses another web format), and the shared chat rows and CRI Lips data when the episode
-uses them. The files are under `assets/`. Add `--json` to any command to get its report as JSON.
+(AAC by default, `--format` chooses another web format), the videos as WebM, and the shared chat rows and CRI Lips
+data when the episode uses them. The files are under `assets/`. Add `--json` to any command to get its report as JSON.
 
 ## Next
 

docs/stages.md

@@ -40,6 +40,7 @@ codec and channel of a USM) and `catalog.index` (version 2: gzip input and Japan
 | `story.crilips` | `default` | `objects` (empty); `apk:lib/arm64-v8a/libcri_lips_unity.so` | `crilips.json` and `crilips.bin` |
 | `story.scene` | `default` | `objects`: the object table of the ADV scene, its prefabs and settings and the built-in graphics settings ([products](products.md#stories)) | `scene.json` and the object table |
 | `story.asset` | a catalog key a story loads | `objects`: the object table of what the key reaches | `asset.json` and the object table |
+| `story.video` | a video key `Cri/Video/<asset>` | `objects`: the object table of the key; `usm:<key>` and `movies`: the key's USM and its `cri.movie` files with their facts ([products](products.md#stories)); the `cri.movie` task takes `format` `webm` | `video.json` and `movie.webm` |
 | `story.chat` | `default` | `objects`: the object table of the shared chat shards | `chat.json` |
 | `story` | a `MasterAdv` id | `objects`: the object table of the episode's command list, its shards and the ADV player settings ([products](products.md#stories)); `master`: the episode's master record | `episode.json` and `story.json` |
 | `link.addresses` | global | catalog index, censuses (the built-in files' too) | key -> objects (a scene key: the objects of the scene's file outside a hierarchy), Resources.Load path -> objects, built-in selector (class, MonoBehaviour script class) -> objects, serialized file -> bundle or built-in file, bundle -> keys |

src/nnnotes/cli/assets.py

@@ -131,6 +131,7 @@ def matches(self, stage: str) -> bool:
     StageSource("story.crilips", "nnnotes.pipeline.story:CriLipsStage", output=True, product=True),
     StageSource("story.scene", "nnnotes.pipeline.story:SceneStage", output=True, product=True),
     StageSource("story.asset", "nnnotes.pipeline.story:AssetStage", output=True, product=True),
+    StageSource("story.video", "nnnotes.pipeline.story:VideoStage", output=True, product=True),
     StageSource("story.chat", "nnnotes.pipeline.story:ChatStage", output=True, product=True),
     StageSource("story", "nnnotes.pipeline.story:StoryStage", output=True, product=True),
     StageSource("view.", "nnnotes.pipeline.views:stages", output=True),
@@ -658,6 +659,9 @@ def views(self) -> list[str]:
     def request(self, products: dict, selection: list[str]) -> None:
         """Ask for the products `products` {product stage: subjects} alone, with the selection `selection` (what
         their census needs) instead of the command's: only what the products reach is exported."""
+        unknown = sorted(n for n in products if n not in self.installed.stages)
+        if unknown:
+            raise ValueError(f"no product stage {', '.join(unknown)} is installed")
         self.products, self.selection = {k: sorted(v) for k, v in products.items()}, list(selection)
 
     def _wanted(self, name: str) -> bool:
@@ -686,6 +690,10 @@ def params(self, pipeline: list[str]) -> dict:
         if fmt is not None:
             for n in [n for n in pipeline if "audio" in stages[n].PARAMS]:
                 out.setdefault(n, {})["audio"] = fmt
+        fmt = getattr(self.args, "movie_format", None)
+        if fmt is not None:
+            for n in [n for n in pipeline if "format" in stages[n].PARAMS]:
+                out.setdefault(n, {})["format"] = fmt
         level = getattr(self.args, "flac_level", None)
         if level is not None:
             if not 0 <= level <= 12:

src/nnnotes/cli/export.py

@@ -288,7 +288,7 @@ def register(sub) -> None:
         "story",
         cmd_story,
         "an ADV episode -> the story product and the products it plays with (scene, assets, Live2D models, "
-        "cue sheets, chat, CRI Lips)",
+        "cue sheets, videos, chat, CRI Lips)",
     )
     c.add_argument("adv_id", type=int, help="MasterAdv id of the episode")
     c.add_argument(
@@ -302,6 +302,7 @@ def register(sub) -> None:
     flac_level_arg(c, "of the decoded sounds")
     out_arg(c, "output directory")
     product_args(c)
+    c.set_defaults(movie_format="webm")  # the story videos: cri.movie's movie.webm
 
     c = command(
         es,

src/nnnotes/content/storyepisode.py

@@ -160,13 +160,14 @@ def chat_ids(episode: dict) -> list[int]:
 
 def parts(episode: dict) -> dict:
     """What the story plays with, from episode.json: the addresses of the ASSET_KINDS resources, the addresses of the
-    live2d models, the cue sheets of the episode, whether it shows chat and whether a voice reaches the CRI Lips
-    analysis. Talk windows and videos are products of their own and are not listed."""
+    live2d models, the cue sheets of the episode, the addresses of its videos, whether it shows chat and whether a
+    voice reaches the CRI Lips analysis. Talk windows are not listed."""
     resources = episode["resources"]
     return {
         "assets": sorted(r["address"] for r in resources if r["kind"] in ASSET_KINDS),
         "models": sorted(r["address"] for r in resources if r["kind"] == "live2d"),
         "sounds": sorted({cs["_cueSheetName"] for cs in episode["cuesheets"].values() if cs.get("_cueSheetName")}),
+        "videos": sorted(r["address"] for r in resources if r["kind"] == "video"),
         "chat": bool(chat_ids(episode)),
         "crilips": lip_synced(episode),
     }
@@ -176,7 +177,7 @@ def story(episode: dict, record: dict, paths: dict) -> dict:
     """The story.json document: the MasterAdvChat row of each chat id of the episode (KeyError when there is none), the
     effect instances (TargetName -> "Adv/Effect/" + TargetAssetName) and `parts`. `paths` are the manifest paths of the
     products the story plays with, given by the caller: {"scene", "assets": {address: path}, "models": {address: path},
-    "sounds": {cue sheet: path}, "chat", "crilips"}."""
+    "sounds": {cue sheet: path}, "videos": {address: path}, "chat", "crilips"}."""
     chats = {}
     for chat_id in chat_ids(episode):
         hits = [r for r in record["MasterAdvChat"] if r["_id"] == chat_id]

src/nnnotes/content/storyvideo.py

@@ -0,0 +1,57 @@
+"""The document of one story video: a Movie / Clip command's `Cri/Video/<asset>` key.
+
+The key loads a CriWare.Assets MonoBehaviour that describes the movie (`movieInfo`: size, display size, frame rate,
+frame count, codec) and the movie itself, a USM. cri.movie converts the USM to `movie.webm` (VP9 as stored, or
+re-encoded as VP9 when the video is in another codec or has an alpha stream, the alpha becoming the WebM's alpha
+channel; the first audio stream as Opus). video.json carries the movie's description and the tracks of `movie.webm`,
+so the product keeps no objects.
+"""
+
+from __future__ import annotations
+
+import re
+
+from .prefabs import Objects, has_json
+
+__all__ = ["CODECS", "KEY_PREFIX", "VIDEO_FORMAT", "manifest_name", "video"]
+
+VIDEO_FORMAT = 1  # video.json "format"
+KEY_PREFIX = "Cri/Video/"
+ASSET = re.compile(r"[A-Za-z0-9_.-]+(/[A-Za-z0-9_.-]+)*")  # the asset part of a video key
+CODECS = {1: "sofdec.prime", 5: "h264", 9: "vp9"}  # movieInfo.codecType (CriMana.CodecType)
+
+
+def manifest_name(key: str) -> str:
+    """The manifest name of the video key `key` (`Cri/Video/<asset>`): `<asset>` with "/" as "~" (ValueError for
+    another key)."""
+    asset = key[len(KEY_PREFIX) :] if key.startswith(KEY_PREFIX) else ""
+    if not ASSET.fullmatch(asset):
+        raise ValueError(f"{key!r}: not a video key ({KEY_PREFIX}<asset>)")
+    return asset.replace("/", "~")
+
+
+def _info(objects: Objects, key: str) -> dict:
+    """The `movieInfo` of the CriWare.Assets MonoBehaviour among the objects of `key`."""
+    hits = [o for o in objects.key(key) if has_json(objects, o) and "movieInfo" in objects.json(o)]
+    if len(hits) != 1:
+        raise ValueError(f"{key}: {len(hits)} MonoBehaviours with a movieInfo, expected one")
+    return objects.json(hits[0])["movieInfo"]
+
+
+def video(objects: Objects, key: str, tracks: list[dict]) -> dict:
+    """video.json of the video key `key`: `format`, `file` (movie.webm), the movie's `codec` (as stored), `width` and
+    `height` (the coded size), `displayWidth` and `displayHeight`, `frameRate` [numerator, denominator], `frames`, and
+    `tracks`, those of movie.webm (cri.movie: kind, channel, codec and, for a converted stream, its `source` codec)."""
+    mi = _info(objects, key)
+    return {
+        "format": VIDEO_FORMAT,
+        "file": "movie.webm",
+        "codec": CODECS.get(mi["codecType"], f"codecType {mi['codecType']}"),
+        "width": mi["width"],
+        "height": mi["height"],
+        "displayWidth": mi["dispWidth"],
+        "displayHeight": mi["dispHeight"],
+        "frameRate": [mi["framerateN"], mi["framerateD"]],
+        "frames": mi["totalFrames"],
+        "tracks": tracks,
+    }