feat(pipeline): add the story and story.chat products

docs/cli.md

@@ -195,18 +195,19 @@ an ADV episode -> JSON
 
 ## nnnotes export story
 
-an ADV episode -> story directory (episode, audio, scene, UI, media, videos) + its Live2D models
+an ADV episode -> the story product and the products it plays with (scene, assets, Live2D models, cue sheets, chat, CRI Lips)
 
 | Argument | Description |
 | --- | --- |
 | `adv_id` | MasterAdv id of the episode |
+| `--format` `FORMAT` | web audio format of the cue sheets (default aac) (one of `aac`, `flac`, `mp3`, `opus`, `vorbis`) |
+| `--flac-level` `N` | FLAC compression level 0-12 of the decoded sounds (default 8; every level decodes to the same samples) |
 | `-o`, `--out` `OUT` | output directory (required) |
-| `--models` `DIR` | directory of the Live2D models, one <model id>/ each (default: OUT/../live2d) |
-| `--force` | export the models again whose directory exists |
-| `--format` `FORMAT` | audio file format (default flac) (one of `flac`, `ogg`, `wav`) |
-| `--flac-level` `N` | FLAC compression level 0-12 of --format flac (default 8; every level decodes to the same samples) |
-| `--no-audio` | do not decode the cue sheets |
-| `--fonts` `FONTS` | open (default): text layout and style only, no font data; game: also the game's TMP fonts (needs the 'fonts' extra) (one of `open`, `game`) |
+| `--stored-catalog` `LABEL\|SHA` | a catalog version of the depot (`catalog versions`; default: the current catalog) |
+| `--workers` `N` | worker processes (default: the usable CPUs) |
+| `--link` `LINK` | as export assets --link (one of `auto`, `clone`, `hard`, `copy`) |
+| `--explain` | print every setting of the run first |
+| `--dry-run` | print the plan, run nothing |
 
 ## nnnotes export live
 

docs/configuration.md

@@ -86,7 +86,7 @@ exist when they are given.
 | `[master] key` | `NNNOTES_MASTER_KEY` | — | Rijndael-256 key of the master data files: 32 bytes as 64 hex digits |
 | `[master] iv` | `NNNOTES_MASTER_IV` | — | Rijndael-256 CBC initialization vector: 32 bytes as 64 hex digits |
 | `[catalog] region` | `NNNOTES_CATALOG_REGION` | `--region` | name of one `[servers.<region>]` table |
-| `[catalog] language` | `NNNOTES_CATALOG_LANGUAGE` | `--language` | catalog language: the `<language>` of `catalog_<version>_<language>.bin`: `ja`, `en`, `zh-Hant`, `zh-Hans` or `ko`; also the client language of `export story` and `site build` (the text field, fonts and line spacing of their UI) and of the model labels of `site build --live2d` |
+| `[catalog] language` | `NNNOTES_CATALOG_LANGUAGE` | `--language` | catalog language: the `<language>` of `catalog_<version>_<language>.bin`: `ja`, `en`, `zh-Hant`, `zh-Hans` or `ko`; also the client language of `site build` (the text field, fonts and line spacing of its story UI) and of the model labels of `site build --live2d` |
 | `[catalog] version` | `NNNOTES_CATALOG_VERSION` | `--catalog-version` | optional international resource version; unset: query the configured region API, or use legacy `main` when no API is configured; JP uses its own discovery |
 | `[servers.<region>] name` | `NNNOTES_SERVERS_<REGION>_NAME` | — | label of the region in `catalog browse` (default: the region name) |
 | `[servers.<region>] provider` | `NNNOTES_SERVERS_<REGION>_PROVIDER` | — | `international` or `jp`; empty selects `jp` for the region named jp, international otherwise |
@@ -156,7 +156,7 @@ in, and bundles that ship inside the APK can only be read with it.
 "Master data" means the decoded master data directory of the region: the `--master` flag, else
 `[servers.<region>] master` of `[catalog] region`, else `[paths] master`.
 
-`--fonts game` (`export story`, `site build`) also needs the optional `fonts` dependencies (`pip install 'nnnotes[fonts]'`),
+`--fonts game` (`site build`) also needs the optional `fonts` dependencies (`pip install 'nnnotes[fonts]'`),
 and so does `site build --story` / `--all-stories` with either font source (it generates the TextMesh Pro font assets of the
 story text); it is not a setting.
 
@@ -170,7 +170,7 @@ story text); it is not a setting.
 | `master version` | `[catalog] region` and that region's `api`; the client version: `[client] version` or `[paths] apk` |
 | `servers` | `[bootstrap] api`; the client version: `[client] version` or `[paths] apk` |
 | `export episode` | catalog, master data |
-| `export story` | catalog, `[catalog] language`, master data, `[paths] apk`, `ffmpeg` (audio and videos), `vgmstream` (not with `--no-audio`) |
+| `export story` | catalog, master data, `[paths] apk`, `vgmstream`, `ffmpeg` |
 | `export live2d` | catalog, `[paths] apk` |
 | `export spot` | catalog, master data, `[paths] apk` |
 | `export room` | catalog |

docs/export.md

@@ -15,14 +15,19 @@ nnnotes export episode 10462 -o out/episode_10462.json     # the episode's comma
 nnnotes export story 10462 -o out/stories/10462            # everything the episode plays
 ```
 
-`export story` writes the episode, its decoded cue sheets, the stage scene, the story UI, frames, effects, stills,
-chats and videos below `-o`, and its Live2D models below the models directory (`--models`, default `live2d/` next to
-the story directory). A model directory that exists is used as it is, so exporting the stories of a season into
-sibling directories exports each model once.
-
-- `--format flac|ogg|wav` and `--flac-level N` choose the audio format; `--no-audio` leaves the cue sheets undecoded.
-- `--fonts game` also writes the game's fonts reduced to the characters shown (needs `pip install 'nnnotes[fonts]'`).
-- `--force` exports the episode's models again.
+`export story` writes the story product of an episode and the products it plays with, as a product layout below `-o`:
+the story index in `stories/<id>.json`, the stage scene in `storyscenes/default.json`, each stage, frame, effect and
+other asset in `storyassets/`, each Live2D model in `models/`, each cue sheet in `cuesheets/`, and, when the episode
+uses them, the shared chat rows in `storychat/default.json` and the CRI Lips data in `crilips/default.json`. The files
+themselves are under `assets/`.
+
+Products that several stories share (the scene, assets, models and cue sheets) are placed once, so exporting several
+stories into one directory keeps one copy of each. Every product goes through the depot, so a second export reuses
+its results.
+
+- `--format NAME` chooses the web audio format of the cue sheets: `aac` (default), `opus`, `vorbis`, `mp3` or `flac`.
+- `--flac-level N` sets the FLAC compression level (0 to 12, default 8) of the decoded sounds.
+- `--dry-run` prints the plan and runs nothing.
 
 `MasterAdv` lists the episode ids; `nnnotes voices list --source story --episode ID` lists the voiced lines of one.
 Details: [export episode](commands.md#export-episode), [export story](commands.md#export-story).

docs/products.md

@@ -359,9 +359,12 @@ A story episode plays with products that several episodes share, made by `nnnote
 | `story.sound` | a cue sheet name (`_cueSheetName` of an episode's cue sheet table) | `cuesheets/<sheet>.json` | the cue sheet's key `Cri/Sound/<sheet>` | `cues.json` and the cues' sounds |
 | `story.crilips` | `default` | `crilips/default.json` | none (it reads the APK) | `crilips.json` and `crilips.bin` |
 | `story.scene` | `default` | `storyscenes/default.json` | the ADV scene `EmbScene/Adv`, the prefabs `EmbCommon/Prefab/CameraManager`, `EmbCommon/Adv/Prefab/AdvCharacterField`, `AdvBackgroundField` and `AdvGlobalVolume`, the settings `EmbCommon/Adv/Settings/AdvPlayerSettings` and `AdvMasterIdSettings`, and the built-in `GraphicsSettings`, `QualitySettings` and `PlayerSettings` | `scene.json` and the object table |
-| `story.asset` | a catalog key a story loads: a stage (`Adv/Stage/<name>`), frame, effect, post effect, still, chat window, chat icon or stamp, transition | `storyassets/<name>.json`, `<name>` the key with `/` as `~` | the key | `asset.json` and the object table |
+| `story.asset` | a catalog key a story loads: a stage (`Adv/Stage/<name>`), frame, effect, post effect, still, timeline, chat window, chat icon or stamp, transition | `storyassets/<name>.json`, `<name>` the key with `/` as `~` | the key | `asset.json` and the object table |
+| `story.chat` | `default` | `storychat/default.json` | the shared chat shards `Adv/Chat/AdvChat-Text`, `Adv/Chat/AdvChat-Sound` and `Adv/Chat/AdvChat-SoundCueSheet` | `chat.json` |
+| `story` | a `MasterAdv` id | `stories/<id>.json` | the episode key `Adv/Episode/<asset>/<asset>` (`<asset>`: the row's `_advEpisodeAsset`), its shard keys (the key with `-Text`, `-Sound`, `-SoundCueSheet`, `-Video`) and `EmbCommon/Adv/Settings/AdvPlayerSettings` | `episode.json` and `story.json` |
 
-Neither keeps objects. `story.sound` reads its cue sheet's `cri.audio` files; `cues.json` maps each cue name to the
+`story.sound`, `story.crilips`, `story.chat` and `story` keep no objects: their files are all they hold.
+`story.sound` reads its cue sheet's `cri.audio` files; `cues.json` maps each cue name to the
 first stream of that name (in stream order): `file` (the sound, next to `cues.json`), `sampleRate`, `channels`,
 `samples` (frames at `sampleRate`, without encoder padding) and, for a looping stream, `loopStart` and `loopEnd`
 (sample frames). The sounds are in the web format of the parameter `audio`, as for `live.music`: `aac` (the default)
@@ -390,3 +393,30 @@ render pipeline asset, its renderer data and their features). `story.asset` hold
 and what they reach; an asset several stories load is one product. `asset.json` has `format` 1, `key`, `objects`
 (the key's object ids), `bindings` and `shaders` as above. The CRI Lips data and the story assets are made only when
 a request names them; the `product` layout makes `story.scene` when the selection holds every key of it.
+
+`story.chat` holds the rows of the chat shards every chat of a story reads. `chat.json` has `format` 1, `text` (the
+chat message texts), `sounds` (their sound rows) and `cueSheets` (the cue sheet rows those name), each the `_allData`
+rows of its shard as stored; a shard key without a TextAsset gives `[]`. The `product` layout makes it when the
+selection holds the three shard keys.
+
+`story` is one episode. Its input `master` is the episode's master record: its `MasterAdv` row, the `MasterText` row
+of its title (none when `_titleTextId` is empty) and every `MasterAdvChat` row. `episode.json` is the episode as
+`export episode` writes it, each resource given by its kind and address: the command list with the lines of every
+language, the voices, sounds, cue sheets and videos the commands name, and the resources the episode loads.
+`story.json` has
+
+| Field | |
+|---|---|
+| `format` | 1 |
+| `advId` | the `MasterAdv` id |
+| `episode` | `"episode.json"`, the file next to it |
+| `chats` | `{chat id: MasterAdvChat row}`: each chat id the commands name (`TargetChatID`) |
+| `effects` | `{TargetName: "Adv/Effect/<asset>"}`: the effect instances, the first asset an Effect command loads under a name |
+| `parts` | the manifests of the products the story plays with: `scene` (`storyscenes/default.json`), `assets` (`{address: manifest}` of each `story.asset`), `models` (`{address: manifest}` of each `live2d.model`), `sounds` (`{cue sheet: manifest}` of each `story.sound`), `chat` (`storychat/default.json` when a command names a chat, else null) and `crilips` (`crilips/default.json` when a voice command has a target and voice cues and sets neither `IgnoreData` nor `IgnoreLipSync`, else null) |
+
+A model's MotionSync (model.json `motionSync`) decides whether a lip-synced voice runs the CRI Lips analysis, so
+`crilips` names the data whenever a voice can reach it. The episode's talk windows and videos are listed in
+`episode.json` `resources`. `parts` depend on the episode's commands, which only the story's result holds: a command
+asks for the `story` product first, reads its `story.json`, then asks for the story with every product `parts` names
+(`nnnotes.pipeline.story.part_products`); `export story` works this way. The `product` layout makes the story of
+every episode whose key the selection holds.

docs/quickstart.md

@@ -1,6 +1,6 @@
 # Quick start
 
-This walk-through takes you from an empty configuration to an exported ADV episode and a story directory. It needs
+This walk-through takes you from an empty configuration to an exported ADV episode and a story export. It needs
 your own installation of the game: its `base.apk`, and the keys and server addresses of your client
 ([where the values come from](configuration.md#where-the-values-come-from)).
 
@@ -66,9 +66,10 @@ nnnotes export story 10462 -o out/story_10462
 ```
 
 `export episode` writes the episode's command list with its lines in five languages and the index of its voices,
-sounds and videos. `export story` writes the whole story directory: the episode, its audio, the stage scene and
-shaders, the story UI, frames, effects, stills, chats and videos; the Live2D models it uses go to `out/live2d/`,
-where the next story reuses them. Add `--json` to any command to get its report as JSON.
+sounds and videos. `export story` writes the story product of the episode and the products it plays with below
+`out/story_10462`: the story index, the stage scene, the assets, the Live2D models, the cue sheets decoded to audio
+(AAC by default, `--format` chooses another web format), and the shared chat rows and CRI Lips data when the episode
+uses them. The files are under `assets/`. Add `--json` to any command to get its report as JSON.
 
 ## Next
 

docs/stages.md

@@ -40,6 +40,8 @@ codec and channel of a USM) and `catalog.index` (version 2: gzip input and Japan
 | `story.crilips` | `default` | `objects` (empty); `apk:lib/arm64-v8a/libcri_lips_unity.so` | `crilips.json` and `crilips.bin` |
 | `story.scene` | `default` | `objects`: the object table of the ADV scene, its prefabs and settings and the built-in graphics settings ([products](products.md#stories)) | `scene.json` and the object table |
 | `story.asset` | a catalog key a story loads | `objects`: the object table of what the key reaches | `asset.json` and the object table |
+| `story.chat` | `default` | `objects`: the object table of the shared chat shards | `chat.json` |
+| `story` | a `MasterAdv` id | `objects`: the object table of the episode's command list, its shards and the ADV player settings ([products](products.md#stories)); `master`: the episode's master record | `episode.json` and `story.json` |
 | `link.addresses` | global | catalog index, censuses (the built-in files' too) | key -> objects (a scene key: the objects of the scene's file outside a hierarchy), Resources.Load path -> objects, built-in selector (class, MonoBehaviour script class) -> objects, serialized file -> bundle or built-in file, bundle -> keys |
 | `view.<name>` | a catalog (`main`) | `master:<Table>` per master table it reads, `rules` (the view's rule and the resolvers it uses), `addresses` (the part of the address table it read: the keys looked up and the prefixes listed); no parameters, atoms or context | `view.<name>:<subject>#view` (facts: rows, entries, gaps, unreferenced) |
 | `view.voices` | a catalog (`main`) | `master:<Table>` per master table its rules read, `rules`, `addresses`, `sheets` (each cue sheet the `cri.audio` results are known by that a voice names or the rules' prefixes list: its task and ACB content id), `acb:<sha256>` (the ACB of each decoded one), `episodes` (per story episode and part, the `unity.export` artifact holding the object its key names), `adv:<sha256>` (those contents); context `audio` (the keys of those `cri.audio` results); runs after `unity.export` and `cri.audio` | `view.voices:<subject>#view` (facts: rows, entries, gaps, unreferenced streams) |

src/nnnotes/cli/assets.py

@@ -131,6 +131,8 @@ def matches(self, stage: str) -> bool:
     StageSource("story.crilips", "nnnotes.pipeline.story:CriLipsStage", output=True, product=True),
     StageSource("story.scene", "nnnotes.pipeline.story:SceneStage", output=True, product=True),
     StageSource("story.asset", "nnnotes.pipeline.story:AssetStage", output=True, product=True),
+    StageSource("story.chat", "nnnotes.pipeline.story:ChatStage", output=True, product=True),
+    StageSource("story", "nnnotes.pipeline.story:StoryStage", output=True, product=True),
     StageSource("view.", "nnnotes.pipeline.views:stages", output=True),
     StageSource("view.voices", "nnnotes.pipeline.voices:VoiceStage", output=True),
 )
@@ -1269,8 +1271,10 @@ def cmd_export(args, cfg):
     export(args, cfg, Workspace(args, cfg, layouts=names))
 
 
-def export(args, cfg, ws: Workspace) -> None:
-    """Run the request of `ws` and place its layouts (ws.layouts) in args.out (cmd_export; export live2d)."""
+def export(args, cfg, ws: Workspace, expand=None) -> None:
+    """Run the request of `ws` and place its layouts (ws.layouts) in args.out (cmd_export; export live2d). `expand`:
+    called with the run when it was not aborted, gives the request of a further run ((products, selection) for
+    Workspace.request) or None; the last run places the layouts (export story: the products a story names)."""
     from ..pipeline import builtins, layout
     from ..pipeline.orchestrate import Orchestrator, failure
 
@@ -1332,6 +1336,11 @@ def export(args, cfg, ws: Workspace) -> None:
         aborted = aborted or str(e)
     if isinstance(facts["raw"], Later) and aborted is None:
         ws.log(f"{sum(p['kind'] == 'raw' for p in ws.problems())} raw files not readable")
+    if expand is not None and aborted is None:
+        more = expand(run)
+        if more is not None:
+            ws.request(*more)
+            return export(args, cfg, ws)
     stage_of = {"bundle": "unity.census", "raw": next((n for n in RAW_STAGES if n in pipeline), "cri.audio")}
     for p in ws.problems():
         if p["code"] == "source.error":