refactor(cli): run export episode through the store

docs/architecture.md

@@ -36,10 +36,10 @@ decrypted bundles in the cache directory.
 
 ## Two ways to export
 
-- **One subject.** `export live2d`, `export spot`, `export room`, `export shader` and `export audio` run the stages
-  of their subject through the store, over the files the subject's catalog keys load, and copy the subject's files
-  from the store into the output directory; a second export of the same subject is a hit of the store. `export
-  episode`, `export story`, `export live`, `export ui`, `site build` and `music-data` read what one subject needs
+- **One subject.** `export episode`, `export live2d`, `export spot`, `export room`, `export shader` and `export
+  audio` run the stages of their subject through the store, over the files the subject's catalog keys load, and copy
+  the subject's files from the store into the output directory; a second export of the same subject is a hit of the
+  store. `export story`, `export live`, `export ui`, `site build` and `music-data` read what one subject needs
   through the Session and write their output directory directly ([Export stories and lives](export.md)).
 - **Every asset.** `export assets` splits the work into stages ([stages](stages.md)). A task is one stage applied to
   one subject (a bundle, a content id, a view); its key covers exactly what its output depends on: the stage and

docs/commands.md

@@ -153,6 +153,8 @@ takes the player settings' default transition, which is read from `[paths] apk`)
 `chatwindow` and `chaticon` from the `MasterAdvChat` row of the chat rows' chat id; `video` from the episode's video
 row of a Movie / Clip row's video id.
 
+The file is the result of the store's `adv.episode` task of the episode ([stages](stages.md)).
+
 ## export story
 
 ```

docs/configuration.md

@@ -164,7 +164,7 @@ story text); it is not a setting.
 | `master download` | `[catalog] region` and that region's `cdn`; `--latest` also that region's `api` and the client version |
 | `master version` | `[catalog] region` and that region's `api`; the client version: `[client] version` or `[paths] apk` |
 | `servers` | `[bootstrap] api`; the client version: `[client] version` or `[paths] apk` |
-| `export episode` | catalog, master data |
+| `export episode` | the store, catalog, master data |
 | `export story` | catalog, `[catalog] language`, master data, `[paths] apk`, `ffmpeg` (audio and videos), `vgmstream` (not with `--no-audio`) |
 | `export live2d` | the store (`[paths] store` or `[paths] cache`), catalog, `[paths] apk` |
 | `export spot` | the store, catalog, master data, `[paths] apk` |

docs/stages.md

@@ -28,6 +28,7 @@ gzip input and Japanese remote asset placeholders).
 | `live2d.model` | a Live2D model key (`Character/Live2D/<group>/<name>/model/<name>`) whose bundle is selected | `bundle:<n>`: the bundles of the key's dependency closure in the order the extractor loads them (the order is part of the key); context `internalId` (the key's location); atom `live2d.extract_model` (the extractor of `nnnotes export live2d`, run unchanged) | `live2d.model:<key>#<path>`: every file `nnnotes export live2d` writes for the model, `<path>` relative to the model's directory; placed under `<key>/` |
 | `room.scene` | a background key a command asks for (`export room`, `export spot`) | `bundle:<n>`: the bundles of the key's closure in load order, as `live2d.model`; atom `room.extract_room` (the glTF writer of `nnnotes export room`) | `room.scene:<key>#room.glb`: the scene (facts: `meshCount`, `inactiveMeshes`); `#room.json`: its report, naming the scene `room.glb` (`export room` names its own file) |
 | `spot.scene` | a Spot id (`MasterHomeSpot`) a command asks for (`export spot`), with master data | `master:<Table>`: `MasterHomeSpot`, `MasterStoryHomeSpotTapTalkEpisode`, `MasterText`; `bundle:<n>`: the bundles of the Spot's situation key in load order; context `resources` (the bundle names of the closures of the situation and background keys, which `spot.json` lists); atom `spot.extract` | `spot.scene:<id>#spot.json` (facts: `tapTargets`, `skeletons`) and `#spine/<file>`: the files `nnnotes export spot` writes for the Spot besides its room and shaders |
+| `adv.episode` | a MasterAdv id a command asks for (`export episode`), with master data | `master:<Table>`: `MasterAdv`, `MasterAdvChat`, `MasterText`; `bundle:<stable name>`: the bundles of the closures of the episode's keys the catalog has (the episode asset, its `-Text`, `-Sound`, `-SoundCueSheet` and `-Video` shards, the ADV player settings); `keys`: the catalog's keys under the prefixes of the episode's resources (whether the catalog has a resource is part of the episode); context `closures` (each key's bundles in load order); atom `adv.extract` | `adv.episode:<id>#episode.json`: the file `nnnotes export episode` writes (facts: `commands`, `resources`) |
 | `spine.skeleton` | a SkeletonDataAsset of a selected bundle, by its stable address `<bundle>:<pathId>` (in a bundle of several serialized files the file's tag follows the bundle name, as in `link.addresses`) | `bundle:<stable name>`: the bundles holding the serialized files the asset's file reaches through its externals; context `object` (the asset's object id); atom `spot.write_skeleton` (the Spine writer of `nnnotes export spot`) | `spine.skeleton:<subject>#<file>`: the skeleton (`.json` or `.skel`), its atlases (`.atlas`) and their page images, the bytes `nnnotes export spot` writes into `spine/`; one item for the asset |
 | `link.addresses` | global | catalog index, censuses | key -> objects, serialized file -> bundle, bundle -> keys |
 | `link.artifacts` | global | the above, the results | object id -> artifact ids |

src/nnnotes/cli/assets.py

@@ -76,7 +76,7 @@
 REPORTS = "_reports"
 SELECT_KINDS = ("group", "key", "bundle")
 RAW_STAGES = ("cri.audio", "cri.movie")
-MASTER_STAGES = ("spot.scene",)  # stages that read master data (besides the views)
+MASTER_STAGES = ("spot.scene", "adv.episode")  # stages that read master data (besides the views)
 EXPORT_STAGE = "unity.export"
 CATALOG_SUBJECT = "main"
 EXIT_FAILED, EXIT_USAGE, EXIT_ABORTED, EXIT_INCOMPATIBLE = 1, 2, 3, 4
@@ -111,6 +111,7 @@ def matches(self, stage: str) -> bool:
     StageSource("spine.skeleton", "nnnotes.pipeline.modelstages:SpineStage", output=True),
     StageSource("room.scene", "nnnotes.pipeline.modelstages:RoomStage", output=True),
     StageSource("spot.scene", "nnnotes.pipeline.modelstages:SpotStage", output=True),
+    StageSource("adv.episode", "nnnotes.pipeline.advstages:EpisodeStage", output=True),
     StageSource("link.artifacts", "nnnotes.pipeline.link:ArtifactsStage"),
     StageSource("view.", "nnnotes.pipeline.views:stages", output=True),
     StageSource("view.voices", "nnnotes.pipeline.voices:VoiceStage", output=True),

src/nnnotes/cli/export.py

@@ -32,16 +32,23 @@
 
 
 def cmd_episode(args, cfg):
-    from ..content import adv
+    from ..pipeline import contract
+    from ..pipeline.advstages import EPISODE, episode_keys
+    from ..source import master
 
     md = Session(cfg).master_dir()
     known_row(args, md, "MasterAdv", args.adv_id, "episode")
-    cat = Session(cfg).catalog()
-    doc = adv.to_json(adv.extract(cat, md, args.adv_id))
+    (row,) = [r for r in master.table(md, "MasterAdv") if r["_id"] == args.adv_id]
+    run = assets.SubjectRun(args, cfg)
+    subject = str(args.adv_id)
+    run.run([EPISODE], keys=episode_keys(row["_advEpisodeAsset"]), requests={EPISODE: [subject]})
+    doc = run.result(contract.task_id(EPISODE, subject))
     out = Path(args.out)
     out.parent.mkdir(parents=True, exist_ok=True)
-    write_json(out, doc)
-    r = {"out": str(out), "commands": doc["commandCount"], "resources": len(doc["resources"])}
+    (art,) = doc["artifacts"]
+    write_file(out, run.store.read(art["content"]["sha256"]))
+    facts = art["semantics"]["facts"]
+    r = {"out": str(out), "commands": facts["commands"], "resources": facts["resources"]}
     report(args, r, lambda r: [f"{r['out']}  ({r['commands']} commands, {r['resources']} resources)"])
 
 

src/nnnotes/pipeline/advstages.py

@@ -0,0 +1,123 @@
+"""The ADV stages: an episode's script and shards as `nnnotes export episode` writes them.
+
+adv.episode, one task per MasterAdv id a command asks for (the fact "requests"), with master data (the fact "master"):
+    inputs     "master:<Table>": MasterAdv, MasterAdvChat and MasterText (the tables adv.extract reads);
+               "bundle:<stable name>": the bundles of the closures of the episode's keys the catalog has (the
+               episode asset Adv/Episode/<asset>/<asset>, its -Text, -Sound, -SoundCueSheet and -Video shards, and
+               the ADV player settings, which name the default transition); "keys": the keys of the catalog that
+               start with a prefix of the episode's resources (key_prefixes: whether the catalog has a resource is
+               part of episode.json)
+    context    {"closures": {key: [stable names of its bundles in load order]}}
+    artifacts  "adv.episode:<id>#episode.json": the episode as `nnnotes export episode` writes it (facts: commands,
+               resources)
+
+The task runs adv.extract through TaskCatalog, its one atom ("adv.extract"); this module imports the extractor (and
+UnityPy) only when it plans or runs a task.
+"""
+
+from __future__ import annotations
+
+import json
+import tempfile
+from pathlib import Path
+
+from . import contract
+from .atoms import impl_id
+from .contract import Cost, Input
+from .modelstages import LIBRARIES, ClosureStage, TaskCatalog, check_atoms, file_artifact, requested, write_master
+from .stages import Output
+
+EPISODE = "adv.episode"
+EPISODE_TABLES = ("MasterAdv", "MasterAdvChat", "MasterText")
+SHARDS = ("", "-Text", "-Sound", "-SoundCueSheet", "-Video")
+
+
+def key_prefixes() -> list[str]:
+    """The prefixes of the addresses adv.closure asks the catalog about (adv.RESOURCE_PREFIX and the resources
+    derived from other fields), sorted."""
+    from ..content import adv
+
+    talk = adv.TALK_WINDOW_PREFIX
+    return sorted(
+        {
+            *(prefix for _kind, prefix in adv.RESOURCE_PREFIX.values()),
+            adv.TRANSITION_PREFIX,
+            talk,
+            adv.EMBEDDED_PREFIX + talk,
+            adv.CHAT_WINDOW_PREFIX,
+            adv.CHAT_ICON_PREFIX,
+            adv.VIDEO_ADDRESS.format(""),
+        }
+    )
+
+
+def episode_keys(asset: str) -> list[str]:
+    """The keys adv.extract reads for the episode asset `asset` (its script and shards, the player settings)."""
+    from ..content import adv
+
+    base = f"Adv/Episode/{asset}/{asset}"
+    return [base + s for s in SHARDS] + [adv.PLAYER_SETTINGS_KEY]
+
+
+def _rows(path: Path) -> list[dict]:
+    return json.loads(path.read_bytes())["_allData"]
+
+
+def _stored(store, role: str, data: bytes) -> Input:
+    return Input(role, store.put(data), len(data), None, ({"kind": "store"},))
+
+
+class EpisodeStage(ClosureStage):
+    """adv.episode: a MasterAdv id -> episode.json of `nnnotes export episode` (module documentation). Subjects:
+    the ids a command asks for (the fact "requests"), when the fact "master" names the master data."""
+
+    name = EPISODE
+    version = 1
+    ATOMS = {"adv.extract": impl_id(LIBRARIES, 1)}
+
+    def subjects(self, env) -> list[str]:
+        if env.facts.get("master") is None:
+            return []
+        return requested(env, self.name)
+
+    def row(self, subject: str, env) -> dict:
+        p = Path(env.fact("master")) / f"{EPISODE_TABLES[0]}.json"
+        for r in _rows(p):
+            if r.get("_id") == int(subject):
+                return r
+        raise ValueError(f"{self.name}: no MasterAdv row {subject}")
+
+    def keys(self, subject: str, env) -> list[str]:
+        """The keys of the episode the catalog has."""
+        c = self.closures(env)
+        return [k for k in episode_keys(self.row(subject, env)["_advEpisodeAsset"]) if k in c.by_key]
+
+    def inputs(self, subject: str, env) -> list[Input]:
+        what = f"{self.name} {subject}"
+        bundles, _ = self.closure_context(self.keys(subject, env), env, what)
+        prefixes = tuple(key_prefixes())
+        known = [k for k in self.closures(env).keys() if k.startswith(prefixes)]
+        keys = _stored(env.store, "keys", contract.encode({"prefixes": list(prefixes), "keys": known}))
+        return self.master_inputs(EPISODE_TABLES, env, what) + bundles + [keys]
+
+    def context(self, subject: str, env) -> dict:
+        return self.closure_context(self.keys(subject, env), env, f"{self.name} {subject}")[1]
+
+    def estimate(self, subject: str, env, inputs: list[Input]) -> Cost:
+        size = sum(i.size for i in inputs)
+        return Cost(0.3 + size * 1e-7, (320 << 20) + 8 * size)
+
+    def run(self, task, store) -> Output:
+        check_atoms(task, self.ATOMS)
+        from ..content import adv
+        from ..core.jsonio import write_json
+
+        known = contract.loads(store.input_bytes(task.input("keys")))["keys"]
+        cat = TaskCatalog.of_closures(task, store, known=known)
+        with tempfile.TemporaryDirectory(prefix="nnnotes-adv-") as d:
+            mdir = write_master(task, store, Path(d, "master"))
+            doc = adv.to_json(adv.extract(cat, mdir, int(task.subject)))  # pyright: ignore[reportArgumentType]
+            path = write_json(Path(d, "episode.json"), doc)
+            facts = {"commands": doc["commandCount"], "resources": len(doc["resources"])}
+            arts = [file_artifact(task, store, "episode.json", path.read_bytes(), "adv.episode", "json", facts)]
+        return Output(arts, [])