feat(pipeline): add the story.sound and story.crilips products

docs/products.md

@@ -35,7 +35,8 @@ What a task reaches: the units holding its roots, then every unit that an item o
 item's `refs`; a SpriteAtlas reached through a reference does not reach the sprites it packs), through the
 `unity.export` results. A sprite whose atlas is in another bundle has its `image` from the
 `sprite.crop` result of that atlas. A reached unit that the request did not export is an error of the request; an
-object that a reached object references and no census holds is listed as `missing`.
+object that a reached object references and no census holds is listed as `missing`. A subject without roots (a
+product made from the APK alone) reaches nothing and reads no census.
 
 | Part of the task | |
 |---|---|
@@ -348,3 +349,24 @@ paths of the parts.
 The object table holds the jacket Sprite and its texture; the chart's `TextAsset` is in `chart.json`. The `product`
 layout makes the live of each chart whose key and jacket the selection holds (with the master data), without a band.
 `export live` asks for one live and its parts ([export live](commands.md#export-live)).
+
+## Stories
+
+A story episode plays with products that several episodes share, made by `nnnotes.pipeline.story`:
+
+| Product | Subject | Manifest | Roots | Files |
+|---|---|---|---|---|
+| `story.sound` | a cue sheet name (`_cueSheetName` of an episode's cue sheet table) | `cuesheets/<sheet>.json` | the cue sheet's key `Cri/Sound/<sheet>` | `cues.json` and the cues' sounds |
+| `story.crilips` | `default` | `crilips/default.json` | none (it reads the APK) | `crilips.json` and `crilips.bin` |
+
+Neither keeps objects. `story.sound` reads its cue sheet's `cri.audio` files; `cues.json` maps each cue name to the
+first stream of that name (in stream order): `file` (the sound, next to `cues.json`), `sampleRate`, `channels`,
+`samples` (frames at `sampleRate`, without encoder padding) and, for a looping stream, `loopStart` and `loopEnd`
+(sample frames). The sounds are in the web format of the parameter `audio`, as for `live.music`: `aac` (the default)
+records `encoderDelay`, the priming samples of the MP4 edit list; `flac` keeps the decoded FLAC. The `product`
+layout makes the `story.sound` product of every cue sheet key the selection holds.
+
+`story.crilips` reads the CRI Lips library of the APK (`apk:lib/arm64-v8a/libcri_lips_unity.so`, in the arm64-v8a
+split of an installed app) and writes the analysis data a player needs to run the CRI Lips analysis on a voice: the
+descriptor `crilips.json` and the float32 weights `crilips.bin`. A library of a version nnnotes does not know makes
+the product `unsupported` (`unsupported.crilips.version`). Only a request makes it.

docs/stages.md

@@ -36,6 +36,8 @@ codec and channel of a USM) and `catalog.index` (version 2: gzip input and Japan
 | `live.noteeffect` | a note effect set (`effect001`, `effect001Light`) | `objects`: the object table of the effect settings | `noteeffect.json` and the object table |
 | `live.scene` | `default` | `objects`: the object table of the scene, the bands' start timelines and backgrounds, the lane skins, the UI assets and the built-in graphics settings ([products](products.md#live-scene)); `master`: the master record of its settings, lane skins and bands | `scene.json` and the object table |
 | `live` | a live: `<musicId>/<difficulty>[/<band>]` | `objects`: the object table of the chart and the jacket ([products](products.md#lives)); `master`: the live's master record and the manifest paths of its parts | `live.json`, `chart.json`, `notes.json`, `notes.mirror.json` and the object table |
+| `story.sound` | a cue sheet name | `objects`; `acb:<key>` and `sheets`: the cue sheet and its `cri.audio` files ([products](products.md#stories)); parameter `audio` (`aac`) | `cues.json` and the cues' sounds in the web format |
+| `story.crilips` | `default` | `objects` (empty); `apk:lib/arm64-v8a/libcri_lips_unity.so` | `crilips.json` and `crilips.bin` |
 | `link.addresses` | global | catalog index, censuses (the built-in files' too) | key -> objects (a scene key: the objects of the scene's file outside a hierarchy), Resources.Load path -> objects, built-in selector (class, MonoBehaviour script class) -> objects, serialized file -> bundle or built-in file, bundle -> keys |
 | `view.<name>` | a catalog (`main`) | `master:<Table>` per master table it reads, `rules` (the view's rule and the resolvers it uses), `addresses` (the part of the address table it read: the keys looked up and the prefixes listed); no parameters, atoms or context | `view.<name>:<subject>#view` (facts: rows, entries, gaps, unreferenced) |
 | `view.voices` | a catalog (`main`) | `master:<Table>` per master table its rules read, `rules`, `addresses`, `sheets` (each cue sheet the `cri.audio` results are known by that a voice names or the rules' prefixes list: its task and ACB content id), `acb:<sha256>` (the ACB of each decoded one), `episodes` (per story episode and part, the `unity.export` artifact holding the object its key names), `adv:<sha256>` (those contents); context `audio` (the keys of those `cri.audio` results); runs after `unity.export` and `cri.audio` | `view.voices:<subject>#view` (facts: rows, entries, gaps, unreferenced streams) |
@@ -296,6 +298,7 @@ The codes of items that are `generic`, `unsupported` or `failed`, and of limits
 | `unsupported.usm.codec` | a USM video or alpha stream in a codec other than VP9, H.264 or MPEG-1, an audio stream in a codec other than ADX or HCA, or an H.264 stream without a frame rate |
 | `unsupported.usm.stream` | a USM stream of a kind other than video, alpha, audio or subtitles |
 | `unsupported.usm.subtitle` | USM subtitles that are not records, or whose text is not UTF-8 (the movie's other artifacts are made) |
+| `unsupported.crilips.version` | a CRI Lips library of a version whose analysis data nnnotes cannot locate |
 | `unsupported.class.AudioClip` | an AudioClip |
 | `unsupported.class.VideoClip` | a VideoClip |
 | `unsupported.class.MovieTexture` | a MovieTexture |

src/nnnotes/cli/assets.py

@@ -127,6 +127,8 @@ def matches(self, stage: str) -> bool:
     StageSource("live.noteeffect", "nnnotes.pipeline.live:NoteEffectStage", output=True, product=True),
     StageSource("live.scene", "nnnotes.pipeline.live:SceneStage", output=True, product=True),
     StageSource("live", "nnnotes.pipeline.live:LiveStage", output=True, product=True),
+    StageSource("story.sound", "nnnotes.pipeline.story:SoundStage", output=True, product=True),
+    StageSource("story.crilips", "nnnotes.pipeline.story:CriLipsStage", output=True, product=True),
     StageSource("view.", "nnnotes.pipeline.views:stages", output=True),
     StageSource("view.voices", "nnnotes.pipeline.voices:VoiceStage", output=True),
 )

src/nnnotes/pipeline/assemble.py

@@ -375,15 +375,13 @@ def manifest(self, subject: str) -> str:
 
     # ---------------------------------------------------------------- planning
     def reach(self, subject: str, env) -> Reach:
-        """What `subject` reaches (kept while the results it read are the same)."""
+        """What `subject` reaches (kept while the results it read are the same); nothing, and no census read, for a
+        subject without roots."""
         keys = self.keys(subject, env)
-        roots = units.root_objects(
-            env,
-            keys,
-            self.root_objects(subject, env),
-            self.resources(subject, env),
-            self.builtins(subject, env),
-        )
+        named = (self.root_objects(subject, env), self.resources(subject, env), self.builtins(subject, env))
+        if not keys and not any(named):
+            return Reach({"schema": contract.OBJECTS, "roots": [], "keys": {}, "objects": {}, "missing": []})
+        roots = units.root_objects(env, keys, *named)
         by_key = {k: units.root_objects(env, [k]) for k in keys}
         sig = (tuple(roots), tuple((k, tuple(v)) for k, v in sorted(by_key.items())), id(units.table(env)))
         memo = self._memo.get(subject)

src/nnnotes/pipeline/story.py

@@ -0,0 +1,153 @@
+"""The story products: the cue sheets a story plays (story.sound) and the CRI Lips analysis data (story.crilips).
+
+Both are product stages (pipeline.assemble) that keep no objects (their files are all they hold):
+
+story.sound    subject a cue sheet name (the `_cueSheetName` of a story's cue sheet table): "<task id>#cues.json"
+               (kind story.sound: cue name -> the cue's first stream: `file`, `sampleRate`, `channels`, `samples`,
+               `loopStart`, `loopEnd` and, in AAC, `encoderDelay`) and "<task id>#<file>" for each of those streams
+               (kind audio.story) in the web format of the parameter `audio` (encoders.WEB_AUDIO; default aac:
+               AAC-LC in MP4, each file recording its encoder delay; flac: the decoded FLAC as it is). Its root is the
+               cue sheet's catalog key `Cri/Sound/<sheet>` and its manifest cuesheets/<sheet>.json. The atom
+               `audio.encode` (ffmpeg) encodes the streams. A cue sheet shared by several stories (a BGM, a sound
+               effect sheet) is one product.
+story.crilips  subject `default`: "<task id>#crilips.json" (kind story.crilips: the descriptor) and
+               "<task id>#crilips.bin" (kind crilips.weights: the float32 weights), read from the CRI Lips library
+               of the APK (formats.crilips, input "apk:lib/arm64-v8a/libcri_lips_unity.so"). Its manifest is
+               crilips/default.json. A library none of the versions formats.crilips knows makes the product
+               `unsupported` (unsupported.crilips.version).
+
+`select` (the `product` layout) makes the story.sound product of every cue sheet key the selection holds; the
+CRI Lips data is made only when a request names it.
+"""
+
+from __future__ import annotations
+
+import tempfile
+from pathlib import Path
+
+from ..content import liveaudio
+from ..formats import crilips
+from ..formats.encoders import DEFAULT_AUDIO_FORMAT, WEB_AUDIO, mp4_priming, web_encode
+from ..unity.atoms import Unsupported
+from . import contract
+from .assemble import Assembly, ProductStage
+from .contract import IncompatibleTask
+from .layout import escape_segment
+from .live import encode_atoms
+
+__all__ = ["CRILIPS", "SOUND", "CriLipsStage", "SoundStage", "cues"]
+
+SOUND, CRILIPS = "story.sound", "story.crilips"
+DEFAULT = "default"
+SHEET_PREFIX = "Cri/Sound/"
+LIBRARY = crilips.LIB_ENTRY_SUFFIX
+
+
+def _ffmpeg() -> str:
+    from ..source.config import tool
+
+    return tool("ffmpeg", "ffmpeg")
+
+
+def cues(streams: list[dict], ext: str) -> tuple[dict, dict]:
+    """The cues of a decoded cue sheet (cri.audio streams.json, in stream order) -> (cues.json: cue name -> its first
+    stream with `file` in the extension `ext`, {that file: the stream's decoded file})."""
+    doc, files = {}, {}
+    for s in streams:
+        if s["name"] in doc:
+            continue
+        name = Path(s["file"]).stem + ext
+        entry = {"file": name, **{k: s[k] for k in ("sampleRate", "channels", "samples")}}
+        if "loopStart" in s:
+            entry["loopStart"], entry["loopEnd"] = s["loopStart"], s["loopEnd"]
+        doc[s["name"]] = entry
+        files[name] = s["file"]
+    return doc, files
+
+
+def _sheet(subject: str) -> str:
+    if not subject or "/" in subject:
+        raise ValueError(f"{subject!r}: not a cue sheet name")
+    return subject
+
+
+class SoundStage(ProductStage):
+    """story.sound: a cue sheet -> cues.json and its streams in the web format (module documentation)."""
+
+    name = SOUND
+    version = 1
+    KIND = "StorySound"
+    PARAMS = {"audio": DEFAULT_AUDIO_FORMAT}
+
+    def normalize(self, params: dict | None) -> dict:
+        p = super().normalize(params)
+        if p["audio"] not in WEB_AUDIO:
+            raise ValueError(f"stage {self.name}: audio {p['audio']!r}: one of {', '.join(WEB_AUDIO)}")
+        return p
+
+    def uses(self, subject: str, env) -> dict:
+        return encode_atoms(env.params["audio"])
+
+    def select(self, keys: list[str], env) -> list[str]:
+        names = (k[len(SHEET_PREFIX) :] for k in keys if k.startswith(SHEET_PREFIX))
+        return sorted(n for n in names if "/" not in n)
+
+    def keys(self, subject: str, env) -> list[str]:
+        return [liveaudio.sheet_key(_sheet(subject))]
+
+    def cue_sheets(self, subject: str, env) -> list[str]:
+        return self.keys(subject, env)
+
+    def manifest(self, subject: str) -> str:
+        return f"cuesheets/{escape_segment(_sheet(subject))}.json"
+
+    def assemble(self, task, objects) -> Assembly:
+        fmt = task.params["audio"]
+        want = encode_atoms(fmt)
+        if task.atoms != want:
+            raise IncompatibleTask(f"task {task.id}: the audio encoder here is {want} (task: {task.atoms})")
+        sheet = objects.sheet(liveaudio.sheet_key(_sheet(task.subject)))
+        ext = WEB_AUDIO[fmt][0]
+        doc, sources = cues(sheet.streams(), ext)
+        files, delays = {}, {}
+        for name, src in sources.items():
+            if WEB_AUDIO[fmt][1] is None:
+                files[name] = (sheet.file(src), ext[1:], "audio.story")
+                continue
+            with tempfile.TemporaryDirectory(prefix="nnnotes-story-sound-") as tmp:
+                flac = Path(tmp) / "in.flac"
+                flac.write_bytes(sheet.bytes(src))
+                data = web_encode(_ffmpeg(), flac, Path(tmp) / f"out{ext}", fmt)
+            if fmt == "aac":
+                delays[name] = mp4_priming(data)
+            files[name] = (data, ext[1:], "audio.story")
+        for entry in doc.values():
+            if entry["file"] in delays:
+                entry["encoderDelay"] = delays[entry["file"]]
+        files["cues.json"] = (contract.encode(doc), "json", SOUND)
+        return Assembly(files, set())
+
+
+class CriLipsStage(ProductStage):
+    """story.crilips: the CRI Lips analysis data of the APK's library (module documentation)."""
+
+    name = CRILIPS
+    version = 1
+    KIND = "StoryCriLips"
+    APK = (LIBRARY,)
+
+    def manifest(self, subject: str) -> str:
+        if subject != DEFAULT:
+            raise ValueError(f"{subject}: the only subject is {DEFAULT}")
+        return f"crilips/{DEFAULT}.json"
+
+    def assemble(self, task, objects) -> Assembly:
+        so = objects.depot.input_bytes(task.input(f"apk:{LIBRARY}"))
+        try:
+            blob, desc = crilips.extract(so, crilips.SUPPORTED)
+        except crilips.CriLipsError as e:
+            raise Unsupported("unsupported.crilips.version", str(e)) from None
+        return Assembly(
+            {"crilips.json": (contract.encode(desc), "json", CRILIPS), "crilips.bin": (blob, "bin", "crilips.weights")},
+            set(),
+        )