diff --git a/docs/products.md b/docs/products.md index afa71fe..eedbc31 100644 --- a/docs/products.md +++ b/docs/products.md @@ -35,7 +35,8 @@ What a task reaches: the units holding its roots, then every unit that an item o item's `refs`; a SpriteAtlas reached through a reference does not reach the sprites it packs), through the `unity.export` results. A sprite whose atlas is in another bundle has its `image` from the `sprite.crop` result of that atlas. A reached unit that the request did not export is an error of the request; an -object that a reached object references and no census holds is listed as `missing`. +object that a reached object references and no census holds is listed as `missing`. A subject without roots (a +product made from the APK alone) reaches nothing and reads no census. | Part of the task | | |---|---| @@ -348,3 +349,24 @@ paths of the parts. The object table holds the jacket Sprite and its texture; the chart's `TextAsset` is in `chart.json`. The `product` layout makes the live of each chart whose key and jacket the selection holds (with the master data), without a band. `export live` asks for one live and its parts ([export live](commands.md#export-live)). + +## Stories + +A story episode plays with products that several episodes share, made by `nnnotes.pipeline.story`: + +| Product | Subject | Manifest | Roots | Files | +|---|---|---|---|---| +| `story.sound` | a cue sheet name (`_cueSheetName` of an episode's cue sheet table) | `cuesheets/.json` | the cue sheet's key `Cri/Sound/` | `cues.json` and the cues' sounds | +| `story.crilips` | `default` | `crilips/default.json` | none (it reads the APK) | `crilips.json` and `crilips.bin` | + +Neither keeps objects. `story.sound` reads its cue sheet's `cri.audio` files; `cues.json` maps each cue name to the +first stream of that name (in stream order): `file` (the sound, next to `cues.json`), `sampleRate`, `channels`, +`samples` (frames at `sampleRate`, without encoder padding) and, for a looping stream, `loopStart` and `loopEnd` +(sample frames). The sounds are in the web format of the parameter `audio`, as for `live.music`: `aac` (the default) +records `encoderDelay`, the priming samples of the MP4 edit list; `flac` keeps the decoded FLAC. The `product` +layout makes the `story.sound` product of every cue sheet key the selection holds. + +`story.crilips` reads the CRI Lips library of the APK (`apk:lib/arm64-v8a/libcri_lips_unity.so`, in the arm64-v8a +split of an installed app) and writes the analysis data a player needs to run the CRI Lips analysis on a voice: the +descriptor `crilips.json` and the float32 weights `crilips.bin`. A library of a version nnnotes does not know makes +the product `unsupported` (`unsupported.crilips.version`). Only a request makes it. diff --git a/docs/stages.md b/docs/stages.md index aa6d268..f3f622a 100644 --- a/docs/stages.md +++ b/docs/stages.md @@ -36,6 +36,8 @@ codec and channel of a USM) and `catalog.index` (version 2: gzip input and Japan | `live.noteeffect` | a note effect set (`effect001`, `effect001Light`) | `objects`: the object table of the effect settings | `noteeffect.json` and the object table | | `live.scene` | `default` | `objects`: the object table of the scene, the bands' start timelines and backgrounds, the lane skins, the UI assets and the built-in graphics settings ([products](products.md#live-scene)); `master`: the master record of its settings, lane skins and bands | `scene.json` and the object table | | `live` | a live: `/[/]` | `objects`: the object table of the chart and the jacket ([products](products.md#lives)); `master`: the live's master record and the manifest paths of its parts | `live.json`, `chart.json`, `notes.json`, `notes.mirror.json` and the object table | +| `story.sound` | a cue sheet name | `objects`; `acb:` and `sheets`: the cue sheet and its `cri.audio` files ([products](products.md#stories)); parameter `audio` (`aac`) | `cues.json` and the cues' sounds in the web format | +| `story.crilips` | `default` | `objects` (empty); `apk:lib/arm64-v8a/libcri_lips_unity.so` | `crilips.json` and `crilips.bin` | | `link.addresses` | global | catalog index, censuses (the built-in files' too) | key -> objects (a scene key: the objects of the scene's file outside a hierarchy), Resources.Load path -> objects, built-in selector (class, MonoBehaviour script class) -> objects, serialized file -> bundle or built-in file, bundle -> keys | | `view.` | a catalog (`main`) | `master:` per master table it reads, `rules` (the view's rule and the resolvers it uses), `addresses` (the part of the address table it read: the keys looked up and the prefixes listed); no parameters, atoms or context | `view.:#view` (facts: rows, entries, gaps, unreferenced) | | `view.voices` | a catalog (`main`) | `master:
` per master table its rules read, `rules`, `addresses`, `sheets` (each cue sheet the `cri.audio` results are known by that a voice names or the rules' prefixes list: its task and ACB content id), `acb:` (the ACB of each decoded one), `episodes` (per story episode and part, the `unity.export` artifact holding the object its key names), `adv:` (those contents); context `audio` (the keys of those `cri.audio` results); runs after `unity.export` and `cri.audio` | `view.voices:#view` (facts: rows, entries, gaps, unreferenced streams) | @@ -296,6 +298,7 @@ The codes of items that are `generic`, `unsupported` or `failed`, and of limits | `unsupported.usm.codec` | a USM video or alpha stream in a codec other than VP9, H.264 or MPEG-1, an audio stream in a codec other than ADX or HCA, or an H.264 stream without a frame rate | | `unsupported.usm.stream` | a USM stream of a kind other than video, alpha, audio or subtitles | | `unsupported.usm.subtitle` | USM subtitles that are not records, or whose text is not UTF-8 (the movie's other artifacts are made) | +| `unsupported.crilips.version` | a CRI Lips library of a version whose analysis data nnnotes cannot locate | | `unsupported.class.AudioClip` | an AudioClip | | `unsupported.class.VideoClip` | a VideoClip | | `unsupported.class.MovieTexture` | a MovieTexture | diff --git a/src/nnnotes/cli/assets.py b/src/nnnotes/cli/assets.py index 91aae8e..2360e8d 100644 --- a/src/nnnotes/cli/assets.py +++ b/src/nnnotes/cli/assets.py @@ -127,6 +127,8 @@ def matches(self, stage: str) -> bool: StageSource("live.noteeffect", "nnnotes.pipeline.live:NoteEffectStage", output=True, product=True), StageSource("live.scene", "nnnotes.pipeline.live:SceneStage", output=True, product=True), StageSource("live", "nnnotes.pipeline.live:LiveStage", output=True, product=True), + StageSource("story.sound", "nnnotes.pipeline.story:SoundStage", output=True, product=True), + StageSource("story.crilips", "nnnotes.pipeline.story:CriLipsStage", output=True, product=True), StageSource("view.", "nnnotes.pipeline.views:stages", output=True), StageSource("view.voices", "nnnotes.pipeline.voices:VoiceStage", output=True), ) diff --git a/src/nnnotes/pipeline/assemble.py b/src/nnnotes/pipeline/assemble.py index 6ca202b..ba8dd74 100644 --- a/src/nnnotes/pipeline/assemble.py +++ b/src/nnnotes/pipeline/assemble.py @@ -375,15 +375,13 @@ def manifest(self, subject: str) -> str: # ---------------------------------------------------------------- planning def reach(self, subject: str, env) -> Reach: - """What `subject` reaches (kept while the results it read are the same).""" + """What `subject` reaches (kept while the results it read are the same); nothing, and no census read, for a + subject without roots.""" keys = self.keys(subject, env) - roots = units.root_objects( - env, - keys, - self.root_objects(subject, env), - self.resources(subject, env), - self.builtins(subject, env), - ) + named = (self.root_objects(subject, env), self.resources(subject, env), self.builtins(subject, env)) + if not keys and not any(named): + return Reach({"schema": contract.OBJECTS, "roots": [], "keys": {}, "objects": {}, "missing": []}) + roots = units.root_objects(env, keys, *named) by_key = {k: units.root_objects(env, [k]) for k in keys} sig = (tuple(roots), tuple((k, tuple(v)) for k, v in sorted(by_key.items())), id(units.table(env))) memo = self._memo.get(subject) diff --git a/src/nnnotes/pipeline/story.py b/src/nnnotes/pipeline/story.py new file mode 100644 index 0000000..bcb977a --- /dev/null +++ b/src/nnnotes/pipeline/story.py @@ -0,0 +1,153 @@ +"""The story products: the cue sheets a story plays (story.sound) and the CRI Lips analysis data (story.crilips). + +Both are product stages (pipeline.assemble) that keep no objects (their files are all they hold): + +story.sound subject a cue sheet name (the `_cueSheetName` of a story's cue sheet table): "#cues.json" + (kind story.sound: cue name -> the cue's first stream: `file`, `sampleRate`, `channels`, `samples`, + `loopStart`, `loopEnd` and, in AAC, `encoderDelay`) and "#" for each of those streams + (kind audio.story) in the web format of the parameter `audio` (encoders.WEB_AUDIO; default aac: + AAC-LC in MP4, each file recording its encoder delay; flac: the decoded FLAC as it is). Its root is the + cue sheet's catalog key `Cri/Sound/` and its manifest cuesheets/.json. The atom + `audio.encode` (ffmpeg) encodes the streams. A cue sheet shared by several stories (a BGM, a sound + effect sheet) is one product. +story.crilips subject `default`: "#crilips.json" (kind story.crilips: the descriptor) and + "#crilips.bin" (kind crilips.weights: the float32 weights), read from the CRI Lips library + of the APK (formats.crilips, input "apk:lib/arm64-v8a/libcri_lips_unity.so"). Its manifest is + crilips/default.json. A library none of the versions formats.crilips knows makes the product + `unsupported` (unsupported.crilips.version). + +`select` (the `product` layout) makes the story.sound product of every cue sheet key the selection holds; the +CRI Lips data is made only when a request names it. +""" + +from __future__ import annotations + +import tempfile +from pathlib import Path + +from ..content import liveaudio +from ..formats import crilips +from ..formats.encoders import DEFAULT_AUDIO_FORMAT, WEB_AUDIO, mp4_priming, web_encode +from ..unity.atoms import Unsupported +from . import contract +from .assemble import Assembly, ProductStage +from .contract import IncompatibleTask +from .layout import escape_segment +from .live import encode_atoms + +__all__ = ["CRILIPS", "SOUND", "CriLipsStage", "SoundStage", "cues"] + +SOUND, CRILIPS = "story.sound", "story.crilips" +DEFAULT = "default" +SHEET_PREFIX = "Cri/Sound/" +LIBRARY = crilips.LIB_ENTRY_SUFFIX + + +def _ffmpeg() -> str: + from ..source.config import tool + + return tool("ffmpeg", "ffmpeg") + + +def cues(streams: list[dict], ext: str) -> tuple[dict, dict]: + """The cues of a decoded cue sheet (cri.audio streams.json, in stream order) -> (cues.json: cue name -> its first + stream with `file` in the extension `ext`, {that file: the stream's decoded file}).""" + doc, files = {}, {} + for s in streams: + if s["name"] in doc: + continue + name = Path(s["file"]).stem + ext + entry = {"file": name, **{k: s[k] for k in ("sampleRate", "channels", "samples")}} + if "loopStart" in s: + entry["loopStart"], entry["loopEnd"] = s["loopStart"], s["loopEnd"] + doc[s["name"]] = entry + files[name] = s["file"] + return doc, files + + +def _sheet(subject: str) -> str: + if not subject or "/" in subject: + raise ValueError(f"{subject!r}: not a cue sheet name") + return subject + + +class SoundStage(ProductStage): + """story.sound: a cue sheet -> cues.json and its streams in the web format (module documentation).""" + + name = SOUND + version = 1 + KIND = "StorySound" + PARAMS = {"audio": DEFAULT_AUDIO_FORMAT} + + def normalize(self, params: dict | None) -> dict: + p = super().normalize(params) + if p["audio"] not in WEB_AUDIO: + raise ValueError(f"stage {self.name}: audio {p['audio']!r}: one of {', '.join(WEB_AUDIO)}") + return p + + def uses(self, subject: str, env) -> dict: + return encode_atoms(env.params["audio"]) + + def select(self, keys: list[str], env) -> list[str]: + names = (k[len(SHEET_PREFIX) :] for k in keys if k.startswith(SHEET_PREFIX)) + return sorted(n for n in names if "/" not in n) + + def keys(self, subject: str, env) -> list[str]: + return [liveaudio.sheet_key(_sheet(subject))] + + def cue_sheets(self, subject: str, env) -> list[str]: + return self.keys(subject, env) + + def manifest(self, subject: str) -> str: + return f"cuesheets/{escape_segment(_sheet(subject))}.json" + + def assemble(self, task, objects) -> Assembly: + fmt = task.params["audio"] + want = encode_atoms(fmt) + if task.atoms != want: + raise IncompatibleTask(f"task {task.id}: the audio encoder here is {want} (task: {task.atoms})") + sheet = objects.sheet(liveaudio.sheet_key(_sheet(task.subject))) + ext = WEB_AUDIO[fmt][0] + doc, sources = cues(sheet.streams(), ext) + files, delays = {}, {} + for name, src in sources.items(): + if WEB_AUDIO[fmt][1] is None: + files[name] = (sheet.file(src), ext[1:], "audio.story") + continue + with tempfile.TemporaryDirectory(prefix="nnnotes-story-sound-") as tmp: + flac = Path(tmp) / "in.flac" + flac.write_bytes(sheet.bytes(src)) + data = web_encode(_ffmpeg(), flac, Path(tmp) / f"out{ext}", fmt) + if fmt == "aac": + delays[name] = mp4_priming(data) + files[name] = (data, ext[1:], "audio.story") + for entry in doc.values(): + if entry["file"] in delays: + entry["encoderDelay"] = delays[entry["file"]] + files["cues.json"] = (contract.encode(doc), "json", SOUND) + return Assembly(files, set()) + + +class CriLipsStage(ProductStage): + """story.crilips: the CRI Lips analysis data of the APK's library (module documentation).""" + + name = CRILIPS + version = 1 + KIND = "StoryCriLips" + APK = (LIBRARY,) + + def manifest(self, subject: str) -> str: + if subject != DEFAULT: + raise ValueError(f"{subject}: the only subject is {DEFAULT}") + return f"crilips/{DEFAULT}.json" + + def assemble(self, task, objects) -> Assembly: + so = objects.depot.input_bytes(task.input(f"apk:{LIBRARY}")) + try: + blob, desc = crilips.extract(so, crilips.SUPPORTED) + except crilips.CriLipsError as e: + raise Unsupported("unsupported.crilips.version", str(e)) from None + return Assembly( + {"crilips.json": (contract.encode(desc), "json", CRILIPS), "crilips.bin": (blob, "bin", "crilips.weights")}, + set(), + ) diff --git a/tests/pipeline/test_story.py b/tests/pipeline/test_story.py new file mode 100644 index 0000000..5c9e460 --- /dev/null +++ b/tests/pipeline/test_story.py @@ -0,0 +1,175 @@ +"""The story products of nnnotes.pipeline.story on synthetic cue sheet decodes and a synthetic CRI Lips library.""" + +import pytest + +from nnnotes.formats import crilips +from nnnotes.pipeline import assemble, contract, cristages, live, story +from nnnotes.pipeline.contract import IncompatibleTask +from nnnotes.pipeline.depot import Depot +from nnnotes.pipeline.stages import Env +from nnnotes.pipeline.story import CriLipsStage, SoundStage +from tests.formats.test_crilips import _elf_with_rodata, _synthetic_version +from tests.pipeline.test_live import run, stored, table +from tests.site.test_story_site import mp4 + +# the streams of a voice sheet as cri.audio lists them: cue v2 has two layers, cue loop loops +STREAMS = [ + {"stream": 1, "name": "v1", "file": "v1.flac", "sampleRate": 48000, "channels": 1, "samples": 100}, + {"stream": 2, "name": "v2", "file": "v2.flac", "sampleRate": 48000, "channels": 1, "samples": 200}, + {"stream": 3, "name": "v2", "file": "v2_3.flac", "sampleRate": 48000, "channels": 1, "samples": 300}, + {"stream": 4, "name": "loop", "file": "loop.flac", "sampleRate": 44100, "channels": 2, "samples": 400, + "loopStart": 10, "loopEnd": 400}, +] # fmt: skip + + +def sheet(depot, name, streams=STREAMS) -> tuple[list, dict]: + """The inputs "acb:" and "sheets" of the cue sheet `name` decoded as `streams` (FLAC bytes per stream).""" + key = f"Cri/Sound/{name}" + acb = stored(depot, f"@UTF {name}".encode(), f"acb:{key}") + files = {} + for s in streams: + data = f"fLaC {name} {s['stream']}".encode() + files[s["file"]] = {"sha256": depot.put(data), "size": len(data), "ext": "flac"} + body = contract.encode(streams) + files["streams.json"] = {"sha256": depot.put(body), "size": len(body), "ext": "json"} + doc = {key: {"acb": acb.sha256, "status": "exported", "files": files}} + return [acb, stored(depot, contract.encode(doc), assemble.SHEETS_INPUT)], doc + + +def read_json(depot, art): + return contract.loads(depot.read(art["content"]["sha256"])) + + +# ---------------------------------------------------------------- subjects, keys and manifests +def test_a_cue_sheet_is_named_by_its_key(): + stage = SoundStage() + assert stage.keys("adv_voice_1", None) == ["Cri/Sound/adv_voice_1"] + assert stage.cue_sheets("adv_voice_1", None) == ["Cri/Sound/adv_voice_1"] + assert stage.manifest("adv_voice_1") == "cuesheets/adv_voice_1.json" + assert stage.manifest("a:b") == "cuesheets/a%3Ab.json" + keys = ["Cri/Sound/se_adv", "Cri/Sound/bgm/x", "Live/MusicScore/1", "Cri/Sound/adv_voice_1"] + assert stage.select(keys, None) == ["adv_voice_1", "se_adv"] + for bad in ("", "a/b"): + with pytest.raises(ValueError, match="not a cue sheet name"): + stage.keys(bad, None) + assert stage.normalize(None) == {"audio": "aac"} + with pytest.raises(ValueError, match="audio 'wav'"): + stage.normalize({"audio": "wav"}) + + +def test_the_crilips_product_has_one_subject_and_reads_the_library(): + stage = CriLipsStage() + assert stage.manifest("default") == "crilips/default.json" + with pytest.raises(ValueError, match="the only subject is default"): + stage.manifest("other") + assert stage.APK == ("lib/arm64-v8a/libcri_lips_unity.so",) + assert stage.select(["Cri/Sound/a"], None) == [] + + +def test_cues_take_the_first_stream_of_each_name(): + doc, files = story.cues(STREAMS, ".m4a") + assert doc == { + "v1": {"file": "v1.m4a", "sampleRate": 48000, "channels": 1, "samples": 100}, + "v2": {"file": "v2.m4a", "sampleRate": 48000, "channels": 1, "samples": 200}, + "loop": {"file": "loop.m4a", "sampleRate": 44100, "channels": 2, "samples": 400, "loopStart": 10, + "loopEnd": 400}, + } # fmt: skip + assert files == {"v1.m4a": "v1.flac", "v2.m4a": "v2.flac", "loop.m4a": "loop.flac"} + + +# ---------------------------------------------------------------- the products +def test_the_sound_product_keeps_the_decoded_flac(tmp_path): + depot = Depot(tmp_path / "d") + inputs, doc = sheet(depot, "adv_voice_1") + _, res, arts = run(depot, SoundStage(), "adv_voice_1", [table(depot), *inputs], {"audio": "flac"}) + assert {p: a["semantics"]["kind"] for p, a in arts.items()} == { + "cues.json": "story.sound", + "loop.flac": "audio.story", + "objects": "product.objects", + "v1.flac": "audio.story", + "v2.flac": "audio.story", + } + files = doc["Cri/Sound/adv_voice_1"]["files"] + for name in ("v1.flac", "v2.flac", "loop.flac"): + assert arts[name]["content"]["sha256"] == files[name]["sha256"] # the decode, not a copy + cues = read_json(depot, arts["cues.json"]) + assert cues == story.cues(STREAMS, ".flac")[0] and "encoderDelay" not in cues["v1"] + assert read_json(depot, arts["objects"])["objects"] == {} + + +def test_the_sound_product_in_aac_records_each_encoder_delay(tmp_path, monkeypatch): + depot = Depot(tmp_path / "d") + monkeypatch.setattr(cristages, "tool_id", lambda which, exe: f"{which}-fake/1") + monkeypatch.setattr(story, "_ffmpeg", lambda: "ffmpeg") + monkeypatch.setattr(live, "_ffmpeg", lambda: "ffmpeg") + seen = [] + + def encode(ffmpeg, src, dst, fmt): + seen.append((src.read_bytes(), fmt)) + return mp4(1024 + len(seen)) + + monkeypatch.setattr(story, "web_encode", encode) + inputs, _ = sheet(depot, "adv_voice_1") + atoms = {"audio.encode": "ffmpeg-fake/1"} + _, _, arts = run(depot, SoundStage(), "adv_voice_1", [table(depot), *inputs], None, atoms) + assert sorted(seen) == [(f"fLaC adv_voice_1 {i}".encode(), "aac") for i in (1, 2, 4)] + cues = read_json(depot, arts["cues.json"]) + assert {c: (e["file"], e["encoderDelay"]) for c, e in cues.items()} == { + "v1": ("v1.m4a", mp4_delay(depot, arts["v1.m4a"])), + "v2": ("v2.m4a", mp4_delay(depot, arts["v2.m4a"])), + "loop": ("loop.m4a", mp4_delay(depot, arts["loop.m4a"])), + } + assert {e["encoderDelay"] for e in cues.values()} == {1025, 1026, 1027} + with pytest.raises(IncompatibleTask, match="audio encoder"): + run(depot, SoundStage(), "adv_voice_1", [table(depot), *inputs], None, {"audio.encode": "ffmpeg-other/1"}) + + +def mp4_delay(depot, art) -> int: + from nnnotes.formats.encoders import mp4_priming + + return mp4_priming(depot.read(art["content"]["sha256"])) + + +def test_a_cue_sheet_that_was_not_decoded_makes_the_sound_product_unsupported(tmp_path): + depot = Depot(tmp_path / "d") + inputs, doc = sheet(depot, "adv_voice_1") + e = doc["Cri/Sound/adv_voice_1"] + doc["Cri/Sound/adv_voice_1"] = {"acb": e["acb"], "status": "unsupported", "reason": "unsupported.cri.awb_external", + "files": {}} # fmt: skip + inputs[-1] = stored(depot, contract.encode(doc), assemble.SHEETS_INPUT) + _, res, _ = run(depot, SoundStage(), "adv_voice_1", [table(depot), *inputs], {"audio": "flac"}) + (item,) = res["items"] + assert item["status"] == "unsupported" and item["reason"]["code"] == "unsupported.cri.awb_external" + + +def test_the_crilips_product_holds_the_descriptor_and_the_weights(tmp_path, monkeypatch): + depot = Depot(tmp_path / "d") + ver, rodata, _ = _synthetic_version() + so = _elf_with_rodata(rodata) + monkeypatch.setattr(crilips, "SUPPORTED", (ver,)) + lib = stored(depot, so, f"apk:{story.LIBRARY}") + _, res, arts = run(depot, CriLipsStage(), "default", [table(depot), lib]) + blob, desc = crilips.extract(so, (ver,)) + assert {p: a["semantics"]["kind"] for p, a in arts.items()} == { + "crilips.bin": "crilips.weights", + "crilips.json": "story.crilips", + "objects": "product.objects", + } + assert depot.read(arts["crilips.bin"]["content"]["sha256"]) == blob + assert read_json(depot, arts["crilips.json"]) == desc and desc["version"] == "9.9.9" + + +def test_a_library_of_an_unknown_version_makes_the_crilips_product_unsupported(tmp_path): + depot = Depot(tmp_path / "d") + lib = stored(depot, _elf_with_rodata(b"\x00" * 64 + b"nothing here"), f"apk:{story.LIBRARY}") + _, res, arts = run(depot, CriLipsStage(), "default", [table(depot), lib]) + (item,) = res["items"] + assert item["status"] == "unsupported" and item["reason"]["code"] == "unsupported.crilips.version" + assert arts == {} + + +def test_the_crilips_product_needs_the_library_in_the_apk(tmp_path): + depot = Depot(tmp_path / "d") + env = Env(depot, {"apk": {}, "products": {story.CRILIPS: ["default"]}}) + with pytest.raises(ValueError, match="libcri_lips_unity.so of the APK"): + CriLipsStage().inputs("default", env)