feat(live)!: assemble a music's BGM and the live sounds as products

docs/contracts.md

@@ -229,7 +229,7 @@ Documents written by particular stages (artifacts of no object: their ids are `<
 | `nnnotes.catalog-index/1` | `catalog.index` | [catalog-index.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/catalog-index.schema.json) |
 | `nnnotes.census/2` | `unity.census` | [census.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/census.schema.json) |
 | `nnnotes.scripts/1` | `link.scripts` | [scripts.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/scripts.schema.json) |
-| `nnnotes.addresses/2` | `link.addresses` | [addresses.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/addresses.schema.json) |
+| `nnnotes.addresses/3` | `link.addresses` | [addresses.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/addresses.schema.json) |
 | `nnnotes.view/1` | `view.<name>` | [view.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/view.schema.json) |
 | `nnnotes.objects/1` | `pipeline.assemble` (a product's object table) | [objects.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/objects.schema.json) |
 

docs/products.md

@@ -14,12 +14,22 @@ objects:
   `link.addresses`;
 - the objects of the Resources.Load paths it gives (`resources`): paths of the ResourceManager in the APK's built-in
   files, such as the materials a runtime loads by name, through the same table and compared ignoring case;
+- the objects of the built-in selectors it gives (`builtins`): objects of the APK's built-in files by class
+  (`GraphicsSettings`) or a MonoBehaviour by its script's class (`MonoBehaviour:<class>`), such as settings assets
+  the game reads without a path, through the same table;
 - the object ids the stage names besides (`root_objects`).
 
+A product stage may also read cue sheets: the catalog keys `cue_sheets` gives (each also one of its keys). A cue
+sheet is the ACB its key loads, held in a bundle (the `acb` file of an object of the key) or a raw file of the
+catalog that `cri.audio` decoded. The task's inputs then hold each ACB (`acb:<key>`) and, as `sheets`, the files
+`cri.audio` made of it (`{key: {acb, status, reason, files: {name: {sha256, size, ext}}}}`); it depends on those
+`cri.audio` tasks, and a product file may be one of those files as stored, without copying its bytes. A stage that
+reads files of the APK names them in `APK`: inputs `apk:<path in the APK>`, from the fact `apk`.
+
 All are read from the planning facts alone, so a command can ask for what its products need before anything runs:
-`assemble.request(stages, products, env)` gives the fact `roots` (`{"keys", "resources", "objects"}`), and the request of
-`unity.export` then holds the units of those objects and every unit reachable from them through the census `refs`
-([units](stages.md#units)), and nothing else of their bundles.
+`assemble.request(stages, products, env)` gives the fact `roots` (`{"keys", "resources", "builtins", "objects"}`), and
+the request of `unity.export` then holds the units of those objects and every unit reachable from them through the
+census `refs` ([units](stages.md#units)), and nothing else of their bundles.
 
 What a task reaches: the units holding its roots, then every unit that an item of a reached unit references (the
 item's `refs`), through the `unity.export` results. A sprite whose atlas is in another bundle has its `image` from the
@@ -29,7 +39,7 @@ object that a reached object references and no census holds is listed as `missin
 | Part of the task | |
 |---|---|
 | inputs | `objects`: the object table of what it reaches (`nnnotes.objects/1`); the stage's own inputs (master tables, data its parameters name) |
-| depends | `link.addresses` (when it has keys or resource paths), the `unity.export` and `sprite.crop` tasks whose results it read |
+| depends | `link.addresses` (when it has keys, resource paths or built-in selectors), the `unity.export` and `sprite.crop` tasks whose results it read, the `cri.audio` tasks of its cue sheets |
 | artifacts | `<task id>#<path>`: the product's own files; `<task id>#objects`: the object table of the objects its files use (the whole table unless the stage narrows it to some objects, or to some files of an object: a shader's programs for one graphics API) |
 | item | one for the subject: `exported`, or `unsupported` with a reason code |
 
@@ -190,3 +200,47 @@ null `focusWorld`.
 | `spine/<skeleton>.json` or `.skel` | the bytes of the SkeletonDataAsset's `skeletonJSON` (JSON when they start with `{`) |
 | `spine/<atlas>.atlas` | the bytes of each SpineAtlasAsset's `atlasFile`, as stored (an atlas two skeletons share is written once) |
 | `spine/<page>` | for each page of the atlas, the PNG of the `_MainTex` of the atlas asset's material of the same index |
+
+## Live sounds
+
+What a live plays besides its chart is two products of `nnnotes.pipeline.live`, each shared by every live that plays
+it: the BGM of a music (`live.music`) and the sounds of an option set (`live.sounds`). Their roots are the objects of
+the keys of their cue sheets, they read the sheets' `cri.audio` files, and neither keeps objects. A sound entry, of
+either, is
+
+| Field | |
+|---|---|
+| `row` | the `MasterSound` row |
+| `sheet`, `cue` | its cue sheet and cue name |
+| `category`, `categories` | the row's category and the CRI categories of the cue (ACB command 65) |
+| `volume`, `busSends` | the cue's volume (with the sheet's) and bus sends (ACB commands 146 and 111) |
+| `lengthMs`, `other` | the cue's length, its other commands as stored |
+| `layers` | one per track of the cue: `file` (`audio/<sheet>/<file>`), the stream's `stream`, `awbId`, `sampleRate`, `channels`, `samples`, `loopStart`, `loopEnd`, the waveform's `loopFlag`, the track's and synth's `volume` and `busSends`, `other` |
+
+`live.music`: its subject is a music id (`MasterLiveMusic._id`), its manifest `music/<id>.json` and its input
+`master` the music's rows of `MasterLiveMusic`, `MasterSound` and `MasterSoundCueSheet`. Its files are `music.json`
+(`format` 1, `musicId`, `music`: the BGM's `soundId`, `volume` and `startTimeSec`, and `sounds`: the BGM's entry)
+and each layer's file in the web format of the parameter `format`: `aac` (AAC-LC 160 kbit/s in MP4, the default),
+`opus`, `vorbis`, `mp3`, or `flac`, the decoded FLAC itself. An AAC layer records `encoderDelay`, the priming samples
+of the MP4 edit list, which a decoder that does not apply the edit list outputs first. The encode is bit-exact; the
+atom `audio.encode` names the ffmpeg build.
+
+`live.sounds`: its subject names the note sound sets offered besides the default, `default` or
+`NoteSePatternId=<id>,<id>,...` (the default set among them), and its manifest is `sounds/<subject>.json`. Its roots
+include the game's sound settings, the `SoundVolumeSettings` and `AppConfigDefaultData` MonoBehaviours of the APK's
+built-in files, and it reads the ACF of the APK (`assets/Cri/Sound/Sirius.acf`). Its input `master` holds the
+option defaults of preset 1 (`MasterOptionDefault`), `MasterLiveNoteSe`, `MasterLiveSe` and the `MasterSound` and
+`MasterSoundCueSheet` rows of its sounds. Its files are each sound's decoded FLAC and `sounds.json`:
+
+| Field of sounds.json | |
+|---|---|
+| `format` | 1 |
+| `sounds` | the entry of each sound by `MasterSound` id: the note sounds of the default options and of every set offered, and the live sounds of the start and finish cheers and of the finish direction of every result |
+| `categories` | the CRI category volumes of a fresh profile: each category's default volume, and the live categories' config volumes |
+| `acf`, `react` | the ACF's `name`, `buses` and `categories`; its REACT (ducking) entries |
+| `noteSe` | the note sounds of option preset 1: `types` (note sound type -> sound id), `volumes`, `mutes`, `patternId`, `useIndividualNoteSe`, `gekisouTraceVolume`; with sets offered `groups` (set id -> type -> sound id) |
+| `liveSe`, `liveSeNames` | live sound type -> sound id, and the types' names |
+| `timeline` | the intro and end timings of the cheers and the start voice |
+
+The `product` layout makes the `live.music` products of the musics whose BGM key the selection holds, and the
+default `live.sounds` when the selection holds every cue sheet of its sounds; both need the master data.

docs/schemas.md

@@ -14,7 +14,7 @@ schemas.load("plan")  # the schema as a dict
 
 | Schema | Format | Document |
 |---|---|---|
-| `addresses` | `nnnotes.addresses/2` | the address table of a catalog version and the censuses of its bundles |
+| `addresses` | `nnnotes.addresses/3` | the address table of a catalog version and the censuses of its bundles |
 | `artifact` | `nnnotes.artifact/1` | the content, provenance and semantics of one output file |
 | `catalog-diff` | `nnnotes.catalog-diff/1` | the difference between two catalog versions (`catalog diff`) |
 | `catalog-index` | `nnnotes.catalog-index/1` | the index of a catalog pair: remote and APK catalogs, every location in catalog order |

docs/stages.md

@@ -13,8 +13,8 @@ The asset export is built from three layers:
 ## Stages
 
 Every stage is at version 1, except `unity.census` (version 4: it lists what every object references, the
-hierarchy holding it and the Resources.Load paths of a ResourceManager), `link.addresses` (version 2: the
-Resources.Load paths), `live2d.model` (version 2: assembled from the exported objects), `unity.export` (version 2: one task per unit), `cri.movie` (version 2: every stream kind,
+hierarchy holding it and the Resources.Load paths of a ResourceManager), `link.addresses` (version 3: the
+Resources.Load paths and the built-in selectors), `live2d.model` (version 2: assembled from the exported objects), `unity.export` (version 2: one task per unit), `cri.movie` (version 2: every stream kind,
 codec and channel of a USM) and `catalog.index` (version 2: gzip input and Japanese remote asset placeholders).
 
 | Stage | Subject | Inputs | Outputs |
@@ -29,7 +29,9 @@ codec and channel of a USM) and `catalog.index` (version 2: gzip input and Japan
 | `live2d.model`, `live2d.cubism` | a Live2D model key (`Character/Live2D/<group>/<name>/model/<name>`) a command asks for | `objects`: the object table of what the model reaches ([products](products.md#live2d-models)); depends on `link.addresses` and the `unity.export` tasks it reads | `model.json` (the player's model) and the object table; the Cubism SDK files |
 | `room.gltf` | a spot background key (`Spot/<place>/Background/<place>`) | `objects`: the object table of what the background reaches ([products](products.md#spots)) | `room.glb` and `room.json` |
 | `spot.files` | a spot situation key (`Spot/<place>/<situation>/<situation>`) | `objects`; `master`: the spot's master record | `spot.json` and the Spine files |
-| `link.addresses` | global | catalog index, censuses (the built-in files' too) | key -> objects, Resources.Load path -> objects, serialized file -> bundle or built-in file, bundle -> keys |
+| `live.music` | a music id (`MasterLiveMusic`) | `objects`; `acb:<key>` and `sheets`: the BGM's cue sheet and its `cri.audio` files ([products](products.md#live-sounds)); `master`: the music's master record; parameter `format` (`aac`) | `music.json` and the BGM in the web format |
+| `live.sounds` | the note sound sets offered: `default` or `NoteSePatternId=<ids>` | `objects` (the sound settings of the APK's built-in files); `acb:<key>` and `sheets` of each cue sheet; `apk:assets/Cri/Sound/Sirius.acf`; `master` | `sounds.json` and the decoded sounds |
+| `link.addresses` | global | catalog index, censuses (the built-in files' too) | key -> objects, Resources.Load path -> objects, built-in selector (class, MonoBehaviour script class) -> objects, serialized file -> bundle or built-in file, bundle -> keys |
 | `view.<name>` | a catalog (`main`) | `master:<Table>` per master table it reads, `rules` (the view's rule and the resolvers it uses), `addresses` (the part of the address table it read: the keys looked up and the prefixes listed); no parameters, atoms or context | `view.<name>:<subject>#view` (facts: rows, entries, gaps, unreferenced) |
 | `view.voices` | a catalog (`main`) | `master:<Table>` per master table its rules read, `rules`, `addresses`, `sheets` (each cue sheet the `cri.audio` results are known by that a voice names or the rules' prefixes list: its task and ACB content id), `acb:<sha256>` (the ACB of each decoded one), `episodes` (per story episode and part, the `unity.export` artifact holding the object its key names), `adv:<sha256>` (those contents); context `audio` (the keys of those `cri.audio` results); runs after `unity.export` and `cri.audio` | `view.voices:<subject>#view` (facts: rows, entries, gaps, unreferenced streams) |
 
@@ -112,7 +114,7 @@ several serialized files the file's tag follows the bundle name, as in `link.add
 bundle's content changes, so a unit keeps its task across catalog versions.
 
 A run exports the units holding an object of the selected bundles, the units of the objects the fact "roots" names
-(catalog keys, Resources.Load paths and object ids: a product's roots, [products](products.md)) and every unit reachable
+(catalog keys, Resources.Load paths, built-in selectors and object ids: a product's roots, [products](products.md)) and every unit reachable
 from those through what their objects reference (the census `refs`, across the censuses of the run); with `--only-class`
 only the units whose root object is of those classes. The selection is not part of any key: a unit's task is the same
 whichever request reaches it.
@@ -174,13 +176,13 @@ whose atlas is in its own bundle.
 
 `cri.audio` and `cri.movie` take their subjects by content: a task is `cri.audio:<sha256 of the ACB>` or
 `cri.movie:<sha256 of the USM>`, so an ACB or a USM stored both as a raw file and in a bundle, or in several bundles,
-is decoded once. Raw files are told apart by their first bytes (`@UTF` ACB, `AFS2` AWB, `CRID` USM). Both read the
-HCA keycode from the `boot` input when the task runs; the key itself is never part of a task, a record or a log.
-Each task has one item (object: the task id; class `ACB` or `USM`): `exported`, or `unsupported` with a reason code.
-Their artifacts belong to no object; the `original` layout places them under each name the stage gives the content:
-the catalog keys that load a raw file, `Cri/Sound/<cue sheet>` for a cue sheet held in a bundle (the sheet's name
-when the catalog has no such key), the keys of its bundle that end in the movie asset's name for a movie held in a
-bundle.
+is decoded once. Raw files are told apart by their first bytes (`@UTF` ACB, `AFS2` AWB, `CRID` USM), read once per
+content and recorded in the depot, so a later plan reads no raw file it has seen. Both read the HCA keycode from the
+`boot` input when the task runs; the key itself is never part of a task, a record or a log. Each task has one item
+(object: the task id; class `ACB` or `USM`): `exported`, or `unsupported` with a reason code. Their artifacts belong
+to no object; the `original` layout places them under each name the stage gives the content: the catalog keys that
+load a raw file, `Cri/Sound/<cue sheet>` for a cue sheet held in a bundle (the sheet's name when the catalog has no
+such key), the keys of its bundle that end in the movie asset's name for a movie held in a bundle.
 
 | Stage | Parameters | Atoms | Artifacts (`<task id>#<role>`) |
 |---|---|---|---|

src/nnnotes/cli/assets.py

@@ -31,6 +31,7 @@
     products   {product stage: subjects} (with a product stage in the pipeline: the `product` layout makes each
                stage's products of the keys the selected bundles hold, held_keys; `export live2d` asks for one)
     roots      the root objects of those products (pipeline.assemble.request)
+    apk        {path in the APK: Input "apk:<path>"} of the APK files the product stages read (their `APK`)
 
 Selection (--select, repeatable; the union; none: everything): `group:G` the bundles of an Addressables group (the
 stable name up to `_assets_`, else up to its first `_`), `key:PREFIX` the bundles and raw files the catalog keys
@@ -119,6 +120,8 @@ def matches(self, stage: str) -> bool:
     StageSource("live2d.cubism", "nnnotes.pipeline.live2d:CubismStage", output=True, product=True),
     StageSource("room.gltf", "nnnotes.pipeline.spot:RoomStage", output=True, product=True),
     StageSource("spot.files", "nnnotes.pipeline.spot:SpotStage", output=True, product=True),
+    StageSource("live.music", "nnnotes.pipeline.live:MusicStage", output=True, product=True),
+    StageSource("live.sounds", "nnnotes.pipeline.live:SoundsStage", output=True, product=True),
     StageSource("view.", "nnnotes.pipeline.views:stages", output=True),
     StageSource("view.voices", "nnnotes.pipeline.voices:VoiceStage", output=True),
 )
@@ -745,11 +748,15 @@ def facts(self, pipeline: list[str], fetch: bool, background: bool = False) -> d
             else:  # without master data, the products that read it have no subjects (as the views)
                 held = held_keys(index, sel.selected)
                 chosen = {
-                    n: [] if n in readers and facts["master"] is None else self.installed.stages[n].select(held)
+                    n: []
+                    if n in readers and facts["master"] is None
+                    else self.installed.stages[n].select(held, Env(self.depot, facts))
                     for n in products
                 }
             facts["products"] = chosen
             facts["roots"] = assemble.request(self.installed.stages, chosen, Env(self.depot, facts))
+            members = {m for n in products if chosen[n] for m in self.installed.stages[n].APK}
+            facts["apk"] = builtins.apk_files(self.depot, self.apk, members)
             if self.products is not None or not set(self.layouts) & set(OBJECT_LAYOUTS):
                 facts["selected"] = []  # products alone: what they reach
         if any(n in pipeline for n in RAW_STAGES):

src/nnnotes/cli/site.py

@@ -5,8 +5,8 @@
 import sys
 from pathlib import Path
 
+from ..formats.encoders import DEFAULT_AUDIO_FORMAT, WEB_AUDIO
 from ..site.compress import DEFAULT_ENCODING, ENCODINGS
-from ..site.webaudio import DEFAULT_AUDIO_FORMAT, WEB_AUDIO
 from ..source.config import ConfigError, use
 from ..source.session import Session
 from .common import (