diff --git a/CHANGELOG.md b/CHANGELOG.md index 3e0f008..617b398 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ All notable changes to this project are listed here. Versions follow [Semantic V over the command line (commands, options, exit statuses and `--json` reports), the configuration keys, the JSON schemas in `nnnotes.schemas` and the documented Python API. Before 1.0 a minor version can change any of them. +## Unreleased + +### Features + +- Build named sites through depot product, player read-set, asset and index tasks. +- Retain unselected regional topics, publish immutable product dependencies in order, and restore complete sites from their saved depot layouts. +- Offer Live variant products from chart manifests and select story UI/font languages explicitly. + ## [0.2.1](https://github.com/MetaSekaiLab/nnnotes/compare/v0.2.0...v0.2.1) - 2026-10-13 ### Build diff --git a/docs/architecture.md b/docs/architecture.md index c6c9af6..f016767 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -34,21 +34,15 @@ The catalog (`nnnotes.source.catalog.Catalog`) resolves an addressable key to it dependency closure, takes a bundle from the APK when the APK has it and from the CDN otherwise, and keeps fetched and decrypted bundles in the depot's `downloads/` directory. -## Two ways to export - -- **One subject.** `export episode`, `export story`, `export shader`, `export audio`, `export ui`, `site build` - and `music-data` read what one subject needs through the Session and write their output directory directly - ([Export stories and lives](export.md)). `export live`, `export live2d`, `export spot` and `export room` ask the - pipeline below for the products of one subject ([products](products.md)). -- **Every asset.** `export assets` splits the work into stages ([stages](stages.md)). A task is one stage applied to - one subject (a bundle, a content id, a view); its key covers exactly what its output depends on: the stage and - converter versions, the input contents and the options ([contracts](contracts.md)). Results and their files are - stored once by content hash in the depot, and the output directory is laid out from the depot. A second run finds - every unchanged result in the depot and only lays out what changed; `depot plan` says what would run and why - ([asset export](assets.md)). - -Both ways use the same readers below the pipeline layer, so a texture, shader, mesh or audio stream is converted by -one implementation whichever command asks for it. +## Product and asset requests + +`export assets` runs stages whose tasks are keyed by their input contents, options, stage versions and converter identities. Results and files are stored in the depot; layouts place their content into output directories. `depot plan` reports which tasks and placements would change. + +`export live`, `export story`, `export live2d`, `export spot`, `export room` and `export ui` request the products of selected subjects through that same pipeline. Stories first produce an episode, then request its referenced products. Shared objects and unchanged products reuse committed tasks. + +`site build` uses the same product requests, then runs player read-set, asset, document and index tasks. A named catalog retains unselected topics and references immutable products. Publication writes dependencies before entry manifests and indexes. The successful run stores the complete layout so a new output directory can be restored through the depot. + +`export episode`, `export shader`, `export audio` and `music-data` expose subject-specific data and conversions through the Session. All commands use the readers and converters below the pipeline layer. ## Outputs diff --git a/docs/cli.md b/docs/cli.md index 07d2755..7f16ea9 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -445,32 +445,32 @@ live charts, Live2D models and stories -> static site for ournotes-player (share | Argument | Description | | --- | --- | | `-o`, `--out` `SITE` | site directory (required) | +| `--name` `NAME` | persistent site name in the depot (default: default) | | `--player` `PLAYER` | ournotes-player checkout (built) or installed package ([paths] player) | | `--pair` `PAIR` | : (repeatable) | | `--all` | every MasterLiveMusic x difficulty | -| `--player-only` | rewrite the player files, charts.json and models.json only | -| `--reingest-json` | store every chart's and model's JSON files again | +| `--player-only` | publish the saved catalog with current player pages | | `--live2d` `MODEL` | Live2D model id or key Character/Live2D///model/ (repeatable) | | `--all-live2d` | every Live2D model of the catalog | | `--format` `FORMAT` | audio format of the charts and stories (default aac) (one of `aac`, `opus`, `vorbis`, `mp3`, `flac`) | | `--no-audio` | export no audio files | | `--compress` `COMPRESS` | encoding of the compressible assets (JSON, shaders, moc3, ...) where it makes them smaller: gzip (default), br (brotli) or none (one of `gzip`, `br`, `none`) | -| `--force` | rebuild charts, models and stories whose manifest exists (stories: also their models) | -| `--tmp` `TMP` | directory for the temporary live and model builds (default .tmp) | -| `--workers` `N` | parallel music / story / model processes (default a quarter of the CPUs, up to 8; models up to 4; 0: this process) | -| `--read-workers` `N` | chart read sets run at a time (default half the CPUs, up to 16) | +| `--workers` `N` | product task worker processes; 0: this process | +| `--memory` `GiB` | product task memory budget | +| `--fetch-workers` `N` | parallel source downloads | +| `--stored-catalog` `LABEL\|SHA` | a stored catalog version of the depot | | `--story` `ADV_ID` | story episode: its MasterAdv id (repeatable) | | `--all-stories` | every story episode (MasterAdv) of the master data | -| `--story-languages` `LANGS` | languages of the stories' text, comma-separated (default: ja,en,zh-Hant,zh-Hans,ko) | +| `--story-languages` `LANGS` | languages of the stories' text, comma-separated (default: the catalog language) | | `--font` `LANG=PATH` | font file the story text of a language is drawn with (repeatable; [paths] fonts.); emoji=PATH: the colour emoji font the emoji sprites are drawn from ([paths] fonts.emoji) | | `--site-region` `REGION` | a region the site serves: a [servers.] table (repeatable; the first is the base; default: [catalog] region) | | `--all-regions` | every configured region | -| `--fonts` `FONTS` | text of the start canvas and the stories: open (default): the start canvas has layout and style only, the stories glyphs from your font files (--font); game: the game's TMP fonts (needs the 'fonts' extra) (one of `open`, `game`) | +| `--fonts` `FONTS` | text of the start canvas and stories: open (default): glyphs from your font files (--font); game: the game's TMP fonts (needs the 'fonts' extra) (one of `open`, `game`) | | `--band` `BAND` | band of the LightWeight background and start timeline (default: band of the music's first vocal character) | | `--leader-card` `LEADER_CARD` | deck centre MasterMemberCard id (its character's band) | | `--live-option` `OPTION[=VALUES]` | also export the files of a Live option's variants (repeatable): MirrorChart, MeasureLineDisplay, NoteDesignId, NoteEffectId, LiveQuality, NoteSePatternId (every value of the master data, or =v,v...), defaults (the option defaults and ranges only) or all | -At most one of `--pair`, `--all`, `--player-only`, `--reingest-json`. +At most one of `--pair`, `--all`, `--player-only`. At most one of `--live2d`, `--all-live2d`. diff --git a/docs/commands.md b/docs/commands.md index a60e5ef..87f2979 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -451,260 +451,51 @@ does. Needs `[paths] apk` (the built-in graphics and sound settings and the CRI ## site build +```sh +nnnotes site build --name NAME -o SITE [--pair MUSIC_ID:DIFFICULTY | --all] + [--live2d MODEL | --all-live2d] [--story ADV_ID | --all-stories] + [--player DIR] [--format aac|opus|vorbis|mp3|flac] [--no-audio] + [--compress gzip|br|none] [--workers N] [--memory GiB] [--fetch-workers N] + [--stored-catalog LABEL|SHA] [--site-region REGION | --all-regions] + [--fonts open|game] [--story-languages LANGS] [--font LANG=PATH] + [--band N | --leader-card ID] [--live-option OPTION[=VALUES]] +nnnotes site build --name NAME -o SITE --player-only ``` -nnnotes site build -o SITE [--pair MUSIC_ID:DIFFICULTY [--pair ...] | --all] [--live2d MODEL [--live2d ...] | --all-live2d] - [--story ADV_ID [--story ...] | --all-stories] [--story-languages LANG,...] [--font LANG=PATH ...] -nnnotes site build -o SITE (--player-only | --reingest-json) - [--player DIR] [--format aac|opus|vorbis|mp3|flac] [--no-audio] [--compress gzip|br|none] - [--force] - [--tmp DIR] [--workers N] [--read-workers N] [--band BAND | --leader-card CARD_ID] - [--site-region REGION [--site-region ...] | --all-regions] [--fonts open|game] - [--live-option OPTION[=VALUES] ...] -``` - -Builds or updates a static [ournotes-player](https://github.com/empty-sekai/ournotes-player) site of charts, Live2D -models, stories or any of them: - -``` -SITE/index.html, chart-list.js ... the player's chart list page; ?music=&difficulty= plays one chart -SITE/ournotes-player.element.min.js the player's built bundle (and its source map) -SITE/charts.json chart index: listing facts (texts in every language), regions, - manifest path, sizes -SITE/charts/_.json chart manifest: every path the player reads -> {asset, size[, stored]}, - or {parts: [[key, asset, size[, stored]], ...], size} for a large JSON - object -SITE/charts//.json the manifest of a region whose chart files differ (see Regions) -SITE/models.json Live2D model index: id, key, group, canvas, manifest path, sizes; with - master data the character and its names -SITE/models/.json model manifest, the same entry forms as a chart manifest -SITE/live2d/ the player's Live2D model page and its bundle (when the player has them) -SITE/stories.json story index: titles and story groups in every language, commands, - languages, manifest path, sizes, regions -SITE/stories/.json story manifest: the common files and one file group per language, the - same entry forms as a chart manifest; the model manifests of its models - and the path to the site root -SITE/stories//.json the manifest of a region whose story files differ (see Regions) -SITE/story/ the player's story page and its bundle (when the player has them) -SITE/assets/.[.gz|.br] content-addressed files shared by all charts, models and stories - (see --compress) -``` - -- `--pair` (repeatable; `:` or `_`) adds the given charts (a chart that - no region of the site has a `MasterLiveMusicScore` row for is a usage error); `--all` adds every music and - difficulty that has a `MasterLiveMusicScore` row (in the master data of any of the site's regions). -- `--live2d` (repeatable; a model id `` or key `Character/Live2D///model/`) adds the given - Live2D models; `--all-live2d` adds every model key of the catalog. A model's id is its ``. Its manifest lists - the files the player's Live2D viewer and story page read: `model.json` (index: moc3, prefab, textures, shader index, - the Cubism mask materials and `motionSync`, whether the prefab's root has the MotionSync controller with its CRI - audio input), the moc3 and prefab as `export live2d` writes them, the atlas pages the drawables use, and the GLSL ES 3.00 - programs of the Live2D shaders that the drawables' materials select, each keyword set also with - `_ADDITIONAL_LIGHTS_VERTEX` (the story player adds it at quality 4), and the mask shader's for a model with masked - drawables. Charts and models can be added in one run; models are built first. -- `--story` (repeatable; a `MasterAdv` id) adds the given story episodes (an id no region of the site has is a usage - error); `--all-stories` adds every `MasterAdv` episode. See [Stories](#stories) below. Stories are built after - models and charts; the Live2D models the stories to build use are built before them, as with `--live2d` (listed in - `models.json`, names from the master data as for `--live2d`). -- A chart, model or story whose manifest exists is skipped unless `--force` (with stories, their models too). A - model manifest whose `model.json` is of an older format (without `motionSync`) is outdated and built again; the - summary lists such models in `modelsRebuilt` (`storyModels.modelsRebuilt` for the models of stories). Assets no - chart, no model and no story (common or language file, or model manifest it names) references are removed. -- `--player-only` rewrites the player files, `charts.json`, `models.json` and `stories.json` only; `--reingest-json` - stores every chart's and model's JSON files again from the site's own assets (then rewrites the player files and - the indexes; story manifests are left as they are). -- `--player`: the ournotes-player checkout (after its build) or installed package; it must contain - `scripts/read-set.mjs`, `dist/ournotes-player.element.min.js` and `examples/chart-list/index.html`. The read-set - script runs under Node.js to list the files the player reads for each chart; only those are stored. When it also has - `examples/live2d/index.html`, that page and `dist/ournotes-player.live2d.element.min.js` go to `SITE/live2d/`; - when it has `examples/story-list/index.html` and `dist/ournotes-player.story.element.min.js`, they go to - `SITE/story/` (a build that adds stories stops when the page is there without its bundle). -- `--compress` (default `gzip`): how the compressible assets are stored: JSON files (split parts included), GLSL, - moc3, atlas, skel, bin, wav and glb files. With `gzip` or `br` (brotli, quality 11) such a file is stored encoded - when that makes it smaller: `assets/..gz` (or `.br`), named by the SHA-256 of the decoded bytes; its - entry's `size` is the decoded size and `stored` the size of the stored file (a part: `[key, asset, size, stored]`). - Other files, and every file with `none`, are stored as they are, without `stored`. `--reingest-json` stores the - JSON files again with the given `--compress`. -- Hosting: the player decodes a `.gz` or `.br` asset when the bytes it receives have the entry's `stored` length and - uses them as they are when they have its `size`. A gzip site can be served as plain files; `Content-Encoding: - gzip` on its `.gz` files is optional (with it the browser decodes them, without it the player does). A brotli site - needs `Content-Encoding: br` on its `.br` files for Chromium-based browsers, which cannot decode brotli in the - page. -- `--format` (default `aac`) is the BGM format of the charts and the format of every story sound (music, sound - effects, voices); the charts' note SE, cheers and voices stay FLAC. `--no-audio` stores no audio (the player then - runs the chart silent on its own clock, and plays stories without their sounds; story videos keep their sound - track). -- `--workers`: parallel music processes (default a quarter of the CPUs, up to 8), model processes (default up - to 4) and story processes (default a quarter of the CPUs, up to 8); `0` or `1` builds in this process. `--read-workers`: chart read sets run at a time, each a Node.js process - (default half the CPUs, up to 16). With a player whose read-set script lists a chart's files from its plan - (without stepping the chart) and serves many charts from one process, the plans run in one long-lived Node.js - process per read-set slot, ended with the build; the full simulations that check a sample of the plans still run - in a process each. -- `--tmp`: directory for the temporary live and model builds (default `SITE.tmp`); the build directories in it are - removed after use. `SITE.tmp/cache/` is kept: decoded cue sheets, encoded PNGs, shader dumps and read sets, each - stored under a hash of everything it was made from (input bytes, settings, the external tools and libraries in use - and nnnotes' own code), so a later build with the same `--tmp` reuses what is unchanged and writes the same files - faster. Deleting it is - safe at any time outside a build; the next build then makes everything again. -- `--band` / `--leader-card`: as for `export live`, for every chart. `--fonts`: for stories, where the glyphs of the - story text come from (see Stories); the site stores no start canvas of a chart. -- `--live-option` (repeatable): as for `export live`, for the charts of this run (with `--pair` or `--all`). The live - directories carry the variants' files, and a chart's read set is the union of its read set with the default - options and its read sets with the settings of each offered variant (every combination of the offered mirror, - note skin, effect set and quality values, the bar lines on, and each other note sound set; the player's read-set - script needs its `--settings`); only the files a variant reads are stored, the shared ones once per site. A chart - manifest lists the offered qualities as `options` (`{"LiveQuality": [0, 1, 2]}`); the other options are offered by - the chart's files. Each variant adds a read-set plan per chart (`all` makes several dozen), so such a build takes - longer. Charts whose manifest exists keep the options they were built with unless `--force`. -- Models need `[paths] apk` (component classes and the mask materials are read from the APK). Master data is - optional: when it is configured (of the site's first region), each model that a `MasterCharacterCostume` row maps - to a character (its `_live2dPath` is the key after `Character/Live2D/`) gets, in its manifest's `model` and its - `models.json` entry, `character` (the `MasterCharacter` id), `names` (`{language: the character's name}`, from the - `MasterText` row of `_nameTextID`, the languages that have a text) and `label` (the name in `[catalog] language`). - A key the rows map to two characters gets none. The manifests of skipped models get this build's fields; without - master data the fields are not written and skipped manifests keep theirs. -- `--site-region` (repeatable) / `--all-regions`: the regions the site serves (see Regions); default: the one - `[catalog] region`. The global `--region` sets `[catalog] region`. + +The [site guide](site.md) covers selection, fonts, read sets and hosting. Chart, model and story requests execute the same depot product stages as their export commands. `--name` (default `default`) is a persistent namespace in the depot. Partial selections retain other topics; complete selections replace that product kind in the requested regions. + +```text +SITE/ + index.html, chart-list.js, ... chart list and player page + live2d/, story/, songs/ player pages and bundles + charts.json, models.json, + stories.json listings and localized facts + site.json complete nnnotes.site/1 catalog + charts/.json live product entry manifests + models/.json model product entry manifests + stories/.json story product entry manifests + charts//.json, + stories//.json regional variants + products/.json immutable nnnotes.product/1 manifests + assets/.[.gz|.br] shared asset bytes + manifest.json output ownership +``` + +Product manifests retain logical file paths, object IDs and file roles. Their declared dependencies point to immutable products. Public aliases have roots relative to their depth. Asset entries support whole and split JSON, with decoded sizes and optional encoded sizes. + +The complete catalog and layout are committed to the depot after publication succeeds. `--player-only` restores that catalog to the chosen output with current player pages. A configured remote supports restoration from a fresh local depot. Layout reachability keeps the retained products and assets available to depot collection. + +Read-set tasks freeze player sources, product inputs, Node identity and settings. Plans are checked against full chart simulations on the first chart and a deterministic sample. Plan failures fall back to full simulation; mismatches switch subsequent reads to full. Variants union every offered setting combination's read set. Structured documents and their image/audio/WebGL2 dependencies remain available; additional binary file roles are selected from the reads. + +Publication orders assets before immutable products, aliases/pages, and indexes. Retired owned files are collected after replacements succeed. Failures preserve the journal and previous successful-run pointer; retry the request after correcting the error. Product failures stop publication and appear in the run report. `--json` prints structured task and placement counts. ### Stories -A story's manifest lists what the player's story page reads (below the site's `assets/`): -`story.json`, `episode.json`, `scene.json`, the scene shaders (GLSL ES 3.00 programs only), textures, per cue sheet -`cues.json` and the file of each cue in the `--format` (an AAC file's encoder delay in -`cues.json` as `encoderDelay`), the media files the episode uses and its videos; per language (`--story-languages`, -default `ja,en,zh-Hant,zh-Hans,ko`) the story UI in that language (`ui/ui.json`), its font data (`ui/fonts.json` with -glyph pages under `ui/fonts/`) and `ui/languages.json`. UI files that are equal in every language are common files. -`scene.json` and `ui/ui.json` are stored split per top-level key, so the parts stories and languages share are stored -once. The Live2D models are the site's: the manifest names the model manifest of each model the story uses -(`models`, `{id: "models/.json"}`, relative to the site root; `root` is the path from the manifest to the site -root), and `stories.json` counts their files in `size.models`. The format is described in the player's -`docs/story-data-format.md`. - -- Story text is laid out with TextMeshPro's rules from TextMeshPro font assets holding exactly the characters the - episode shows in the language (its text table, title and UI labels). `--fonts open` (default) generates them from - a font file per language: `--font LANG=PATH` (repeatable) or `fonts.` in the `[paths]` table of the config - file (`NNNOTES_PATHS_FONTS_`, e.g. `NNNOTES_PATHS_FONTS_ZH_HANT`); every language of `--story-languages` - needs one. Any OpenType or TrueType font (the first face of a collection) with the characters of the language - works. The game's fallback chains are mirrored: where the game takes a character from a fallback font asset of - another language's font (Korean text falls back to the Japanese font, for example), the generated asset falls - back to an asset of that language's font file, which then needs to be given too (the build stops with the setting - it needs when a character reaches it); characters no font file of the chain has are listed in `ui/fonts.json` - `coverage.missing`. Characters the game's own font assets lack are drawn as the game draws them, as its missing - glyph (U+25A1 where the game's font has it), not from the font file (`coverage.missingGlyph`). The simple talk - window of an Overlay story is handled the same way. Each generated asset takes the - point size, padding, style settings and render mode of the game font asset the texts use in that language, so the - game's text materials apply unchanged; those values, the game's fallback chains, the characters the game's assets - have and the text materials come from the table of them shipped with nnnotes (`src/nnnotes/unity/fontsettings.json`, - `fontsettings.FORMAT`), so an open-font export reads no game font asset, material or font bundle; - its face info and glyph metrics come from the font file (FreeType, no - hinting), its distance field from the generator `--fonts game` uses for runtime glyphs (supersampled render modes - at up to 8x). The asset records the font's family, version, license and SHA-256. `--fonts game` uses the game's - font assets instead (the `fonts` extra); the format is the same. No font file is stored in the site other than as - the generated glyph pages. -- Emoji are laid out as the game lays them out: its UI texts (the talk and chat texts) draw a character their font - assets lack from the game's emoji sprite asset, and emoji sequences (several code points with U+200D or U+FE0F) - become sprite tags through the game's emoji search. The layout keeps the game's sprite metrics; the images come - from a colour emoji font of your own: `--font emoji=PATH` or `fonts.emoji` in the `[paths]` table - (`NNNOTES_PATHS_FONTS_EMOJI`), a font with PNG bitmap glyphs (CBDT or sbix tables). Noto Color Emoji (SIL Open Font - License 1.1) is the tested font. Only the sprites an episode's texts can draw are generated, each at the game's - sprite size, into one page under `ui/fonts/`; `ui/fonts.json` records the emoji font as it records a text font. - Without an emoji font the sprites keep their layout with empty glyphs, and `ui/fonts.json` - `coverage.sprites.missing` lists them. `--fonts game` uses the game's sprite asset. -- Stories need `[paths] apk` and the optional `fonts` dependencies (`pip install 'nnnotes[fonts]'`) in both modes. -- The story index's groups come from the story tables of the master data (`MasterStoryEpisode` and its chapter, - `MasterStoryFriendshipEpisode`, `MasterStoryHomeSpotTapTalkEpisode`, `MasterStoryLiveResultEpisode`, - `MasterHomeSpot`), with chapter, character and spot names in every language. -- An Overlay episode (`MasterAdv._playbackMode` 1: the game plays it in its simple ADV player over the screen that - opened it) is listed with `playbackMode` 1. With `--fonts open` its story also has the host screen (`host/`: the - home spot or the live result screen of its story group) and, per language, the simple talk window - (`ui/simple/ui.json`, `ui/simple/fonts.json` and its glyph pages); the manifest names them in `host`. The host has - no game-font variant: with `--fonts game` an Overlay story has neither. -- With `--fonts open` the chat window texts of an episode with chat rows get their bindings in `ui/fonts.json` - `chatTexts`, and the font assets also hold the chat windows' status texts and the texts they format at run time. - The text nodes of the episode's frames get theirs in `frameTexts`, and the font assets also hold the texts a frame - that receives texts is given at run time (the texts of its Frame rows' `TargetTextIDs`, as the frame formats them). -- Stories that fail are listed in the printed summary and in `SITE.story-failures.json`; the exit status is then 1. - -`ui/ui.json` holds the parts of the game's ADV widget the story draws: the front canvas (talk window, speaker plate, -location caption, title, rule transition, curtains, flash, subtitles caption, next indicator, menu entry button, menu -panel and video buttons, backlog, choices, tap areas), the canvases of stills, frames and videos with the screen -image and the camera they render with (`videoAndStillCamera`), the UI camera that renders the frame canvas -(`uiCamera`), and the letterbox bands; per node its transform, rect -and uGUI components, the views' serialized references as node paths (the menu view with its fast-forward sprites), -the other game components' serialized fields (`behaviours`), the widget's canvas sort orders (`widget`), the ADV -screen's dialogs (`dialogs`: the skip confirm and common dialog prefabs) and their texts of the language -(`masterIdTexts`), the chat phone (`chatWidget`), the text -records of the episode's chat windows (`chatTexts`) with their status texts (`chatStatusTexts`). The talk windows -sit under `TalkView`: the default window, -then each other window the episode's TalkWindow rows attach; when one of them dims and blurs the screen behind its -talk (`UICenterTalkWindow`), the front canvas' `CenterTalkBackdrop` node, the renderer's blur settings (`blur`) and -the blur shader come with it. - -Each language's `ui/` holds the localized fonts and materials, the line spacing LocalizeText applies, and, with -`--fonts game`, the characters of the lines (Talk, Location and Subtitles rows, ruby readings included), speaker -names, title and UI texts in that language (`ui/ui.json` `language`: `mode`, `field`, `lineSpacing`). - -`--fonts` (default `open`) chooses how `ui/` handles text. Both modes write a text record `textStyle` for each text -node of `ui/ui.json`, and a document-level `textStyle` that holds the units and the line metrics of each font role. -With `open`, `ui/` contains no font data: no font atlases, glyph tables, font materials or text shader. `game` also -exports the game's TextMesh Pro fonts: per node the serialized text settings with the localized font and material -(`text`), and in the document the font assets reduced to the characters the episode shows (`fonts`), runtime glyphs, -text materials, `glyphCoverage`, `tmpSettings` and the text shader. `game` needs the optional `fonts` dependencies -(`pip install -e ".[fonts]"`). - -The `ui/ui.json` text record (`textStyle` of a node; the node's `rect` and transform give its layout box): - -| Field | Value | -|---|---| -| `class`, `enabled` | TMP component class, enabled flag | -| `fontRole` | font slot of the game's localized fonts: `primary` (main text face), `number` (Latin / numeral face) | -| `materialType` | material style name of the text (`Default`, `OutlineAdvCommon`, ...) | -| `localized`, `textKey` | LocalizeText enabled; its master text id, or null when the runtime sets the text (talk, speaker, location and title texts come from `episode.json`, one field per language) | -| `text`, `richText`, `parseControlCharacters` | serialized text, rich text tags on, `\n`-style escapes parsed | -| `fontSize`, `autoSize` | size in canvas units; auto-size `enabled`, `min`, `max`, `maxCharWidthAdjust`, `maxLineSpacingAdjust` | -| `fontStyle`, `fontWeight` | style flags (`bold`, `italic`, `underline`, `strikethrough`, `lowerCase`, `upperCase`, `smallCaps`, `superscript`, `subscript`, `highlight`), weight 100-900 | -| `alignment` | `horizontal` (`left`, `center`, `right`, `justified`, `flush`, `geometry`), `vertical` (`top`, `middle`, `bottom`, `baseline`, `geometry`, `capline`) | -| `wrapping`, `overflow` | `noWrap` / `normal` / `preserveWhitespace` / `preserveWhitespaceNoWrap`; `overflow` / `ellipsis` / `masking` / `truncate` / `scrollRect` / `page` / `linked` | -| `margin` | `left`, `top`, `right`, `bottom` insets of the rect | -| `lineSpacing` | `serialized`, `applied` (the value for the export language), `byLanguage` (the per-language line spacing LocalizeText applies; null when not localized) | -| `paragraphSpacing`, `characterSpacing`, `wordSpacing` | spacing in 1/100 em | -| `characterHorizontalScale`, `kerning`, `rightToLeft`, `orthographic` | as serialized | -| `color`, `colorMode`, `colorGradient`, `overrideHtmlColors` | vertex colour, gradient mode, four-corner gradient (null when off) | -| `face` | `color` (fill = `color` x `face.color`), `dilateEm`, `boldDilateEm` (glyph edge moved outwards), `softnessEm` (edge ramp width) | -| `outline` | null, or `color`, `widthEm` (band on each side of the glyph edge: stroke width 2 x `widthEm`, drawn over the fill), `softnessEm` | -| `underlay` | null, or `color`, `offsetEm` [x, y] (+x right, +y down), `dilateEm`, `softnessEm`, `inner` (shadow inside the glyph) | - -`*Em` values are fractions of the drawn font size, converted from the text material's distance-field properties the -way the TMP shader applies them. The document-level `textStyle.roles` gives, per role, `lineHeightEm`, `ascentEm`, -`descentEm` (line pitch = `lineHeightEm` + `lineSpacing` / 100 em) and the font's `spacingOffset` / `boldSpacing` -(1/100 em) for the export language. +Story entry products reference their episode, scene, models, assets, cue sheets, videos, localized UI/font products and optional host products. The selected languages default to the catalog language. Story models also appear in the model listing. `--no-audio` retains cue metadata while omitting the sound streams; playback uses silent timing. ### Regions and languages -One site serves several regions and every language: - -- The regions serve the same catalog for a language, so a chart's files depend on the region's master data only. - Regions whose chart tables (the master tables the chart build reads) are identical share one manifest, - `charts/.json`, built once from the first region's data. A region with other chart tables is built on its own - into `charts//.json`; such a manifest whose files equal the shared one's is dropped and the region joins - the shared manifest. Models read no master data: one build serves every region. Stories follow the charts' rule - with the master tables the story build reads (`stories/.json`, `stories//.json`); a region offers - the stories of its `MasterAdv`. -- Each region's master data: `[servers.] master` (else `[paths] master`, for at most one of the regions; the - global `--master` flag is refused with more than one region). A region offers the charts its master data has a - `MasterLiveMusicScore` row for. -- Chart files come from the catalog of `[catalog] language`; the listing texts come from the text tables in every - language: a manifest's `chart` has `title` and `bands` in `[catalog] language` (`language`), and `titles` / - `bandNames` in `ja`, `en`, `zh-Hant`, `zh-Hans` and `ko`. The manifest's `regions` lists the regions it serves; - regions accumulate over builds (a skipped chart gains the regions of the build), so a region is dropped by building - the site again from scratch. -- `charts.json` has one entry per manifest (an id appears once per region) with `regions`, `titles`, `bandNames`, - and at the top `language` (the default listing language: `[catalog] language` of the latest build), `languages` - and `regions` (`id`, `name` from `[servers.] name`, `languages` from `[servers.] languages`). The - chart list page switches with `?region=&lang=`. - -The same inputs with the same versions of nnnotes, its libraries and tools (and the same font files) give -byte-identical outputs; the encoded assets also need the same zlib and brotli versions. Charts that fail are listed -in the printed summary and in `SITE.failures.json`, models that fail in the summary and in -`SITE.model-failures.json`, stories in `SITE.story-failures.json`; the exit status is then 1. +Configured regions use their own master data. Products with identical immutable manifests and listing metadata share an alias and list all regions they serve. Other versions use regional paths. JP stories/models and international releases use separate sites; conflicting contents for the same model ID require separate sites. Chart and story listings include localized titles, supported languages and region metadata for the player pages. ## music-data diff --git a/docs/contracts.md b/docs/contracts.md index 85e2aa3..d323c51 100644 --- a/docs/contracts.md +++ b/docs/contracts.md @@ -313,3 +313,7 @@ Every request names one object; only `depot gc --remote` lists the remote. - [ui-pack.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/ui-pack.schema.json) - [ui-library.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/ui-library.schema.json) + +## Site catalog + +[`site.schema.json`](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/site.schema.json) describes `nnnotes.site/1`. A named site stores its complete region-topic catalog in `site.json`. Each topic names an immutable product manifest and the metadata of its listing entry. Partial selections retain other topics; complete selections replace the requested region and product kind. The successful run points at the complete site layout so depot collection retains every published dependency. diff --git a/docs/schemas.md b/docs/schemas.md index f1537ab..8ed6374 100644 --- a/docs/schemas.md +++ b/docs/schemas.md @@ -33,6 +33,7 @@ schemas.load("plan") # the schema as a dict | `result` | `nnnotes.result/1` | the result of a task, a function of its key alone | | `run` | `nnnotes.run/1` | a run manifest: context, selection, tasks with keys and statuses, layouts | | `scripts` | `nnnotes.scripts/1` | every MonoScript of the censuses, by object id | +| `site` | `nnnotes.site/1` | a named site's complete region-topic catalog and immutable product references | | `story-fonts` | `nnnotes.story-fonts/1` (`format`) | an episode's text bindings, font records, glyph pages and text shaders | | `story-host` | `nnnotes.story-host/1` (`format`) | an Overlay story's host screen, layout and initial state | | `story-simple-ui` | `nnnotes.story-simple-ui/1` (`format`) | localized Overlay talk-window and system-message prefabs | diff --git a/docs/site.md b/docs/site.md index 668bb57..6987475 100644 --- a/docs/site.md +++ b/docs/site.md @@ -1,77 +1,46 @@ # Build an ournotes-player site -`nnnotes site build` writes a static site of charts, Live2D models and stories that -[ournotes-player](https://github.com/empty-sekai/ournotes-player) plays in the browser. The site is a directory of -plain files: the player's pages and bundles, one manifest per chart, model and story, and the files they read, -stored once by content hash. +`nnnotes site build` publishes a static site from depot products. The site contains player pages, chart/model/story listings, public entry manifests, immutable product manifests, and content-addressed assets. -## What it needs +## Requirements -- The settings of the export commands: keys, the region's CDN, `[paths] apk`, and the decoded master data of every - region the site serves ([configuration](configuration.md)). -- A built ournotes-player: its checkout after `npm run build`, or its installed package, given with `--player` or - `[paths] player`. -- Node.js 20+, which runs the player's read-set script to list the files each chart reads; only those are stored. -- vgmstream and FFmpeg for the audio. -- For stories: `pip install 'nnnotes[fonts]'` and a font file per story language (below). +Configure `[paths] depot`, the catalog and APK, and the decoded master data for the selected charts and stories. Supply a built [ournotes-player](https://github.com/empty-sekai/ournotes-player) checkout or package through `--player` or `[paths] player`. Node.js 22 or newer runs chart read sets. Audio conversion uses vgmstream and FFmpeg. Text rendering needs the `fonts` extra and either `--fonts game` or a font file for every requested language. -## Build a few charts +## Select products ```sh -nnnotes site build -o out/site --player path/to/ournotes-player \ - --pair 100001:expert --pair 100001:hard +nnnotes site build --name example -o out/site --player path/to/ournotes-player \ + --pair 100001:expert --font ja=fonts/NotoSansJP.otf +nnnotes site build --name example -o out/site --story 10004 --story-languages ja,en \ + --font ja=fonts/NotoSansJP.otf --font en=fonts/NotoSans.ttf ``` -`--pair MUSIC_ID:DIFFICULTY` (repeatable) adds charts; `--all` adds every chart of the master data. Serve the -directory with any static file server and open `index.html`: the chart list, with `?music=&difficulty=` -playing one chart. +`--pair`, `--live2d`, and `--story` are repeatable. Each updates those topics and retains the site's other topics. `--all`, `--all-live2d`, and `--all-stories` replace the corresponding kind's complete selection in the requested regions. Story products include their model dependencies, which also appear in the model listing. -The build is incremental. A chart, model or story whose manifest exists is skipped unless `--force`, and files no -manifest references any more are removed, so the same command run again after a game update adds what is new. -`--tmp DIR` (default `SITE.tmp`) keeps a cache of decoded audio, encoded images, shader dumps and read sets keyed by -everything they were made from; deleting it is safe outside a build. - -## Add models and stories +`--name` identifies the site in the depot and defaults to `default`. Use different names for independent sites sharing a depot. The latest successful run records the complete catalog and layout. A new output directory or machine can restore that state from the same depot, including its configured remote. ```sh -nnnotes site build -o out/site --player path/to/ournotes-player \ - --all --all-live2d --all-stories \ - --font ja=fonts/NotoSansJP.otf --font en=fonts/NotoSans.ttf \ - --font zh-Hant=fonts/NotoSansTC.otf --font zh-Hans=fonts/NotoSansSC.otf --font ko=fonts/NotoSansKR.otf +nnnotes site build --name example -o restored/site --player-only ``` -- `--live2d MODEL` (repeatable) or `--all-live2d` adds Live2D models; the site gets the player's model page. -- `--story ADV_ID` (repeatable) or `--all-stories` adds story episodes, and the models they use; the site gets the - player's story page. -- Story text is laid out with font assets generated from your font files, holding exactly the characters each - episode shows. Give one per language of `--story-languages` (default all five) with `--font LANG=PATH` or - `fonts.` in `[paths]`. `--font emoji=PATH` gives a colour emoji font for the emoji of chat and talk texts. - `--fonts game` uses the game's own font assets instead. +`--player-only` publishes the saved products and current player pages. Product extraction and site transformations use task keys: unchanged inputs reuse their committed results, while changed inputs produce new manifests. Depot collection follows the successful layout's complete dependency closure. -## Several regions +## Regions and languages -```sh -nnnotes site build -o out/site --player path/to/ournotes-player --all --all-regions -``` +`--site-region REGION` selects a configured region and is repeatable; `--all-regions` selects all configured regions. Each region uses its own master data. Equal product content and listing metadata share an alias; other regional versions get an alias under their region. Chart and story pages select them with `?region=&lang=`. JP stories/models and international stories/models require separate sites; differing content for one model ID also requires separate sites. + +The catalog language supplies default text. `--story-languages ja,en,...` selects story text/font products and defaults to the catalog language. Supply fonts with `--font LANG=PATH` or `[paths.fonts]`; `--font emoji=PATH` supplies a colour emoji font. `--fonts game` selects the game's font assets. + +## Player read sets and variants -`--site-region REGION` (repeatable) or `--all-regions` chooses the regions the site serves; each region's master -data is `[servers.] master`. Regions whose chart tables are equal share one manifest per chart; a region -whose tables differ gets its own. The list texts are written in all five languages, and the chart list page switches -with `?region=&lang=`. +A chart read set is a depot task whose inputs include the player sources, product files, requested settings and Node implementation identity. The first chart and a deterministic sample compare the player's plan with a full simulation. A failed plan uses a full simulation; a mismatch switches subsequent reads to full simulation. The site retains structured product documents, their image/audio/WebGL2 dependencies, and additional binary roles read by the selected settings. -## Audio, compression and hosting +`--live-option` adds note skins, effect sets, qualities, mirrored charts, bar lines and note sound sets. Read sets cover the default settings and offered combinations. Each chart manifest includes its immutable variant product paths and supported quality values. -- `--format aac|opus|vorbis|mp3|flac` (default `aac`) is the BGM and story audio format; `--no-audio` stores none. -- `--compress gzip|br|none` (default `gzip`) stores the compressible files encoded when that makes them smaller. A - gzip site works from any static host; a brotli site needs `Content-Encoding: br` on its `.br` files. -- `--workers N` sets the parallel music, model and story processes (`0` builds in this process) and - `--read-workers N` the read sets run at a time. +## Publication and hosting -## When something fails +`--format aac|opus|vorbis|mp3|flac` selects audio encoding; `--no-audio` omits sound streams. `--compress gzip|br|none` selects storage encoding. gzip assets work as plain static files; brotli assets need `Content-Encoding: br`. -Charts, models and stories that fail are listed in the summary and in `SITE.failures.json`, -`SITE.model-failures.json` and `SITE.story-failures.json`, and the exit status is 1; the others are built. Fix the -cause and run the same command again: what was built is skipped. +Publication writes assets, immutable products, aliases/pages, and then indexes. It collects retired owned files after successful replacements and retains unowned files. A failed publication keeps its journal and successful-run pointer for retry. Serve the directory with a static server and open `index.html`, `live2d/`, or `story/`. -The full option list is in the [command reference](cli.md#nnnotes-site-build), the site layout and every rule in -[Command outputs](commands.md#site-build). +`--workers`, `--memory`, and `--fetch-workers` control product task execution. `--stored-catalog LABEL|SHA` selects a catalog already registered in the depot. Reports include product task counts, site task counts, read-set checks, and file placement counts. See the [command reference](cli.md#nnnotes-site-build) for all flags and [site snapshots](products.md#site-snapshots) for product references. diff --git a/docs/stages.md b/docs/stages.md index 07732b2..144db7e 100644 --- a/docs/stages.md +++ b/docs/stages.md @@ -353,3 +353,11 @@ and file roles agree with the source. It also records the relative site root and The snapshot assembler consumes these task results to build immutable product graphs; see [site snapshots](products.md#site-snapshots). + +## site.readset, site.selection and site.index + +`site.readset` freezes the player's sources and product input tree together with a request (entry path, plan/full mode, settings and variant manifests). The Node/V8 version, platform and architecture are implementation atoms. Its `readset.json` contains sorted logical file paths and object roles plus deterministic simulation counters. A result requires the planned state or the end of the full chart simulation. + +`site.selection` takes a source product and an explicit read set. It retains structured documents, their image/audio/WebGL2 dependencies and additional read binary roles, preserving object IDs and metadata. With audio disabled it omits sound streams and marks the entry product accordingly. `site.manifest` can record variant paths and supported Live options. + +`site.index` takes the complete `nnnotes.site/1` catalog and emits chart, model and story listings, alias mappings and the catalog itself. Equal regional products with equal listing facts share aliases. The site layout traverses every retained product dependency before publishing and recording its successful run. diff --git a/src/nnnotes/cli/assets.py b/src/nnnotes/cli/assets.py index 3815644..d77d215 100644 --- a/src/nnnotes/cli/assets.py +++ b/src/nnnotes/cli/assets.py @@ -143,6 +143,9 @@ def matches(self, stage: str) -> bool: StageSource("site.asset", "nnnotes.pipeline.siteassets:AssetStage"), StageSource("site.references", "nnnotes.pipeline.siteproducts:ReferencesStage"), StageSource("site.manifest", "nnnotes.pipeline.siteproducts:ManifestStage"), + StageSource("site.readset", "nnnotes.pipeline.readsets:ReadSetStage"), + StageSource("site.index", "nnnotes.pipeline.siteindex:IndexStage"), + StageSource("site.selection", "nnnotes.pipeline.siteproducts:SelectionStage"), StageSource("view.", "nnnotes.pipeline.views:stages", output=True), StageSource("view.voices", "nnnotes.pipeline.voices:VoiceStage", output=True), ) @@ -1312,17 +1315,11 @@ def cmd_export(args, cfg): export(args, cfg, Workspace(args, cfg, layouts=names)) -def export(args, cfg, ws: Workspace, expand=None) -> None: - """Run the request of `ws` and place its layouts (ws.layouts) in args.out (cmd_export; export live2d). `expand`: - called with the run when it was not aborted, gives the request of a further run ((products, selection) for - Workspace.request) or None; the last run places the layouts (export story: the products a story names).""" - from ..pipeline import builtins, layout - from ..pipeline.orchestrate import Orchestrator, failure +def request_orchestrator(ws, cfg): + """The shared executor for product requests.""" + from ..pipeline.orchestrate import Orchestrator - names = ws.layouts - pipeline = ws.pipeline() - params = ws.params(pipeline) - orch = Orchestrator( + return Orchestrator( ws.depot, ws.scheduled, workers=ws.workers(), @@ -1333,19 +1330,19 @@ def export(args, cfg, ws: Workspace, expand=None) -> None: initargs=(cfg,), log=ws.log, ) - out = Path(args.out) - placement = None - if not args.dry_run: # before any work: an explicit method that cannot work stops it - try: - placement = layout.probe(out, ws.root / "cas" / "sha256", link_mode(args, cfg)) - except layout.LayoutError as e: - usage(args, str(e)) - if args.explain: - for line in ws.explain(pipeline, params, orch, names, placement): - print(line, file=sys.stderr) - if placement is None: # --dry-run - sys.stdout.write(ws.plan(pipeline, params, out=out, layouts=names).text()) - return + + +def run_request(args, cfg, ws: Workspace, expand=None, prepared=None): + """Execute a product request and optional expansion without materializing an output layout.""" + from ..pipeline import builtins + from ..pipeline.orchestrate import failure + + if prepared is None: + pipeline = ws.pipeline() + params = ws.params(pipeline) + orch = request_orchestrator(ws, cfg) + else: + pipeline, params, orch = prepared ws.run_index() facts = ws.facts(pipeline, fetch=True, background=True) sel = ws.selected @@ -1381,7 +1378,7 @@ def export(args, cfg, ws: Workspace, expand=None) -> None: more = expand(run) if more is not None: ws.request(*more) - return export(args, cfg, ws) + return run_request(args, cfg, ws) stage_of = {"bundle": "unity.census", "raw": next((n for n in RAW_STAGES if n in pipeline), "cri.audio")} for p in ws.problems(): if p["code"] == "source.error": @@ -1394,6 +1391,33 @@ def export(args, cfg, ws: Workspace, expand=None) -> None: p["name"], ) ) + return run, orch, aborted + + +def export(args, cfg, ws: Workspace, expand=None) -> None: + """Run the request of `ws` and place its layouts (ws.layouts) in args.out (cmd_export; export live2d). `expand`: + called with the run when it was not aborted, gives the request of a further run ((products, selection) for + Workspace.request) or None; the last run places the layouts (export story: the products a story names).""" + from ..pipeline import layout + + names = ws.layouts + pipeline = ws.pipeline() + params = ws.params(pipeline) + orch = request_orchestrator(ws, cfg) + out = Path(args.out) + placement = None + if not args.dry_run: # before any work: an explicit method that cannot work stops it + try: + placement = layout.probe(out, ws.root / "cas" / "sha256", link_mode(args, cfg)) + except layout.LayoutError as e: + usage(args, str(e)) + if args.explain: + for line in ws.explain(pipeline, params, orch, names, placement): + print(line, file=sys.stderr) + if placement is None: # --dry-run + sys.stdout.write(ws.plan(pipeline, params, out=out, layouts=names).text()) + return + run, orch, aborted = run_request(args, cfg, ws, expand, (pipeline, params, orch)) done = {tid: t["key"] for tid, t in run.tasks.items() if t["status"] != "failed"} ws.done(done) output = {tid for tid in done if ws._output(contract.parse_task_id(tid)[0])} @@ -1413,7 +1437,7 @@ def visit(tid, doc): for name, d in layout_dirs(out, names).items(): entries, report = ws.layout_entries(name, placed) w = layout.materialize(d, name, {}, entries, layout.from_depot(ws.depot), link=placement[0]) - shas[name] = w["sha256"] + shas[name] = ws.depot.put(contract.encode(w["manifest"])) written[name] = { "dir": str(d), "files": len(w["manifest"]["entries"]), diff --git a/src/nnnotes/cli/export.py b/src/nnnotes/cli/export.py index 6c2233c..b288d3d 100644 --- a/src/nnnotes/cli/export.py +++ b/src/nnnotes/cli/export.py @@ -46,7 +46,7 @@ def cmd_episode(args, cfg): report(args, r, lambda r: [f"{r['out']} ({r['commands']} commands, {r['resources']} resources)"]) -def cmd_story(args, cfg): +def prepare_story(args, cfg): from ..pipeline import contract from ..pipeline import story as sp from ..pipeline.stages import Env @@ -112,10 +112,15 @@ def parts(run): # the products story.json names, with the story return (*request(products), inputs) ws.request(*request({sp.STORY: [subject]})) + return ws, parts + + +def cmd_story(args, cfg): + ws, parts = prepare_story(args, cfg) assets.export(args, cfg, ws, expand=parts) -def cmd_live(args, cfg): +def prepare_live(args, cfg): from ..content import live, liveaudio, livenotes, liveoptions from ..pipeline import live as lp from ..pipeline.stages import Env @@ -174,10 +179,14 @@ def cmd_live(args, cfg): stages = {n: classes[n]() for n in products} keys = {k for n, subjects in products.items() for s in subjects for k in stages[n].keys(s, env)} ws.request(products, sorted(f"key:{k}" for k in keys)) - assets.export(args, cfg, ws) + return ws -def cmd_live2d(args, cfg): +def cmd_live(args, cfg): + assets.export(args, cfg, prepare_live(args, cfg)) + + +def prepare_live2d(args, cfg): from ..content import live2d from ..pipeline.live2d import CUBISM, MODEL @@ -191,7 +200,11 @@ def cmd_live2d(args, cfg): except ValueError as e: usage(args, str(e)) ws.request({MODEL if args.player else CUBISM: [key]}, [f"key:{key}"]) - assets.export(args, cfg, ws) + return ws + + +def cmd_live2d(args, cfg): + assets.export(args, cfg, prepare_live2d(args, cfg)) def cmd_spot(args, cfg): diff --git a/src/nnnotes/cli/site.py b/src/nnnotes/cli/site.py index c0f0557..d58f536 100644 --- a/src/nnnotes/cli/site.py +++ b/src/nnnotes/cli/site.py @@ -1,14 +1,20 @@ -"""`nnnotes site build`: live charts, Live2D models and stories -> a static site for ournotes-player.""" +"""Build a named static site from depot products and publish its complete layout.""" from __future__ import annotations -import sys +import argparse +import re from pathlib import Path +from ..content import sitefacts from ..formats.compression import DEFAULT_ENCODING, ENCODINGS from ..formats.encoders import DEFAULT_AUDIO_FORMAT, WEB_AUDIO +from ..pipeline import contract +from ..pipeline.depot import open_depot from ..source.config import ConfigError, use +from ..source.languages import check from ..source.session import Session +from . import assets, export from .common import ( band_args, command, @@ -26,138 +32,196 @@ ) +def product_entries(args, cfg, ws, expand=None): + run, _, aborted = assets.run_request(args, cfg, ws, expand=expand) + saved = run.write() + if aborted or run.exit_code: + report(args, {"run": saved["id"], "tasks": saved["summary"], "failures": run.failures(), "aborted": aborted}) + raise SystemExit(assets.EXIT_ABORTED if aborted else assets.EXIT_FAILED) + done = {tid: t["key"] for tid, t in run.tasks.items() if t["status"] != "failed"} + ws.done(done) + output = {tid for tid in done if ws._output(contract.parse_task_id(tid)[0])} + placed = ws.layout_input({tid: done[tid] for tid in output}) + for tid, doc in run.results(): + if tid in output: + placed.add(tid, doc) + entries, _ = ws.layout_entries("product", placed) + return entries, saved["summary"] + + def cmd_build(args, cfg): - from ..content import liveoptions - from ..site import sitefiles, siteindex, storysite, web, webmodel + from ..content import live2d, liveoptions + from ..pipeline import live as lp + from ..pipeline import story as sp + from ..site import catalog, pages + from ..site.build import Builder, document, documents, model_stats, size + from ..source.master import table - request = live_options(args) - charts, models = bool(args.pair or args.all), bool(args.live2d or args.all_live2d) + charts = bool(args.pair or args.all) + models = bool(args.live2d or args.all_live2d) stories = bool(args.story or args.all_stories) - if not (charts or models or stories or args.player_only or args.reingest_json): - usage( - args, - "one of the arguments --pair --all --live2d --all-live2d --story --all-stories --player-only " - "--reingest-json is required", - ) - if models and (args.player_only or args.reingest_json): - usage(args, "--player-only and --reingest-json build nothing: leave out --live2d / --all-live2d") - if stories and (args.player_only or args.reingest_json): - usage(args, "--player-only and --reingest-json build nothing: leave out --story / --all-stories") - if request and not charts: - usage(args, "--live-option applies to charts: give --pair or --all") - player = cfg.path("paths", "player") - out = Path(args.out) - if args.player_only or args.reingest_json: - r = { - **(sitefiles.reingest_json(out, args.compress) if args.reingest_json else {}), - **sitefiles.write_player(out, sitefiles.check_player(player)), - **siteindex.write_index(out), - } - else: - sitefiles.check_player(player) - regions = None # the one [catalog] region - if args.site_regions or args.all_regions: - regions = sitefiles.site_regions(cfg, args.site_regions, args.all_regions) - if (stories or models) and regions and len({cfg.provider(r) for r in regions}) > 1: - raise ConfigError("build JP stories/models in a separate site directory from international releases") - if (stories or models) and regions: - cfg = use(cfg.for_region(regions[0])) - base = {"region": regions[0]} if regions else {} - unknown = web.unknown_pairs(cfg, args.pair, regions) if args.pair else [] - if unknown: - usage( - args, - f"chart{'s' if len(unknown) > 1 else ''} {', '.join(f'{m}:{d}' for m, d in unknown)}: no " - f"MasterLiveMusicScore row in the master data of the site's region(s)", - ) - if request: # a value a region's master data does not have - try: - for md in sitefiles.region_masters(cfg, sitefiles.site_regions(cfg, regions)).values(): - liveoptions.resolve(request, md) - except liveoptions.OptionSpecError as e: - usage(args, str(e)) - r = {} - encoding = args.compress - if models: - cfg.require_path("paths", "apk") # the Cubism component classes and mask materials: the APK - try: - selected = webmodel.catalog_models(Session(cfg).catalog(bundles=False, **base), args.live2d) - except ValueError as e: - usage(args, str(e)) - r.update( - webmodel.build( - out, - selected, - cfg, - player, - force=args.force, - tmp_dir=args.tmp, - workers=args.workers, - encoding=encoding, - **base, - ) - ) - if charts: - fonts_extra(args.fonts) - r.update( - web.build( - out, - None if args.all else args.pair, - cfg, - player, - args.format, - audio=not args.no_audio, - force=args.force, - tmp_dir=args.tmp, - workers=args.workers, - band=args.band, - leader_card=args.leader_card, - regions=regions, - fonts=args.fonts, - read_workers=args.read_workers, - live_options=request, - encoding=args.compress, - ) - ) - if stories: - from ..unity.tmpfont import require_extra + if not (charts or models or stories or args.player_only): + usage(args, "one of --pair --all --live2d --all-live2d --story --all-stories --player-only is required") + if args.player_only and (models or stories): + usage(args, "--player-only cannot be combined with a product selection") + if not re.fullmatch(r"[a-zA-Z0-9_-]+", args.name): + usage(args, "--name must contain letters, digits, underscores or hyphens") + requested_options = live_options(args) + if requested_options and not charts: + usage(args, "--live-option requires --pair or --all") + player = pages.check_player(cfg.path("paths", "player")) + depot = open_depot(cfg) + if args.player_only: + saved, _ = catalog.load(depot, args.name) + if saved is None: + usage(args, f"site {args.name}: no saved catalog; select products for the first build") + builder = Builder(depot, args.name, saved["language"], saved["regions"], player) + report(args, builder.finish(Path(args.out))) + return + regions = pages.site_regions(cfg, args.site_regions, args.all_regions) + if (stories or models) and len({cfg.provider(r) for r in regions}) > 1: + raise ConfigError("build JP stories/models in a separate site from international releases") + language = check(cfg.get("catalog", "language")) + builder = Builder(depot, args.name, language, pages.region_meta(cfg, regions), player, encoding=args.compress) + for region in regions: + cfg.for_region(region).require_path("paths", "apk") + if charts: + fonts_extra(args.fonts) + if stories: + from ..unity.tmpfont import require_extra - require_extra("--story / --all-stories") # the font assets of the story text (open and game) - cfg.require_path("paths", "apk") # the ADV settings and UI are embedded content of the APK - unknown = storysite.unknown_stories(cfg, args.story, regions) if args.story else [] - if unknown: - noun = "stories" if len(unknown) > 1 else "story" - usage( - args, - f"{noun} {', '.join(map(str, unknown))}: no MasterAdv row in the master data of the " - f"site's region(s)", - ) - r.update( - storysite.build( - out, - args.story, - cfg, - player, - args.format, - audio=not args.no_audio, - force=args.force, - tmp_dir=args.tmp, - workers=args.workers, - regions=regions, - story_languages=args.story_languages, - fonts=args.fonts, - fonts_flags=dict(args.font or []), - encoding=encoding, - ) - ) - report(args, r) - if ( - r.get("failed") - or r.get("modelsFailed") - or r.get("storiesFailed") - or (r.get("storyModels") or {}).get("modelsFailed") - ): - sys.exit(1) + require_extra("--story / --all-stories") + masters = ( + pages.region_masters(cfg, regions) + if charts or stories + else {r: Session(cfg).master_dir(r) if cfg.path(*cfg.master(r)) is not None else None for r in regions} + ) + pairs = {r: set(sitefacts.all_pairs(md)) for r, md in masters.items()} if charts else {} + episodes = ( + {r: {row["_id"] for row in table(md, "MasterAdv")} for r, md in masters.items() if md is not None} + if stories + else {} + ) + if args.pair and (missing := set(args.pair) - set.union(*pairs.values())): + usage(args, "charts not in the site's master data: " + ", ".join(f"{m}:{d}" for m, d in sorted(missing))) + if args.story and (missing := set(args.story) - set.union(*episodes.values())): + usage(args, "stories not in the site's master data: " + ", ".join(map(str, sorted(missing)))) + summaries = [] + try: + for region in regions: + regional = use(cfg.for_region(region)) + md = masters[region] + lang = check(regional.get("catalog", "language") or language) + wanted = args.story_languages or [lang] + base = { + **vars(args), + "audio_format": args.format, + "movie_format": "webm", + "views": "none", + "live_languages": [lang], + "story_languages": wanted, + } + if charts: + options = liveoptions.resolve(requested_options, md) + selected = pairs[region] if args.all else set(args.pair) & pairs[region] + if args.all: + builder.replace.add((region, "live")) + for music, difficulty in sorted(selected): + request = argparse.Namespace(**{**base, "music_id": music, "difficulty": difficulty}) + ws = export.prepare_live(request, regional) + entries, summary = product_entries(request, regional, ws) + summaries.append(summary) + assert ws.products is not None + subject = ws.products[lp.LIVE][0] + source = lp.LiveStage().manifest(subject) + manifests = documents(builder.depot, entries) + live = document(builder.depot, manifests, source, "live.json") + music_doc = document(builder.depot, manifests, live["parts"]["music"], "music.json") + facts = { + **sitefacts.chart(live, music_doc, lang), + "audio": not args.no_audio, + "audioFormat": args.format, + "flows": ["normal"], + } + snapshot = builder.add( + entries, + source, + "live", + f"{music}_{difficulty}", + region, + facts, + languages=[lang], + options=options, + audio=not args.no_audio, + ) + facts["bytes"] = size(snapshot, source) + if models: + ws = assets.Workspace(argparse.Namespace(**base), regional, layouts=["product"]) + ws.run_index() + keys = live2d.model_keys(loc["primaryKey"] for loc in ws.index()["locations"]) + try: + selected = live2d.select_models(args.live2d, keys) + except ValueError as error: + usage(args, str(error)) + names = sitefacts.optional_model_names(md, lang) + if args.all_live2d: + builder.replace.add((region, "model")) + for mid, key in sorted(selected.items()): + request = argparse.Namespace(**{**base, "key": key, "player": True}) + model_ws = export.prepare_live2d(request, regional) + entries, summary = product_entries(request, regional, model_ws) + summaries.append(summary) + source = f"models/{mid}.json" + facts = {"key": key, **names.get(key, {})} + snapshot = builder.add(entries, source, "model", mid, region, facts) + facts.update(model_stats(snapshot, source)) + if stories: + assert md is not None + groups = sitefacts.story_groups(md) + selected = episodes[region] if args.all_stories else set(args.story) & episodes[region] + if args.all_stories: + builder.replace.add((region, "story")) + for adv_id in sorted(selected): + request = argparse.Namespace(**{**base, "adv_id": adv_id}) + ws, expand = export.prepare_story(request, regional) + entries, summary = product_entries(request, regional, ws, expand) + summaries.append(summary) + source = sp.StoryStage().manifest(str(adv_id)) + manifests = documents(builder.depot, entries) + episode = document(builder.depot, manifests, source, "episode.json") + facts = { + **sitefacts.story(episode, groups.get(adv_id, []), wanted, lang), + "audio": not args.no_audio, + "audioFormat": args.format, + "fonts": args.fonts, + } + snapshot = builder.add( + entries, source, "story", adv_id, region, facts, languages=wanted, audio=not args.no_audio + ) + facts["size"] = { + "common": 0, + "models": 0, + "languages": {code: size(snapshot, source, code) for code in wanted}, + } + names = sitefacts.optional_model_names(md, lang) + for path, model in manifests.items(): + if model["product"] != "live2d.model" or path not in snapshot.paths: + continue + key = model["subject"] + builder.topics.append( + { + "kind": "model", + "id": live2d.model_id(key), + "region": region, + "manifest": snapshot.paths[path], + "entry": {"key": key, **names.get(key, {}), **model_stats(snapshot, path)}, + } + ) + result = builder.finish(Path(args.out)) + result["products"] = {key: sum(s[key] for s in summaries) for key in ["tasks", "hit", "ran", "failed"]} + report(args, result) + finally: + use(cfg) def register(sub) -> None: @@ -170,14 +234,12 @@ def register(sub) -> None: "per-model / per-story data)", ) c.add_argument("-o", "--out", required=True, metavar="SITE", help="site directory") + c.add_argument("--name", default="default", help="persistent site name in the depot (default: default)") c.add_argument("--player", help="ournotes-player checkout (built) or installed package ([paths] player)") g = c.add_mutually_exclusive_group() g.add_argument("--pair", type=parse_pair, action="append", help=": (repeatable)") g.add_argument("--all", action="store_true", help="every MasterLiveMusic x difficulty") - g.add_argument( - "--player-only", action="store_true", help="rewrite the player files, charts.json and models.json only" - ) - g.add_argument("--reingest-json", action="store_true", help="store every chart's and model's JSON files again") + g.add_argument("--player-only", action="store_true", help="publish the saved catalog with current player pages") m = c.add_mutually_exclusive_group() m.add_argument( "--live2d", @@ -200,25 +262,10 @@ def register(sub) -> None: help="encoding of the compressible assets (JSON, shaders, moc3, ...) where it makes them " "smaller: gzip (default), br (brotli) or none", ) - c.add_argument( - "--force", - action="store_true", - help="rebuild charts, models and stories whose manifest exists (stories: also their models)", - ) - c.add_argument("--tmp", help="directory for the temporary live and model builds (default .tmp)") - c.add_argument( - "--workers", - type=nonnegative, - metavar="N", - help="parallel music / story / model processes (default a quarter of the CPUs, up to 8; models up to 4; " - "0: this process)", - ) - c.add_argument( - "--read-workers", - type=positive, - metavar="N", - help="chart read sets run at a time (default half the CPUs, up to 16)", - ) + c.add_argument("--workers", type=nonnegative, metavar="N", help="product task worker processes; 0: this process") + c.add_argument("--memory", type=assets._gib, metavar="GiB", help="product task memory budget") + c.add_argument("--fetch-workers", type=positive, metavar="N", help="parallel source downloads") + c.add_argument("--stored-catalog", metavar="LABEL|SHA", help="a stored catalog version of the depot") t = c.add_mutually_exclusive_group() t.add_argument( "--story", type=int, action="append", metavar="ADV_ID", help="story episode: its MasterAdv id (repeatable)" @@ -228,7 +275,7 @@ def register(sub) -> None: "--story-languages", type=parse_languages, metavar="LANGS", - help="languages of the stories' text, comma-separated (default: ja,en,zh-Hant,zh-Hans,ko)", + help="languages of the stories' text, comma-separated (default: the catalog language)", ) c.add_argument( "--font", @@ -252,9 +299,8 @@ def register(sub) -> None: "--fonts", default="open", choices=("open", "game"), - help="text of the start canvas and the stories: open (default): the start canvas has layout and " - "style only, the stories glyphs from your font files (--font); game: the game's TMP fonts " - "(needs the 'fonts' extra)", + help="text of the start canvas and stories: open (default): glyphs from your font files (--font); " + "game: the game's TMP fonts (needs the 'fonts' extra)", ) band_args(c) live_option_arg(c) diff --git a/src/nnnotes/content/sitefacts.py b/src/nnnotes/content/sitefacts.py new file mode 100644 index 0000000..af50544 --- /dev/null +++ b/src/nnnotes/content/sitefacts.py @@ -0,0 +1,183 @@ +"""Localized listing facts from master tables and product documents.""" + +from pathlib import Path + +from ..source import languages, master +from .live2d import LIVE2D_PREFIX + +COSTUME_TABLE = "MasterCharacterCostume" + + +def optional_model_names(master_dir, language): + if master_dir is None: + return {} + try: + return model_names(master_dir, language) + except FileNotFoundError: + return {} + + +def all_pairs(master_dir): + from .score import DIFFICULTIES + + scores = {r["_id"] for r in master.table(master_dir, "MasterLiveMusicScore")} + return [ + (m["_id"], d) + for m in sorted(master.table(master_dir, "MasterLiveMusic"), key=lambda r: r["_id"]) + for d in DIFFICULTIES + if m.get(f"_{d}ID") in scores + ] + + +def chart(live, music, language): + rows, difficulty = live["master"], live["difficulty"] + row, score = rows["MasterLiveMusic"], rows["MasterLiveMusicScore"][difficulty] + title = rows.get("title") or {} + bands = [b.get("name") or {} for b in rows.get("bands") or []] + layer = music["sounds"][str(music["music"]["soundId"])]["layers"][0] + return { + "musicId": live["musicId"], + "difficulty": difficulty, + "title": title.get(language), + "bands": [b.get(language) for b in bands], + "language": language, + "titles": {c: title[c] for c in languages.LANGUAGES if isinstance(title.get(c), str)}, + "bandNames": { + c: [b[c] for b in bands] for c in languages.LANGUAGES if all(isinstance(b.get(c), str) for b in bands) + }, + "bandIds": list(row.get("_bandIDs") or []), + "stageBand": live["band"]["band"], + "level": score["_musicScoreLevel"], + "displayLevel": score["_musicScoreDisplayLevel"], + "notes": live["judgementNoteCount"], + "fullComboCount": live["fullComboCount"], + "durationMs": layer["samples"] * 1000 // layer["sampleRate"], + "sortOrder": row.get("_sortOrder"), + } + + +def story(episode, groups, selected_languages, language): + title, row = episode.get("title") or {}, episode["master"] + return { + "advId": episode["advId"], + "asset": episode["asset"], + "sheetName": row.get("_sheetName", ""), + "playbackMode": row["_playbackMode"], + "titles": { + c: title[languages.column(c)[1:]] + for c in languages.LANGUAGES + if isinstance(title.get(languages.column(c)[1:]), str) and title[languages.column(c)[1:]] + }, + "groups": groups, + "commands": sorted({c["cmd"] for c in episode["commands"] if not c.get("IgnoreData")}), + "commandCount": episode["commandCount"], + "language": language, + "languages": list(selected_languages), + } + + +def story_groups(master_dir: Path) -> dict[int, list[dict]]: + """{advId: [group]}: the rows of the story tables that name an episode (docs/story-data-format.md, story + facts), in table order (MasterStoryEpisode, MasterStoryFriendshipEpisode, MasterStoryHomeSpotTapTalkEpisode, + MasterStoryLiveResultEpisode, MasterHomeSpot), rows of a table by id. Names in every language with a text; + a table the master data lacks gives no groups.""" + md = Path(master_dir) + + def table(name: str) -> list[dict]: + try: + return sorted(master.table(md, name), key=lambda r: r["_id"]) + except FileNotFoundError: + return [] + + texts = {r["_id"]: r for r in table("MasterText")} + + def names(text_id) -> dict: + row = texts.get(text_id) + return {c: t for c, t in languages.texts(row).items() if isinstance(t, str) and t} if row else {} + + chars = {r["_id"]: r for r in table("MasterCharacter")} + + def character(cid: int) -> dict: + return {"id": cid, "names": names((chars.get(cid) or {}).get("_nameTextID"))} + + chapters = {r["_id"]: r for r in table("MasterStoryChapter")} + friendships = {r["_id"]: r for r in table("MasterCharacterFriendship")} + spots = {r["_id"]: r for r in table("MasterHomeSpot")} + + def spot(sid: int) -> dict: + return {"id": sid, "names": names((spots.get(sid) or {}).get("_nameTextId"))} + + out: dict[int, list[dict]] = {} + for r in table("MasterStoryEpisode"): + ch = chapters.get(r["_chapterId"]) or {} + g = { + "kind": "chapter", + "id": r["_id"], + "episodeNumber": r["_episodeNumber"], + "chapter": { + "id": r["_chapterId"], + "names": names(ch.get("_nameTextId")), + "bandId": ch.get("_bandId", 0), + "special": bool(ch.get("_isSpecialStory")), + }, + "characters": [character(r["_characterId"])] if r.get("_characterId") else [], + } + out.setdefault(r["_advId"], []).append(g) + for r in table("MasterStoryFriendshipEpisode"): + f = friendships.get(r["_characterFriendshipId"]) or {} + ids = [f[k] for k in ("_masterCharacterIdA", "_masterCharacterIdB") if f.get(k)] + out.setdefault(r["_advId"], []).append( + { + "kind": "friendship", + "id": r["_id"], + "episodeNumber": r["_episodeNumber"], + "characters": [character(c) for c in ids], + } + ) + for r in table("MasterStoryHomeSpotTapTalkEpisode"): + out.setdefault(r["_advId"], []).append( + { + "kind": "spotTalk", + "id": r["_id"], + "spot": spot(r["_spotId"]), + "characters": [character(r["_characterId"])] if r.get("_characterId") else [], + } + ) + for r in table("MasterStoryLiveResultEpisode"): + out.setdefault(r["_advId"], []).append( + {"kind": "liveResult", "id": r["_id"], "characters": [character(c) for c in r.get("_characterIds") or []]} + ) + for r in table("MasterHomeSpot"): + if r.get("_advId"): + out.setdefault(r["_advId"], []).append( + {"kind": "spot", "id": r["_id"], "spot": spot(r["_id"]), "characters": []} + ) + return out + + +def model_names(master_dir: Path, language: str | None = None) -> dict[str, dict]: + """{model key: {"character", "names", "label"}} of the models the master data maps to one character: every + MasterCharacterCostume row maps the key LIVE2D_PREFIX + `_live2dPath` to `_characterID` (the MasterCharacter id, + `character`); `names`: {language code: the MasterText of the character's `_nameTextID`}, the languages with a + text; `label`: the name in `language` (left out without `language` or a text in it). A key the rows map to two + characters, or a character without a name text, is left out.""" + chars_of: dict[str, set] = {} + for r in master.table(master_dir, COSTUME_TABLE): + if r.get("_live2dPath") and r.get("_characterID") is not None: + chars_of.setdefault(LIVE2D_PREFIX + r["_live2dPath"], set()).add(r["_characterID"]) + chars = {r["_id"]: r for r in master.table(master_dir, "MasterCharacter")} + texts = {r["_id"]: r for r in master.table(master_dir, "MasterText")} + out = {} + for key, ids in sorted(chars_of.items()): + if len(ids) != 1: + continue + (cid,) = ids + row = texts.get((chars.get(cid) or {}).get("_nameTextID")) + names = {c: t for c, t in languages.texts(row).items() if isinstance(t, str) and t} if row else {} + if not names: + continue + entry = {"character": cid, "names": names} + if language in names: + entry["label"] = names[language] + out[key] = entry + return out diff --git a/src/nnnotes/pipeline/readsets.py b/src/nnnotes/pipeline/readsets.py new file mode 100644 index 0000000..2dc2ef0 --- /dev/null +++ b/src/nnnotes/pipeline/readsets.py @@ -0,0 +1,180 @@ +"""Read-set tasks keyed by player source files, product files and the Node implementation.""" + +from __future__ import annotations + +import json +import os +import re +import shutil +import subprocess +import tempfile +from pathlib import Path + +from ..source.config import tool +from . import contract +from .contract import Cost, IncompatibleTask, Input +from .stages import Output, Stage + +READ_SET = "site.readset" + + +def relative(path: str) -> str: + if ( + not isinstance(path, str) + or re.search(r"[\\:?#\x00]", path) + or any(p in {"", ".", ".."} for p in path.split("/")) + ): + raise ValueError(f"read-set path must be relative: {path!r}") + return path + + +def node_identity(executable: str) -> str: + code = "JSON.stringify([process.version,process.versions.v8,process.platform,process.arch])" + result = subprocess.run([executable, "-p", code], check=True, capture_output=True, text=True, timeout=15) + return "node:" + json.dumps(json.loads(result.stdout), separators=(",", ":")) + + +def request(entry: str, *, mode="plan", settings=None, keys=False, variants=()) -> dict: + relative(entry) + if mode not in {"plan", "full"}: + raise ValueError("read-set mode must be plan or full") + if settings is not None and not isinstance(settings, dict): + raise ValueError("read-set settings must be an object") + return { + "entry": entry, + "mode": mode, + "settings": settings or {}, + "keys": bool(keys), + "variants": sorted({relative(p) for p in variants}), + } + + +def player_inputs(depot, directory) -> list[Input]: + """The player's source tree and read-set script, frozen into the depot.""" + root = Path(directory) + files = {root / "package.json", root / "scripts" / "read-set.mjs"} + for name in ["src", "scripts/lib"]: + files.update(p for p in (root / name).rglob("*") if p.is_file()) + out = [] + for path in sorted(files): + data = path.read_bytes() + role = "player:" + relative(path.relative_to(root).as_posix()) + out.append(Input(role, depot.put(data), len(data), None, ({"kind": "depot"},))) + return out + + +def data_inputs(entries) -> list[Input]: + by_path = {} + for entry in entries: + path = relative(entry["path"]) + inp = Input("data:" + path, entry["sha256"], entry["size"], None, ({"kind": "depot"},)) + if path in by_path and by_path[path] != inp: + raise ValueError(f"read-set file has two contents: {path}") + by_path[path] = inp + return [by_path[path] for path in sorted(by_path)] + + +class ReadSetError(RuntimeError): + """The player did not return a complete read set for the requested mode.""" + + +class ReadSetStage(Stage): + name, version = READ_SET, 1 + + def subjects(self, env): + return sorted(env.facts.get(self.name, {})) + + def inputs(self, subject, env): + return env.fact(self.name)[subject] + + def uses(self, subject, env): + return {"player.node": node_identity(tool("node", "node"))} + + def estimate(self, subject, env, inputs): + return Cost(1, max(256 << 20, sum(i.size for i in inputs) * 3)) + + def run(self, task, depot): + executable = tool("node", "node") + if task.atoms != {"player.node": node_identity(executable)}: + raise IncompatibleTask("read-set Node implementation differs from the task") + spec = json.loads(depot.input_bytes(task.input("request"))) + spec = request(**spec) + work = depot.root / "tmp" + work.mkdir(parents=True, exist_ok=True) + with tempfile.TemporaryDirectory(prefix="readset-", dir=work) as directory: + base = Path(directory) + seen = set() + for inp in task.inputs: + if inp.role == "request": + continue + prefix, _, path = inp.role.partition(":") + if prefix not in {"player", "data"}: + raise ValueError(f"unknown read-set input role: {inp.role}") + path = relative(path) + key = (prefix + "/" + path).casefold() + if key in seen: + raise ValueError(f"read-set paths differ only by case: {path}") + seen.add(key) + dst = base / prefix / path + dst.parent.mkdir(parents=True, exist_ok=True) + src = depot.open_input(inp) + if os.name == "nt": + shutil.copyfile(src, dst) + else: + try: + os.link(src, dst) + except OSError: + shutil.copyfile(src, dst) + args = [executable, str(base / "player/scripts/read-set.mjs"), str(base / "data" / spec["entry"])] + if spec["mode"] == "plan": + args.append("--plan") + if spec["keys"]: + args.append("--keys") + if spec["settings"]: + args.append("--settings=" + json.dumps(spec["settings"], separators=(",", ":"))) + if spec["variants"]: + args.append("--variants=" + json.dumps(spec["variants"], separators=(",", ":"))) + result = subprocess.run(args, capture_output=True, text=True, encoding="utf-8", timeout=1200) + if result.returncode: + error = (result.stderr or result.stdout).strip().replace(str(base), "") + raise ReadSetError(f"{spec['mode']} read set failed: {error[-8000:]}") + try: + data = json.loads(result.stdout) + except ValueError as e: + raise ReadSetError("player returned an invalid read-set document") from e + expected = "planned" if spec["mode"] == "plan" else "ended" + if data.get("mode") != spec["mode"] or data.get("state") != expected: + raise ReadSetError(f"read set did not reach {expected}") + files = data.get("files") + if not isinstance(files, list) or any(not isinstance(p, str) or not p for p in files): + raise ReadSetError("read set has no file list") + if data.get("read") != len(set(files)): + raise ReadSetError("read-set file count differs from its list") + for p in files: + if not re.fullmatch(r"[^/\\]+:-?[0-9]+#.+", p): + relative(p) + for name in ["frames", "readAfterLoad", "read"]: + if type(data.get(name)) is not int or data[name] < 0: + raise ReadSetError(f"read-set {name} must be a nonnegative integer") + # Keep only deterministic player results. Execution paths and durations belong to run logs. + out = { + "mode": spec["mode"], + "state": expected, + "frames": data["frames"], + "readAfterLoad": data["readAfterLoad"], + "read": data["read"], + "files": sorted(set(files)), + } + if spec["settings"]: + out["settings"] = spec["settings"] + if spec["keys"]: + if not isinstance(data.get("keys"), list) or any(not isinstance(p, str) for p in data["keys"]): + raise ReadSetError("read set has no key list") + out["keys"] = sorted(set(data["keys"])) + art = contract.artifact( + contract.artifact_id(task.id, "readset.json"), + depot.add(contract.encode(out), "json"), + contract.provenance(task), + {"kind": "site.readset"}, + ) + return Output([art]) diff --git a/src/nnnotes/pipeline/siteindex.py b/src/nnnotes/pipeline/siteindex.py new file mode 100644 index 0000000..f14e558 --- /dev/null +++ b/src/nnnotes/pipeline/siteindex.py @@ -0,0 +1,82 @@ +"""Site catalog and listing indexes from explicit region-topic records.""" + +from __future__ import annotations + +from collections import defaultdict + +from . import contract +from .stages import Output, Stage + +INDEX = "site.index" +CATALOG = "nnnotes.site/1" +KINDS = {"live": ("charts", 2), "model": ("models", 2), "story": ("stories", "ournotes.stories/1")} + + +class IndexStage(Stage): + name, version = INDEX, 2 + + def subjects(self, env): + return sorted(env.facts.get(self.name, {})) + + def inputs(self, subject, env): + return env.fact(self.name)[subject] + + def run(self, task, depot): + catalog = contract.loads(depot.input_bytes(task.input("catalog"))) + if catalog.get("format") != CATALOG: + raise ValueError("site index requires a site catalog") + regions = {r["id"]: i for i, r in enumerate(catalog["regions"])} + topics = sorted(catalog["topics"], key=lambda t: (t["kind"], t["id"], regions[t["region"]])) + groups = defaultdict(dict) + for topic in topics: + sig = contract.digest({"manifest": topic["manifest"], "entry": topic["entry"]}) + group = groups[(topic["kind"], topic["id"])].setdefault(sig, {"topic": topic, "regions": []}) + group["regions"].append(topic["region"]) + rows = {kind: [] for kind in KINDS} + aliases = {} + for (kind, tid), versions in sorted(groups.items()): + folder, _ = KINDS[kind] + if kind == "model" and len(versions) > 1: + raise ValueError(f"model {tid}: differing contents require separate sites") + for index, group in enumerate(versions.values()): + topic, rs = group["topic"], group["regions"] + alias = f"{folder}/{'' if index == 0 else rs[0] + '/'}{tid}.json" + aliases[alias] = topic["manifest"] + rows[kind].append({**topic["entry"], "id": tid, "manifest": alias, "regions": rs}) + artifacts = [] + + def emit(path, doc): + artifacts.append( + contract.artifact( + contract.artifact_id(task.id, path), + depot.add(contract.encode(doc), "json"), + contract.provenance(task), + {"kind": INDEX}, + ) + ) + + for kind, (folder, fmt) in KINDS.items(): + listing = rows[kind] + languages = sorted( + { + catalog["language"], + *( + code + for entry in listing + for code in [*entry.get("languages", []), *entry.get("titles", {}), *entry.get("names", {})] + ), + } + ) + emit( + folder + ".json", + { + "format": fmt, + "language": catalog["language"], + "languages": languages, + "regions": catalog["regions"], + folder: listing, + }, + ) + emit("site.json", {**catalog, "topics": topics}) + emit("aliases.json", aliases) + return Output(artifacts) diff --git a/src/nnnotes/pipeline/siteproducts.py b/src/nnnotes/pipeline/siteproducts.py index 26dadea..a5f88a9 100644 --- a/src/nnnotes/pipeline/siteproducts.py +++ b/src/nnnotes/pipeline/siteproducts.py @@ -112,4 +112,37 @@ def transform(self, source, spec): doc["root"] = spec.get("root", "../") if spec.get("language"): doc["language"] = spec["language"] + if spec.get("variants"): + doc["variants"] = spec["variants"] + if spec.get("options"): + doc["options"] = spec["options"] + return doc + + +class SelectionStage(_DocumentStage): + """Retain read roles and the files required by the product's structured documents.""" + + name, version = "site.selection", 2 + + def transform(self, source, spec): + if source.get("schema") != contract.PRODUCT: + raise ValueError("site selection requires a product") + doc = copy.deepcopy(source) + reads = None if spec.get("reads") is None else set(spec["reads"]) + audio = spec.get("audio", True) + + def keep(path, entry): + asset = entry.get("asset", "") + if asset.rsplit(".", 1)[-1] in {"flac", "ogg", "wav", "mp3", "m4a"}: + return audio + # Structured products describe every band, sprite and shader variant they offer. + # Keep their image and WebGL2 program dependencies even when this chart does not draw them. + return reads is None or asset.endswith((".json", ".png")) or "#program:gles3/" in path or path in reads + + directory = spec["path"][:-5] + doc["files"] = {p: e for p, e in doc["files"].items() if keep(f"{directory}/{p}", e)} + for oid, obj in doc["objects"].items(): + obj["files"] = {role: e for role, e in obj["files"].items() if keep(f"{oid}#{role}", e)} + if doc["product"] in {"live", "story", "live.music", "live.sounds", "story.sound"}: + doc["audio"] = audio return doc diff --git a/src/nnnotes/schemas/product.schema.json b/src/nnnotes/schemas/product.schema.json index ce2eaba..21c4678 100644 --- a/src/nnnotes/schemas/product.schema.json +++ b/src/nnnotes/schemas/product.schema.json @@ -63,6 +63,38 @@ "zh-Hans", "ko" ] + }, + "options": { + "type": "object", + "description": "Optional: Live option name -> the values the files support (options that select files).", + "properties": { + "LiveQuality": { + "type": "array", + "items": { + "enum": [ + 0, + 1, + 2 + ] + }, + "minItems": 1 + } + }, + "additionalProperties": { + "type": "array", + "minItems": 1 + } + }, + "variants": { + "type": "array", + "items": { + "type": "string", + "minLength": 1 + } + }, + "audio": { + "type": "boolean", + "description": "Whether audio streams are included." } }, "$defs": { diff --git a/src/nnnotes/schemas/site.schema.json b/src/nnnotes/schemas/site.schema.json new file mode 100644 index 0000000..68d9cec --- /dev/null +++ b/src/nnnotes/schemas/site.schema.json @@ -0,0 +1,91 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "urn:nnnotes:schema:site", + "title": "nnnotes.site/1", + "description": "The complete catalog of region topics published by a named site.", + "type": "object", + "required": [ + "format", + "name", + "language", + "regions", + "topics" + ], + "properties": { + "format": { + "const": "nnnotes.site/1" + }, + "name": { + "type": "string", + "pattern": "^[a-zA-Z0-9_-]+$" + }, + "language": { + "type": "string" + }, + "regions": { + "type": "array", + "items": { + "type": "object", + "required": [ + "id", + "name" + ], + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "name": { + "type": "string" + }, + "languages": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "additionalProperties": false + } + }, + "topics": { + "type": "array", + "items": { + "type": "object", + "required": [ + "kind", + "id", + "region", + "manifest", + "entry" + ], + "properties": { + "kind": { + "enum": [ + "live", + "model", + "story" + ] + }, + "id": { + "type": "string", + "minLength": 1 + }, + "region": { + "type": "string", + "minLength": 1 + }, + "manifest": { + "type": "string", + "pattern": "^products/[0-9a-f]{64}\\.json$" + }, + "entry": { + "type": "object" + } + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false +} diff --git a/src/nnnotes/site/build.py b/src/nnnotes/site/build.py new file mode 100644 index 0000000..fd7b328 --- /dev/null +++ b/src/nnnotes/site/build.py @@ -0,0 +1,174 @@ +"""Build and publish a named site's complete product and page layout.""" + +from __future__ import annotations + +import hashlib +import json + +from ..pipeline import contract +from ..pipeline.orchestrate import Run +from ..pipeline.siteproducts import at +from . import catalog, pages, publish +from .products import REFERENCE_FIELDS, Snapshot, _strings +from .reads import Reader + + +def combine(run, other): + """Combine content-addressed site tasks, counting repeated tasks once.""" + for tid, task in other.tasks.items(): + old = run.tasks.get(tid) + if old and old["key"] != task["key"]: + raise ValueError(f"site task {tid} has conflicting keys") + if not old or task["status"] != "hit": + run.tasks[tid] = task + run.stages.update(other.stages) + run.params.update(other.params) + run.log.extend(other.log) + run.failures_.extend(other.failures_) + + +def documents(depot, entries): + """Decoded product manifests from an unencoded product layout.""" + out = {} + for e in entries: + if e["path"].startswith("assets/") or not e["path"].endswith(".json"): + continue + doc = json.loads(depot.read(e["sha256"])) + if doc.get("schema") == contract.PRODUCT: + out[e["path"]] = doc + return out + + +def document(depot, manifests, path, logical): + entry = manifests[path]["files"][logical] + return json.loads(depot.read(entry["asset"].split("/")[-1].split(".")[0])) + + +def model_stats(snapshot, source): + manifest = snapshot.documents[source] + files = [*manifest["files"].values(), *(e for o in manifest["objects"].values() for e in o["files"].values())] + return {"bytes": sum(e["size"] for e in files), "files": len(files)} + + +def size(snapshot, source, language=None): + """Unique decoded asset bytes in a product's closure for one optional language.""" + from ..formats import compression + + by_target = {snapshot.paths[p]: m for p, m in snapshot.documents.items()} + visited, files = set(), {} + + def raw(path): + return compression.decode(path, snapshot.depot.read(snapshot.entries[path]["sha256"])) + + def visit(path): + if path in visited: + return + visited.add(path) + man = by_target[path] + for group in [man["files"], *(o["files"] for o in man["objects"].values())]: + for e in group.values(): + if "parts" in e: + files.update({p: n for _, p, n, *_ in e["parts"]}) + else: + files[e["asset"]] = e["size"] + for logical, paths in REFERENCE_FIELDS.get(man["product"], {}).items(): + e = man["files"][logical] + doc = {k: json.loads(raw(p)) for k, p, *_ in e["parts"]} if "parts" in e else json.loads(raw(e["asset"])) + for field in paths: + refs = at(doc, field) + if language is not None and isinstance(refs, dict): + refs = { + k: ({language: v[language]} if isinstance(v, dict) and language in v else v) + for k, v in refs.items() + } + for ref in _strings(refs): + visit(ref) + for ref in man.get("variants", []): + visit(ref) + + visit(snapshot.paths[source]) + return sum(files.values()) + + +class Builder: + def __init__(self, depot, name, language, regions, player, *, encoding="gzip"): + self.depot, self.name, self.language = depot, name, language + self.old, self.entries = catalog.load(depot, name) + self.regions, self.player, self.encoding = regions, player, encoding + self.topics, self.replace = [], set() + self.run = Run(depot, {}, ["site:" + name], {}, {}) + self.reader = None + self.live_count = 0 + + def add(self, entries, source, kind, tid, region, facts, *, languages=None, options=None, audio=True): + manifests = documents(self.depot, entries) + if languages and self.language in languages: + manifests[source]["language"] = self.language + variants, reads = [], None + if kind == "live": + variants = sorted( + p for p, m in manifests.items() if m["product"] in {"live.noteskin", "live.noteeffect", "live.sounds"} + ) + if self.reader is None: + self.reader = Reader(self.depot, self.player) + sample = int.from_bytes(hashlib.sha256(source.encode()).digest()[:4], "big") % 10 == 0 + reads = self.reader.read( + entries, + source, + products=variants, + variants=options.read_variants() if options else (), + check=self.live_count == 0 or sample, + ) + self.live_count += 1 + combine(self.run, self.reader.run) + snapshot = Snapshot( + self.depot, + manifests, + languages=languages, + encoding=self.encoding, + variants=variants, + live_options=options.manifest_options() if options else None, + read_files=reads, + audio=audio, + ) + target = snapshot.product(source) + combine(self.run, snapshot.run) + self.entries.update(snapshot.entries) + self.topics.append({"kind": kind, "id": str(tid), "region": region, "manifest": target, "entry": facts}) + return snapshot + + def finish(self, directory): + doc = catalog.merge(self.old, self.name, self.language, self.regions, self.topics, self.replace) + entries, run = catalog.assemble(self.depot, doc, self.entries) + combine(self.run, run) + files, stats = pages.gather(self.player, stories=any(t["kind"] == "story" for t in doc["topics"])) + for path, body in files.items(): + entries[path] = { + "path": path, + "sha256": self.depot.put(body), + "size": len(body), + "id": contract.artifact_id(contract.task_id("site.page", path), "file"), + } + placed = publish.materialize(directory, list(entries.values()), self.depot, params={"name": self.name}) + sha = self.depot.put(contract.encode(placed["manifest"])) + if sha != placed["sha256"]: + raise ValueError("published layout digest differs") + manifest = self.run.write({"site": sha}) + return { + **stats, + "name": self.name, + "run": manifest["id"], + "tasks": manifest["summary"], + "topics": len(doc["topics"]), + "write": placed["write"], + "keep": placed["keep"], + "removed": placed["remove"], + "files": len(entries), + "readSets": { + "checked": self.reader.checked, + "mismatches": self.reader.mismatches, + "planErrors": self.reader.plan_errors, + } + if self.reader + else None, + } diff --git a/src/nnnotes/site/catalog.py b/src/nnnotes/site/catalog.py new file mode 100644 index 0000000..d58fe03 --- /dev/null +++ b/src/nnnotes/site/catalog.py @@ -0,0 +1,123 @@ +"""Persistent site topics and the complete file closure of their product snapshots.""" + +from __future__ import annotations + +import json +import re + +from ..formats import compression +from ..pipeline import contract, siteindex, siteproducts +from ..pipeline.contract import Input +from .products import REFERENCE_FIELDS, Snapshot, _strings + + +def load(depot, name): + """The last successfully materialized catalog and its layout entries, fetched through the depot.""" + run = depot.latest_run(["site:" + name]) + if not run: + return None, {} + sha = run.get("layouts", {}).get("site") + if not sha: + raise ValueError(f"site {name}: previous run has no site layout") + layout = contract.loads(depot.read(sha)) + entries = {e["path"]: e for e in layout["entries"]} + catalog = contract.loads(depot.read(entries["site.json"]["sha256"])) + if catalog.get("format") != siteindex.CATALOG or catalog.get("name") != name: + raise ValueError("site catalog identity differs") + return catalog, entries + + +def merge(old, name, language, regions, topics, replace=()): + """Update named topics; complete selections replace their region-kind group.""" + if not re.fullmatch(r"[a-zA-Z0-9_-]+", name): + raise ValueError("site name must contain letters, digits, underscores or hyphens") + replace = set(replace) + known = { + (t["region"], t["kind"], t["id"]): t + for t in (old or {}).get("topics", []) + if (t["region"], t["kind"]) not in replace + } + known.update({(t["region"], t["kind"], t["id"]): t for t in topics}) + rs = {r["id"]: r for r in (old or {}).get("regions", [])} + rs.update({r["id"]: r for r in regions}) + return { + "format": siteindex.CATALOG, + "name": name, + "language": language, + "regions": list(rs.values()), + "topics": [known[k] for k in sorted(known)], + } + + +def assemble(depot, catalog, entries): + """Index tasks, aliases and the exact reachable files of the complete catalog.""" + runner = Snapshot(depot, {}) + body = contract.encode(catalog) + inp = Input("catalog", depot.put(body), len(body), None, ({"kind": "depot"},)) + result = runner._task(siteindex.IndexStage(), catalog["name"], [inp]) + all_entries = dict(entries) + output = {} + aliases = {} + for art in result["artifacts"]: + role = contract.parse_artifact_id(art["id"])[1] + c = art["content"] + if role == "aliases.json": + aliases = json.loads(depot.read(c["sha256"])) + else: + output[role] = {"path": role, "sha256": c["sha256"], "size": c["size"], "id": art["id"]} + visited = set() + + def keep(path): + if path not in all_entries: + raise ValueError(f"site reference has no file: {path}") + output[path] = all_entries[path] + + def content(file): + if "parts" in file: + return ( + b"{" + + b",".join(json.dumps(k, ensure_ascii=False).encode() + b":" + raw(p) for k, p, *_ in file["parts"]) + + b"}" + ) + return raw(file["asset"]) + + def raw(path): + keep(path) + return compression.decode(path, depot.read(all_entries[path]["sha256"])) + + def product(path): + if path in visited: + return + visited.add(path) + keep(path) + manifest = json.loads(depot.read(all_entries[path]["sha256"])) + if manifest.get("schema") != contract.PRODUCT: + raise ValueError(f"{path}: expected a product manifest") + for files in [manifest["files"], *(obj["files"] for obj in manifest["objects"].values())]: + for file in files.values(): + for p in [r[1] for r in file["parts"]] if "parts" in file else [file["asset"]]: + keep(p) + for logical, paths in REFERENCE_FIELDS.get(manifest["product"], {}).items(): + doc = json.loads(content(manifest["files"][logical])) + for field in paths: + for reference in _strings(siteproducts.at(doc, field)): + product(reference) + for reference in manifest.get("variants", []): + product(reference) + + for path in aliases.values(): + product(path) + for path, target in aliases.items(): + manifest = json.loads(depot.read(all_entries[target]["sha256"])) + art = runner._document( + siteproducts.ManifestStage(), + contract.encode(manifest), + { + "files": manifest["files"], + "objects": {oid: obj["files"] for oid, obj in manifest["objects"].items()}, + "root": "../" * (len(path.split("/")) - 1), + }, + ) + c = art["content"] + output[path] = {"path": path, "sha256": c["sha256"], "size": c["size"], "id": art["id"]} + return output, runner.run diff --git a/src/nnnotes/site/pages.py b/src/nnnotes/site/pages.py new file mode 100644 index 0000000..05e275c --- /dev/null +++ b/src/nnnotes/site/pages.py @@ -0,0 +1,184 @@ +"""Player page files, bundles and region metadata for site publication.""" + +from __future__ import annotations + +import hashlib +import re +from pathlib import Path + +from ..source.config import Config, ConfigError + +READ_SET_SCRIPT = "scripts/read-set.mjs" # node