feat(link)!: find objects by their Resources.Load path

docs/contracts.md

@@ -229,7 +229,7 @@ Documents written by particular stages (artifacts of no object: their ids are `<
 | `nnnotes.catalog-index/1` | `catalog.index` | [catalog-index.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/catalog-index.schema.json) |
 | `nnnotes.census/2` | `unity.census` | [census.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/census.schema.json) |
 | `nnnotes.scripts/1` | `link.scripts` | [scripts.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/scripts.schema.json) |
-| `nnnotes.addresses/1` | `link.addresses` | [addresses.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/addresses.schema.json) |
+| `nnnotes.addresses/2` | `link.addresses` | [addresses.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/addresses.schema.json) |
 | `nnnotes.view/1` | `view.<name>` | [view.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/view.schema.json) |
 | `nnnotes.objects/1` | `pipeline.assemble` (a product's object table) | [objects.schema.json](https://github.com/MetaSekaiLab/nnnotes/blob/main/src/nnnotes/schemas/objects.schema.json) |
 

docs/products.md

@@ -12,10 +12,12 @@ objects:
 
 - the objects of the catalog keys the stage gives for the subject (`keys`), through the address table of
   `link.addresses`;
+- the objects of the Resources.Load paths it gives (`resources`): paths of the ResourceManager in the APK's built-in
+  files, such as the materials a runtime loads by name, through the same table and compared ignoring case;
 - the object ids the stage names besides (`root_objects`).
 
-Both are read from the planning facts alone, so a command can ask for what its products need before anything runs:
-`assemble.request(stages, products, env)` gives the fact `roots` (`{"keys", "objects"}`), and the request of
+All are read from the planning facts alone, so a command can ask for what its products need before anything runs:
+`assemble.request(stages, products, env)` gives the fact `roots` (`{"keys", "resources", "objects"}`), and the request of
 `unity.export` then holds the units of those objects and every unit reachable from them through the census `refs`
 ([units](stages.md#units)), and nothing else of their bundles.
 
@@ -27,7 +29,7 @@ object that a reached object references and no census holds is listed as `missin
 | Part of the task | |
 |---|---|
 | inputs | `objects`: the object table of what it reaches (`nnnotes.objects/1`); the stage's own inputs (master tables, data its parameters name) |
-| depends | `link.addresses` (when it has keys), the `unity.export` and `sprite.crop` tasks whose results it read |
+| depends | `link.addresses` (when it has keys or resource paths), the `unity.export` and `sprite.crop` tasks whose results it read |
 | artifacts | `<task id>#<path>`: the product's own files; `<task id>#objects`: the object table of the objects its files use (the whole table unless the stage narrows it) |
 | item | one for the subject: `exported`, or `unsupported` with a reason code |
 

docs/schemas.md

@@ -14,7 +14,7 @@ schemas.load("plan")  # the schema as a dict
 
 | Schema | Format | Document |
 |---|---|---|
-| `addresses` | `nnnotes.addresses/1` | the address table of a catalog version and the censuses of its bundles |
+| `addresses` | `nnnotes.addresses/2` | the address table of a catalog version and the censuses of its bundles |
 | `artifact` | `nnnotes.artifact/1` | the content, provenance and semantics of one output file |
 | `catalog-diff` | `nnnotes.catalog-diff/1` | the difference between two catalog versions (`catalog diff`) |
 | `catalog-index` | `nnnotes.catalog-index/1` | the index of a catalog pair: remote and APK catalogs, every location in catalog order |

docs/stages.md

@@ -12,22 +12,23 @@ The asset export is built from three layers:
 
 ## Stages
 
-Every stage is at version 1, except `unity.census` (version 3: it lists what every object references and the
-hierarchy holding it), `unity.export` (version 2: one task per unit), `cri.movie` (version 2: every stream kind,
+Every stage is at version 1, except `unity.census` (version 4: it lists what every object references, the
+hierarchy holding it and the Resources.Load paths of a ResourceManager), `link.addresses` (version 2: the
+Resources.Load paths), `unity.export` (version 2: one task per unit), `cri.movie` (version 2: every stream kind,
 codec and channel of a USM) and `catalog.index` (version 2: gzip input and Japanese remote asset placeholders).
 
 | Stage | Subject | Inputs | Outputs |
 |---|---|---|---|
 | `catalog.index` | a catalog (remote + APK) | the catalog files | every location (with the bundle hash, crc and size), bundles and raw remote files |
-| `unity.census` | a bundle, or a built-in file of the APK | the bundle | serialized files, externals, objects (file, path id, class, size, type hash, name, script, the scripts an AnimationClip's bindings name, the objects it references, the hierarchy holding it), container, scene hashes, dependencies, script fields |
+| `unity.census` | a bundle, or a built-in file of the APK | the bundle | serialized files, externals, objects (file, path id, class, size, type hash, name, script, the scripts an AnimationClip's bindings name, the objects it references, the hierarchy holding it), container, scene hashes, dependencies, script fields, the ResourceManager's paths (Resources.Load) |
 | `link.scripts` | global | the censuses holding MonoScripts | serialized file and path id -> assembly, namespace, class |
 | `unity.export` | a unit of a bundle or of a built-in file of the APK (below), by the stable address of its root object | the bundle; context `unit` (the root's object id), `bundle`, the script entries its objects use and, for files without type trees, the type trees nnnotes ships for their type hashes | its objects' artifacts and one item per object |
 | `sprite.crop` | a SpriteAtlas | the atlas' JSON, the images of its textures, the metas of the sprites of other bundles it packs | the sprites' images |
 | `cri.audio` | ACB content | `acb` (a raw file of the catalog, or the `acb` artifact of a cue sheet held in a bundle), `awb` (an AWB the catalog pairs with it), `boot` (the game's boot data: the key's source) | FLAC per stream, `cues.json`, `streams.json` |
 | `cri.movie` | USM content | `usm` (a raw file of the catalog, or the `usm` artifact of a movie held in a bundle), `boot` | every stream as stored (video, alpha, audio, subtitles), `movie.mkv` (or `movie.webm`) |
 | `live2d.model` | a Live2D model key (`Character/Live2D/<group>/<name>/model/<name>`) whose bundle is selected | `bundle:<n>`: the bundles of the key's dependency closure in the order the extractor loads them (the order is part of the key); context `internalId` (the key's location); atom `live2d.extract_model` (the extractor of `nnnotes export live2d`, run unchanged) | `live2d.model:<key>#<path>`: every file `nnnotes export live2d` writes for the model, `<path>` relative to the model's directory; placed under `<key>/` |
 | `spine.skeleton` | a SkeletonDataAsset of a selected bundle, by its stable address `<bundle>:<pathId>` (in a bundle of several serialized files the file's tag follows the bundle name, as in `link.addresses`) | `bundle:<stable name>`: the bundles holding the serialized files the asset's file reaches through its externals; context `object` (the asset's object id); atom `spot.write_skeleton` (the Spine writer of `nnnotes export spot`) | `spine.skeleton:<subject>#<file>`: the skeleton (`.json` or `.skel`), its atlases (`.atlas`) and their page images, the bytes `nnnotes export spot` writes into `spine/`; one item for the asset |
-| `link.addresses` | global | catalog index, censuses (the built-in files' too) | key -> objects, serialized file -> bundle or built-in file, bundle -> keys |
+| `link.addresses` | global | catalog index, censuses (the built-in files' too) | key -> objects, Resources.Load path -> objects, serialized file -> bundle or built-in file, bundle -> keys |
 | `view.<name>` | a catalog (`main`) | `master:<Table>` per master table it reads, `rules` (the view's rule and the resolvers it uses), `addresses` (the part of the address table it read: the keys looked up and the prefixes listed); no parameters, atoms or context | `view.<name>:<subject>#view` (facts: rows, entries, gaps, unreferenced) |
 | `view.voices` | a catalog (`main`) | `master:<Table>` per master table its rules read, `rules`, `addresses`, `sheets` (each cue sheet the `cri.audio` results are known by that a voice names or the rules' prefixes list: its task and ACB content id), `acb:<sha256>` (the ACB of each decoded one), `episodes` (per story episode and part, the `unity.export` artifact holding the object its key names), `adv:<sha256>` (those contents); context `audio` (the keys of those `cri.audio` results); runs after `unity.export` and `cri.audio` | `view.voices:<subject>#view` (facts: rows, entries, gaps, unreferenced streams) |
 
@@ -110,9 +111,10 @@ several serialized files the file's tag follows the bundle name, as in `link.add
 bundle's content changes, so a unit keeps its task across catalog versions.
 
 A run exports the units holding an object of the selected bundles, the units of the objects the fact "roots" names
-(a product's roots: [products](products.md)) and every unit reachable from those through what their objects reference
-(the census `refs`, across the censuses of the run); with `--only-class` only the units whose root object is of those
-classes. The selection is not part of any key: a unit's task is the same whichever request reaches it.
+(catalog keys, Resources.Load paths and object ids: a product's roots, [products](products.md)) and every unit reachable
+from those through what their objects reference (the census `refs`, across the censuses of the run); with `--only-class`
+only the units whose root object is of those classes. The selection is not part of any key: a unit's task is the same
+whichever request reaches it.
 
 A task's key includes the atoms `reader` and `sniff`, the converter `generic.json`, the converters (with their
 atoms) of the classes of the unit's objects, its bundle, its root and the script entries and shipped type trees its

src/nnnotes/pipeline/assemble.py

@@ -1,8 +1,9 @@
 """Product stages: products assembled from the unity.export results of the objects they reach, never from a bundle.
 
 A product (a Live2D model, a story, a live, ...) is a stage whose task per subject starts from root objects: the
-objects of the catalog keys `keys(subject)` gives and the objects `root_objects(subject)` names (both read from the
-planning facts alone, so a command can ask for the units its products need before anything runs: `request`). The
+objects of the catalog keys `keys(subject)` gives, of the Resources.Load paths `resources(subject)` gives and the
+objects `root_objects(subject)` names (all read from the planning facts alone, so a command can ask for the units its
+products need before anything runs: `request`). The
 objects it reaches are those of the units holding its roots and every unit an item of a reached unit references
 (contract.item `refs`), through the unity.export results; a sprite whose atlas is in another bundle has its image
 from sprite.crop. A reached unit that the request did not export is an error of the request; a referenced object
@@ -34,15 +35,16 @@
 
 
 def request(stages: dict, products: dict, env) -> dict:
-    """The fact "roots" (pipeline.units) of a request for `products` {stage name: subjects}: the catalog keys and the
-    root objects of every subject, sorted."""
-    keys, objects = set(), set()
+    """The fact "roots" (pipeline.units) of a request for `products` {stage name: subjects}: the catalog keys, the
+    resource paths and the root objects of every subject, sorted."""
+    keys, resources, objects = set(), set(), set()
     for name, subjects in sorted(products.items()):
         stage = stages[name]
         for s in subjects:
             keys.update(stage.keys(s, env))
+            resources.update(stage.resources(s, env))
             objects.update(stage.root_objects(s, env))
-    return {"keys": sorted(keys), "objects": sorted(objects)}
+    return {"keys": sorted(keys), "resources": sorted(resources), "objects": sorted(objects)}
 
 
 def _entry(item: dict, arts: dict) -> dict:
@@ -211,7 +213,8 @@ class Assembly:
 
 class ProductStage(Stage):
     """Base of the product stages (module documentation). Subclasses set `name`, `version` and `KIND`, and define
-    `keys` and/or `root_objects` and `assemble`; `subjects` defaults to the subjects of the fact "products"."""
+    `keys`, `resources` and/or `root_objects` and `assemble`; `subjects` defaults to the subjects of the fact
+    "products"."""
 
     after = (ADDRESSES, EXPORT, CROP)
     KIND = "product"
@@ -235,8 +238,13 @@ def keys(self, subject: str, env) -> list[str]:
         """The catalog keys whose objects are roots of `subject` (from the facts alone)."""
         return []
 
+    def resources(self, subject: str, env) -> list[str]:
+        """The Resources.Load paths whose objects are roots of `subject` (from the facts alone)."""
+        return []
+
     def root_objects(self, subject: str, env) -> list[str]:
-        """Object ids that are roots of `subject` besides those of its keys (from the facts alone)."""
+        """Object ids that are roots of `subject` besides those of its keys and resource paths (from the facts
+        alone)."""
         return []
 
     def more_inputs(self, subject: str, env) -> list[Input]:
@@ -253,7 +261,9 @@ def manifest(self, subject: str) -> str:
     # ---------------------------------------------------------------- planning
     def reach(self, subject: str, env) -> Reach:
         """What `subject` reaches (kept while the results it read are the same)."""
-        roots = units.root_objects(env, self.keys(subject, env), self.root_objects(subject, env))
+        roots = units.root_objects(
+            env, self.keys(subject, env), self.root_objects(subject, env), self.resources(subject, env)
+        )
         sig = (tuple(roots), id(units.table(env)))
         memo = self._memo.get(subject)
         if memo is not None and memo[0] == sig and all(env.key(t) == k for t, k in memo[1]):
@@ -268,7 +278,8 @@ def inputs(self, subject: str, env) -> list[Input]:
         return [table, *self.more_inputs(subject, env)]
 
     def depends(self, subject: str, env) -> list[str]:
-        own = [contract.task_id(ADDRESSES, s) for s in sorted(env.fact("catalogs"))] if self.keys(subject, env) else []
+        named = self.keys(subject, env) or self.resources(subject, env)
+        own = [contract.task_id(ADDRESSES, s) for s in sorted(env.fact("catalogs"))] if named else []
         return own + self.reach(subject, env).read
 
     def estimate(self, subject: str, env, inputs: list[Input]) -> Cost:

src/nnnotes/pipeline/census.py

@@ -17,6 +17,9 @@
     assetBundles   the AssetBundle objects: {file, pathId, name, assetBundleName, dependencies,
                    isStreamedSceneAssetBundle, sceneHashes [[scene, hash]], container [{path, file, pathId,
                    preloadIndex, preloadSize}] (stored order)}
+    resourcePaths  the container of the ResourceManager objects (the paths Resources.Load finds objects at, in the
+                   APK's global managers): [{path, file, pathId}] in stored order; a path names one object per
+                   class it was saved as
     scripts        the MonoScript objects: {file, pathId, assembly, namespace, class, name}
     classes        objects per class; `objects` the total
     errors         [{object, message}] of the objects whose typetree could not be read (their record has no name,
@@ -140,6 +143,14 @@ def _asset_bundle(d: dict, refs: _Refs, oid: tuple[str, int]) -> dict:
     }
 
 
+def _resource_paths(d: dict, refs: _Refs) -> list:
+    out = []
+    for path, p in _pairs(d.get("m_Container")):
+        target = refs.ref(p) or {}
+        out.append({"path": path, "file": target.get("file"), "pathId": target.get("pathId")})
+    return out
+
+
 def _binding_scripts(d: dict, refs: _Refs) -> list:
     """The MonoScripts an AnimationClip's generic bindings name, {file, pathId} (or {fileId, pathId}), sorted."""
     out = {}
@@ -170,7 +181,7 @@ def census_env(env, shipped: bool = False) -> dict:
     by_file: dict[str, list] = {}
     for i in ctx.infos:
         by_file.setdefault(i.file, []).append(i)
-    files, bundles, scripts, errors, classes = [], [], [], [], {}
+    files, bundles, paths, scripts, errors, classes = [], [], [], [], [], {}
     for name in sorted(by_file):
         sf = view.serialized_file(name)
         refs = _Refs(name, ctx.ext_files[name])
@@ -204,6 +215,8 @@ def census_env(env, shipped: bool = False) -> dict:
                     rec["refs"] = sorted(contract.object_id(*k) for k in found)
                 if i.class_name == "AssetBundle":
                     bundles.append(_asset_bundle(d, refs, key))
+                elif i.class_name == "ResourceManager":
+                    paths.extend(_resource_paths(d, refs))
                 elif i.class_name == "MonoScript":
                     scripts.append(
                         {
@@ -241,6 +254,7 @@ def census_env(env, shipped: bool = False) -> dict:
         "files": files,
         "resources": resources,
         "assetBundles": sorted(bundles, key=lambda b: (b["file"], b["pathId"])),
+        "resourcePaths": paths,
         "scripts": sorted(scripts, key=lambda s: (s["file"], s["pathId"])),
         "classes": dict(sorted(classes.items())),
         "objects": sum(classes.values()),
@@ -332,7 +346,7 @@ class CensusStage(Stage):
     MonoBehaviours with."""
 
     name = "unity.census"
-    version = 3
+    version = 4
     ATOMS = {"reader": impl_id(("UnityPy",), 1)}
 
     def subjects(self, env) -> list[str]: