feat(depot): share results, contents and runs through a remote

.github/workflows/ci.yml

@@ -32,7 +32,7 @@ jobs:
       - name: Install (with the deck model and the fonts extra)
         run: |
           python -m pip install --upgrade pip
-          python -m pip install -e ".[test,deck,fonts]"
+          python -m pip install -e ".[test,deck,fonts,remote]"
       - name: Lint
         run: |
           python -m ruff check .
@@ -55,7 +55,7 @@ jobs:
       - name: Install
         run: |
           python -m pip install --upgrade pip
-          python -m pip install -e ".[test,deck,fonts]"
+          python -m pip install -e ".[test,deck,fonts,remote]"
       - name: Type check
         run: python -m pyright
 

docs/assets.md

@@ -126,7 +126,9 @@ Removes what no kept run reaches. The kept runs are, for each selection, the `--
 the latest run of a selection is always kept). A kept run reaches the results of its tasks, the contents those
 results name (their artifacts, and the inputs their task keys name when the depot holds them) and its layout
 manifests with the files they list. Every other run, result (with its measured cost) and content is removed.
-`--dry-run` counts what would be removed. A layout made of hard links or clones keeps its files. gc runs only when
+`--dry-run` counts what would be removed. A layout made of hard links or clones keeps its files. `--remote`
+collects the depot's remote the same way, by the runs on the remote; its results and contents younger than a day stay,
+since a run on another machine may be writing them. gc runs only when
 no other command uses the depot (it exits with an error otherwise), and commands that start meanwhile wait for it.
 The downloads are bounded separately, by `[depot] downloads` ([commands.md](commands.md#downloads)).
 

docs/cli.md

@@ -356,6 +356,7 @@ remove the runs, results and contents no kept run reaches
 | --- | --- |
 | `--keep` `N` | keep the N latest runs of each selection (default: every run; the latest run of a selection is kept) |
 | `--dry-run` | count what would be removed, remove nothing |
+| `--remote` | also collect the depot's remote, by its runs (its results and contents younger than a day stay) |
 
 ## nnnotes voices list
 

docs/configuration.md

@@ -111,6 +111,11 @@ exist when they are given.
 | `[paths] fonts.<language>` | `NNNOTES_PATHS_FONTS_<LANGUAGE>` (e.g. `NNNOTES_PATHS_FONTS_ZH_HANT`) | `--font <language>=PATH` (of `site build`) | font file (OpenType or TrueType) the story text of a language is drawn with (`--fonts open`); also read for the other story languages whose game fonts fall back to that language's font |
 | `[paths] fonts.emoji` | `NNNOTES_PATHS_FONTS_EMOJI` | `--font emoji=PATH` (of `site build`) | colour emoji font (PNG bitmap glyphs: CBDT or sbix) the stories' emoji sprites are drawn from (`--fonts open`); Noto Color Emoji (SIL Open Font License 1.1) is the tested font. Optional: without it the sprites keep their layout with empty glyphs, listed in `ui/fonts.json` `coverage.sprites.missing` |
 | `[depot] downloads` | `NNNOTES_DEPOT_DOWNLOADS` | — | how much of the depot's downloaded bundles and raw files to keep once no command uses the depot: a size (`500MB`, `20GiB`; units B, KB, MB, GB, TB and KiB, MiB, GiB, TiB), `all`, or `0` (unset) ([commands.md](commands.md#downloads)) |
+| `[depot] remote` | `NNNOTES_DEPOT_REMOTE` | — | a remote of the depot shared by the machines that use it: `s3://<bucket>/<prefix>` (S3-compatible object storage; `pip install 'nnnotes[remote]'`) or `file:///<directory>` ([contracts.md](contracts.md#depot-remote)) |
+| `[depot] endpoint` | `NNNOTES_DEPOT_ENDPOINT` | — | endpoint URL of the S3-compatible object storage; unset: AWS S3 |
+| `[depot] access_key_id` | `NNNOTES_DEPOT_ACCESS_KEY_ID` | — | access key id of the object storage; unset: boto3 looks for credentials itself (`AWS_ACCESS_KEY_ID`, `~/.aws`) |
+| `[depot] secret_access_key` | `NNNOTES_DEPOT_SECRET_ACCESS_KEY` | — | secret access key of the object storage |
+| `[depot] signing_region` | `NNNOTES_DEPOT_SIGNING_REGION` | — | region the requests to the object storage are signed for; unset: `us-east-1` |
 | `[export] link` | `NNNOTES_EXPORT_LINK` | `--link` (of `export assets`) | how `export assets` makes the layout files from the depot: `auto` (unset), `clone`, `hard` or `copy` ([assets.md](assets.md)) |
 
 Hex values are case-insensitive; a `0x` prefix and surrounding whitespace are ignored. All values are TOML strings

docs/contracts.md

@@ -256,5 +256,29 @@ The depot is a directory: the downloaded game files, and the results of all stag
 
 Every file is written to a temporary name and renamed into place, so concurrent writers never see or leave a
 partial file and an interrupted task leaves no result; a record that changes is one file, replaced whole. `cas/` and
-`ac/` depend only on the tasks run: two runs of the same tasks, with any number of workers, leave identical trees.
-`downloads/`, `inputs/`, `runs/*.log.jsonl`, `costs/` and `leases/` are local and not deterministic.
+`ac/` depend only on the tasks run: two runs of the same tasks, with any number of workers, leave identical trees;
+`inputs/`, `runs/`, `costs/` and `leases/` are not deterministic.
+
+### Depot remote
+
+`[depot] remote` gives the depot a remote shared by the machines that use it: `s3://<bucket>/<prefix>` in
+S3-compatible object storage (`pip install 'nnnotes[remote]'`; path-style requests, so stores other than AWS answer
+them) or `file:///<directory>`. The remote holds the same keys as the depot directory, below its prefix:
+
+```
+cas/sha256/ab/<sha256>          contents
+ac/ab/<key>.json                results
+inputs/<kind>/ab/<digest>.json  input identities by name
+runs/<run>.json                 run manifests
+runs/selections/<digest>        the id of the latest run of a selection
+```
+
+The depot directory is the remote's cache. A content, result, input identity or run the depot lacks is read from the
+remote and kept; a content read from the remote is checked against its content id. What the depot adds goes to the
+remote as well: the bytes a result names before the result, a run manifest before the pointer to it. A result that
+differs from the remote's result of the same key is a conflict, as in the depot itself. The latest run of a selection
+is the one any machine wrote last, and the results of its tasks are known to be on the remote without asking for each
+of them. Downloads, file identities by path (`inputs/paths/`), catalog versions, measured costs, run logs and leases
+stay in the depot directory.
+
+Every request names one object; only `depot gc --remote` lists the remote.

pyproject.toml

@@ -29,6 +29,8 @@ dynamic = ["version"]
 test = ["pytest>=8", "ruff>=0.15", "jsonschema>=4.18", "import-linter>=2.1", "pyright>=1.1.414"]
 fonts = ["fonttools>=4.50", "freetype-py>=2.5", "scipy>=1.13"]
 docs = ["mkdocs>=1.6,<2", "mkdocs-material>=9.5", "mkdocstrings[python]>=0.29"]
+# a depot remote in S3-compatible object storage ([depot] remote = "s3://...")
+remote = ["boto3>=1.35"]
 # the deck model of music-data: one nnnotes version carries one ournotes-deck release (.github/workflows/deck.yml moves
 # the pin when ournotes-deck publishes a newer one)
 deck = ["ournotes-sim==0.0.8"]

src/nnnotes/cli/assets.py

@@ -456,13 +456,13 @@ class Workspace:
 
     def __init__(self, args, cfg, log=_log):
         from ..pipeline.catalogdb import CatalogDB
-        from ..pipeline.depot import Depot
+        from ..pipeline.depot import open_depot
 
         self.args, self.cfg, self.log = args, cfg, log
         self.root = Session(cfg).depot_root()
         self.apk = cfg.path("paths", "apk")
         self.fetcher = CatalogFetcher(self.root, cfg)
-        self.depot = Depot(self.root, fetch=self.fetcher)
+        self.depot = open_depot(cfg, fetch=self.fetcher)
         self.db = CatalogDB(self.root)
         try:
             self.selection = parse_selection(getattr(args, "select", None))
@@ -1399,7 +1399,7 @@ def _task_files(items) -> list[Path]:
 
 
 def cmd_run_stage(args, cfg):
-    from ..pipeline.depot import Depot
+    from ..pipeline.depot import open_depot
     from ..pipeline.orchestrate import run_one
 
     problems, tasks = [], []
@@ -1425,7 +1425,7 @@ def cmd_run_stage(args, cfg):
             print(f"nnnotes: incompatible task: {m}", file=sys.stderr)
         sys.exit(EXIT_INCOMPATIBLE)
     root = Session(cfg).depot_root()
-    depot = Depot(root, fetch=CatalogFetcher(root, cfg) if args.fetch else None)
+    depot = open_depot(cfg, fetch=CatalogFetcher(root, cfg) if args.fetch else None)
     use(cfg)
     out, bad = [], 0
     for t in tasks:

src/nnnotes/cli/catalog.py

@@ -190,7 +190,7 @@ def cmd_fetch(args, cfg):
 
 def cmd_diff(args, cfg):
     from ..pipeline import catalogdb
-    from ..pipeline.depot import Depot
+    from ..pipeline.depot import open_depot
     from .assets import addresses_of
 
     db = _db(cfg)
@@ -200,7 +200,7 @@ def cmd_diff(args, cfg):
             vs.append(db.get(ref))
         except (KeyError, ValueError) as e:
             usage(args, f"{ref}: {e.args[0] if e.args else e}")
-    depot = Depot(Session(cfg).depot_root())
+    depot = open_depot(cfg)
     a, b = vs
     d = catalogdb.diff(db.index(a), db.index(b), addresses_of(depot, a["id"]), addresses_of(depot, b["id"]))
     if args.json:

src/nnnotes/cli/depot.py

@@ -23,7 +23,7 @@ def cmd_verify(args, cfg):
 
 def cmd_gc(args, cfg):
     from ..core.downloads import exclusive
-    from ..pipeline.depot import Depot
+    from ..pipeline.depot import open_depot
 
     if args.keep is not None and args.keep < 1:
         usage(args, "--keep must be at least 1")
@@ -34,17 +34,20 @@ def cmd_gc(args, cfg):
                 f"nnnotes depot gc: another command uses the depot ({root / 'leases'}); "
                 "gc runs when no other command does"
             )
-        r = Depot(root).gc(keep=args.keep, dry_run=args.dry_run)
-    report(
-        args,
-        r,
-        lambda r: [
-            f"{'would remove' if args.dry_run else 'removed'}: {r['runs']['removed']} runs, "
+        depot = open_depot(cfg)
+        if args.remote and depot.remote is None:
+            usage(args, "--remote: the depot has no remote (setting depot.remote)")
+        r = depot.gc(keep=args.keep, dry_run=args.dry_run, remote=args.remote)
+
+    def lines(r: dict, where: str = "") -> list[str]:
+        return [
+            f"{where}{'would remove' if args.dry_run else 'removed'}: {r['runs']['removed']} runs, "
             f"{r['results']['removed']} results, {r['contents']['removed']} contents "
             f"({r['bytesRemoved'] / 1e6:.1f} MB)",
-            f"kept: {r['runs']['kept']} runs, {r['results']['kept']} results, {r['contents']['kept']} contents",
-        ],
-    )
+            f"{where}kept: {r['runs']['kept']} runs, {r['results']['kept']} results, {r['contents']['kept']} contents",
+        ]
+
+    report(args, r, lambda r: lines(r) + (lines(r["remote"], "remote ") if "remote" in r else []))
 
 
 def register(sub) -> None:
@@ -81,3 +84,8 @@ def register(sub) -> None:
         help="keep the N latest runs of each selection (default: every run; the latest run of a selection is kept)",
     )
     c.add_argument("--dry-run", action="store_true", help="count what would be removed, remove nothing")
+    c.add_argument(
+        "--remote",
+        action="store_true",
+        help="also collect the depot's remote, by its runs (its results and contents younger than a day stay)",
+    )

src/nnnotes/pipeline/cristages.py

@@ -92,7 +92,7 @@ def boot_input(depot, apk) -> Input | None:
     if known is None or not depot.has(*known):
         with ApkSet(apk) as z:
             data = z.read(BOOT_IN_APK)
-        known = depot.identify(depot.path(depot.put(data)), "files", name)
+        known = depot.identify(depot.local(depot.put(data)), "files", name)
     return Input("boot", known[0], known[1], "data.unity3d", ({"kind": "depot"},))