Skip to content

feat(depot): share results, contents and runs through a remote - #53

Merged
nichinichisou0609 merged 1 commit into
redesign/depotfrom
redesign/depot-remote
Oct 14, 2026
Merged

nichinichisou0609 merged 1 commit into
redesign/depotfrom
redesign/depot-remote

Conversation

@nichinichisou0609

Copy link
Copy Markdown
Collaborator

Summary

  • [depot] remote (NNNOTES_DEPOT_REMOTE) gives the depot a remote shared by the machines that use it: s3://<bucket>/<prefix> in S3-compatible object storage (the remote extra, through boto3; path-style requests) or file:///<directory>. [depot] endpoint, access_key_id, secret_access_key (both hidden in prompts) and signing_region configure the object storage.
  • The remote holds the same keys as the depot directory: cas/, ac/, inputs/<kind>/, runs/. Downloads, file identities by path, catalog versions, costs, run logs and leases stay local.
  • The depot directory is the remote's cache: a content, result, input identity or run it lacks is read from the remote and kept; a content read from the remote is checked against its id. What the depot adds goes to the remote as well: the bytes of a result before the result, a run manifest before the pointer to it. A result that differs from the remote's is a conflict.
  • The results of the tasks of the latest run read from the remote are known to be there, so planning asks no question per task.
  • depot gc --remote collects the remote by its runs; its results and contents younger than a day stay.
  • Layouts read contents through the depot, so a layout placed by a depot without the contents reads them from the remote.
  • Docs: configuration, contracts (Depot remote), assets (depot gc --remote); docs/cli.md regenerated. CI installs the remote extra.

Test plan

  • pytest (whole suite, synthetic fixtures)
  • ruff check, ruff format --check
  • pyright (basic, and strict for core/source), import-linter layers
  • mkdocs build --strict
  • new tests: a second depot with the same remote exports with every task a hit and no download; results, contents, identities and runs found through the remote; a conflicting result; a content of the remote checked against its id; contents stored before the remote pushed with their result; the latest run answering which results the remote holds; pickling with a remote; S3 requests (bucket, prefixed keys, list times, deletes) through a botocore stub; settings checks; depot gc --remote from the command line

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

[depot] remote gives the depot a remote shared by the machines that use
it: s3://<bucket>/<prefix> in S3-compatible object storage (the remote
extra, through boto3) or file:///<directory>, with [depot] endpoint,
access_key_id, secret_access_key and signing_region. The remote holds
the same keys as the depot directory: cas/, ac/, inputs/<kind>/, runs/.

The depot directory is the remote's cache. A content, result, input
identity or run it lacks is read from the remote and kept, and a content
read from the remote is checked against its id. What the depot adds goes
to the remote as well: the bytes of a result before the result, a run
manifest before the pointer to it. A result that differs from the
remote's is a conflict. The results of the tasks of the latest run read
from the remote are known to be there without a request for each.

depot gc --remote collects the remote by its runs; its results and
contents younger than a day stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nichinichisou0609
nichinichisou0609 merged commit a7e040e into redesign/depot Oct 14, 2026
8 checks passed
@nichinichisou0609
nichinichisou0609 deleted the redesign/depot-remote branch October 14, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant