Repository navigation
feat(depot): share results, contents and runs through a remote - #53
Merged
Merged
Conversation
[depot] remote gives the depot a remote shared by the machines that use it: s3://<bucket>/<prefix> in S3-compatible object storage (the remote extra, through boto3) or file:///<directory>, with [depot] endpoint, access_key_id, secret_access_key and signing_region. The remote holds the same keys as the depot directory: cas/, ac/, inputs/<kind>/, runs/. The depot directory is the remote's cache. A content, result, input identity or run it lacks is read from the remote and kept, and a content read from the remote is checked against its id. What the depot adds goes to the remote as well: the bytes of a result before the result, a run manifest before the pointer to it. A result that differs from the remote's is a conflict. The results of the tasks of the latest run read from the remote are known to be there without a request for each. depot gc --remote collects the remote by its runs; its results and contents younger than a day stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[depot] remote(NNNOTES_DEPOT_REMOTE) gives the depot a remote shared by the machines that use it:s3://<bucket>/<prefix>in S3-compatible object storage (theremoteextra, through boto3; path-style requests) orfile:///<directory>.[depot] endpoint,access_key_id,secret_access_key(both hidden in prompts) andsigning_regionconfigure the object storage.cas/,ac/,inputs/<kind>/,runs/. Downloads, file identities by path, catalog versions, costs, run logs and leases stay local.depot gc --remotecollects the remote by its runs; its results and contents younger than a day stay.depot gc --remote);docs/cli.mdregenerated. CI installs theremoteextra.Test plan
pytest(whole suite, synthetic fixtures)ruff check,ruff format --checkmkdocs build --strictdepot gc --remotefrom the command lineCo-Authored-By: Claude Opus 5.5 noreply@anthropic.com