feat(storage): import retained manifest snapshots - #611
Conversation
github-actions
Bot
added
type/docs
634b94b to
a27a220
Compare
Plan and replay canonical view-bundle v1 bytes through retained publication readers without reopening paths or materializing temporary archives. Bind replay to exact subtree authority, drain successful partial consumers, abort failed consumers, and preserve cancellation-aware cleanup. Verified with Python 3.10 and 3.12 view-bundle tests, the atomic/CAS/contract regression surface, and pinned formatting/lint checks.
Reject hostile scalar subclasses nested inside otherwise exact directory ownership tokens and compare canonical built-in snapshots instead of attacker-controlled dataclass equality. Verified with the view-bundle suite and pinned formatting, lint, and compile checks.
Add backend-neutral namespace and retained-import capabilities, canonical response attestation, bounded compound generations, and SQLite pre-publication response checks. Oversized aggregate responses now fail inside the publication transaction before refs move. Verified with the complete storage test suite and focused aggregate rollback probes.
Validate strict context receipts against retained repository sources, stream canonical view bundles and members into CAS, publish identity-closed catalog snapshots, and retain a reversible v2 manifest projection. Keep all artifact validation ahead of CAS and all receipt checks ahead of catalog publication. Verified with real BM25/vector imports, failure injection, retry, source/receipt mutation, backend attestation, and near-limit profile tests.
Add a callback-shaped object retention contract and keep LocalCAS receipt verification, catalog registration, snapshot publication, and attestation inside one cancellation-safe lifecycle fence. This closes the gap where a point-in-time receipt could be reclaimed before its catalog reference became durable.
Document authenticated bundle replay, direct schema-v4 bootstrap import, projection closure, and the remaining export, runtime, GC, and fenced-job work.
Assert that every catalog mutation and attestation occurs after object retention begins, and that both success and failure release the scope.
Document the direct importer at-least-once boundary and prove that an exact retry converges after interruption following the SQLite commit. Verified with the retained manifest importer tests, strict MkDocs build, pre-commit, and the full local unit tier.
a532ad7 to
a97b0a8
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a97b0a8700
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if len(raw_members) > MAX_VIEW_GENERATION_MEMBERS: | ||
| raise StorageValidationError("view generation has too many member objects") |
There was a problem hiding this comment.
When opening a schema-v4 catalog created by the previous release, a generation may legally contain more than 32,768 member objects because the old normalizer imposed no limit and view bundles support up to 100,000 files by default. This new read-time check makes _validate_view_generation_input() classify that persisted generation as an identity conflict, so resolve_ref() and get_manifest_summary() can no longer serve an already-published snapshot after upgrade. Because the schema remains v4 and no migration handles these rows, apply the limit only when accepting new data or add an explicit migration/versioning strategy for existing catalogs.
Useful? React with 👍 / 👎.
| if not receipt.zip_members or len(receipt.zip_members) != len(receipt.members) + 1: | ||
| raise StorageIntegrityError("planned view bundle ZIP inventory is incomplete") | ||
| if receipt.manifest_bytes != _manifest_bytes( | ||
| receipt.view_type, | ||
| receipt.members, | ||
| ): |
There was a problem hiding this comment.
When a caller modifies both the public plan and its accessible _receipt, this validation proves only that the forged ZIP fields are internally consistent; it never checks that receipt.members still equals the file records in receipt.source_ownership. A caller can therefore replace the inventory with an empty or partial tuple, rebuild the manifest/layout and digest, and successfully consume a bundle that omits authenticated source files—the later subtree projection check still passes because it validates the unchanged ownership token independently. Reconstruct and compare the expected ArtifactMember tuple from the retained ownership before accepting the receipt.
Useful? React with 👍 / 👎.
1 participant
Summary
Adds the direct retained RepoManifest M1 bootstrap path on top of the merged
storage authority foundations. The importer consumes authenticated strict
context and source identities, streams canonical BM25/vector artifacts into
CAS, retains their exact receipts through catalog publication, and publishes
an identity-closed schema-v4 snapshot/ref without trusting caller paths or
mutable public projections.
Changes
readers, including exact token/layout validation and bounded stream cleanup.
portable and strict artifact validation.
budgets, backend-neutral namespace identities, and SQLite pre-ref aggregate
response validation with transactional rollback.
reversible internal manifest v2 projection, and attest the final ref and
summary.
verification, snapshot publication, and post-publication attestation so a
compliant GC cannot reclaim an object during publication.
converges after interruption following the SQLite commit.
implementation/policy, and M2 fenced-publication work.
Type of Change
Testing
deselection is an exact-main host-umask baseline failure.
deselected.
git diff --checkpassed.
Checklist