Skip to content

feat(storage): plan retained manifest imports - #609

Merged
fishmingyu merged 3 commits into
mainfrom
feat/manifest-import-planner-v2
Aug 16, 2026
Merged

feat(storage): plan retained manifest imports#609
fishmingyu merged 3 commits into
mainfrom
feat/manifest-import-planner-v2

Conversation

@fishmingyu

@fishmingyu fishmingyu commented Aug 12, 2026

Copy link
Copy Markdown
Member

Summary

Add a pure, backend-neutral planner that converts bounded RepoManifest v1.1 input into immutable retained-import intent. This layer validates and selects portable artifacts, but deliberately does not read artifacts, write CAS objects, mutate the catalog, or publish refs.

Changes

  • Add immutable source, view, selection, and import-plan value types plus mapping and byte-oriented planning entry points.
  • Parse manifest JSON under strict UTF-8, ambiguity, size, depth, node, key, atom, integer, and selection bounds.
  • Admit only current BM25 schema 8 and vector schema 6 artifacts with complete generation and builder-profile identity; required views fail closed while optional views record explicit skip decisions.
  • Treat source fingerprint v2 as eligible input for later authentication and keep v1 plans inert; reject diagnostic and authority claims embedded anywhere in the manifest.
  • Charge remaining encoded-byte budgets before reading caller-owned mapping children, and expose the planner through the lazy compiler API.
  • Record the pure planning boundary while keeping retained reads, CAS upload, catalog mutation, and ref publication explicitly outstanding.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Refactoring
  • Performance improvement
  • Tests

Testing

  • Tests pass locally

  • Added new tests for the changes

  • pytest -q test/compiler/test_manifest_storage.py --tb=short: 256 passed.

  • pytest -q test/compiler test/storage/test_models.py --tb=short: 640 passed.

  • Local unit tier on the equivalent restacked patch: 5815 passed, 71 skipped, 191 deselected.

  • pre-commit run --from-ref origin/main --to-ref HEAD: passed.

  • python -m mkdocs build --strict: passed.

  • git diff --check origin/main...HEAD: passed.

Checklist

  • My code follows the project style guidelines
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

#606 is merged and its exact-head and post-merge gates are green. This PR is restacked directly on the resulting main. The planner is intentionally authority-free and does not enable a production retained-manifest importer.

@github-actions github-actions Bot added type/docs Documentation type/test Test-related scope/compiler Index compiler, manifest, build integration labels Aug 12, 2026
@fishmingyu
fishmingyu force-pushed the feat/retained-manifest-prerequisites-v2 branch from 3e3b413 to 92465c9 Compare August 16, 2026 00:07
Base automatically changed from feat/retained-manifest-prerequisites-v2 to main August 16, 2026 00:43
Add a pure bounded planner for current RepoManifest v1.1 BM25 and vector projections. The planner freezes source, selection, profile, and generation intent without filesystem, native parser, CAS, catalog, or environment access.

Required views fail closed, optional incompatibilities are explicit, and all public construction and JSON entry points enforce bounded canonical values before copying or traversal.

Verified with 255 focused tests and 615 tests across compiler plus storage models, along with Black, repository-profile isort, flake8, py_compile, and diff checks.
Record the pure RepoManifest planning boundary and keep retained artifact reads, CAS upload, snapshot publication, export, and M2 job publication explicitly outstanding.
Propagate the remaining encoded-byte budget through recursive manifest snapshots so object keys, separators, and minimum child size are charged before a caller-owned child mapping is consumed. Bound scalar and container values at each recursion level.

Verified with 256 focused planner tests and 616 tests across compiler plus storage models, including a hostile child mapping that remains unconsumed after the parent budget is exhausted.
@fishmingyu
fishmingyu force-pushed the feat/manifest-import-planner-v2 branch from d6fcc2a to c07df9a Compare August 16, 2026 00:44
@fishmingyu
fishmingyu marked this pull request as ready for review August 16, 2026 00:56
@fishmingyu
fishmingyu merged commit bb3ea3e into main Aug 16, 2026
23 checks passed
@fishmingyu
fishmingyu deleted the feat/manifest-import-planner-v2 branch August 16, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope/compiler Index compiler, manifest, build integration type/docs Documentation type/test Test-related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant