WIP: content-addressed code-image pool (phases 6-8)#688
Draft
FileSystemGuy wants to merge 46 commits into
Draft
Conversation
…name
RED-first per D-52/D-59 of Plan 06-01. Locks the 5 NEW helper contracts
before their production implementation lands in the GREEN commit:
- _POINTER_FILENAME (module-level constant)
- PointerMalformed (exception, subclass of CodeImageError)
- _write_pointer_atomic(run_leaf, full_hash, log)
- _read_pointer(run_leaf, log) -> (algorithm, full_hash)
- _pool_dir_name(full_hash) -> str
TestPointerWrite (6 tests) — locks writer contract:
- literal `md5-tree-v2:<32-hex>` output (SC#2, D-61)
- BaseException cleanup of tmp sibling (SC#5)
- stale tmp sibling cleanup before write
- dot-prefixed tmp sibling name (Pitfall 4)
- no trailing newline (SC#9)
- idempotency across repeated calls
TestPointerRead (10 tests) — locks reader contract + malformed variants
from RESEARCH Pitfall 3:
- happy-path round-trip via writer (SC#3)
- trailing whitespace + newline tolerated (SC#8)
- missing pointer file → FileNotFoundError
- empty tail, short hex, uppercase hex, no colon, unknown algorithm
→ PointerMalformed (SC#6, SC#7)
- every PointerMalformed message names the pointer path
- PointerMalformed IS-A CodeImageError
TestPoolDirName (2 tests) — locks `code-<first-8-hex>` shape (SC#4, D-62).
MockLogger is reused verbatim from
test_capture_or_verify_code_image.py:33-64 per the plan's read_first spec.
Collection currently fails with ImportError — the RED state. Task 2 (GREEN)
adds the production symbols to
mlpstorage_py/submission_checker/tools/code_image.py.
…hecker/tools/code_image.py
Introduces the 5 NEW helper symbols consumed by Plan 06-02's content-
addressed pool capture rewrite. No wiring into `capture_or_verify_code_image`
yet — this commit only turns Task 1's RED tests GREEN.
Symbols added to mlpstorage_py.submission_checker.tools.code_image:
- _POINTER_FILENAME = ".mlps-code-image"
Module-level constant. Leading dot keeps the sentinel invisible to a
naive `Path.glob("code-*")` pool scan (RESEARCH Pitfall 4).
- class PointerMalformed(CodeImageError)
Subclass of the existing CodeImageError so main.py's exit-code
mapping surfaces it as EXIT_CODE.CODE_IMAGE_ERROR without a new
handler. Distinct from MalformedHashFile (which covers .code-hash.json).
- _pool_dir_name(full_hash) -> str
Returns `code-<first-8-hex>` per D-62. No validation of its own so
callers can inline it into path assembly.
- _write_pointer_atomic(run_leaf, full_hash, log) -> None
Write-tmp + os.rename atomicity per D-65. `except BaseException`
(not `except Exception`) ensures KeyboardInterrupt / SystemExit
also trigger tmp cleanup. Pattern extracted verbatim from the
soon-to-be-deleted mlpstorage_py/results_dir/code_image.py:160-186.
Tmp sibling is dot-prefixed as `.mlps-code-image.tmp.<pid>`
(Pitfall 4).
- _read_pointer(run_leaf, log) -> tuple[str, str]
Splits on the first `:` via str.partition, validates the algorithm
prefix is exactly "md5-tree-v2", and validates the hex tail with
`re.fullmatch(r"[0-9a-f]{32}", …)` — the same regex the existing
_read_hash_file uses at :473. Rejects every malformed variant from
RESEARCH Pitfall 3 (empty tail, short hex, uppercase, no colon,
unknown algorithm). PointerMalformed always names the offending
pointer path in its message. Missing pointer file surfaces as
the built-in FileNotFoundError so callers can distinguish
"pool image not linked" from "pool image linked but corrupted".
Verification:
- pytest mlpstorage_py/tests/test_pointer_file.py -v → 18 passed
- pytest mlpstorage_py/tests/test_capture_or_verify_code_image.py
mlpstorage_py/tests/test_cli_code_image.py
mlpstorage_py/tests/test_code_image.py
mlpstorage_py/tests/test_main_code_image_wiring.py -v → 100 passed
- Full runnable-subset regression (2885 tests) passes with no new
failures. 8 pre-existing collection errors in tests/unit/ come from
missing numpy/pyarrow imports in the environment — verified identical
on the parent RED commit before this change.
Reuses the existing `_ALGORITHM = "md5-tree-v2"` constant at :152 instead
of introducing a new `_POINTER_PREFIX` — cross-verification for D-61 stays
in one place.
…plan - 06-01-SUMMARY.md: RED-first execution record (2 atomic commits + zero deviations), 10-entry coverage block (D1..D10), full RED/GREEN gate compliance, self-check PASSED. - STATE.md: advance current_plan → 1; add 4 decisions from the plan execution (algorithm-prefix reuse, FileNotFoundError not wrapped, no-trailing-newline writer, Q2 resolved); record session continuity pointing to Plan 06-02 as the next action. Requirements PTR-01, PTR-02, POOL-01, POOL-02 are marked complete for Phase 6.
RED phase for Plan 06-02 Task 1. Locks the D-63 refusal semantics:
capture_or_verify_code_image must raise LegacyLayoutDetected before any
pool writes when a legacy 'code/' directory exists under
<results_dir>/{closed,open}/<orgname>/.
Tests cover:
- CLOSED-mode and OPEN-mode legacy 'code/' detection
- Deep OPEN legacy tree caught via parent scan
- Multiple offenders: message names first + counts extras
- LegacyLayoutDetected is a CodeImageError subclass (main.py exit-code map)
- Message contains Phase 7 auto-migrate hint (D-63 spec text)
- No-legacy path: LegacyLayoutDetected not raised
- 'code-<hash8>' pool dirs are NOT flagged as legacy
Fails at collection with ImportError on LegacyLayoutDetected.
RED phase for Plan 06-02 Task 2. Locks CAPVER-01/02/03 + POOL-01/02/03/04 + PTR-01 + D-63/D-64/D-65/D-66/D-67 semantics for the rewritten capture_or_verify_code_image. Coverage: - Match branch: existing pool_dir reused; no duplicate capture - Two-run pointer atomicity: fresh leaf per run, same hash reference - No-match branch: fresh tree captures pool + pointer - Source change: second pool dir alongside first; no CodeImageError raised - CAPVER-03 direct: hash mismatch does not raise CodeImageError - UX-01: retired reject strings not emitted on hash mismatch - POOL-04: mode-agnostic dedup (closed<->open reuse pool) - POOL-03: per-org isolation (Acme vs Beta separate pool dirs) - PTR-01: pointer written after run leaf; no tmp sibling leaked - Pointer content: 'md5-tree-v2:<full32hex>' matches .code-hash.json.hash - D-66 loser branch: winner-hash-match → silent success - D-66 loser branch: winner-hash-mismatch → PoolCorruption raised - D-67: --skip-validation flag does not bypass capture - Gating: whatif/configview preserved - Path safety: orgname='..' raises ConfigurationError Fails at collection with ImportError on LegacyLayoutDetected/PoolCorruption.
…ject strings absent RED phase for Plan 06-02 Task 3. Locks UX-01 + D-63 requirement: - The CLOSED-mode reject string 'changes to the codebase are not allowed' must not appear in submission_checker/tools/code_image.py source. - The OPEN-mode sibling 'all runs of this type must use the same codebase' must not appear either. Both are the user-facing UX that CAPVER-03 + UX-01 retire. The negative-grep covers code AND comment lines (both retire together per D-63). Fails as expected: both strings are still present at OLD :150 (comment), :748 (CLOSED reject), :750 (OPEN reject).
…ed pool + pointer semantics; refuse legacy layout; retire reject strings; update Rules.md
GREEN phase for Plan 06-02 Task 4. Implements the semantic core of Phase 6:
9 of 10 phase REQ-IDs land here (CAPVER-01/02/03, POOL-01/02/03/04, PTR-01,
UX-01). Consumes the Plan 06-01 pointer + pool-dir-name helpers.
New symbols in submission_checker/tools/code_image.py:
- class LegacyLayoutDetected(CodeImageError) — D-63 refusal marker
- class PoolCorruption(CodeImageError) — D-66 loser-branch integrity marker
- _scan_legacy_layout(results_dir, orgname) — 2-syscall legacy code/ scan
- _find_matching_pool_image(org_root, live_hash, log) — glob + hash-file read;
catches MissingHashFile / MalformedHashFile at DEBUG
- _capture_new_pool_image(org_root, source_root, live_hash, log) — write-tmp
+ os.rename (D-66 first-writer-wins); .code-hash.json inside tmp BEFORE
rename so loser hits ENOTEMPTY (Pitfall 1); leading-dot tmp name so
pool scan glob does not pick it up mid-write (Pitfall 4)
capture_or_verify_code_image body rewrite (KEEP + REPLACE surgical map):
- KEEP the gating prelude verbatim (mode/command gating, orgname/systemname
validation, args._validated_* stash, results_dir existence gate IN-03)
- REPLACE the mode-branching image_parent computation and the
capture-vs-verify + reject-string block with:
1. Mode-agnostic org_root = results_dir/orgname (D-64)
2. _scan_legacy_layout → raise LegacyLayoutDetected on offender (D-63)
3. Hash live source; raise CodeTreeUnreadable on None
4. _find_matching_pool_image → reuse OR _capture_new_pool_image
5. Compute run leaf via generate_output_location (lightweight shim)
6. _write_pointer_atomic(run_leaf, live_hash, log) — PTR-01, D-65
7. Return pool_dir
Rules.md line 82 rewritten to describe the pool-and-pointer semantics
without quoting either retired reject string.
Retired strings deleted (UX-01):
- 'changes to the codebase are not allowed in a CLOSED run'
- 'all runs of this type must use the same codebase'
Tests turning GREEN:
- mlpstorage_py/tests/test_pointer_file.py (Plan 06-01 helpers still green)
- mlpstorage_py/tests/test_capture_or_verify_pool.py (18 tests)
- mlpstorage_py/tests/test_legacy_layout_refuse.py (8 tests)
- mlpstorage_py/tests/test_ux01_reject_string_retired.py (2 tests)
18 pre-existing tests in test_capture_or_verify_code_image.py and
test_cli_code_image.py FAIL as expected — they lock the retired CAPVER-03
mismatch-raise behavior and legacy code/ layout paths. Plan 06-02 Task 5
retires those tests.
…UX-01 Retires tests that locked the pre-Phase-6 legacy behavior: - test_capture_or_verify_code_image.py: TestCapturePath + TestVerifyPath (legacy code/ tree-shape assertions and hash-mismatch-raise assertions) - test_cli_code_image.py: TestClosedFirstCapture, TestOpenFirstCapture, TestRuntimeMatchPasses, TestRuntimeMismatchCLOSED, TestRuntimeMismatchOPEN, TestBadImageRecovery (same categories) Retirement rationale documented per file: - CAPVER-03: hash mismatch no longer raises CodeImageError; it captures a new pool image alongside the existing one. - UX-01: the retired reject strings no longer appear anywhere in the module. - D-63: legacy code/ layouts are now refused via LegacyLayoutDetected before any capture writes; Phase 7 owns the migration. The D-21 'delete code/ and re-run' recovery workflow no longer applies. - D-64: pool images are mode-agnostic under <results_dir>/<orgname>/, so the per-mode leaf assertions no longer hold. Replacement coverage lives in mlpstorage_py/tests/test_capture_or_verify_pool.py under class TestCaptureOrVerifyPool (18 tests). Surviving classes in the edited files still cover gating (D-10), env-var validation (D-04, D-05), and the T-02-02-05 path-traversal guard (REVIEWS.md consensus). pytest mlpstorage_py/tests -v: 839 passed, 0 failed. pytest tests/unit -v (with 8 pre-existing collection failures ignored, per 06-RESEARCH.md ## Validation Architecture): 2056 passed.
- 06-02-SUMMARY.md: 5-task execution record (RED×3 + GREEN + cleanup), zero deviations, 5-entry coverage block (D1..D5), full retired-tests inventory with rationale, Rules.md line-82 rewrite before/after, self-check PASSED. - .planning/STATE.md: advance current_plan 1 → 2, completed_plans 1 → 2 (v1.1 milestone 17% complete), stopped_at + last_activity_desc updated. Plan 06-02 delivers 9 of 10 phase REQ-IDs (CAPVER-01/02/03, POOL-01..04, PTR-01, UX-01); D-60's legacy retire is Plan 06-03.
… (D-60) Deletes the second (dead) code-image capture path so only one path remains: submission_checker/tools/code_image.py::capture_or_verify_code_image, called once from main.py:224 BEFORE Benchmark construction. Two coexisting capture paths was a stale artifact of an incremental rollout; Plan 06-02 rewrote the surviving one, and this plan removes the dead one. - Delete mlpstorage_py/results_dir/code_image.py (190 lines). - Remove capture_code_image re-export from mlpstorage_py/results_dir/__init__.py. - Delete the self.code_image_path init block from mlpstorage_py/benchmarks/base.py and update the surrounding LAY-06 comment to reference the surviving main.py capture site. self.code_image_path attribute no longer exists. - Delete the autouse _suppress_capture_code_image fixture from tests/unit/conftest.py — the module it patched no longer exists. - Delete tests/unit/test_code_image.py (511 lines): every test targeted the retired module; the surviving archival copy engine is covered by mlpstorage_py/tests/test_code_image.py. - Update a stale line-number reference in submission_checker/tools/code_image.py's docstring so no dangling string reference to the retired path remains anywhere in the tree. Grep gates (all zero): from mlpstorage_py.results_dir.code_image; mlpstorage_py.results_dir.code_image; self.code_image_path; _suppress_capture_code_image. Refs: 06-03-PLAN.md; D-60; RESEARCH Assumption A6.
…d.py for pool layout
Rewrites the three Category B methods that exercised the legacy
`<rd>/{closed,open}/<orgname>/code/` per-mode copy path (retired in
Task 1 alongside `mlpstorage_py.results_dir.code_image`). They now
target the surviving content-addressed pool at
`<rd>/<orgname>/code-<hash8>/` via `capture_or_verify_code_image`
(Phase 6 D-63..D-67, POOL-01, PTR-01, UX-01).
Preserves Category A methods unchanged:
- TestDirectoryCheckRegression (LAY-08)
- TestUninitializedErrorMessage (LAY-03)
- TestInitThenRunFullCliDispatch (HARDEN-03)
Rewritten methods (smoke-level; exhaustive pool-layout coverage is
Plan 06-04's ROADMAP scope):
- TestInitThenRunLayout::test_init_then_run_closed — asserts pool
image at <rd>/<org>/code-<hash8>/ with a valid .code-hash.json
sidecar whose first-8-hex prefix matches the pool dir name (D-62).
- TestWhatifLayoutShape::test_whatif_path_shape — asserts the D-10
mode gate returns None for whatif and no code-* pool dir is
created under <rd>/<org>/.
- TestOpenLayoutShape::test_open_path_shape — asserts OPEN honors
the same mode-agnostic pool (D-64) and that a second OPEN capture
with the same source hash reuses the same pool image (CAPVER-01,
POOL-04). No methods were skipped.
Grep gates (both zero): `from mlpstorage_py.results_dir import
capture_code_image`; `capture_code_image` (retired symbol).
Refs: 06-03-PLAN.md; 06-04-PLAN.md; D-60.
…etire Plan 06-03 landed in two atomic task commits: - refactor(06-03) 8c010c4: retire legacy results_dir/code_image.py + its callers/tests/fixture - test(06-03) 7b0c07f: rewrite tests/integration/test_canonical_layout_end_to_end.py for pool layout Codebase now has exactly one code-image capture entrypoint: mlpstorage_py.submission_checker.tools.code_image::capture_or_verify_code_image, called once from mlpstorage_py/main.py:224 BEFORE Benchmark construction. STATE.md advances current_plan 2 → 3; UX-01 marked complete. No deviations.
…-1) + D-63 refuse
Task 1 of Plan 06-04. Adds shared `tests/integration/conftest.py` fixture
(MockLogger, capture_args_factory, fake_source_root, init_results_dir,
pool_dirs helper) reused across the six pool-layout integration test
files planned for 06-04.
Two test classes in the new file:
- TestFreshTreePoolCapture:
- test_fresh_tree_writes_pool_and_pointer — asserts POOL-01 (one
code-<hash8>/ under <rd>/<orgname>/), POOL-02 (dir name matches
hash prefix), D-64 (mode-agnostic), PTR-01 + D-61 (pointer content
md5-tree-v2:<full-hash>).
- test_fresh_tree_hash_json_schema_fields_all_present — POOL-02 +
D-07 schema fields all present in .code-hash.json.
- TestFreshTreeLegacyRefuse (SC#11 / D-63):
- test_fresh_tree_refuses_when_closed_legacy_code_present —
LegacyLayoutDetected raised for <rd>/closed/Acme/code/ before any
pool write.
- test_fresh_tree_refuses_when_open_legacy_code_present — symmetric
check for the OPEN-mode legacy layout.
Runtime: 0.07s for all 4 tests.
…image + UX-01 negative-grep (SC-3)
gsd-verifier goal-backward audit of the content-addressed pool + capture-or-verify rewrite. All 10 REQ-IDs (POOL-01..04, PTR-01..02, CAPVER-01..03, UX-01) verified against codebase evidence. Structural gates D-60 (legacy retire), D-63 (LegacyLayoutDetected), D-65 (except BaseException), D-66 (first-writer-wins) all verified. Test evidence: - pytest mlpstorage_py/tests -v --tb=short → 839 passed, 0 failed - pytest tests/integration/test_pool_*.py -v → 15 passed - Pre-existing collection errors unchanged from branch base 76edfd3. No BLOCKER or WARNING findings.
10/10 REQ-IDs delivered per gsd-verifier goal-backward audit. Ready for /gsd-transition to phase 7.
- Add legacy_tree_factory fixture supporting orgname/mode/n_run_leaves/hand_edit/benchmark_shape params - Supports training (5-level), checkpointing (4-level), and vector_database (6-level) run-leaf shapes - hand_edit=True mutates file1.py after .code-hash.json write to simulate D-73 tamper detection - Stamps .code-hash.json via compute_code_tree_md5 matching _read_hash_file schema - Validates mode and benchmark_shape with ValueError on unknown values
…enarios - test_migration_flow.py: 8 stubs covering MIG-01 happy-path, dedup, D-74 log output, all 3 benchmark shapes, empty/fresh tree edge cases - test_migration_hand_edit.py: 6 stubs covering MIG-03 hand-edit abort + MissingHashFile/MalformedHashFile edge cases - test_migration_multi_org.py: 2 stubs covering D-70 per-org isolation - test_migration_idempotency.py: 6 stubs covering MIG-02a sentinel short-circuit + MIG-02b 4-checkpoint crash-safety - All 22 stubs marked xfail(strict=True, raises=NotImplementedError) - No production symbol imports (Wave-2 planners add these)
- tests/unit/test_legacy_migration.py: 9 stubs in 6 classes covering _verify_all_legacy_dirs, _write_sentinel_atomic, _read_sentinel, _enumerate_run_leaves (all 3 shapes), _check_and_migrate_legacy_layout, HandEditedCodeImage subclass chain - mlpstorage_py/tests/test_legacy_migration_source.py: 6 structural (grep-testable) stubs covering fixed step order, two-pass separation, no-try/except-around-pass-1, sentinel atomic write, HandEditedCodeImage inheritance, exactly-two-log.status invariants - All 15 stubs marked xfail(strict=True, raises=NotImplementedError) - No production symbol imports in either file
…uctural test (D-73) - Add HandEditedCodeImage(CodeImageError) after LegacyLayoutDetected, before PoolCorruption - Class docstring anchors D-73: aborts migration before any writes; handles missing/malformed .code-hash.json via MissingHashFile/MalformedHashFile chain - Promote test_HandEditedCodeImage_subclasses_CodeImageError from xfail to real assertion: importability, issubclass check, str round-trip
…or (D-70..D-74) - VerifiedLegacyImage frozen dataclass: pass-1 to pass-2 hand-off (D-73) - _verify_all_legacy_dirs: re-hashes all legacy code/ dirs; raises HandEditedCodeImage on mismatch before any writes (D-73 two-pass) - _materialize_pool_images: dedup-aware pool materialization via _capture_new_pool_image (D-71 step 1) - _write_pointers_for_migrated_leaves: bounded glob run-leaf enumerator writes .mlps-code-image in each leaf (D-71 step 2) - _delete_legacy_dirs: idempotent rmtree with FileNotFoundError swallow (D-71 step 3) - _write_sentinel_atomic: write-tmp + os.rename D-65 pattern (D-71 step 4, D-72 format) - _read_sentinel: forward-compatible key=value parser - _enumerate_run_leaves: bounded 4/5/6-level globs for training/ checkpointing/vector_database shapes (no os.walk, no recursive glob) - migrate_legacy_layout: entry point with strict two-pass, fixed step order, no try/except around pass-1, exactly two log.status() calls (D-74) - _check_and_migrate_legacy_layout: pre-check helper mirroring capture_or_verify_code_image signature (args, env, log); sentinel fast-path; fresh-tree A3b: no sentinel written on no-legacy-dirs trees
…test - Add import of _check_and_migrate_legacy_layout from legacy_migration - Call pre-check immediately before capture_or_verify_code_image in run_benchmark - Both calls inside same progress_context block (single spinner per RESEARCH §7) - No try/except LegacyLayoutDetected (D-70 explicit pre-check pattern) - Add test_main_precheck.py: 4 structural assertions for the D-70 wiring - Phase 6 wiring test (test_main_code_image_wiring.py) still passes (9/9)
… passing) - test_legacy_migration_source.py: replace 5 xfail stubs with real source-grep assertions (D-71 fixed step order, D-73 two-pass separation, D-73 no-try/except, D-65 atomic sentinel, D-74 exactly-two-log.status); 6/6 pass - test_legacy_migration.py: replace 9 xfail stubs with real unit tests covering _verify_all_legacy_dirs, _write_sentinel_atomic, _read_sentinel, _enumerate_run_leaves (all 3 shapes), _check_and_migrate_legacy_layout (sentinel fast-path + non-submission no-op), HandEditedCodeImage subclass; 9/9 pass - No @pytest.mark.xfail decorators or NotImplementedError bodies remain
…tests - Replace 8 xfail stubs in test_migration_flow.py with real assertions - TestMigrateEndToEnd: canonical happy path (pool + 3 pointers + sentinel + D-74 exactly-2-status), dedup (N=2 legacy -> M=1 pool), exactly-2-status lines - TestMigrateBenchmarkShapes: training (5-level), checkpointing (4-level), vector_database (6-level) shapes all produce per-leaf pointers - TestMigrateEmptyRunLeaves: n_run_leaves=0 still writes sentinel; fresh-tree no-legacy returns without writing sentinel (A3b) - All 8 tests pass; no xfail/xpassed/skip
…on tests - test_migration_hand_edit.py: 6 tests covering D-73 hand-edit abort (raises HandEditedCodeImage, no sentinel, tree byte-identical, no pool images) plus missing .code-hash.json and malformed .code-hash.json edge cases - test_migration_multi_org.py: 2 tests for D-70 per-org isolation (migrate Acme leaves Bravo untouched; migrate both independently produces separate sentinels + separate pool dirs in each org root) - All 8 tests pass; no xfail/xpassed/skip
…passing - TestSentinelShortCircuit (2 tests): sentinel fast-path spy confirms _scan_legacy_layout call_count==0 on second invocation; all log accumulators remain empty after sentinel short-circuit - TestCrashResume (4 tests): monkeypatch fault injection at each D-71 checkpoint — step1→step2, step2→step3, step3→step4, mid-step2 partial pointer write; each test verifies per-checkpoint partial state then monkeypatch.undo() + re-invoke to full converged state - _AbortAtStep module-level sentinel exception class - stateful closure for mid-step-2 partial pointer test capturing orig _write_pointer_atomic before patching - 0 xfail decorators, 0 NotImplementedError, 4 explicit monkeypatch.undo() - Full Phase 7 suite: 41 passing (>= 38 required); Phase 6 baseline: 15 passed
…ase complete Co-Authored-By: Curtis Anderson <curtis.anderson@hammerspace.com>
- New pool_structure_checks.py: PoolStructureCheck(BaseCheck) with four @rule-decorated methods for v1.1 pool layout verification - CHECK-01 (poolPointerResolution): validates .mlps-code-image pointers in every datetime run-leaf resolve to an existing pool image; D-84 one-error- per-org if .mlps-image-pool sentinel is absent - CHECK-02 (poolImageSelfConsistency): calls verify_image_self_consistent on every code-<hash8>/ subdirectory under each org pool root - CHECK-03 (poolOrphanCheck): detects pool images not referenced by any run leaf across closed/ and open/ (D-92 cross-division union) - CHECK-04 (poolLegacyCheck): D-81 migrate-first message for legacy code/ dirs; D-91 partial-migration failure; D-90 sentinel-but-empty advisory warning
…l layout
- Remove code_directory_contents_check (STRUCT-06) from init_checks and delete
the method body entirely (D-80: v1.1-only checker, STRUCT-06 retired)
- Remove now-unused imports: compute_code_tree_md5, verify_image_self_consistent,
CodeImageError, MissingHashFile, MalformedHashFile
- Update _REQUIRED_SUBMITTER_SUBDIRS_CLOSED from {code,results,systems} to
{results,systems} matching v1.1 layout where code/ is in org-level pool
- Update top_level_subdirectories_check (D-83/D-85): dirs containing
.mlps-image-pool sentinel are recognized pool roots and skip violation
…-checksum (D-82, D-88) BREAKING CHANGE: --reference-checksum CLI flag removed from mlpstorage validate. Per-image REFERENCE_CHECKSUMS lookup (D-86) is now the only supported path. Any CI script using --reference-checksum must be updated. - main.py: import PoolStructureCheck; instantiate as pre-loop check after SystemYamlSchemaCheck (D-82); remove --reference-checksum argument from get_args(); drop reference_checksum_override= from Config() call - configuration.py: remove reference_checksum_override param from Config.__init__; remove get_reference_checksum() method entirely (D-88); REFERENCE_CHECKSUMS constant retained for Plan 08-02 per-image lookup - training_checks.py: stub closed_submission_checksum to return True with TODO(Phase8-Plan2) comment; Plan 08-02 replaces with D-89 pool-image flow - vdb_checks.py: stub vdb_closed_submission_checksum identically
- Imports _read_pointer, _pool_dir_name, _read_hash_file, PointerMalformed from code_image (alongside existing MissingHashFile/MalformedHashFile) - Adds resolve_run_pool_image(run_leaf, results_dir, orgname, log) as a module-level function after _check_code_image_layered - Implements D-89 steps 1-4: pointer read → pool path assembly → hash-file read - Propagates FileNotFoundError, PointerMalformed, MissingHashFile, MalformedHashFile to caller (CHECK-05 flow; callers own error logging) - Shared by TrainingCheck and VdbCheck to avoid duplication (Plan 08-02)
…ecks Replace return True stubs in closed_submission_checksum (3.6.1) and vdb_closed_submission_checksum (5.6.1) with the full six-step D-89 flow: 1. For each run leaf, call resolve_run_pool_image to read .mlps-code-image and resolve to pool image path + .code-hash.json hash_data 2. catch FileNotFoundError → log_violation (CHECK-01 style, D-93) catch PointerMalformed / MissingHashFile / MalformedHashFile → log_violation 3. call verify_image_self_consistent on pool image (self-consistency check) 4. read mlpstorage_version from hash_data; look up REFERENCE_CHECKSUMS 5. if expected is None: warn once per pool image via _warned_pool_images set (D-87) 6. if expected is not None: compute_code_tree_md5 and compare; mismatch → log_violation Walk-up depths: training parents[5/7], vdb parents[6/8] from datetime leaf. Both files add Path, resolve_run_pool_image, verify_image_self_consistent, PointerMalformed, MissingHashFile, MalformedHashFile, compute_code_tree_md5, REFERENCE_CHECKSUMS imports.
…CK-02 dir-name check - Add tests/unit/test_submission_checker_pool_structure.py (13 tests): TestPoolPointerResolutionCheck: missing/dangling/valid pointer (CHECK-01) TestPoolImageSelfConsistencyCheck: pass/modified/renamed (CHECK-02) TestPoolOrphanCheck: orphan/all-referenced (CHECK-03) TestPoolLegacyCheck: legacy-code/, partial-migration, D-90 empty pool (CHECK-04) TestTopLevelSubdirPoolRoot: sentinel-recognized/unexpected-dir (D-83/D-85) - Rule 1 bug fix in pool_structure_checks.py: pool_image_self_consistency_check was missing the directory-name-vs-hash check (Part 1 of CHECK-02 spec). Added dir name verification before content self-consistency check so that a renamed pool dir (different hash8 suffix) is correctly caught by CHECK-02.
- Add tests/integration/test_submission_checker_pool_v11.py (8 tests): SC-1: valid v1.1 tree → run() returns 0 SC-2a: missing .mlps-code-image pointer → run() returns 1 SC-2b: dangling pointer (non-existent pool hash) → run() returns 1 SC-3a: pool image content modified → run() returns 1 (CHECK-02) SC-3b: pool dir renamed to wrong hash8 → run() returns 1 (CHECK-02) SC-4a: unreferenced pool image (orphan) → run() returns 1 (CHECK-03) SC-4b: legacy code/ directory → run() returns 1 (CHECK-04) SC-5: two-version two-pool-image tree with patched REFERENCE_CHECKSUMS → run() returns 0 Non-pool checks (SubmissionStructureCheck, SystemYamlSchemaCheck, DirectoryCheck) are monkeypatched to no-op so tests focus on pool behavior. _PoolAwareTrainingCheck runs only closed_submission_checksum (CHECK-05) for SC-5.
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
These files are gitignored and belong only in local dev context.
# Conflicts: # mlpstorage_py/submission_checker/checks/training_checks.py # mlpstorage_py/submission_checker/checks/vdb_checks.py # tests/integration/test_submission_checker_pool_v11.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
This branch implements a content-addressed code-image pool for MLPerf Storage submissions. Instead of capturing a copy of the submitter's source tree inside each run result directory, a single pool image is stored at
<results_dir>/<orgname>/code-<hash8>/and each run leaf carries a pointer file (.mlps-code-image) referencing it by hash. Duplicate captures of the same code tree are deduplicated automatically.What's in this branch (phases 6–8)
Known open question — pool location
The pool currently lives at
<results_dir>/<orgname>/code-<hash8>/(mode-agnostic, per design decision D-64). Run results live at<results_dir>/<mode>/<orgname>/results/.... This asymmetry — orgname appearing at the top level for the pool, but under mode for runs — is under discussion. Two alternatives are being evaluated before this branch is finished:<results_dir>/<mode>/<orgname>/(aligned with runs, two separate pools per org)<results_dir>/<orgname>/<mode>/...(orgname becomes the top-level unit, pool location unchanged)No code changes will be made on this topic until a decision is reached.
Integration work ahead
Main has diverged significantly since this branch was started. A rebase/merge integration pass is required before this can be considered for merge.
Test plan
pytest tests/unit -v— all unit tests pass on this branchpytest tests/integration -v— integration tests pass on this branch