From 196ee28559191b7fda1034936116c941379ec98d Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sat, 11 Jul 2026 18:01:52 +0900 Subject: [PATCH 01/11] spec: self-improve-checkpoint-resume Phase 1a/1b (behavioral-spec + verification-architecture) --- .vcsdd/active-feature.txt | 2 +- .../specs/behavioral-spec.md | 241 ++++++++++++++++++ .../specs/verification-architecture.md | 102 ++++++++ .../self-improve-checkpoint-resume/state.json | 31 +++ .vcsdd/history.jsonl | 3 + .vcsdd/index.json | 10 +- 6 files changed, 387 insertions(+), 2 deletions(-) create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/state.json diff --git a/.vcsdd/active-feature.txt b/.vcsdd/active-feature.txt index c51e0989..964ee8c5 100644 --- a/.vcsdd/active-feature.txt +++ b/.vcsdd/active-feature.txt @@ -1 +1 @@ -franklin-alwaysact-skill-router +self-improve-checkpoint-resume diff --git a/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md b/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md new file mode 100644 index 00000000..d553340f --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md @@ -0,0 +1,241 @@ +--- +feature: self-improve-checkpoint-resume +phase: 1a/1b +mode: lean +sources: + - skills/earn/self-improve/run_evolve.sh (read directly, this worktree) — the launchd-triggered + recurring cycle this feature edits; confirms today's `openevolve-run` invocation passes a fresh + `--output "$RUN_DIR"` every cycle and NEVER passes `--checkpoint` + - `openevolve-run --help` (run live in this worktree's venv, `~/.anicca-venvs/self-improve/bin/ + openevolve-run`) — confirms the real, installed `openevolve==0.3.0` CLI flag is `--checkpoint + CHECKPOINT Path to checkpoint directory to resume from (e.g., + openevolve_output/checkpoints/checkpoint_50)` — this feature adds exactly this flag, no new + mechanism invented + - skills/earn/self-improve/config.yaml line 11 (read directly) — `checkpoint_interval: 10 # + top-level Config field (NOT database.checkpoint_interval)` and line 128's own comment "a + rate-limit-interrupted run can be resumed with `--checkpoint`" — confirms checkpoint resume was + already an anticipated, designed-for capability of this harness, just never wired into + `run_evolve.sh`'s automatic recurring invocation + - direct log inspection this session (real `runs/*/assessment.json` files under this worktree's + launchd-produced production body, read by the parent agent-economy loop, not hypothesized): + three consecutive ~6h cycles (`run-20260710T190919Z`, `run-20260711T011638Z`, + `run-20260711T072156Z`) each produced a best-candidate `combined_score` numerically IDENTICAL + to `baseline_stage2_score` (e.g. `4.015488840463803` to many decimal places) — `stage2_pass` + was `False` (a tie never counts as "beats baseline") every time, so zero candidates reached the + fresh-adversary review step for 18+ hours despite the loop firing, iterating, and logging + normally + - .vcsdd/features/anicca-self-improve-harness/specs/behavioral-spec.md (same repo, prior phase) + — REQ-OE1/OE5/OE6/OE7, INV-1..8 — this feature extends REQ-OE7's recurring-trigger wiring and + MUST NOT contradict any of them + - .vcsdd/features/self-improve-real-ledger/specs/behavioral-spec.md (same repo, prior phase) — + REQ-RL group, INV-RL1..6, EDGE-RL5a — this feature's own EDGE-CR1 mirrors EDGE-RL5a's + already-established "runs/ and state/ are gitignored, dev worktrees have neither" pattern + (confirmed live: this worktree's own `skills/earn/self-improve/.gitignore` contains `runs/` and + `state/`) + - skills/earn/self-improve/lib/ledger_reader.py's own module docstring (read directly) — this + repo's established convention for a pure-Python core called from bash via `"$PY_BIN" "$SKILL_DIR/ + lib/.py"`, the exact pattern this feature's own `lib/checkpoint_resume.py` follows for + its shell call site +integration: + extends: + - anicca-self-improve-harness (REQ-OE7's recurring-trigger wiring gains one new pre-invocation + step; REQ-OE1/OE5/OE6, INV-1..8 all stay in force, none weakened) + does_not_replace_or_modify: + - openevolve's own checkpoint internals (`openevolve/controller.py`'s checkpoint save/load + code) — this feature only ever passes an EXISTING CLI flag pointing at a path openevolve + itself already knows how to read; it never reads or writes checkpoint file contents + - config.yaml's `checkpoint_interval`/`database` settings (unchanged) + - promote_gate.sh / lib/promote_gate.py / lib/promote.py (the promotion-gate step downstream of + openevolve-run is untouched; this feature only affects what population openevolve STARTS + from, never what gets promoted or how) + - strategies/pm_backtest_strategy.py's EVOLVE-BLOCK content or fixed region + - any live-money path, wallet key, `.env`, ledger file, spend cap, or the launchd plist itself + (`ai.anicca.self-improve-evolve.plist` is not edited by this feature — it already invokes + `run_evolve.sh` with no arguments this feature needs to change) + out_of_scope_this_phase: + - Any change to `--iterations`/`SELF_IMPROVE_ITERATIONS` (REQ-OE5's per-run budget is + unaffected; this feature makes MORE of each run's iterations count toward NEW exploration by + not re-deriving the same starting population, but does not itself raise the per-run cap) + - Cross-run checkpoint garbage collection / retention policy for `runs/*/checkpoints/` (old + checkpoint directories accumulate on disk exactly as they do today; a future feature may + prune them — not specified here) +--- + +# Behavioral Specification — self-improve-checkpoint-resume (Phase 1a) + +## Purpose + +`run_evolve.sh` fires on a recurring ~6h launchd timer (REQ-OE7). Every firing creates a brand-new +`runs/run-/` directory and invokes `openevolve-run` with `initial_program = +strategies/pm_backtest_strategy.py` (the single committed baseline) and NO `--checkpoint` flag. +`openevolve-run` itself already checkpoints its MAP-Elites population under `/checkpoints/ +checkpoint_/` every `checkpoint_interval` iterations (config.yaml line 11) — but because the +next cycle never points at ANY prior run's checkpoint, that population is discarded at the end of +every cycle. Each cycle re-starts from a population of exactly one program (the seed) and gets only +`ITERATIONS` (today 20-40) total iterations to both rediscover and try to beat the same +already-known-optimum baseline from a cold start — which is the directly-observed, confirmed-live +root cause of three consecutive ~6h cycles producing a tied (never-improving) `combined_score` and +zero forward progress for 18+ hours (see sources above). + +This feature adds ONE pre-invocation step to `run_evolve.sh`: before calling `openevolve-run`, look +for the most recent PRIOR run's highest-numbered checkpoint and pass it via `--checkpoint ` — +an existing, already-installed, already-documented `openevolve-run` CLI flag, not a new mechanism. +When no prior checkpoint exists (first-ever run, or every prior run crashed before +`checkpoint_interval`), the command runs exactly as it does today — this feature strictly ADDS an +optional flag, it never removes or alters the existing fallback path. + +This is a harness-only change: no strategy file, promotion gate, live-money path, or launchd plist +is touched. + +## Purity Boundary Analysis + +- **Pure core (this feature's entire testable surface):** `lib/checkpoint_resume.py:: + find_latest_checkpoint(runs_dir: str, current_run_id: str) -> Optional[str]`. Given a `runs_dir` + path and the CURRENT run's own ID (so it never picks itself), it decides which checkpoint path + (if any) to resume from. It is deterministic path/existence logic ONLY: `os.listdir`, + `os.path.isdir`, string parsing/sorting — no subprocess, no network, no mutation of anything on + disk, no reading of ANY file's byte contents (it never opens/parses a checkpoint's internal + files — see REQ-CR5). Same input directory tree → same output, every time, in-process. +- **Effectful shell:** `run_evolve.sh`'s existing bash — it already creates `$RUN_DIR`, invokes + `openevolve-run`, and appends to `$LOG` (REQ-OE6/OE7, unchanged conventions). This feature adds + exactly one new effectful step to that shell script: shell out to the pure function (via + `"$PY_BIN" -c` or a thin CLI entrypoint, mirroring `lib/ledger_reader.py`'s existing + `"$PY_BIN" "$SKILL_DIR/lib/.py"` call convention) BEFORE the `openevolve-run` invocation, + capture its stdout (a path or empty string), log the outcome, and conditionally append + `--checkpoint ""` to the existing command array. `openevolve-run` itself (a separate + process, already effectful, unchanged by this feature) is what actually reads the checkpoint + directory's contents and resumes from it — that read is entirely outside this feature's own + code. + +## EARS-Format Functional Requirements + +### Group CR-FIND — pure checkpoint discovery + +- **REQ-CR1** `lib/checkpoint_resume.py` SHALL expose a PURE function `find_latest_checkpoint( + runs_dir: str, current_run_id: str) -> Optional[str]` that returns the absolute path to a + checkpoint directory to resume from, or `None` when no prior run has any usable checkpoint. + +- **REQ-CR2** WHEN selecting which PRIOR RUN to resume from, THE SYSTEM SHALL: list the immediate + subdirectories of `runs_dir`, EXCLUDE any entry equal to `current_run_id`, sort the REMAINING + candidate names lexicographically DESCENDING (run directory names are `run-YYYYMMDDTHHMMSSZ` UTC + timestamps, which sort correctly as plain strings — REQ-OE7's existing naming convention, + unchanged), and select the FIRST (most recent) candidate that has a non-empty `checkpoints/` + subdirectory (per REQ-CR3's definition of "has a checkpoint"). Runs with no `checkpoints/` + subdirectory, or an EMPTY one, SHALL be skipped over (not selected, but also not treated as a + fatal error) in favor of the next-most-recent candidate. + +- **REQ-CR3** WITHIN the selected prior run's `checkpoints/` directory, THE SYSTEM SHALL consider + only entries whose name matches the literal pattern `checkpoint_` where `` is one or more + ASCII digits, parse `` as an INTEGER (not a string), and select the entry with the LARGEST + integer `` (e.g. `checkpoint_100` SHALL be selected over `checkpoint_20` — NEVER a lexicographic + comparison, which would incorrectly rank `checkpoint_20` above `checkpoint_100`). Any entry inside + `checkpoints/` that does NOT match this pattern (a stray file, a differently-named directory, a + hidden dotfile) SHALL be ignored — its presence SHALL NEVER raise an exception or abort the scan. + +- **REQ-CR4** WHEN `runs_dir` does not exist at all, is empty, or contains ONLY the excluded + `current_run_id` entry, THE SYSTEM SHALL return `None` — this is the ordinary, expected "first-ever + run" case, not an error condition. + +- **REQ-CR5** `find_latest_checkpoint` SHALL perform PATH/EXISTENCE logic only (directory listing, + name pattern matching, integer comparison). It SHALL NEVER open, read, or validate the byte + contents of any file inside a `checkpoint_` directory (no checking for openevolve's own + expected internal files, no deserializing any checkpoint state). Whether a selected checkpoint + directory is internally complete/valid enough for `openevolve-run` to actually resume from is + ENTIRELY `openevolve-run`'s own concern, at a LATER, separate point in the pipeline (REQ-CR7) — + this function's contract ends at "a directory matching the expected shape and naming convention + exists at this path." + +### Group CR-WIRE — wiring into run_evolve.sh + +- **REQ-CR6** `run_evolve.sh` SHALL invoke the REQ-CR1 function, via `"$PY_BIN"`, AFTER `$RUN_DIR` + is created (so `current_run_id` = the just-created `$RUN_ID`, correctly excluding itself even + though its own now-empty directory already exists under `runs/`) and BEFORE the existing + `"$OPENEVOLVE_BIN"` invocation. + +- **REQ-CR7** WHEN the REQ-CR6 call returns a non-empty checkpoint path, THE SYSTEM SHALL append + exactly one additional argument pair, `--checkpoint ""`, to the EXISTING `openevolve-run` + invocation (unchanged otherwise: same `initial_program`, `evaluation_file`, `--config`, + `--iterations`, `--output` arguments, in the same order) — `openevolve-run`'s OWN internal + behavior when given a `--checkpoint` path (including how it handles an internally + incomplete/corrupt one) is entirely `openevolve-run`'s concern; if it fails or exits non-zero for + that reason, that failure SHALL surface through the EXISTING REQ-OE6 "crashed run is inconclusive" + handling in `run_evolve.sh` — this feature introduces NO new failure-handling branch for that + case. + +- **REQ-CR8** WHEN the REQ-CR6 call returns empty/`None` (no prior checkpoint found, for ANY reason + — first-ever run, all prior runs lack a `checkpoints/` dir, or `runs_dir` itself is missing), THE + SYSTEM SHALL invoke `openevolve-run` with EXACTLY the same argument list it uses today (no + `--checkpoint` flag at all) — i.e. this feature's fallback path is byte-for-byte the pre-existing + behavior, never a degraded or different variant of it. + +- **REQ-CR9** THE SYSTEM SHALL append a log line to `$LOG` (the same file/convention every other + `run_evolve.sh` step already writes to) stating EITHER the checkpoint path selected and used, OR + that none was found — this decision SHALL NEVER be silent. This restates this repo's existing + "loud, visible logging, never a silent branch" convention (every other step in `run_evolve.sh` + already follows it; REQ-CR9 makes it explicit for this feature's own new step). + +## Global Invariants + +| # | Invariant | +|---|---| +| INV-CR1 | `find_latest_checkpoint` NEVER writes, deletes, renames, or mutates anything on disk — it is a pure read-only path/existence query (REQ-CR5) | +| INV-CR2 | This feature never touches `strategies/pm_backtest_strategy.py`, `promote_gate.sh`, `lib/promote_gate.py`, `lib/promote.py`, `config.yaml`, or `ai.anicca.self-improve-evolve.plist` | +| INV-CR3 | This feature never reads or writes any wallet key, `.env`, ledger file, or spend-cap value — it is scoped exclusively to `runs/*/checkpoints/*` path selection | +| INV-CR4 | The fallback path (no checkpoint found) is byte-for-byte identical, in its `openevolve-run` invocation, to `run_evolve.sh`'s pre-feature behavior (REQ-CR8) — this feature can only ADD a `--checkpoint` argument, never change any other existing argument | +| INV-CR5 | REQ-OE6's "a crashed/killed/timed-out openevolve run is inconclusive, no candidate promoted, baseline untouched" invariant is preserved unchanged — this feature adds no new promotion path and does not weaken that handling for a `--checkpoint`-resumed run | + +## Edge Cases + +- **EDGE-CR1** (mirrors the prior phase's already-established EDGE-RL5a pattern) `runs/` is + gitignored (confirmed: this worktree's own `skills/earn/self-improve/.gitignore` contains + `runs/`) — a fresh dev worktree checkout has NO `runs/` directory at all. Tests for + `find_latest_checkpoint` MUST construct a synthetic `runs_dir` tree under `tmp_path` (pytest + fixture), never rely on or read this worktree's own (nonexistent) `runs/` directory, and MUST + NEVER symlink or copy any real production `runs/` directory into a worktree. +- **EDGE-CR2** No `runs/` directory exists at all (very first invocation ever) → REQ-CR4 → `None` → + REQ-CR8 fallback. +- **EDGE-CR3** `runs/` exists but is empty, or contains ONLY the current run's own just-created + directory → REQ-CR4 → `None` → REQ-CR8 fallback. +- **EDGE-CR4** A prior run directory exists but has no `checkpoints/` subdirectory at all (e.g. it + crashed before `checkpoint_interval` iterations, or predates this feature) → skipped per REQ-CR2, + next-most-recent candidate considered instead; if none qualifies, REQ-CR4 → `None`. +- **EDGE-CR5** A prior run directory has a `checkpoints/` subdirectory that is present but EMPTY → + same treatment as EDGE-CR4 (skipped, not selected). +- **EDGE-CR6** A `checkpoints/` directory contains non-`checkpoint_`-shaped entries (a stray + file, a hidden dotfile, a directory with a non-numeric suffix) interleaved with valid + `checkpoint_` entries → REQ-CR3 ignores the non-matching entries and selects correctly among + the valid ones; their mere presence never raises an exception. +- **EDGE-CR7** Multiple prior runs each have checkpoints; the most recent run BY DIRECTORY NAME has + fewer/lower-numbered checkpoints than an older run (e.g. it crashed early). REQ-CR2 still selects + the MOST RECENT run (by directory-name recency), NOT the run with the globally highest checkpoint + number — "resume from the last cycle's population" is defined as most-recent-cycle, not + highest-ever-checkpoint-count, so this feature keeps extending the SAME chronological lineage + rather than possibly reaching further back in time. +- **EDGE-CR8** The selected checkpoint directory exists (by REQ-CR3's path logic) but is internally + incomplete/corrupt (missing files `openevolve-run` itself needs to resume) — explicitly OUT of + this function's responsibility (REQ-CR5); if `openevolve-run` then fails to resume, that surfaces + as REQ-OE6's existing "crashed run is inconclusive" handling, not a new failure mode this feature + must detect or prevent. +- **EDGE-CR9** `current_run_id` happens to already have its OWN (freshly created, necessarily empty) + directory present under `runs_dir` at scan time (true in practice, since `run_evolve.sh` creates + `$RUN_DIR` before calling this function, per REQ-CR6) — REQ-CR2's explicit exclusion of + `current_run_id` prevents a run from ever "resuming from itself." + +## "Done" / 4-D Convergence + +| dimension | condition | +|---|---| +| spec | this document + verification-architecture.md | +| test | RED: unit tests for `find_latest_checkpoint` covering REQ-CR1-5 and EDGE-CR1-9 all written and failing (module does not yet exist) before any implementation; GREEN: all passing | +| impl | `lib/checkpoint_resume.py::find_latest_checkpoint` + `run_evolve.sh`'s new pre-invocation step (REQ-CR6-9), both present and runnable | +| impl review | fresh-context `vcsdd-adversary` review of both files returns PASS (lean mode: no BLOCKING findings) | +| verification | Phase 5: proof obligations in verification-architecture.md discharged; a real, hand-constructed two-run `tmp_path` fixture demonstrates `run_evolve.sh`'s new step selecting and logging the correct `--checkpoint` path end-to-end (or a documented reason it could not be exercised without touching the production `runs/` tree, per EDGE-CR1) | + +## UNVERIFIED + +- Whether resuming from a stale/older checkpoint (REQ-CR2's "most recent run by directory name," + EDGE-CR7) ever meaningfully out-performs the alternative "highest checkpoint number across all + runs" policy in practice is an empirical question about openevolve's own MAP-Elites dynamics, not + something this feature's pure path-selection logic can prove — the chronological-lineage choice + is a documented, defensible design decision (see EDGE-CR7's rationale), not an empirically-tuned + one. A future feature may revisit this if real cycle-over-cycle score data suggests otherwise. diff --git a/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md b/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md new file mode 100644 index 00000000..588571b1 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md @@ -0,0 +1,102 @@ +--- +feature: self-improve-checkpoint-resume +phase: 1b +mode: lean +sources: + - behavioral-spec.md (this feature, same directory) — REQ-CR/INV-CR/EDGE-CR IDs referenced below + - .vcsdd/features/self-improve-real-ledger/specs/verification-architecture.md (prior phase) — + Purity Boundary Map / Proof Obligations table conventions, and its EDGE-RL5a-driven "live tier + must run from the merged checkout, never a worktree" execution-locus note, whose EDGE-CR1 + equivalent this document restates for `runs/` +--- + +# Verification Architecture — self-improve-checkpoint-resume (Phase 1b) + +## Purity Boundary Map + +### Pure Core + +| function | signature | why it is pure | REQ traced | +|---|---|---|---| +| `find_latest_checkpoint(runs_dir, current_run_id)` | `(str, str) -> Optional[str]` | deterministic directory-listing + string/integer comparison over an already-given path; no subprocess, no network, no file-content reads, no writes — same input tree yields same output every call | REQ-CR1, REQ-CR2, REQ-CR3, REQ-CR4, REQ-CR5 | + +This is the feature's ENTIRE testable surface. There is no additional pure helper to extract — the +function is small and single-purpose by design (lean mode; do not add abstractions the spec does +not require). + +### Effectful Shell + +| module / function | primary I/O surface | REQ traced | +|---|---|---| +| `run_evolve.sh` (EXTENDED, existing script) | shells out to `"$PY_BIN"` to call `find_latest_checkpoint` against the REAL `runs_dir` (`$SKILL_DIR/runs`) and the real `$RUN_ID`; appends a log line to `$LOG`; conditionally appends `--checkpoint ""` to the existing `"$OPENEVOLVE_BIN"` invocation | REQ-CR6, REQ-CR7, REQ-CR8, REQ-CR9 | +| `openevolve-run` (external, unchanged binary, not part of this feature) | reads the checkpoint directory's own internal contents and resumes from it — entirely outside this feature's code; this feature only ever hands it a PATH | REQ-CR7 (downstream consumer only) | + +The pure `find_latest_checkpoint` never itself invokes `openevolve-run` or writes to `$LOG` — those +are `run_evolve.sh`'s own effectful responsibilities, exactly mirroring how `lib/ledger_reader.py`'s +pure predicates are called from, but never call into, the shell script that logs their results. + +--- + +## Proof Obligations + +Tier legend (lean mode — this feature's whole surface is Tier 0/1; no Tier 2/3 formal-methods +machinery is warranted for ~20 lines of path/existence logic): **Tier 0** = trivial/no-proof-needed +(constant-shaped fallback behavior). **Tier 1** = property/unit test over synthetic `tmp_path` +directory trees, no mocking beyond building a fake filesystem tree. **Tier "wire"** = a shell-level +check that `run_evolve.sh`'s new step is actually invoked and its output actually reaches the +`openevolve-run` argument list (no orphan implementation). + +| ID | Description | Tier | Required | Tool | +|---|---|---|---|---| +| PROP-CR1 | `find_latest_checkpoint` returns `None` for a nonexistent `runs_dir` (REQ-CR4, EDGE-CR2) | 0 | true | pytest | +| PROP-CR2 | `find_latest_checkpoint` returns `None` for an empty `runs_dir`, and for a `runs_dir` containing ONLY `current_run_id`'s own directory (REQ-CR4, EDGE-CR3, EDGE-CR9) | 1 | true | pytest, `tmp_path` | +| PROP-CR3 | Given two prior run directories with checkpoints, the MOST RECENT one (by lexicographic-descending `run-YYYYMMDDTHHMMSSZ` name) is selected, never the older one, even when the older one has a higher checkpoint number (REQ-CR2, EDGE-CR7) | 1 | true | pytest, `tmp_path`, parametrized names | +| PROP-CR4 | Within the selected run, `checkpoint_100` is selected over `checkpoint_20` (integer comparison, not lexicographic) (REQ-CR3) | 1 | true | pytest, `tmp_path` | +| PROP-CR5 | A prior run with no `checkpoints/` subdirectory, or an empty one, is skipped in favor of the next-most-recent qualifying run (REQ-CR2, EDGE-CR4, EDGE-CR5) | 1 | true | pytest, `tmp_path` | +| PROP-CR6 | Non-`checkpoint_`-shaped entries inside `checkpoints/` (a stray file, a hidden dotfile, a non-numeric-suffixed directory) are ignored without raising an exception, interleaved with valid entries (REQ-CR3, EDGE-CR6) | 1 | true | pytest, `tmp_path` | +| PROP-CR7 | `current_run_id` is always excluded from candidate selection, even when its own (empty) directory already exists under `runs_dir` at scan time (REQ-CR2, EDGE-CR9) | 1 | true | pytest, `tmp_path` | +| PROP-CR8 | `find_latest_checkpoint` performs zero file-content reads — a `checkpoint_` directory that is empty or contains arbitrary/garbage file contents is still selected purely on its NAME/existence, never inspected further (REQ-CR5, EDGE-CR8) | 1 | true | pytest, `tmp_path` (checkpoint dirs created with no real openevolve state inside) | +| PROP-CR9 | Static source-text check: `find_latest_checkpoint`'s module source contains no `open(`, `subprocess`, `requests`, `socket`, or `urllib` reference (purity restated as an executable check, mirrors this repo's existing `gate_math.py` AST/text-scan convention) (INV-CR1) | 0 | true | pytest source-text scan | +| PROP-CR-WIRE1 | `run_evolve.sh`'s source text: the `find_latest_checkpoint` call (via `$PY_BIN`) appears BEFORE the `"$OPENEVOLVE_BIN"` invocation, and the invocation's argument list conditionally includes `--checkpoint` (REQ-CR6, REQ-CR7) | wire | true | grep/text-position check over `run_evolve.sh` | +| PROP-CR-WIRE2 | WHEN no checkpoint is found, the resulting `openevolve-run` argument list is IDENTICAL (same arguments, same order) to the pre-feature invocation — no `--checkpoint` flag appears (REQ-CR8, INV-CR4) | 1 | true | a synthetic re-assembly test of the argument array (bash unit check or a small harness that captures the composed command line under both branches) | +| PROP-CR-WIRE3 | Every code path through the new step (found / not-found) appends exactly one line to `$LOG` before proceeding to invoke `openevolve-run` (REQ-CR9) | 1 | true | shell-level check: run the new step in isolation against both a with-checkpoint and without-checkpoint fixture, assert `$LOG` gained exactly one new line each time | +| PROP-CR-LIVE1 | End-to-end: a hand-constructed two-run `tmp_path` tree (an older run with `checkpoints/checkpoint_10/` and `checkpoints/checkpoint_20/`, and a current run with no checkpoints yet) fed through `run_evolve.sh`'s new step (invoked directly, not via the full openevolve subprocess) produces the log line naming `checkpoint_20`'s path (REQ-CR6, REQ-CR9) | live (Tier 1, no real openevolve invocation required) | true | integration test against a synthetic fixture tree, per EDGE-CR1's worktree constraint below | + +**Execution locus note (mirrors the prior phase's EDGE-RL5a pattern, EDGE-CR1):** this worktree's +`skills/earn/self-improve/.gitignore` ignores `runs/` and `state/` — a fresh checkout has no +`runs/` directory. Every test above (PROP-CR1 through PROP-CR-LIVE1) MUST build its own synthetic +`runs_dir` under pytest's `tmp_path`, and MUST NEVER read, symlink, or copy any real production +`runs/` directory into this worktree. A true end-to-end run of the FULL `run_evolve.sh` script +(including a real `openevolve-run` invocation actually resuming from a real checkpoint) is +optional, out-of-band verification evidence for Phase 5 — not a Phase 2 gating requirement — and, +if performed, MUST run from a real instance body (e.g. this feature's own production `runs/` +directory after merge), never fabricated inside a worktree. + +## Verification Strategy + +- **Tier 0**: `find_latest_checkpoint`'s trivial no-input-tree fallback cases (missing/empty + `runs_dir`) and the static purity source-scan — no formal proof needed, a single assertion each + settles them. +- **Tier 1**: the bulk of this feature's proof burden — deterministic unit/property tests over + synthetic `tmp_path` directory trees covering every REQ-CR/EDGE-CR combination (recency ordering, + integer-vs-lexicographic checkpoint comparison, malformed-entry tolerance, self-exclusion, + no-file-content-read). This is standard `pytest`, matching this repo's existing convention for + `gate_math.py`/`ledger_reader.py` pure-function tests — no `hypothesis`/`kani`/`fast-check` + property-fuzzing harness is introduced for ~20 lines of path logic (lean mode: do not + over-specify formal machinery this feature's size does not warrant). +- **Tier 2**: not used. This feature has no numeric/financial invariant, no concurrency, and no + security-sensitive parsing surface that would justify lightweight formal methods (contrast with + `gate_math.py`'s `is_implausible_jump`/reward-hacking trip-wire math in the parent harness, which + DOES warrant Tier 2 in that feature's own spec). +- **Tier 3**: not used. No cryptographic, concurrency-safety, or strong-correctness claim is made by + this feature. + +## Regression Table + +The pre-existing `skills/earn/self-improve/tests/` suite (14 files, covering ledger resolution, +gate math, adversary wiring, denylist, reward-hacking trip-wire, etc.) MUST remain green after this +feature's changes. This feature adds ONE new file (`lib/checkpoint_resume.py`) and edits ONE +existing file (`run_evolve.sh`, bash only — no Python module this feature touches is imported by +any pre-existing test), so the regression risk surface is the full existing pytest suite re-run +unmodified, plus a bash syntax/execution smoke check of `run_evolve.sh` itself (`bash -n +run_evolve.sh`). diff --git a/.vcsdd/features/self-improve-checkpoint-resume/state.json b/.vcsdd/features/self-improve-checkpoint-resume/state.json new file mode 100644 index 00000000..edc18dfe --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/state.json @@ -0,0 +1,31 @@ +{ + "featureName": "self-improve-checkpoint-resume", + "mode": "lean", + "language": "python", + "currentPhase": "1b", + "phaseHistory": [ + { + "from": "init", + "to": "1a", + "timestamp": "2026-07-11T09:01:42.930Z" + }, + { + "from": "1a", + "to": "1b", + "timestamp": "2026-07-11T09:01:42.933Z" + } + ], + "iterations": { + "1a": 1, + "1b": 1 + }, + "proofObligations": [], + "traceability": { + "epicId": "VCSDD-self-improve-checkpoint-resume-1783760284512", + "beads": [] + }, + "gates": {}, + "sprintCount": 0, + "createdAt": "2026-07-11T08:58:04.512Z", + "updatedAt": "2026-07-11T09:01:42.933Z" +} diff --git a/.vcsdd/history.jsonl b/.vcsdd/history.jsonl index 8bf6099b..0fba9977 100644 --- a/.vcsdd/history.jsonl +++ b/.vcsdd/history.jsonl @@ -345,3 +345,6 @@ {"timestamp":"2026-07-11T01:16:39.171Z","event":"phase_transition","featureName":"franklin-alwaysact-skill-router","from":"6","to":"6","reason":"RECONCILIATION: Iteration-3 findings fixed (commit fb7fc4de: mechanical citation audit, evidence/citation-audit-2026-07-11.md, 56/57 rows corrected: FIND-005/006/007 resolved). Phase 6 convergence iteration 4 verdict: NOT_CONVERGED — 1 new blocking finding (FIND-008: REQ-508's own fix-sentence introduced a false function-attribution claim — index.mjs:878 misattributed to writeAlwaysActEscalation, actually inside writeWakeErrorAndSleep) on spec_fidelity/verification_readiness; duplicate_detection/four_dimensional_convergence/residuals_honest FAIL (4th consecutive 'exhaustive fix leaves one further undisclosed same-class defect' recurrence); finding_diminishment FAIL despite raw count dropping 3->1 (criterion judged on whether the mechanical audit's own blind spot — enclosing-function attribution vs. line-existence — was closed, and it was not yet). reviewed_commit fb7fc4de. reviews/converge/output/verdict-iteration-4.json.","reconciled":true} {"timestamp":"2026-07-11T01:16:39.171Z","event":"phase_transition","featureName":"franklin-alwaysact-skill-router","from":"6","to":"complete","reason":"RECONCILIATION: Iteration-4 finding fixed (commit a4720db: FIND-008 — REQ-508 EARS clause corrected to attribute index.mjs:914 to writeAlwaysActEscalation (starts :899) and index.mjs:878 to the separate writeWakeErrorAndSleep (starts :868); evidence/citation-audit-2026-07-11.md row 27 synced). Phase 6 convergence iteration 5 verdict: CONVERGED — 0 blocking findings, all 6 criteria PASS (finding_diminishment, finding_specificity, criteria_coverage, duplicate_detection, four_dimensional_convergence, residuals_honest). Method extended per iteration 4's own recommendation to check every other enclosing-function claim across the full document set (7 additional citations independently re-verified); zero further misattributions found. findingCount trend across converge iterations: 2 -> 2 -> 3 -> 1 -> 0. reviewed_commit a4720db. reviews/converge/output/verdict-iteration-5.json. Gate 6 PASSED (CONVERGED) at commit a4720db. Entering phase complete.","reconciled":true} {"timestamp":"2026-07-11T01:16:39.171Z","event":"gate_recorded","featureName":"franklin-alwaysact-skill-router","phase":"6","verdict":"PASS","reviewedBy":"adversary","reconciled":true} +{"timestamp":"2026-07-11T08:58:04.515Z","event":"feature_created","featureName":"self-improve-checkpoint-resume","mode":"lean","language":"python"} +{"timestamp":"2026-07-11T09:01:42.933Z","event":"phase_transition","featureName":"self-improve-checkpoint-resume","from":"init","to":"1a","iteration":1,"phase":"1a"} +{"timestamp":"2026-07-11T09:01:42.934Z","event":"phase_transition","featureName":"self-improve-checkpoint-resume","from":"1a","to":"1b","iteration":1,"phase":"1b"} diff --git a/.vcsdd/index.json b/.vcsdd/index.json index 85aa93ec..af8f9f97 100644 --- a/.vcsdd/index.json +++ b/.vcsdd/index.json @@ -238,7 +238,15 @@ "language": "typescript", "currentPhase": "complete", "updatedAt": "2026-07-11T01:16:39.166Z" + }, + "self-improve-checkpoint-resume": { + "status": "active", + "createdAt": "2026-07-11T08:58:04.512Z", + "mode": "lean", + "language": "python", + "currentPhase": "1b", + "updatedAt": "2026-07-11T09:01:42.933Z" } }, - "activeFeature": "franklin-alwaysact-skill-router" + "activeFeature": "self-improve-checkpoint-resume" } From 6a27a4401f375b819372717e645f607d2bf42809 Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sat, 11 Jul 2026 18:14:15 +0900 Subject: [PATCH 02/11] spec: resolve spec-review iteration-1 FIND-001..004 (self-improve-checkpoint-resume) FIND-001 (blocking): add REQ-CR10 run-directory-name shape filter (^run-\d{8}T\d{6}Z$) so runs/db (config.yaml database.db_path) and any other non-run sibling under runs_dir is never a candidate; add EDGE-CR10 and PROP-CR10. FIND-002 (blocking): rewrite REQ-CR2/REQ-CR3 into one unambiguous iterate-until-valid-checkpoint-or-exhausted algorithm. A checkpoints/ dir that is non-empty at the OS-listing level but has zero checkpoint_ entries (e.g. a stray .DS_Store) now explicitly falls through to the next-older run-shaped candidate rather than returning early; add EDGE-CR11 and PROP-CR11. FIND-003 (major): commit to one shell-invocation shape. checkpoint_resume.py gets a __main__ CLI entrypoint mirroring lib/ledger_reader.py's convention exactly (no argv, reads RUNS_DIR/RUN_ID from os.environ, prints one line to stdout). Pinned exact call site: RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py". REQ-CR6 and PROP-CR-WIRE1 updated to check this exact string. FIND-004 (minor): PROP-CR9 no longer claims to mirror an unverified gate_math.py AST/text-scan test file (none exists); it now states the purity check on its own terms. Adds reviews/spec/iteration-2/input/manifest.json for the next spec-review pass. Does not transition VCSDD phase or run the review. --- .../spec/iteration-1/input/manifest.json | 5 + .../iteration-1/output/findings/FIND-001.json | 13 ++ .../iteration-1/output/findings/FIND-002.json | 13 ++ .../iteration-1/output/findings/FIND-003.json | 13 ++ .../iteration-1/output/findings/FIND-004.json | 13 ++ .../spec/iteration-1/output/verdict.json | 43 +++++ .../spec/iteration-2/input/manifest.json | 5 + .../specs/behavioral-spec.md | 153 +++++++++++++----- .../specs/verification-architecture.md | 27 +++- 9 files changed, 238 insertions(+), 47 deletions(-) create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/input/manifest.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-001.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-002.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-003.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-004.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/verdict.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/input/manifest.json diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/input/manifest.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/input/manifest.json new file mode 100644 index 00000000..d99d842f --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/input/manifest.json @@ -0,0 +1,5 @@ +{ + "reviewType": "spec", + "artifactsToReview": ["specs/behavioral-spec.md", "specs/verification-architecture.md"], + "reviewDimensions": ["spec_fidelity", "verification_readiness"] +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-001.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-001.json new file mode 100644 index 00000000..ebf185f3 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-001.json @@ -0,0 +1,13 @@ +{ + "findingId": "FIND-001", + "dimension": "spec_fidelity", + "category": "spec_gap", + "severity": "blocking", + "description": "REQ-CR2 instructs `find_latest_checkpoint` to 'list the immediate subdirectories of runs_dir' and select among them, but the spec never accounts for a real, always-present non-run-named sibling entry that DOES live directly under runs_dir in production: `config.yaml`'s `database.db_path: \"runs/db\"` (skills/earn/self-improve/config.yaml:129), which `run_evolve.sh`'s own comment (run_evolve.sh:28-32) confirms resolves to `$SKILL_DIR/runs/db` after the script's `cd \"$SKILL_DIR\"`. This is not hypothetical: a prior, already-reviewed sibling feature's live verification logs show openevolve actually writing/loading this exact directory ('Loaded database with 82 programs from runs/db', .vcsdd/features/self-improve-real-ledger/verification/live-results/PROP-RL-LIVE3-fresh-log-excerpt.txt:12). So the REAL runs_dir this feature's function will scan in production contains at least one entry ('db') that is not a `run-YYYYMMDDTHHMMSSZ` directory and has no `checkpoints/` subdirectory of its own. REQ-CR2 (behavioral-spec.md:118-125) and the full EDGE-CR1..CR9 list (behavioral-spec.md:189-222) never mention or test this entry. The algorithm as literally written happens to tolerate it only by lexicographic accident ('d' < 'r', so 'db' always sorts after every 'run-*' name in descending order, and it lacks checkpoints/ so REQ-CR2's skip rule applies) -- an undocumented, untested invariant rather than a specified one. verification-architecture.md's Proof Obligations table (lines 49-63) has no PROP-CR* entry exercising a runs_dir that contains this real sibling directory, so a test-writer following this spec exactly would plausibly never construct a fixture containing it, leaving the real production shape of runs_dir completely unverified by Phase 2's own test suite.", + "evidence": { + "filePath": "skills/earn/self-improve/config.yaml", + "lineRange": "124-129", + "snippet": "database:\n db_path: \"runs/db\"" + }, + "routeToPhase": "1a/1b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-002.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-002.json new file mode 100644 index 00000000..fc651369 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-002.json @@ -0,0 +1,13 @@ +{ + "findingId": "FIND-002", + "dimension": "spec_fidelity", + "category": "spec_gap", + "severity": "blocking", + "description": "The composed REQ-CR2/REQ-CR3 selection algorithm has an unresolved case. REQ-CR2 (behavioral-spec.md:118-125) selects a single prior-run candidate based on it having 'a non-empty checkpoints/ subdirectory,' then REQ-CR3 (behavioral-spec.md:127-133) filters entries WITHIN that already-selected directory down to only `checkpoint_`-shaped names, ignoring everything else. These are two different predicates: a `checkpoints/` directory can be non-empty at the OS-listing level while containing ZERO `checkpoint_`-shaped entries (for example a stray macOS Finder `.DS_Store` file -- a concrete, environment-grounded risk given this repo's own CLAUDE.md states the production execution host is 'Mac Mini (anicca-mac-mini-1)', where Finder/Spotlight can and does drop `.DS_Store` into any directory that gets browsed or indexed). EDGE-CR5 (behavioral-spec.md:202-203) only covers a checkpoints/ dir that is literally EMPTY ('same treatment as EDGE-CR4'). EDGE-CR6 (behavioral-spec.md:204-207) only covers stray entries INTERLEAVED WITH at least one genuinely valid `checkpoint_` entry. Neither edge case -- nor any PROP-CR* proof obligation in verification-architecture.md:49-63 -- specifies the actual case in dispute: what must `find_latest_checkpoint` do when the run REQ-CR2 selected turns out, after REQ-CR3's filtering, to have NO valid `checkpoint_` entry at all? Two materially different, both textually plausible, implementations exist: (a) return `None` immediately, even if an OLDER prior run has a genuinely valid checkpoint (silently forfeiting a usable resume), or (b) fall through and re-run REQ-CR2's selection against the next-most-recent candidate. The spec must pin exactly one of these and add a corresponding EDGE-CR id and PROP-CR obligation before Phase 2 TDD, or two independent implementers following this spec verbatim could produce silently divergent, both-'spec-compliant' behavior.", + "evidence": { + "filePath": ".vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "118-133 and 199-207", + "snippet": "REQ-CR2: '...select the FIRST (most recent) candidate that has a non-empty checkpoints/ subdirectory...' / REQ-CR3: '...consider only entries whose name matches...checkpoint_...'" + }, + "routeToPhase": "1a/1b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-003.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-003.json new file mode 100644 index 00000000..0076c2da --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-003.json @@ -0,0 +1,13 @@ +{ + "findingId": "FIND-003", + "dimension": "verification_readiness", + "category": "spec_gap", + "severity": "major", + "description": "The Purity Boundary Analysis section (behavioral-spec.md:100-108) leaves the shell-to-pure-function call convention genuinely open: it offers TWO alternative invocation shapes ('via \"$PY_BIN\" -c' OR 'a thin CLI entrypoint, mirroring lib/ledger_reader.py's existing... call convention') without committing to one, and REQ-CR6 (behavioral-spec.md:150-153) only says the function is invoked 'via \"$PY_BIN\"' without pinning the exact command line. Confirmed live: `lib/ledger_reader.py`'s actual convention (run_evolve.sh:60, `\"$PY_BIN\" \"$SKILL_DIR/lib/ledger_reader.py\"`) is a full-script invocation relying on a `__main__`-style entrypoint that reads no argv and prints JSON to stdout -- a materially different code shape from an inline `-c` snippet (which would need the module imported and called inline from the shell, with `checkpoint_resume.py` needing no `__main__` block at all, taking `runs_dir`/`current_run_id` as function arguments passed through `-c`'s own argument string). Since PROP-CR-WIRE1 (verification-architecture.md:60) only checks 'the call... appears BEFORE the invocation' via a text-position/grep check, this ambiguity will not surface as a test failure at Phase 2/5 either -- it is left to whichever the implementer happens to pick, undermining REQ-CR6/CR9's own stated goal of mirroring an established convention.", + "evidence": { + "filePath": ".vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "100-108", + "snippet": "\"$PY_BIN\" -c\" or a thin CLI entrypoint, mirroring `lib/ledger_reader.py`'s existing... call convention" + }, + "routeToPhase": "1a/1b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-004.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-004.json new file mode 100644 index 00000000..5a73afc5 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/findings/FIND-004.json @@ -0,0 +1,13 @@ +{ + "findingId": "FIND-004", + "dimension": "verification_readiness", + "category": "spec_gap", + "severity": "minor", + "description": "PROP-CR9 (verification-architecture.md:59) claims its source-text purity scan for `checkpoint_resume.py` 'mirrors this repo's existing `gate_math.py` AST/text-scan convention.' A grep of `skills/earn/self-improve/tests/` for an AST/text-import-scan test targeting `gate_math.py`'s imports did not locate any such dedicated test file -- only `gate_math.py`'s own module docstring (gate_math.py:1-21) asserts the no-I/O-imports property, without a located enforcing test to confirm the claimed precedent actually exists as a runnable check today. This citation should be corrected to a concrete file:line, or the 'mirrors an existing convention' claim should be dropped so PROP-CR9 stands on its own rather than an unverified precedent.", + "evidence": { + "filePath": ".vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md", + "lineRange": "59", + "snippet": "mirrors this repo's existing `gate_math.py` AST/text-scan convention" + }, + "routeToPhase": "1a/1b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/verdict.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/verdict.json new file mode 100644 index 00000000..d0556ebd --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-1/output/verdict.json @@ -0,0 +1,43 @@ +{ + "feature": "self-improve-checkpoint-resume", + "reviewContext": { + "reviewType": "spec", + "phase": "1c", + "mode": "lean", + "iterationDir": "reviews/spec/iteration-1" + }, + "overallVerdict": "FAIL", + "timestamp": "2026-07-11T00:00:00Z", + "iteration": 1, + "dimensions": [ + { + "name": "spec_fidelity", + "verdict": "FAIL", + "findings": ["FIND-001", "FIND-002"], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/run_evolve.sh:82-89", "description": "Confirmed live: today's openevolve-run invocation never passes --checkpoint; --output \"$RUN_DIR\" is fresh every cycle, matching the spec's stated root-cause narrative."}, + {"type": "file", "location": "skills/earn/self-improve/config.yaml:11,128-129", "description": "Confirmed checkpoint_interval is a top-level Config field (not database.checkpoint_interval) and its own comment already anticipates --checkpoint resume, exactly as the spec's sources claim."}, + {"type": "file", "location": "/Users/anicca/.anicca-venvs/self-improve/lib/python3.14/site-packages/openevolve/cli.py:49-50,116-121,166-171", "description": "Confirmed live in the actual installed openevolve==0.3.0 venv: --checkpoint is a real CLI flag, a missing/invalid checkpoint path returns exit code 1 (non-zero), and any exception during checkpoint load is caught and also returns 1 -- validating REQ-CR7/EDGE-CR8's claim that failure surfaces through the existing non-zero-exit 'crashed run is inconclusive' handling already in run_evolve.sh (lines 78-91), with no new failure-handling branch needed."}, + {"type": "file", "location": "/Users/anicca/.anicca-venvs/self-improve/lib/python3.14/site-packages/openevolve/controller.py:451-462", "description": "Confirmed checkpoint directories are actually named checkpoint_{iteration} under /checkpoints/, matching REQ-CR3's assumed checkpoint_ naming pattern exactly."}, + {"type": "file", "location": "skills/earn/self-improve/.gitignore:1-2", "description": "Confirmed runs/ and state/ are gitignored, validating EDGE-CR1's worktree-fixture-only testing constraint."}, + {"type": "file", "location": "skills/earn/self-improve/lib/ledger_reader.py:1-38, run_evolve.sh:59-64", "description": "Confirmed the '$PY_BIN $SKILL_DIR/lib/.py' shell-call convention this feature claims to mirror is real and already in use for the OBSERVE step."}, + {"type": "grep", "location": "skills/_search for wallet/.env/ledger/spend-cap references inside the two spec files", "description": "No occurrence found; INV-CR3's claim of zero wallet/ledger/spend-cap touch is consistent with the spec's own text and with promote_gate.sh/lib/promote.py/config.yaml/plist being untouched per the frontmatter's does_not_replace_or_modify list."} + ] + }, + { + "name": "verification_readiness", + "verdict": "FAIL", + "findings": ["FIND-002", "FIND-003", "FIND-004"], + "evidence": [ + {"type": "file", "location": "verification-architecture.md:75-92", "description": "Tier assignment (0/1/wire only, no Tier 2/3) is proportionate and justified for ~20 lines of path/existence logic; no unjustified formal machinery introduced, consistent with lean mode."}, + {"type": "file", "location": "verification-architecture.md:94-102", "description": "Regression table correctly scopes the pre-existing 14-file pytest suite plus a bash -n smoke check; accurate given no pre-existing test imports run_evolve.sh's Python surface."} + ] + } + ], + "convergenceSignals": { + "findingCount": 4, + "allCriteriaEvaluated": true, + "evaluatedCriteria": [], + "duplicateFindings": [] + } +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/input/manifest.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/input/manifest.json new file mode 100644 index 00000000..d99d842f --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/input/manifest.json @@ -0,0 +1,5 @@ +{ + "reviewType": "spec", + "artifactsToReview": ["specs/behavioral-spec.md", "specs/verification-architecture.md"], + "reviewDimensions": ["spec_fidelity", "verification_readiness"] +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md b/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md index d553340f..2a060b6d 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md +++ b/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md @@ -98,14 +98,21 @@ is touched. files — see REQ-CR5). Same input directory tree → same output, every time, in-process. - **Effectful shell:** `run_evolve.sh`'s existing bash — it already creates `$RUN_DIR`, invokes `openevolve-run`, and appends to `$LOG` (REQ-OE6/OE7, unchanged conventions). This feature adds - exactly one new effectful step to that shell script: shell out to the pure function (via - `"$PY_BIN" -c` or a thin CLI entrypoint, mirroring `lib/ledger_reader.py`'s existing - `"$PY_BIN" "$SKILL_DIR/lib/.py"` call convention) BEFORE the `openevolve-run` invocation, - capture its stdout (a path or empty string), log the outcome, and conditionally append - `--checkpoint ""` to the existing command array. `openevolve-run` itself (a separate - process, already effectful, unchanged by this feature) is what actually reads the checkpoint - directory's contents and resumes from it — that read is entirely outside this feature's own - code. + exactly one new effectful step to that shell script: shell out to the pure function via a thin + CLI entrypoint living INSIDE `lib/checkpoint_resume.py` itself (its own `if __name__ == + "__main__":` block) — this is the ONE committed invocation shape (no alternative form is left + open; see REQ-CR6 for the exact command line). That entrypoint takes NO argv, reads `RUNS_DIR` + and `RUN_ID` from the process environment, calls `find_latest_checkpoint(runs_dir=os.environ[ + "RUNS_DIR"], current_run_id=os.environ["RUN_ID"])`, and prints EXACTLY one line to stdout — the + resulting path, or an empty string when `None` — nothing else. This is a byte-for-byte mirror of + `lib/ledger_reader.py`'s own established convention (confirmed live, `lib/ledger_reader.py:222- + 225`: no argv, reads its own env-derived config internally, prints one line of JSON to stdout; + called from `run_evolve.sh:60` as `"$PY_BIN" "$SKILL_DIR/lib/ledger_reader.py"`, no `-c`, no + extra args). `run_evolve.sh` invokes it BEFORE the `openevolve-run` invocation, captures its + stdout, logs the outcome, and conditionally appends `--checkpoint ""` to the existing + command array. `openevolve-run` itself (a separate process, already effectful, unchanged by this + feature) is what actually reads the checkpoint directory's contents and resumes from it — that + read is entirely outside this feature's own code. ## EARS-Format Functional Requirements @@ -116,25 +123,64 @@ is touched. checkpoint directory to resume from, or `None` when no prior run has any usable checkpoint. - **REQ-CR2** WHEN selecting which PRIOR RUN to resume from, THE SYSTEM SHALL: list the immediate - subdirectories of `runs_dir`, EXCLUDE any entry equal to `current_run_id`, sort the REMAINING - candidate names lexicographically DESCENDING (run directory names are `run-YYYYMMDDTHHMMSSZ` UTC - timestamps, which sort correctly as plain strings — REQ-OE7's existing naming convention, - unchanged), and select the FIRST (most recent) candidate that has a non-empty `checkpoints/` - subdirectory (per REQ-CR3's definition of "has a checkpoint"). Runs with no `checkpoints/` - subdirectory, or an EMPTY one, SHALL be skipped over (not selected, but also not treated as a - fatal error) in favor of the next-most-recent candidate. - -- **REQ-CR3** WITHIN the selected prior run's `checkpoints/` directory, THE SYSTEM SHALL consider - only entries whose name matches the literal pattern `checkpoint_` where `` is one or more - ASCII digits, parse `` as an INTEGER (not a string), and select the entry with the LARGEST - integer `` (e.g. `checkpoint_100` SHALL be selected over `checkpoint_20` — NEVER a lexicographic - comparison, which would incorrectly rank `checkpoint_20` above `checkpoint_100`). Any entry inside - `checkpoints/` that does NOT match this pattern (a stray file, a differently-named directory, a - hidden dotfile) SHALL be ignored — its presence SHALL NEVER raise an exception or abort the scan. - -- **REQ-CR4** WHEN `runs_dir` does not exist at all, is empty, or contains ONLY the excluded - `current_run_id` entry, THE SYSTEM SHALL return `None` — this is the ordinary, expected "first-ever - run" case, not an error condition. + entries of `runs_dir`, filter that list down to ONLY entries whose name matches REQ-CR10's + run-directory-name shape (this step is unconditional and happens BEFORE any other filtering — an + entry that does not match REQ-CR10's shape, such as the real, always-present `runs/db` sibling + directory created by `config.yaml`'s `database.db_path: "runs/db"` — see EDGE-CR10 — is NEVER a + candidate, regardless of what it contains or how it sorts), EXCLUDE any remaining entry equal + to `current_run_id`, sort the REMAINING run-shaped candidate names lexicographically DESCENDING + (run directory names are `run-YYYYMMDDTHHMMSSZ` UTC timestamps, which sort correctly as plain + strings — REQ-OE7's existing naming convention, unchanged), and THEN iterate that sorted list IN + ORDER, calling REQ-CR3's per-run checkpoint selector on each candidate in turn: the FIRST + candidate for which REQ-CR3 returns a non-`None` checkpoint path is the one used, and iteration + STOPS there (REQ-CR3's result for that candidate IS the function's overall result, modulo + REQ-CR1's absolute-path contract). A candidate for which REQ-CR3 returns `None` (for ANY reason — + no `checkpoints/` subdirectory at all, an empty one, or one that is non-empty at the OS-listing + level but contains ZERO valid `checkpoint_`-shaped entries — see EDGE-CR11) is skipped (not + selected, not a fatal error) and iteration CONTINUES to the next-most-recent candidate. If the + entire sorted candidate list is exhausted with every candidate returning `None` from REQ-CR3 (or + the list was empty to begin with), THE SYSTEM SHALL return `None` (REQ-CR4). This iterate-until- + a-real-checkpoint-or-exhausted rule is the ONLY selection algorithm this function implements — + there is no early return on "first candidate encountered," only on "first candidate that + actually yields a usable checkpoint." + +- **REQ-CR3** GIVEN a single candidate run directory (as selected by REQ-CR2's iteration), THE + SYSTEM SHALL: if that directory has no `checkpoints/` subdirectory at all, return `None` + immediately. Otherwise, list `checkpoints/`'s immediate entries and consider ONLY those whose + name matches the literal pattern `checkpoint_` where `` is one or more ASCII digits — any + entry that does NOT match this pattern (a stray file such as `.DS_Store`, a differently-named + directory, a hidden dotfile) SHALL be ignored and SHALL NEVER raise an exception or abort the + scan. If, after this filter, there are ZERO matching entries — whether because `checkpoints/` was + literally empty (zero entries of any kind) OR because it was non-empty at the OS-listing level + but every entry present failed the `checkpoint_` shape test (EDGE-CR11) — THE SYSTEM SHALL + return `None` for this candidate (these two cases are INDISTINGUISHABLE in their outcome: both + mean "this candidate has no usable checkpoint," and REQ-CR2's caller treats both identically as a + reason to move on to the next-most-recent candidate). Otherwise (at least one matching entry + exists), parse each matching entry's `` as an INTEGER (not a string) and return the absolute + path of the entry with the LARGEST integer `` (e.g. `checkpoint_100` SHALL be selected over + `checkpoint_20` — NEVER a lexicographic comparison, which would incorrectly rank `checkpoint_20` + above `checkpoint_100`). + +- **REQ-CR10** A `runs_dir` entry SHALL be considered a candidate run directory (eligible for + REQ-CR2's iteration at all) ONLY IF its name matches the literal shape `run-` followed by exactly + the UTC timestamp format this repo's own `run_evolve.sh:50` already produces (`run-$(date -u + +%Y%m%dT%H%M%SZ)`, i.e. `run-YYYYMMDDTHHMMSSZ`: 8 ASCII digits, literal `T`, 6 ASCII digits, + literal `Z`) — concretely, a regex equivalent to `^run-\d{8}T\d{6}Z$`. This is the ONLY test used + to decide "is this a run directory" — it is applied BEFORE, and independently of, the + `current_run_id` exclusion, and independently of whatever the entry contains (a directory or a + file, empty or not). Any entry that does not match this shape — including, concretely, the real + `runs/db` sibling directory that `config.yaml`'s `database.db_path: "runs/db"` causes + `openevolve-run` to create directly under `runs_dir` on every real production cycle (confirmed: + `skills/earn/self-improve/config.yaml:129`) — is EXCLUDED from candidacy entirely, never + considered, never inspected for a `checkpoints/` subdirectory, regardless of its lexicographic + sort position relative to any `run-*` name. + +- **REQ-CR4** WHEN `runs_dir` does not exist at all, is empty, contains ONLY the excluded + `current_run_id` entry, contains ONLY entries that fail REQ-CR10's run-directory-name shape + filter (e.g. `runs_dir` contains only `db`, per EDGE-CR10, and no `run-*` entry at all), OR every + run-shaped candidate's REQ-CR3 result is `None` (REQ-CR2's iteration exhausted), THE SYSTEM SHALL + return `None` — this is the ordinary, expected "no usable prior checkpoint" case, not an error + condition. - **REQ-CR5** `find_latest_checkpoint` SHALL perform PATH/EXISTENCE logic only (directory listing, name pattern matching, integer comparison). It SHALL NEVER open, read, or validate the byte @@ -147,10 +193,16 @@ is touched. ### Group CR-WIRE — wiring into run_evolve.sh -- **REQ-CR6** `run_evolve.sh` SHALL invoke the REQ-CR1 function, via `"$PY_BIN"`, AFTER `$RUN_DIR` - is created (so `current_run_id` = the just-created `$RUN_ID`, correctly excluding itself even - though its own now-empty directory already exists under `runs/`) and BEFORE the existing - `"$OPENEVOLVE_BIN"` invocation. +- **REQ-CR6** `run_evolve.sh` SHALL invoke `lib/checkpoint_resume.py`'s CLI entrypoint via EXACTLY + the command `RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py"` + — no `-c` inline snippet, no positional argv, environment-variable assignment ONLY, mirroring + `lib/ledger_reader.py`'s existing `"$PY_BIN" "$SKILL_DIR/lib/ledger_reader.py"` call shape + (`run_evolve.sh:60`) exactly, with the sole addition of the two inline env-var assignments this + module needs that `ledger_reader.py` does not. This call happens AFTER `$RUN_DIR` is created (so + `RUN_ID` = the just-created `$RUN_ID`, correctly excluding itself even though its own now-empty + directory already exists under `runs/`) and BEFORE the existing `"$OPENEVOLVE_BIN"` invocation. + `run_evolve.sh` SHALL capture this call's stdout verbatim as the checkpoint path (an empty string + meaning "no checkpoint found," per REQ-CR8). - **REQ-CR7** WHEN the REQ-CR6 call returns a non-empty checkpoint path, THE SYSTEM SHALL append exactly one additional argument pair, `--checkpoint ""`, to the EXISTING `openevolve-run` @@ -197,14 +249,17 @@ is touched. - **EDGE-CR3** `runs/` exists but is empty, or contains ONLY the current run's own just-created directory → REQ-CR4 → `None` → REQ-CR8 fallback. - **EDGE-CR4** A prior run directory exists but has no `checkpoints/` subdirectory at all (e.g. it - crashed before `checkpoint_interval` iterations, or predates this feature) → skipped per REQ-CR2, - next-most-recent candidate considered instead; if none qualifies, REQ-CR4 → `None`. -- **EDGE-CR5** A prior run directory has a `checkpoints/` subdirectory that is present but EMPTY → - same treatment as EDGE-CR4 (skipped, not selected). + crashed before `checkpoint_interval` iterations, or predates this feature) → REQ-CR3 returns + `None` for it → REQ-CR2's iteration skips it and continues to the next-most-recent run-shaped + candidate; if none qualifies, REQ-CR4 → `None`. +- **EDGE-CR5** A prior run directory has a `checkpoints/` subdirectory that is present but literally + EMPTY (zero entries of any kind) → REQ-CR3 returns `None` for it (same treatment as EDGE-CR4: + skipped, iteration continues). - **EDGE-CR6** A `checkpoints/` directory contains non-`checkpoint_`-shaped entries (a stray - file, a hidden dotfile, a directory with a non-numeric suffix) interleaved with valid - `checkpoint_` entries → REQ-CR3 ignores the non-matching entries and selects correctly among - the valid ones; their mere presence never raises an exception. + file, a hidden dotfile, a directory with a non-numeric suffix) interleaved with AT LEAST ONE + valid `checkpoint_` entry → REQ-CR3 ignores the non-matching entries and selects correctly + among the valid ones; their mere presence never raises an exception. (Contrast EDGE-CR11, where + ZERO valid entries survive the filter.) - **EDGE-CR7** Multiple prior runs each have checkpoints; the most recent run BY DIRECTORY NAME has fewer/lower-numbered checkpoints than an older run (e.g. it crashed early). REQ-CR2 still selects the MOST RECENT run (by directory-name recency), NOT the run with the globally highest checkpoint @@ -220,13 +275,33 @@ is touched. directory present under `runs_dir` at scan time (true in practice, since `run_evolve.sh` creates `$RUN_DIR` before calling this function, per REQ-CR6) — REQ-CR2's explicit exclusion of `current_run_id` prevents a run from ever "resuming from itself." +- **EDGE-CR10** `runs_dir` contains a real, always-present, non-run-shaped sibling entry that is NOT + a run directory at all: `runs/db`, created directly under `runs_dir` by `config.yaml`'s + `database.db_path: "runs/db"` (confirmed: `skills/earn/self-improve/config.yaml:129`; this is not + hypothetical — it is a real directory openevolve itself writes on every production cycle, e.g. + "Loaded database with 82 programs from runs/db" per prior-phase live logs). REQ-CR10's + run-directory-name shape filter (`^run-\d{8}T\d{6}Z$`) EXCLUDES `db` from candidacy entirely, + before any lexicographic sort and before any `checkpoints/` inspection — it is NEVER selected, + NEVER treated as "the most recent candidate," regardless of its name's sort position relative to + `run-*` names. +- **EDGE-CR11** A run-shaped candidate directory's `checkpoints/` subdirectory is non-empty at the + OS-listing level (`os.listdir` returns >= 1 entry) but contains ZERO entries matching the + `checkpoint_` shape — for example, only a stray macOS `.DS_Store` file (a concrete, + environment-grounded risk: this repo's own CLAUDE.md names the production execution host as a + Mac Mini, where Finder/Spotlight can drop `.DS_Store` into any browsed/indexed directory). This is + NOT the same test as "non-empty `checkpoints/` dir" (contrast EDGE-CR6, where at least one valid + entry survives). Per REQ-CR3, this candidate returns `None` (treated identically to EDGE-CR4/ + EDGE-CR5, NOT as an error) and REQ-CR2's iteration falls through to the next-most-recent + run-shaped candidate, continuing until a candidate yields a valid `checkpoint_` entry or the + sorted candidate list is exhausted (→ REQ-CR4 → `None`). This function SHALL NEVER return early + or raise merely because `checkpoints/` was non-empty at the OS-listing level. ## "Done" / 4-D Convergence | dimension | condition | |---|---| | spec | this document + verification-architecture.md | -| test | RED: unit tests for `find_latest_checkpoint` covering REQ-CR1-5 and EDGE-CR1-9 all written and failing (module does not yet exist) before any implementation; GREEN: all passing | +| test | RED: unit tests for `find_latest_checkpoint` covering REQ-CR1-5, REQ-CR10, and EDGE-CR1-11 all written and failing (module does not yet exist) before any implementation; GREEN: all passing | | impl | `lib/checkpoint_resume.py::find_latest_checkpoint` + `run_evolve.sh`'s new pre-invocation step (REQ-CR6-9), both present and runnable | | impl review | fresh-context `vcsdd-adversary` review of both files returns PASS (lean mode: no BLOCKING findings) | | verification | Phase 5: proof obligations in verification-architecture.md discharged; a real, hand-constructed two-run `tmp_path` fixture demonstrates `run_evolve.sh`'s new step selecting and logging the correct `--checkpoint` path end-to-end (or a documented reason it could not be exercised without touching the production `runs/` tree, per EDGE-CR1) | diff --git a/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md b/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md index 588571b1..12130a57 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md +++ b/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md @@ -18,7 +18,7 @@ sources: | function | signature | why it is pure | REQ traced | |---|---|---|---| -| `find_latest_checkpoint(runs_dir, current_run_id)` | `(str, str) -> Optional[str]` | deterministic directory-listing + string/integer comparison over an already-given path; no subprocess, no network, no file-content reads, no writes — same input tree yields same output every call | REQ-CR1, REQ-CR2, REQ-CR3, REQ-CR4, REQ-CR5 | +| `find_latest_checkpoint(runs_dir, current_run_id)` | `(str, str) -> Optional[str]` | deterministic directory-listing + name-shape filtering + string/integer comparison over an already-given path; no subprocess, no network, no file-content reads, no writes — same input tree yields same output every call | REQ-CR1, REQ-CR2, REQ-CR3, REQ-CR4, REQ-CR5, REQ-CR10 | This is the feature's ENTIRE testable surface. There is no additional pure helper to extract — the function is small and single-purpose by design (lean mode; do not add abstractions the spec does @@ -28,12 +28,21 @@ not require). | module / function | primary I/O surface | REQ traced | |---|---|---| -| `run_evolve.sh` (EXTENDED, existing script) | shells out to `"$PY_BIN"` to call `find_latest_checkpoint` against the REAL `runs_dir` (`$SKILL_DIR/runs`) and the real `$RUN_ID`; appends a log line to `$LOG`; conditionally appends `--checkpoint ""` to the existing `"$OPENEVOLVE_BIN"` invocation | REQ-CR6, REQ-CR7, REQ-CR8, REQ-CR9 | +| `lib/checkpoint_resume.py`'s `if __name__ == "__main__":` CLI entrypoint (thin wrapper, same file as the pure core, but itself effectful) | reads `RUNS_DIR`/`RUN_ID` from `os.environ` (no argv), calls `find_latest_checkpoint`, prints exactly one line to stdout (the path, or empty string for `None`) — no other output | REQ-CR6 | +| `run_evolve.sh` (EXTENDED, existing script) | invokes EXACTLY `RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py"` (mirroring `lib/ledger_reader.py:60`'s existing no-`-c`, no-argv call shape) against the REAL `runs_dir` (`$SKILL_DIR/runs`) and the real `$RUN_ID`; captures its stdout; appends a log line to `$LOG`; conditionally appends `--checkpoint ""` to the existing `"$OPENEVOLVE_BIN"` invocation | REQ-CR6, REQ-CR7, REQ-CR8, REQ-CR9 | | `openevolve-run` (external, unchanged binary, not part of this feature) | reads the checkpoint directory's own internal contents and resumes from it — entirely outside this feature's code; this feature only ever hands it a PATH | REQ-CR7 (downstream consumer only) | -The pure `find_latest_checkpoint` never itself invokes `openevolve-run` or writes to `$LOG` — those -are `run_evolve.sh`'s own effectful responsibilities, exactly mirroring how `lib/ledger_reader.py`'s -pure predicates are called from, but never call into, the shell script that logs their results. +The pure `find_latest_checkpoint` never itself invokes `openevolve-run`, writes to `$LOG`, or +touches `os.environ`/stdout — those are the CLI entrypoint's and `run_evolve.sh`'s own effectful +responsibilities, exactly mirroring how `lib/ledger_reader.py`'s pure predicates are called from, +but never call into, the `__main__` block / shell script that logs their results. + +**One committed invocation shape (resolves the ambiguity a prior review round flagged):** there is +no `"$PY_BIN" -c` alternative left open anywhere in this feature. The ONLY call site is +`RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py"` — a full- +script invocation of `lib/checkpoint_resume.py`'s own `__main__` block, taking its two inputs from +the environment (not argv, not an inline `-c` snippet), printing its one-line result to stdout. This +is pinned so two independent implementers cannot diverge on code shape. --- @@ -52,12 +61,14 @@ check that `run_evolve.sh`'s new step is actually invoked and its output actuall | PROP-CR2 | `find_latest_checkpoint` returns `None` for an empty `runs_dir`, and for a `runs_dir` containing ONLY `current_run_id`'s own directory (REQ-CR4, EDGE-CR3, EDGE-CR9) | 1 | true | pytest, `tmp_path` | | PROP-CR3 | Given two prior run directories with checkpoints, the MOST RECENT one (by lexicographic-descending `run-YYYYMMDDTHHMMSSZ` name) is selected, never the older one, even when the older one has a higher checkpoint number (REQ-CR2, EDGE-CR7) | 1 | true | pytest, `tmp_path`, parametrized names | | PROP-CR4 | Within the selected run, `checkpoint_100` is selected over `checkpoint_20` (integer comparison, not lexicographic) (REQ-CR3) | 1 | true | pytest, `tmp_path` | -| PROP-CR5 | A prior run with no `checkpoints/` subdirectory, or an empty one, is skipped in favor of the next-most-recent qualifying run (REQ-CR2, EDGE-CR4, EDGE-CR5) | 1 | true | pytest, `tmp_path` | +| PROP-CR5 | A prior run with no `checkpoints/` subdirectory, or a literally empty one, is skipped in favor of the next-most-recent qualifying run (REQ-CR2, EDGE-CR4, EDGE-CR5; contrast PROP-CR11's non-empty-but-zero-valid-entries variant, EDGE-CR11) | 1 | true | pytest, `tmp_path` | | PROP-CR6 | Non-`checkpoint_`-shaped entries inside `checkpoints/` (a stray file, a hidden dotfile, a non-numeric-suffixed directory) are ignored without raising an exception, interleaved with valid entries (REQ-CR3, EDGE-CR6) | 1 | true | pytest, `tmp_path` | | PROP-CR7 | `current_run_id` is always excluded from candidate selection, even when its own (empty) directory already exists under `runs_dir` at scan time (REQ-CR2, EDGE-CR9) | 1 | true | pytest, `tmp_path` | | PROP-CR8 | `find_latest_checkpoint` performs zero file-content reads — a `checkpoint_` directory that is empty or contains arbitrary/garbage file contents is still selected purely on its NAME/existence, never inspected further (REQ-CR5, EDGE-CR8) | 1 | true | pytest, `tmp_path` (checkpoint dirs created with no real openevolve state inside) | -| PROP-CR9 | Static source-text check: `find_latest_checkpoint`'s module source contains no `open(`, `subprocess`, `requests`, `socket`, or `urllib` reference (purity restated as an executable check, mirrors this repo's existing `gate_math.py` AST/text-scan convention) (INV-CR1) | 0 | true | pytest source-text scan | -| PROP-CR-WIRE1 | `run_evolve.sh`'s source text: the `find_latest_checkpoint` call (via `$PY_BIN`) appears BEFORE the `"$OPENEVOLVE_BIN"` invocation, and the invocation's argument list conditionally includes `--checkpoint` (REQ-CR6, REQ-CR7) | wire | true | grep/text-position check over `run_evolve.sh` | +| PROP-CR9 | Static source-text check: `find_latest_checkpoint`'s module source (`lib/checkpoint_resume.py`, excluding its own `__main__` CLI-entrypoint block, which legitimately touches `os.environ`/stdout per REQ-CR6) contains no `open(`, `subprocess`, `requests`, `socket`, or `urllib` reference — purity restated as an executable check, standing on its own: it asserts INV-CR1 directly, over this module's own source text, rather than by comparison to any other file's test (INV-CR1) | 0 | true | pytest source-text scan | +| PROP-CR10 | A `runs_dir` entry that does NOT match REQ-CR10's `^run-\d{8}T\d{6}Z$` shape (concretely, an entry named `db`, matching the real production `runs/db` sibling from `config.yaml`'s `database.db_path`) is NEVER selected as a candidate, even when it is lexicographically ordered such that a naive unfiltered sort would place it first, and even when it is given its own `checkpoints/checkpoint_1/` subdirectory as an adversarial fixture (REQ-CR10, EDGE-CR10) | 1 | true | pytest, `tmp_path` (fixture includes a non-run-shaped `db` entry alongside real `run-*` entries) | +| PROP-CR11 | A run-shaped candidate whose `checkpoints/` subdirectory is non-empty at the OS-listing level (`os.listdir` returns >= 1 entry) but contains ZERO `checkpoint_`-shaped entries (fixture: a single stray file, e.g. `.DS_Store`, and nothing else) causes `find_latest_checkpoint` to fall through to the next-most-recent run-shaped candidate (which DOES have a valid `checkpoint_` entry) and return THAT candidate's checkpoint path — never `None` and never an exception, even though the "non-empty checkpoints/ dir" test alone would wrongly treat this candidate as usable (REQ-CR2, REQ-CR3, EDGE-CR11) | 1 | true | pytest, `tmp_path`, two-run fixture (newer run: stray-file-only `checkpoints/`; older run: valid `checkpoint_/`) | +| PROP-CR-WIRE1 | `run_evolve.sh`'s source text contains the EXACT substring `RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py"` (no `-c`, no other argv), that call's line number is BEFORE the `"$OPENEVOLVE_BIN"` invocation's line number, and the invocation's argument list conditionally includes `--checkpoint` (REQ-CR6, REQ-CR7) | wire | true | grep/text-position check over `run_evolve.sh` for the exact pinned command string | | PROP-CR-WIRE2 | WHEN no checkpoint is found, the resulting `openevolve-run` argument list is IDENTICAL (same arguments, same order) to the pre-feature invocation — no `--checkpoint` flag appears (REQ-CR8, INV-CR4) | 1 | true | a synthetic re-assembly test of the argument array (bash unit check or a small harness that captures the composed command line under both branches) | | PROP-CR-WIRE3 | Every code path through the new step (found / not-found) appends exactly one line to `$LOG` before proceeding to invoke `openevolve-run` (REQ-CR9) | 1 | true | shell-level check: run the new step in isolation against both a with-checkpoint and without-checkpoint fixture, assert `$LOG` gained exactly one new line each time | | PROP-CR-LIVE1 | End-to-end: a hand-constructed two-run `tmp_path` tree (an older run with `checkpoints/checkpoint_10/` and `checkpoints/checkpoint_20/`, and a current run with no checkpoints yet) fed through `run_evolve.sh`'s new step (invoked directly, not via the full openevolve subprocess) produces the log line naming `checkpoint_20`'s path (REQ-CR6, REQ-CR9) | live (Tier 1, no real openevolve invocation required) | true | integration test against a synthetic fixture tree, per EDGE-CR1's worktree constraint below | From d81c52bfa66f5b9e2c9be8cb8921dd0637f35eb9 Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sat, 11 Jul 2026 18:25:50 +0900 Subject: [PATCH 03/11] spec: resolve spec-review iteration-2 FIND-001/FIND-002 (self-improve-checkpoint-resume) FIND-001 (blocking): INV-CR1 was untraceable to any REQ/test. Added REQ-CR11 (Group CR-SAFETY) explicitly prohibiting find_latest_checkpoint and the __main__ entrypoint from calling any filesystem-mutating API (os.remove, os.rmdir, os.mkdir, os.rename, shutil.*, non-read-only open(), etc). Added PROP-CR12: (a) expanded static denylist scan over checkpoint_resume.py source sibling to PROP-CR9, (b) dynamic tmp_path directory-snapshot before/after comparison across both None-returning and path-returning calls. FIND-002 (major): REQ-CR9 not-found branch was only line-count verified. Pinned concrete log-line wording for both branches (found: literal substring checkpoint_resume: resuming from checkpoint ; not-found: literal substring checkpoint_resume: no prior checkpoint found) and updated PROP-CR-WIRE3/PROP-CR-LIVE1 to assert actual log content, not just count. Also committing iteration-2 review output (verdict + findings) as evidence, and iteration-3 input manifest for the next review round. No phase transition; state.json currentPhase remains 1b. --- .../iteration-2/output/findings/FIND-001.json | 13 +++++ .../iteration-2/output/findings/FIND-002.json | 13 +++++ .../spec/iteration-2/output/verdict.json | 41 ++++++++++++++++ .../spec/iteration-3/input/manifest.json | 5 ++ .../specs/behavioral-spec.md | 49 ++++++++++++++++--- .../specs/verification-architecture.md | 24 +++++---- 6 files changed, 127 insertions(+), 18 deletions(-) create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/findings/FIND-001.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/findings/FIND-002.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/verdict.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/input/manifest.json diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/findings/FIND-001.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/findings/FIND-001.json new file mode 100644 index 00000000..66e278f7 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/findings/FIND-001.json @@ -0,0 +1,13 @@ +{ + "findingId": "FIND-001", + "dimension": "verification_readiness", + "category": "purity_boundary", + "severity": "blocking", + "description": "INV-CR1 (behavioral-spec.md:233) claims `find_latest_checkpoint` 'NEVER writes, deletes, renames, or mutates anything on disk' and cites REQ-CR5 as its basis -- but REQ-CR5's own text (behavioral-spec.md:185-192) only forbids reading/validating a checkpoint directory's BYTE CONTENTS ('SHALL NEVER open, read, or validate the byte contents of any file'). Nowhere does REQ-CR5 (or any other REQ-CR id) actually say the function must never CREATE, DELETE, or RENAME a directory/file -- so INV-CR1's write/delete/rename/mutate clause is an untraced addition with no REQ backing it. Compounding this, the ONLY proof obligation offered for INV-CR1 is PROP-CR9 (verification-architecture.md:68), a static source-text denylist scan for `open(`, `subprocess`, `requests`, `socket`, `urllib` references. That denylist contains zero filesystem-mutation APIs: `os.remove`, `os.unlink`, `os.rmdir`, `os.rename`, `os.replace`, `os.mkdir`, `os.makedirs`, `shutil.*`, or a pathlib `Path(...).write_text()/.unlink()/.rmdir()/.mkdir()` call would ALL pass PROP-CR9's scan untouched. No other row in the Proof Obligations table (verification-architecture.md:60-74) asserts anything about the synthetic `tmp_path` tree being byte-identical (same file list/mtimes/contents) before and after a `find_latest_checkpoint` call, and no PROP-CR-SAFE-style static scan (the convention the SAME document's own cited prior phase, self-improve-real-ledger/specs/verification-architecture.md:91 PROP-RL-SAFE2, already established for exactly this kind of no-mutation guarantee) is carried forward here. Net effect: an implementation that, say, defensively calls `os.makedirs(candidate_checkpoints_dir, exist_ok=True)` or performs a stray 'cleanup' of a malformed entry inside `checkpoints/` would satisfy every single PROP-CR1 through PROP-CR-LIVE1 obligation in this table while silently violating INV-CR1/INV-CR2 against a LIVE, launchd-shared production `runs/` tree that an actively-running ~6h recurring openevolve cycle may be reading/writing concurrently. This must be fixed before Phase 2 TDD: either (a) add an explicit REQ that grounds the write/delete/rename prohibition, plus a dynamic proof obligation (e.g. snapshot the synthetic `tmp_path` tree's full file listing + mtimes before and after every property test's call to `find_latest_checkpoint` and assert byte-identical), or (b) expand PROP-CR9's static denylist to also cover the mutation-capable API surface named above.", + "evidence": { + "filePath": ".vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "185-192, 233", + "snippet": "INV-CR1 | `find_latest_checkpoint` NEVER writes, deletes, renames, or mutates anything on disk ... (REQ-CR5) / REQ-CR5 ... SHALL NEVER open, read, or validate the byte contents of any file inside a checkpoint_ directory" + }, + "routeToPhase": "1a/1b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/findings/FIND-002.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/findings/FIND-002.json new file mode 100644 index 00000000..18345a63 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/findings/FIND-002.json @@ -0,0 +1,13 @@ +{ + "findingId": "FIND-002", + "dimension": "verification_readiness", + "category": "test_coverage", + "severity": "major", + "description": "REQ-CR9 (behavioral-spec.md:223-227) requires the log line to state 'EITHER the checkpoint path selected and used, OR that none was found' and explicitly says 'this decision SHALL NEVER be silent' -- a requirement on BOTH branches. Of the two proof obligations that touch this requirement, only PROP-CR-LIVE1 (verification-architecture.md:74) actually checks log-line CONTENT, and only for the found branch ('produces the log line naming checkpoint_20's path'). The only obligation covering the not-found branch, PROP-CR-WIRE3 (verification-architecture.md:73), checks solely that '$LOG gained exactly one new line each time' -- it makes no assertion about that line's actual text. As specified, an implementation that appends a blank line, a generic timestamp-only line, or any other content-free placeholder on the not-found branch would satisfy PROP-CR-WIRE3's line-count check while directly violating REQ-CR9's 'never silent' requirement for that branch, and no other row in the Proof Obligations table would catch it. Fix: add (or extend PROP-CR-LIVE1 to cover) a not-found-branch fixture whose expected log line is asserted to actually name the outcome (e.g. contains a recognizable 'no checkpoint found' marker), mirroring the found-branch treatment PROP-CR-LIVE1 already gives.", + "evidence": { + "filePath": ".vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md", + "lineRange": "73-74", + "snippet": "PROP-CR-WIRE3 | Every code path through the new step (found / not-found) appends exactly one line to $LOG ... | 1 | true | shell-level check ... assert $LOG gained exactly one new line each time" + }, + "routeToPhase": "1a/1b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/verdict.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/verdict.json new file mode 100644 index 00000000..9a62dfe6 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-2/output/verdict.json @@ -0,0 +1,41 @@ +{ + "feature": "self-improve-checkpoint-resume", + "reviewContext": { + "reviewType": "spec", + "phase": "1c", + "mode": "lean", + "iterationDir": "reviews/spec/iteration-2" + }, + "overallVerdict": "FAIL", + "timestamp": "2026-07-11T00:00:00Z", + "iteration": 2, + "dimensions": [ + { + "name": "spec_fidelity", + "verdict": "FAIL", + "findings": ["FIND-001"], + "evidence": [ + {"type": "file", "location": "behavioral-spec.md REQ-CR10 / EDGE-CR10 / verification-architecture.md PROP-CR10", "description": "Iteration-1 FIND-001 (runs/db sibling-directory collision, blocking) is RESOLVED: REQ-CR10 pins an explicit regex `^run-\\d{8}T\\d{6}Z$` applied unconditionally before any other filtering; EDGE-CR10 names the real `runs/db` sibling and its exclusion by name-shape alone, independent of sort order or contents; PROP-CR10 gives it a dedicated adversarial fixture (a `db` entry given its own `checkpoints/checkpoint_1/`) and asserts it is still never selected. Cross-checked live against skills/earn/self-improve/run_evolve.sh:50 (`RUN_ID=\"run-$(date -u +%Y%m%dT%H%M%SZ)\"`) and config.yaml:129 (`db_path: \"runs/db\"`) -- both citations are accurate."}, + {"type": "file", "location": "behavioral-spec.md REQ-CR2/REQ-CR3/EDGE-CR11 / verification-architecture.md PROP-CR11", "description": "Iteration-1 FIND-002 (ambiguity: non-empty-but-zero-valid-entries checkpoints/ dir, blocking) is RESOLVED: REQ-CR2 now explicitly states a REQ-CR3 None result 'for ANY reason' (no checkpoints/, empty one, or zero-valid-entries) is skipped and iteration CONTINUES to the next-most-recent candidate -- pinning fall-through (option b), never immediate None. EDGE-CR11 and PROP-CR11 (two-run fixture: newer run has stray-file-only checkpoints/, older run has a valid checkpoint_/) directly exercise this exact scenario and assert the correct fall-through outcome."}, + {"type": "grep", "location": "behavioral-spec.md, verification-architecture.md (full text scan for '-c' inline-snippet alternative)", "description": "No remaining reference to a `\"$PY_BIN\" -c` alternative invocation shape anywhere in either file; REQ-CR6 and the Purity Boundary Analysis both now state the ONE committed shape verbatim and explicitly say no alternative form is left open -- Iteration-1 FIND-003 (major) is RESOLVED."}, + {"type": "file", "location": "verification-architecture.md:68 (PROP-CR9), skills/earn/self-improve/tests/ (globbed, no import-scan test targeting gate_math.py found)", "description": "Iteration-1 FIND-004 (minor, unverified gate_math.py citation) is RESOLVED: PROP-CR9's reworded text drops the comparison entirely and stands on its own ('asserts INV-CR1 directly ... rather than by comparison to any other file's test'). Confirmed independently: no dedicated AST/text-import-scan test for gate_math.py exists among the 13 files in skills/earn/self-improve/tests/, so dropping the unverifiable precedent claim was the correct fix."}, + {"type": "file", "location": "/Users/anicca/.anicca-venvs/self-improve/lib/python3.14/site-packages/openevolve/cli.py:48-52,116-124; controller.py:462", "description": "Live re-verification: the installed openevolve==0.3.0 --checkpoint flag, its missing-path/exception-caught non-zero-exit behavior (REQ-CR7/EDGE-CR8's basis), and checkpoint_{iteration} (plain-int, no zero-padding) naming (REQ-CR3's checkpoint_ assumption) all still hold exactly as iteration-1 confirmed."} + ] + }, + { + "name": "verification_readiness", + "verdict": "FAIL", + "findings": ["FIND-001", "FIND-002"], + "evidence": [ + {"type": "file", "location": "verification-architecture.md:51-56, 88-103", "description": "Tier assignment (0/1/wire only) remains proportionate for ~20 lines of path/existence logic; Tier 2/3 correctly declined and justified by contrast with gate_math.py's own risk_adjusted_score/is_implausible_jump, which DO warrant it in that file's own spec."}, + {"type": "file", "location": "verification-architecture.md:105-113", "description": "Regression table correctly scopes the pre-existing 14-file pytest suite (confirmed via Glob: conftest.py + 13 test_*.py files) plus a bash -n smoke check; accurate, no pre-existing test imports checkpoint_resume.py (module does not exist yet, confirmed)."} + ] + } + ], + "convergenceSignals": { + "findingCount": 2, + "allCriteriaEvaluated": true, + "evaluatedCriteria": [], + "duplicateFindings": [] + } +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/input/manifest.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/input/manifest.json new file mode 100644 index 00000000..d99d842f --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/input/manifest.json @@ -0,0 +1,5 @@ +{ + "reviewType": "spec", + "artifactsToReview": ["specs/behavioral-spec.md", "specs/verification-architecture.md"], + "reviewDimensions": ["spec_fidelity", "verification_readiness"] +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md b/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md index 2a060b6d..4f033671 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md +++ b/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md @@ -104,7 +104,11 @@ is touched. open; see REQ-CR6 for the exact command line). That entrypoint takes NO argv, reads `RUNS_DIR` and `RUN_ID` from the process environment, calls `find_latest_checkpoint(runs_dir=os.environ[ "RUNS_DIR"], current_run_id=os.environ["RUN_ID"])`, and prints EXACTLY one line to stdout — the - resulting path, or an empty string when `None` — nothing else. This is a byte-for-byte mirror of + resulting path, or an empty string when `None` — nothing else. This entrypoint's own effects are + strictly limited to reading `os.environ` and writing to stdout: it NEVER itself creates, deletes, + renames, or writes bytes to any file or directory (REQ-CR11) — its only output channel is that one + stdout line; any subsequent logging of that output to `$LOG` is `run_evolve.sh`'s own + responsibility (REQ-CR9), not this Python entrypoint's. This is a byte-for-byte mirror of `lib/ledger_reader.py`'s own established convention (confirmed live, `lib/ledger_reader.py:222- 225`: no argv, reads its own env-derived config internally, prints one line of JSON to stdout; called from `run_evolve.sh:60` as `"$PY_BIN" "$SKILL_DIR/lib/ledger_reader.py"`, no `-c`, no @@ -220,17 +224,46 @@ is touched. `--checkpoint` flag at all) — i.e. this feature's fallback path is byte-for-byte the pre-existing behavior, never a degraded or different variant of it. -- **REQ-CR9** THE SYSTEM SHALL append a log line to `$LOG` (the same file/convention every other - `run_evolve.sh` step already writes to) stating EITHER the checkpoint path selected and used, OR - that none was found — this decision SHALL NEVER be silent. This restates this repo's existing - "loud, visible logging, never a silent branch" convention (every other step in `run_evolve.sh` - already follows it; REQ-CR9 makes it explicit for this feature's own new step). +- **REQ-CR9** THE SYSTEM SHALL append EXACTLY one log line to `$LOG` (the same + `echo "$(now) " >> "$LOG"` file/convention every other `run_evolve.sh` step already + writes to, e.g. `run_evolve.sh:91`'s `echo "$(now) run_id=$RUN_ID status=$STATUS" >> "$LOG"`) on + EVERY invocation of this feature's new step — this decision SHALL NEVER be silent, and its content + SHALL NEVER be merely "some line got written" (a bare line-count check is not sufficient evidence + this requirement holds; the line's TEXT is pinned below so a test can assert on it directly). + WHEN the REQ-CR6 call returns a non-empty checkpoint path, the log line's message SHALL contain the + literal substring `checkpoint_resume: resuming from checkpoint` immediately followed by the actual + resolved path. WHEN the REQ-CR6 call returns empty/`None` (no prior checkpoint found, for any + reason), the log line's message SHALL contain the literal substring + `checkpoint_resume: no prior checkpoint found` — a concrete, non-blank, informative string, not a + generic placeholder. Both wordings are pinned here, verbatim, precisely so two independent + implementers cannot diverge on wording and so a test can assert on this exact content in BOTH + branches (not just the found branch), never merely `count(lines) == 1`. + +### Group CR-SAFETY — no filesystem mutation, anywhere in this feature's code + +- **REQ-CR11** Neither `find_latest_checkpoint` NOR `lib/checkpoint_resume.py`'s `__main__` CLI + entrypoint (REQ-CR6) SHALL EVER call any filesystem-mutating API. Concretely, this module's own + source code (in both the pure function and the `__main__` block) SHALL NEVER call: `os.remove`, + `os.unlink`, `os.rmdir`, `os.removedirs`, `os.mkdir`, `os.makedirs`, `os.rename`, `os.replace`, + `os.truncate`, any `shutil.*` function (`shutil.rmtree`, `shutil.copy*`, `shutil.move`, etc.), or + `open(...)` with any mode other than the implicit read-only default (i.e. never `'w'`, `'a'`, + `'x'`, `'w+'`, or any other write/append/create/truncate mode) — nor any equivalent operation that + creates, deletes, renames, or writes bytes to a file or directory. THE ONLY filesystem operations + this module's runtime code MAY perform are READ-ONLY directory traversal: `os.listdir`, + `os.scandir`, and `os.path.*` queries (`isdir`, `isfile`, `exists`, `join`, `abspath`, etc.) — + listing directory entries and inspecting their names/existence, NEVER opening or reading the byte + contents of any file (this restates and sharpens REQ-CR5's "no file-content reads" rule to also + explicitly cover the `__main__` entrypoint, not only the pure function). The `__main__` + entrypoint's only permitted output channel is stdout (REQ-CR6); it never writes to any file. This + requirement makes INV-CR1 directly traceable to an explicit, testable REQ (closing the gap where + INV-CR1 previously had no REQ or test enforcing it) and is verified by PROP-CR12 (static + denylist scan + dynamic before/after directory-snapshot comparison). ## Global Invariants | # | Invariant | |---|---| -| INV-CR1 | `find_latest_checkpoint` NEVER writes, deletes, renames, or mutates anything on disk — it is a pure read-only path/existence query (REQ-CR5) | +| INV-CR1 | `find_latest_checkpoint` AND `lib/checkpoint_resume.py`'s `__main__` entrypoint NEVER write, delete, rename, or mutate anything on disk — both are pure/read-only with respect to the filesystem (REQ-CR5, REQ-CR11) | | INV-CR2 | This feature never touches `strategies/pm_backtest_strategy.py`, `promote_gate.sh`, `lib/promote_gate.py`, `lib/promote.py`, `config.yaml`, or `ai.anicca.self-improve-evolve.plist` | | INV-CR3 | This feature never reads or writes any wallet key, `.env`, ledger file, or spend-cap value — it is scoped exclusively to `runs/*/checkpoints/*` path selection | | INV-CR4 | The fallback path (no checkpoint found) is byte-for-byte identical, in its `openevolve-run` invocation, to `run_evolve.sh`'s pre-feature behavior (REQ-CR8) — this feature can only ADD a `--checkpoint` argument, never change any other existing argument | @@ -301,7 +334,7 @@ is touched. | dimension | condition | |---|---| | spec | this document + verification-architecture.md | -| test | RED: unit tests for `find_latest_checkpoint` covering REQ-CR1-5, REQ-CR10, and EDGE-CR1-11 all written and failing (module does not yet exist) before any implementation; GREEN: all passing | +| test | RED: unit tests for `find_latest_checkpoint` covering REQ-CR1-5, REQ-CR10, REQ-CR11, and EDGE-CR1-11 all written and failing (module does not yet exist) before any implementation; GREEN: all passing | | impl | `lib/checkpoint_resume.py::find_latest_checkpoint` + `run_evolve.sh`'s new pre-invocation step (REQ-CR6-9), both present and runnable | | impl review | fresh-context `vcsdd-adversary` review of both files returns PASS (lean mode: no BLOCKING findings) | | verification | Phase 5: proof obligations in verification-architecture.md discharged; a real, hand-constructed two-run `tmp_path` fixture demonstrates `run_evolve.sh`'s new step selecting and logging the correct `--checkpoint` path end-to-end (or a documented reason it could not be exercised without touching the production `runs/` tree, per EDGE-CR1) | diff --git a/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md b/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md index 12130a57..83338ffd 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md +++ b/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md @@ -18,7 +18,7 @@ sources: | function | signature | why it is pure | REQ traced | |---|---|---|---| -| `find_latest_checkpoint(runs_dir, current_run_id)` | `(str, str) -> Optional[str]` | deterministic directory-listing + name-shape filtering + string/integer comparison over an already-given path; no subprocess, no network, no file-content reads, no writes — same input tree yields same output every call | REQ-CR1, REQ-CR2, REQ-CR3, REQ-CR4, REQ-CR5, REQ-CR10 | +| `find_latest_checkpoint(runs_dir, current_run_id)` | `(str, str) -> Optional[str]` | deterministic directory-listing + name-shape filtering + string/integer comparison over an already-given path; no subprocess, no network, no file-content reads, no writes — same input tree yields same output every call | REQ-CR1, REQ-CR2, REQ-CR3, REQ-CR4, REQ-CR5, REQ-CR10, REQ-CR11 | This is the feature's ENTIRE testable surface. There is no additional pure helper to extract — the function is small and single-purpose by design (lean mode; do not add abstractions the spec does @@ -28,7 +28,7 @@ not require). | module / function | primary I/O surface | REQ traced | |---|---|---| -| `lib/checkpoint_resume.py`'s `if __name__ == "__main__":` CLI entrypoint (thin wrapper, same file as the pure core, but itself effectful) | reads `RUNS_DIR`/`RUN_ID` from `os.environ` (no argv), calls `find_latest_checkpoint`, prints exactly one line to stdout (the path, or empty string for `None`) — no other output | REQ-CR6 | +| `lib/checkpoint_resume.py`'s `if __name__ == "__main__":` CLI entrypoint (thin wrapper, same file as the pure core, but itself effectful) | reads `RUNS_DIR`/`RUN_ID` from `os.environ` (no argv), calls `find_latest_checkpoint`, prints exactly one line to stdout (the path, or empty string for `None`) — no other output; NEVER creates, deletes, renames, or writes bytes to any file or directory | REQ-CR6, REQ-CR11 | | `run_evolve.sh` (EXTENDED, existing script) | invokes EXACTLY `RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py"` (mirroring `lib/ledger_reader.py:60`'s existing no-`-c`, no-argv call shape) against the REAL `runs_dir` (`$SKILL_DIR/runs`) and the real `$RUN_ID`; captures its stdout; appends a log line to `$LOG`; conditionally appends `--checkpoint ""` to the existing `"$OPENEVOLVE_BIN"` invocation | REQ-CR6, REQ-CR7, REQ-CR8, REQ-CR9 | | `openevolve-run` (external, unchanged binary, not part of this feature) | reads the checkpoint directory's own internal contents and resumes from it — entirely outside this feature's code; this feature only ever hands it a PATH | REQ-CR7 (downstream consumer only) | @@ -68,10 +68,11 @@ check that `run_evolve.sh`'s new step is actually invoked and its output actuall | PROP-CR9 | Static source-text check: `find_latest_checkpoint`'s module source (`lib/checkpoint_resume.py`, excluding its own `__main__` CLI-entrypoint block, which legitimately touches `os.environ`/stdout per REQ-CR6) contains no `open(`, `subprocess`, `requests`, `socket`, or `urllib` reference — purity restated as an executable check, standing on its own: it asserts INV-CR1 directly, over this module's own source text, rather than by comparison to any other file's test (INV-CR1) | 0 | true | pytest source-text scan | | PROP-CR10 | A `runs_dir` entry that does NOT match REQ-CR10's `^run-\d{8}T\d{6}Z$` shape (concretely, an entry named `db`, matching the real production `runs/db` sibling from `config.yaml`'s `database.db_path`) is NEVER selected as a candidate, even when it is lexicographically ordered such that a naive unfiltered sort would place it first, and even when it is given its own `checkpoints/checkpoint_1/` subdirectory as an adversarial fixture (REQ-CR10, EDGE-CR10) | 1 | true | pytest, `tmp_path` (fixture includes a non-run-shaped `db` entry alongside real `run-*` entries) | | PROP-CR11 | A run-shaped candidate whose `checkpoints/` subdirectory is non-empty at the OS-listing level (`os.listdir` returns >= 1 entry) but contains ZERO `checkpoint_`-shaped entries (fixture: a single stray file, e.g. `.DS_Store`, and nothing else) causes `find_latest_checkpoint` to fall through to the next-most-recent run-shaped candidate (which DOES have a valid `checkpoint_` entry) and return THAT candidate's checkpoint path — never `None` and never an exception, even though the "non-empty checkpoints/ dir" test alone would wrongly treat this candidate as usable (REQ-CR2, REQ-CR3, EDGE-CR11) | 1 | true | pytest, `tmp_path`, two-run fixture (newer run: stray-file-only `checkpoints/`; older run: valid `checkpoint_/`) | +| PROP-CR12 | Filesystem-mutation prohibition (REQ-CR11, INV-CR1), verified BOTH ways: **(a) static** — `lib/checkpoint_resume.py`'s FULL module source (including its `__main__` block; only its legitimate `os.environ` read and stdout write are exempt) contains no reference to `os.remove`, `os.unlink`, `os.rmdir`, `os.removedirs`, `os.mkdir`, `os.makedirs`, `os.rename`, `os.replace`, `os.truncate`, any `shutil.` attribute access, or `open(` used with a non-default (write/append/create/truncate) mode — this expands PROP-CR9's network/subprocess-only denylist to also cover filesystem-mutation APIs. **(b) dynamic** — snapshot a synthetic `tmp_path` `runs/` tree (a recursive listing of every path plus its `os.stat` mtime, or a full byte-for-byte directory copy) BEFORE calling `find_latest_checkpoint` at least twice against it (once where it returns `None` — e.g. an empty tree — and once where it returns a real checkpoint path — e.g. a populated two-run fixture), then re-snapshot the SAME tree AFTER both calls and assert it is byte-for-byte/mtime-for-mtime identical to the BEFORE snapshot. This closes the gap a static-only or unit-test-only check cannot: a function could pass every REQ-CR1-10 test above by returning the right paths while ALSO, incidentally, mutating disk — (a) catches that via source inspection, (b) catches it via observed behavior, independently of each other. | 0/1 | true | pytest source-text scan (a) + pytest `tmp_path` before/after directory-tree snapshot comparison (b) | | PROP-CR-WIRE1 | `run_evolve.sh`'s source text contains the EXACT substring `RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py"` (no `-c`, no other argv), that call's line number is BEFORE the `"$OPENEVOLVE_BIN"` invocation's line number, and the invocation's argument list conditionally includes `--checkpoint` (REQ-CR6, REQ-CR7) | wire | true | grep/text-position check over `run_evolve.sh` for the exact pinned command string | | PROP-CR-WIRE2 | WHEN no checkpoint is found, the resulting `openevolve-run` argument list is IDENTICAL (same arguments, same order) to the pre-feature invocation — no `--checkpoint` flag appears (REQ-CR8, INV-CR4) | 1 | true | a synthetic re-assembly test of the argument array (bash unit check or a small harness that captures the composed command line under both branches) | -| PROP-CR-WIRE3 | Every code path through the new step (found / not-found) appends exactly one line to `$LOG` before proceeding to invoke `openevolve-run` (REQ-CR9) | 1 | true | shell-level check: run the new step in isolation against both a with-checkpoint and without-checkpoint fixture, assert `$LOG` gained exactly one new line each time | -| PROP-CR-LIVE1 | End-to-end: a hand-constructed two-run `tmp_path` tree (an older run with `checkpoints/checkpoint_10/` and `checkpoints/checkpoint_20/`, and a current run with no checkpoints yet) fed through `run_evolve.sh`'s new step (invoked directly, not via the full openevolve subprocess) produces the log line naming `checkpoint_20`'s path (REQ-CR6, REQ-CR9) | live (Tier 1, no real openevolve invocation required) | true | integration test against a synthetic fixture tree, per EDGE-CR1's worktree constraint below | +| PROP-CR-WIRE3 | Every code path through the new step (found / not-found) appends exactly one line to `$LOG` before proceeding to invoke `openevolve-run`, AND that line's CONTENT (not merely its count) is asserted in BOTH branches: the found-branch line contains the literal substring `checkpoint_resume: resuming from checkpoint` followed by the actual selected path; the not-found-branch line contains the literal substring `checkpoint_resume: no prior checkpoint found` (REQ-CR9) | 1 | true | shell-level check: run the new step in isolation against both a with-checkpoint and without-checkpoint fixture, assert `$LOG` gained exactly one new line each time AND that the new line's text contains the required substring for that branch — never merely `count(lines) == 1` | +| PROP-CR-LIVE1 | End-to-end: a hand-constructed two-run `tmp_path` tree (an older run with `checkpoints/checkpoint_10/` and `checkpoints/checkpoint_20/`, and a current run with no checkpoints yet) fed through `run_evolve.sh`'s new step (invoked directly, not via the full openevolve subprocess) produces a log line containing the literal substring `checkpoint_resume: resuming from checkpoint` immediately followed by `checkpoint_20`'s path (REQ-CR6, REQ-CR9) | live (Tier 1, no real openevolve invocation required) | true | integration test against a synthetic fixture tree, per EDGE-CR1's worktree constraint below | **Execution locus note (mirrors the prior phase's EDGE-RL5a pattern, EDGE-CR1):** this worktree's `skills/earn/self-improve/.gitignore` ignores `runs/` and `state/` — a fresh checkout has no @@ -86,15 +87,18 @@ directory after merge), never fabricated inside a worktree. ## Verification Strategy - **Tier 0**: `find_latest_checkpoint`'s trivial no-input-tree fallback cases (missing/empty - `runs_dir`) and the static purity source-scan — no formal proof needed, a single assertion each - settles them. + `runs_dir`) and PROP-CR12(a)'s static filesystem-mutation-API source-text denylist scan — no + formal proof needed, a single assertion each settles them. - **Tier 1**: the bulk of this feature's proof burden — deterministic unit/property tests over synthetic `tmp_path` directory trees covering every REQ-CR/EDGE-CR combination (recency ordering, integer-vs-lexicographic checkpoint comparison, malformed-entry tolerance, self-exclusion, - no-file-content-read). This is standard `pytest`, matching this repo's existing convention for - `gate_math.py`/`ledger_reader.py` pure-function tests — no `hypothesis`/`kani`/`fast-check` - property-fuzzing harness is introduced for ~20 lines of path logic (lean mode: do not - over-specify formal machinery this feature's size does not warrant). + no-file-content-read), PLUS PROP-CR12(b)'s dynamic before/after directory-snapshot comparison + (REQ-CR11) and PROP-CR-WIRE3/PROP-CR-LIVE1's log-content substring assertions (REQ-CR9) — none of + these are "count-only" or "ran without crashing" checks; each asserts a specific, pinned piece of + content (a byte-identical tree, or a literal log substring). This is standard `pytest`, matching + this repo's existing convention for `gate_math.py`/`ledger_reader.py` pure-function tests — no + `hypothesis`/`kani`/`fast-check` property-fuzzing harness is introduced for ~20 lines of path logic + (lean mode: do not over-specify formal machinery this feature's size does not warrant). - **Tier 2**: not used. This feature has no numeric/financial invariant, no concurrency, and no security-sensitive parsing surface that would justify lightweight formal methods (contrast with `gate_math.py`'s `is_implausible_jump`/reward-hacking trip-wire math in the parent harness, which From 8da70c31ab46686057de1a66c7ec6857d93a1332 Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sat, 11 Jul 2026 18:36:38 +0900 Subject: [PATCH 04/11] spec: self-improve-checkpoint-resume iteration-3 fixes (REQ-CR12 crash handling, REQ-CR11 pathlib/symlink hardening, REQ-CR1 absolute-path normalization, REQ-CR7 found-branch arg stability proof) --- .../iteration-3/output/findings/FIND-001.json | 20 +++++ .../iteration-3/output/findings/FIND-002.json | 20 +++++ .../iteration-3/output/findings/FIND-003.json | 20 +++++ .../iteration-3/output/findings/FIND-004.json | 13 +++ .../spec/iteration-3/output/verdict.json | 57 +++++++++++++ .../spec/iteration-4/input/manifest.json | 5 ++ .../specs/behavioral-spec.md | 81 +++++++++++++------ .../specs/verification-architecture.md | 5 +- 8 files changed, 197 insertions(+), 24 deletions(-) create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-001.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-002.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-003.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-004.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/verdict.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/input/manifest.json diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-001.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-001.json new file mode 100644 index 00000000..d354f90b --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-001.json @@ -0,0 +1,20 @@ +{ + "findingId": "FIND-001", + "dimension": "spec_fidelity", + "category": "spec_gap", + "severity": "high", + "description": "REQ-CR6/REQ-CR8/REQ-CR9 (behavioral-spec.md:200-240) collectively define exactly two outcomes for the new checkpoint-resume step: 'the REQ-CR6 call returns a non-empty checkpoint path' (REQ-CR7) and 'the REQ-CR6 call returns empty/None' (REQ-CR8), with REQ-CR9 requiring one of two fixed log substrings depending on which of those two outcomes fired. No requirement, and no EDGE-CR, addresses a third, distinct outcome: the CLI entrypoint invocation itself fails to complete cleanly (an uncaught Python exception inside checkpoint_resume.py, `$PY_BIN` missing/not executable, a syntax error in the module). REQ-CR6 only specifies 'run_evolve.sh SHALL capture this call's stdout verbatim as the checkpoint path' -- it says nothing about the invocation's exit code or its stderr. A crashed invocation necessarily produces empty captured stdout (nothing is printed before a mid-function exception, and command-not-found also yields empty $() output), and REQ-CR8's own wording -- 'WHEN the REQ-CR6 call returns empty/None (no prior checkpoint found, for ANY reason ...)' -- explicitly folds a crash into the SAME branch as a genuine, expected cold start, which REQ-CR9 then requires logging as the literal string 'checkpoint_resume: no prior checkpoint found'. This is indistinguishable, in the one artifact an unsupervised operator would ever look at ($LOG), from ordinary first-run/no-checkpoint behavior -- masking a real bug or environment breakage (e.g. a permissions error walking runs_dir, a broken/rotated venv) as harmless steady-state, in a script that fires autonomously on a ~6h launchd timer with no human in the loop. This is not a hypothetical concern for this specific harness: the SAME file already contains a precedent for exactly this failure mode -- the pre-existing OBSERVE step explicitly guards `if [ -x \"$PY_BIN\" ]` and emits a DISTINCT log message ('OBSERVE SKIPPED (python not found at $PY_BIN) -- proceeding to EVOLVE anyway') specifically so a broken interpreter is never silently reinterpreted as 'nothing to observe' (run_evolve.sh:59-64). REQ-CR6 claims to be 'a byte-for-byte mirror of lib/ledger_reader.py's own established convention,' but on this one dimension -- distinguishing 'call failed' from 'call succeeded and returned a negative/empty answer' -- it does not carry that safeguard over, and no requirement pins whether run_evolve.sh must add an equivalent guard or a third, distinct log message for this new step. Two independent implementers could diverge here: one lets a checkpoint_resume.py crash go completely unremarked (falls straight into the 'no prior checkpoint found' log line), another adds its own extra guard/log line -- both are consistent with the letter of REQ-CR6/CR8/CR9 as currently written.", + "evidence": { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "200-240", + "snippet": "REQ-CR6: '...run_evolve.sh SHALL capture this call's stdout verbatim as the checkpoint path (an empty string meaning \"no checkpoint found,\" per REQ-CR8).' / REQ-CR8: 'WHEN the REQ-CR6 call returns empty/None (no prior checkpoint found, for ANY reason -- first-ever run, all prior runs lack a checkpoints/ dir, or runs_dir itself is missing) ...' -- no branch for 'the call itself failed to run'." + }, + "groundTruthEvidence": [ + { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/skills/earn/self-improve/run_evolve.sh", + "lineRange": "59-64", + "snippet": "if [ -x \"$PY_BIN\" ]; then\n OBSERVE_JSON=\"$(\"$PY_BIN\" \"$SKILL_DIR/lib/ledger_reader.py\" 2>>\"$LOG\")\"\n echo \"$(now) OBSERVE realized_ledger=${OBSERVE_JSON:-'{\"error\":\"ledger_reader produced no output\"}'}\" >> \"$LOG\"\nelse\n echo \"$(now) OBSERVE SKIPPED (python not found at $PY_BIN) -- proceeding to EVOLVE anyway\" >> \"$LOG\"\nfi" + } + ], + "routeToPhase": "1a" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-002.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-002.json new file mode 100644 index 00000000..81ef9b6a --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-002.json @@ -0,0 +1,20 @@ +{ + "findingId": "FIND-002", + "dimension": "verification_readiness", + "category": "purity_boundary", + "severity": "high", + "description": "REQ-CR11 (behavioral-spec.md:244-260) itself states the requirement in a strong ALLOWLIST form: 'THE ONLY filesystem operations this module's runtime code MAY perform are READ-ONLY directory traversal: os.listdir, os.scandir, and os.path.* queries.' The Proof Obligation this requirement names as its own verification (PROP-CR12, verification-architecture.md:71) implements only the (a) static half of this as a NEGATIVE DENYLIST scan -- grepping the module source for the absence of the literal names os.remove/os.unlink/os.rmdir/os.removedirs/os.mkdir/os.makedirs/os.rename/os.replace/os.truncate, any shutil.* attribute, and open(...) in a non-default mode. This denylist omits an entire class of filesystem-mutating APIs that create/delete/rename/write bytes just as effectively as the named ones: pathlib.Path mutating methods (.mkdir(), .unlink(), .rmdir(), .rename(), .replace(), .write_text(), .write_bytes(), .touch(), Path.open('w', ...)), and os.symlink/os.link (both literally CREATE a filesystem entry, matching REQ-CR11's own 'creates ... a file or directory' clause verbatim). A conforming implementation that used pathlib for its read-side logic (e.g. Path(runs_dir).iterdir()/glob()) and, on some untested error-handling branch, called Path(...).mkdir() or os.symlink(...) would pass PROP-CR12(a)'s static scan cleanly -- none of its denylisted literal strings appear in the source -- while directly violating REQ-CR11's own allowlist prose. PROP-CR12(b)'s dynamic before/after directory-snapshot comparison is a real, independent, API-agnostic safety net, but it only observes the SPECIFIC code paths the two named fixtures exercise (an empty tree returning None; a populated two-run tree returning a real path) -- a mutation on a branch neither fixture reaches (e.g. an exception handler, or a third code path introduced later) would escape both PROP-CR12(a) and PROP-CR12(b). Given that INV-CR1/REQ-CR11 is this feature's single money-adjacent safety claim (the entire reason FIND-001 from the prior review iteration required an explicit REQ+PROP in the first place), the static half of its own designated verification mechanism does not actually discharge the requirement as written -- it discharges a strictly narrower one (absence of ~10 named os/shutil identifiers), not 'the only filesystem ops this module may perform are 3 named read-only forms.'", + "evidence": { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md", + "lineRange": "71", + "snippet": "PROP-CR12 | ... (a) static -- lib/checkpoint_resume.py's FULL module source ... contains no reference to os.remove, os.unlink, os.rmdir, os.removedirs, os.mkdir, os.makedirs, os.rename, os.replace, os.truncate, any shutil. attribute access, or open( used with a non-default ... mode ..." + }, + "groundTruthEvidence": [ + { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "251-256", + "snippet": "THE ONLY filesystem operations this module's runtime code MAY perform are READ-ONLY directory traversal: os.listdir, os.scandir, and os.path.* queries (isdir, isfile, exists, join, abspath, etc.) -- listing directory entries and inspecting their names/existence, NEVER opening or reading the byte contents of any file" + } + ], + "routeToPhase": "1b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-003.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-003.json new file mode 100644 index 00000000..f2288996 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-003.json @@ -0,0 +1,20 @@ +{ + "findingId": "FIND-003", + "dimension": "verification_readiness", + "category": "test_coverage", + "severity": "medium", + "description": "REQ-CR7 (behavioral-spec.md:211-219) requires that when a checkpoint IS found, run_evolve.sh appends --checkpoint \"\" while leaving 'the EXISTING openevolve-run invocation (unchanged otherwise: same initial_program, evaluation_file, --config, --iterations, --output arguments, in the same order).' No Proof Obligation in the table actually asserts this for the FOUND branch. PROP-CR-WIRE1 (verification-architecture.md:72) is a static grep/text-position check that only confirms (a) the exact checkpoint_resume.py call substring is present, (b) that call's line number precedes the openevolve-run call's line number, and (c) that 'the invocation's argument list conditionally includes --checkpoint' -- a static grep cannot actually verify a runtime conditional-inclusion property, and the table names no additional mechanism for it. It does not pin down that the other five pre-existing arguments remain present, byte-identical, and in the same relative order once --checkpoint is appended. PROP-CR-WIRE2 (verification-architecture.md:73) is explicitly scoped to the NOT-found branch only ('WHEN no checkpoint is found, the resulting openevolve-run argument list is IDENTICAL ... to the pre-feature invocation'). There is no found-branch counterpart to PROP-CR-WIRE2 anywhere in the table. An implementation that, when appending --checkpoint, accidentally drops --iterations, reorders --output ahead of --config, or duplicates an existing flag would satisfy every currently-listed Proof Obligation (PROP-CR1 through PROP-CR-LIVE1) while directly violating REQ-CR7's 'unchanged otherwise ... same order' clause.", + "evidence": { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md", + "lineRange": "72-73", + "snippet": "PROP-CR-WIRE1 | ... the invocation's argument list conditionally includes --checkpoint (REQ-CR6, REQ-CR7) | wire | true | grep/text-position check ... // PROP-CR-WIRE2 | WHEN no checkpoint is found, the resulting openevolve-run argument list is IDENTICAL ... (REQ-CR8, INV-CR4) | 1 | true | a synthetic re-assembly test ..." + }, + "groundTruthEvidence": [ + { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "211-219", + "snippet": "REQ-CR7 ... THE SYSTEM SHALL append exactly one additional argument pair, --checkpoint \"\", to the EXISTING openevolve-run invocation (unchanged otherwise: same initial_program, evaluation_file, --config, --iterations, --output arguments, in the same order)" + } + ], + "routeToPhase": "1b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-004.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-004.json new file mode 100644 index 00000000..83d9d7b8 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/findings/FIND-004.json @@ -0,0 +1,13 @@ +{ + "findingId": "FIND-004", + "dimension": "spec_fidelity", + "category": "spec_gap", + "severity": "low", + "description": "REQ-CR1 (behavioral-spec.md:125-127) declares find_latest_checkpoint's return contract as 'the absolute path to a checkpoint directory,' but no requirement states whether the function itself must normalize its result via os.path.abspath independent of whether the caller-supplied runs_dir argument is itself absolute, versus 'absolute' being merely an inherited property that happens to hold because run_evolve.sh's own RUNS_DIR (derived via `cd ... && pwd`) is always absolute in production. REQ-CR2/REQ-CR3's own selection-algorithm text never mentions abspath normalization. Every Proof Obligation that exercises this function (PROP-CR1 through PROP-CR11) is built over pytest's tmp_path fixture, which is always already an absolute path -- so no test in the current table would distinguish an implementation that actively normalizes its result from one that merely echoes back whatever path shape the caller supplied (e.g. one that would silently return a relative path if ever called with a relative runs_dir). This is a minor, non-safety-affecting ambiguity given the one real production call site, but it means the function's own stated 'absolute path' contract is accidentally, not verifiably, true.", + "evidence": { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "125-149", + "snippet": "REQ-CR1 ... find_latest_checkpoint(runs_dir: str, current_run_id: str) -> Optional[str] that returns the absolute path to a checkpoint directory to resume from, or None ..." + }, + "routeToPhase": "1b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/verdict.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/verdict.json new file mode 100644 index 00000000..c2ecbd60 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/output/verdict.json @@ -0,0 +1,57 @@ +{ + "sprintNumber": null, + "feature": "self-improve-checkpoint-resume", + "reviewType": "spec", + "phase": "1c", + "overallVerdict": "FAIL", + "timestamp": "2026-07-11T00:00:00Z", + "iteration": 3, + "reviewContext": { + "targetSpec": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "targetVerificationArchitecture": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md", + "manifest": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-3/input/manifest.json (reviewDimensions restricted to spec_fidelity + verification_readiness)", + "crossCheckedImplementationContext": [ + "skills/earn/self-improve/run_evolve.sh (full file read)", + "skills/earn/self-improve/lib/ledger_reader.py (full file read -- the module this feature explicitly claims to mirror)", + "skills/earn/self-improve/config.yaml (full file read -- confirms checkpoint_interval/db_path claims)", + "skills/earn/self-improve/lib/gate_math.py (full file read -- confirms this repo's pure-core/no-I/O convention)" + ], + "priorIterationFindingsStatus": { + "FIND-001 (iteration 2, blocking -- INV-CR1 untraceable to any enforcing REQ/test)": "RESOLVED -- REQ-CR11 (behavioral-spec.md:244-260) now explicitly names the exact denylisted API surface for BOTH the pure function and the __main__ entrypoint, and states 'This requirement makes INV-CR1 directly traceable to an explicit, testable REQ ... and is verified by PROP-CR12.' PROP-CR12 (verification-architecture.md:71) is a new, dedicated proof obligation (static denylist scan + dynamic tmp_path snapshot-before/after). Traceability gap closed. HOWEVER a residual, narrower defect in this exact mechanism is found this iteration: see FIND-002 (PROP-CR12(a)'s static half is a denylist, not the allowlist REQ-CR11 itself claims, and misses pathlib/os.symlink/os.link mutation paths).", + "FIND-002 (iteration 2, major -- not-found log branch only line-count-checked)": "RESOLVED -- REQ-CR9 (behavioral-spec.md:227-240) now pins literal substrings for BOTH branches verbatim ('checkpoint_resume: resuming from checkpoint' / 'checkpoint_resume: no prior checkpoint found') and explicitly states 'a bare line-count check is not sufficient evidence.' PROP-CR-WIRE3 (verification-architecture.md:74) and PROP-CR-LIVE1 (verification-architecture.md:75) were both updated to assert on these exact substrings in both branches, not merely count(lines)==1. No residual defect found in this specific fix." + }, + "newFindingsThisIteration": ["FIND-001", "FIND-002", "FIND-003", "FIND-004"] + }, + "dimensions": [ + { + "name": "spec_fidelity", + "verdict": "FAIL", + "findings": ["FIND-001", "FIND-004"], + "evidence": [ + {"type": "file", "location": "specs/behavioral-spec.md:227-240", "description": "REQ-CR9's iteration-2 fix (pinned literal log substrings for both branches) verified present and correctly worded -- no finding raised on this specific point."}, + {"type": "file", "location": "specs/behavioral-spec.md:90-119", "description": "Purity Boundary Analysis section correctly identifies the pure core as find_latest_checkpoint only, with the __main__ CLI entrypoint and run_evolve.sh both explicitly effectful -- no hidden coupling found in this section."}, + {"type": "file", "location": "specs/behavioral-spec.md (frontmatter, integration.does_not_replace_or_modify)", "description": "Confirmed this feature's own scope statement excludes any live-money path, wallet key, .env, ledger file, spend cap, and the launchd plist -- cross-checked against run_evolve.sh's real OPENEVOLVE_BIN/promote_gate.sh invocations (run_evolve.sh:82-110), which this feature's spec correctly never touches. INV-CR3/INV-CR2 remain true of the current spec text: this is still a paper-only backtest-infrastructure change with no live-money surface."} + ] + }, + { + "name": "verification_readiness", + "verdict": "FAIL", + "findings": ["FIND-002", "FIND-003"], + "evidence": [ + {"type": "file", "location": "specs/verification-architecture.md:58-70", "description": "PROP-CR1 through PROP-CR11 verified individually traceable to a specific REQ-CR/EDGE-CR combination each, with concrete (non-tautological) assertions -- e.g. PROP-CR4/PROP-CR11 specifically construct adversarial fixtures (integer-vs-lexicographic checkpoint numbering, a stray .DS_Store-only checkpoints/ dir) rather than happy-path-only cases. No finding raised on PROP-CR1-CR11's own individual content."}, + {"type": "file", "location": "specs/verification-architecture.md:109-117", "description": "Regression Table correctly scopes the pre-existing 14-file pytest suite as required-green and correctly notes no existing test imports the new module -- accurate, grounded claim (checkpoint_resume.py does not exist yet, confirmed via direct filesystem check, so it cannot be imported by any pre-existing test)."}, + {"type": "file", "location": "specs/verification-architecture.md:87-107", "description": "Tier 0/1-only classification (no Tier 2/3) is a defensible, non-inflated choice for a ~20-line path-selection function with no numeric/financial invariant and no concurrency -- correctly contrasted against gate_math.py's own Tier-2-warranting risk_adjusted_score/is_implausible_jump in the same Verification Strategy section."} + ] + } + ], + "blockingFindings": ["FIND-001", "FIND-002"], + "convergenceSignals": { + "findingCount": 4, + "allCriteriaEvaluated": true, + "evaluatedCriteria": [], + "duplicateFindings": [], + "iteration2ResolvedCount": 2, + "iteration2NotFixedCount": 0, + "newFindingsThisIteration": 4 + } +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/input/manifest.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/input/manifest.json new file mode 100644 index 00000000..d99d842f --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/input/manifest.json @@ -0,0 +1,5 @@ +{ + "reviewType": "spec", + "artifactsToReview": ["specs/behavioral-spec.md", "specs/verification-architecture.md"], + "reviewDimensions": ["spec_fidelity", "verification_readiness"] +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md b/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md index 4f033671..05c85715 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md +++ b/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md @@ -124,7 +124,13 @@ is touched. - **REQ-CR1** `lib/checkpoint_resume.py` SHALL expose a PURE function `find_latest_checkpoint( runs_dir: str, current_run_id: str) -> Optional[str]` that returns the absolute path to a - checkpoint directory to resume from, or `None` when no prior run has any usable checkpoint. + checkpoint directory to resume from, or `None` when no prior run has any usable checkpoint. This + function SHALL itself normalize its return value to an absolute path via `os.path.abspath` (never + relying on the caller happening to pass an already-absolute `runs_dir`) — i.e. calling it with a + RELATIVE `runs_dir` (e.g. `"runs"`, resolved against the current working directory) SHALL still + return an absolute path string, not a relative one, making REQ-CR1's "absolute path" contract true + by construction rather than true only by accident of production's `RUNS_DIR` always being + absolute. - **REQ-CR2** WHEN selecting which PRIOR RUN to resume from, THE SYSTEM SHALL: list the immediate entries of `runs_dir`, filter that list down to ONLY entries whose name matches REQ-CR10's @@ -206,7 +212,24 @@ is touched. `RUN_ID` = the just-created `$RUN_ID`, correctly excluding itself even though its own now-empty directory already exists under `runs/`) and BEFORE the existing `"$OPENEVOLVE_BIN"` invocation. `run_evolve.sh` SHALL capture this call's stdout verbatim as the checkpoint path (an empty string - meaning "no checkpoint found," per REQ-CR8). + meaning "no checkpoint found," per REQ-CR8). `run_evolve.sh` SHALL also capture this call's own + exit status separately from its stdout (mirroring the existing pattern at `run_evolve.sh:58-64`'s + `OBSERVE_JSON="$(...)"` capture, which does not abort on a non-zero `ledger_reader.py` exit + either) — see REQ-CR12 for what happens on a non-zero exit here. + +- **REQ-CR12** WHEN the REQ-CR6 call itself exits non-zero (the `checkpoint_resume.py` process + crashed, raised an uncaught exception, or `$PY_BIN` was not executable) — as DISTINCT from the + call succeeding and simply returning an empty string — THE SYSTEM SHALL treat this exactly like + the REQ-CR8 "no checkpoint found" fallback for the PURPOSES of the `openevolve-run` invocation + (no `--checkpoint` flag; run proceeds as a cold start — resuming is a pure optimization to this + harness, never a precondition for it to run at all, the same "degrade, never block" posture the + existing OBSERVE/`ledger_reader.py` step already takes at `run_evolve.sh:58-64`), BUT the log + line this case produces (per REQ-CR9) SHALL be a THIRD, distinct wording — the literal substring + `checkpoint_resume: resume-check crashed` followed by the captured exit status — so a crashed + lookup is never silently conflated with, or indistinguishable in the log from, an ordinary + "no prior run exists yet" cold start. This is the one and only new failure-handling branch + REQ-CR7 defers to; it governs solely the REQ-CR6 helper call's own exit status, never + `openevolve-run`'s (that remains REQ-OE6's existing concern, unchanged). - **REQ-CR7** WHEN the REQ-CR6 call returns a non-empty checkpoint path, THE SYSTEM SHALL append exactly one additional argument pair, `--checkpoint ""`, to the EXISTING `openevolve-run` @@ -230,14 +253,19 @@ is touched. EVERY invocation of this feature's new step — this decision SHALL NEVER be silent, and its content SHALL NEVER be merely "some line got written" (a bare line-count check is not sufficient evidence this requirement holds; the line's TEXT is pinned below so a test can assert on it directly). - WHEN the REQ-CR6 call returns a non-empty checkpoint path, the log line's message SHALL contain the - literal substring `checkpoint_resume: resuming from checkpoint` immediately followed by the actual - resolved path. WHEN the REQ-CR6 call returns empty/`None` (no prior checkpoint found, for any - reason), the log line's message SHALL contain the literal substring - `checkpoint_resume: no prior checkpoint found` — a concrete, non-blank, informative string, not a - generic placeholder. Both wordings are pinned here, verbatim, precisely so two independent - implementers cannot diverge on wording and so a test can assert on this exact content in BOTH - branches (not just the found branch), never merely `count(lines) == 1`. + There are exactly THREE mutually-exclusive branches, each with its own pinned, verbatim substring + (so two independent implementers cannot diverge on wording, and a test can assert on this exact + content in ALL THREE branches, never merely `count(lines) == 1`): + 1. REQ-CR6 call exits zero AND returns a non-empty checkpoint path: the log line's message SHALL + contain the literal substring `checkpoint_resume: resuming from checkpoint` immediately + followed by the actual resolved path. + 2. REQ-CR6 call exits zero AND returns empty/`None` (no prior checkpoint found, for any reason): + the log line's message SHALL contain the literal substring + `checkpoint_resume: no prior checkpoint found`. + 3. REQ-CR6 call itself exits non-zero (REQ-CR12's crash case): the log line's message SHALL + contain the literal substring `checkpoint_resume: resume-check crashed` immediately followed + by the captured exit status — a concrete, non-blank, informative string in every branch, not a + generic placeholder. ### Group CR-SAFETY — no filesystem mutation, anywhere in this feature's code @@ -245,19 +273,26 @@ is touched. entrypoint (REQ-CR6) SHALL EVER call any filesystem-mutating API. Concretely, this module's own source code (in both the pure function and the `__main__` block) SHALL NEVER call: `os.remove`, `os.unlink`, `os.rmdir`, `os.removedirs`, `os.mkdir`, `os.makedirs`, `os.rename`, `os.replace`, - `os.truncate`, any `shutil.*` function (`shutil.rmtree`, `shutil.copy*`, `shutil.move`, etc.), or - `open(...)` with any mode other than the implicit read-only default (i.e. never `'w'`, `'a'`, - `'x'`, `'w+'`, or any other write/append/create/truncate mode) — nor any equivalent operation that - creates, deletes, renames, or writes bytes to a file or directory. THE ONLY filesystem operations - this module's runtime code MAY perform are READ-ONLY directory traversal: `os.listdir`, - `os.scandir`, and `os.path.*` queries (`isdir`, `isfile`, `exists`, `join`, `abspath`, etc.) — - listing directory entries and inspecting their names/existence, NEVER opening or reading the byte - contents of any file (this restates and sharpens REQ-CR5's "no file-content reads" rule to also - explicitly cover the `__main__` entrypoint, not only the pure function). The `__main__` - entrypoint's only permitted output channel is stdout (REQ-CR6); it never writes to any file. This - requirement makes INV-CR1 directly traceable to an explicit, testable REQ (closing the gap where - INV-CR1 previously had no REQ or test enforcing it) and is verified by PROP-CR12 (static - denylist scan + dynamic before/after directory-snapshot comparison). + `os.truncate`, `os.symlink`, `os.link`, any `shutil.*` function (`shutil.rmtree`, `shutil.copy*`, + `shutil.move`, etc.), `open(...)` with any mode other than the implicit read-only default (i.e. + never `'w'`, `'a'`, `'x'`, `'w+'`, or any other write/append/create/truncate mode), NOR any + `pathlib.Path` mutating method (`Path.unlink`, `Path.rmdir`, `Path.mkdir`, `Path.rename`, + `Path.replace`, `Path.touch`, `Path.write_text`, `Path.write_bytes`, `Path.symlink_to`, + `Path.hardlink_to`, or any other `Path` method that creates/deletes/renames/writes) — nor any + equivalent operation that creates, deletes, renames, or writes bytes to a file or directory. THE + ONLY filesystem operations this module's runtime code MAY perform are READ-ONLY directory + traversal: `os.listdir`, `os.scandir`, and `os.path.*` queries (`isdir`, `isfile`, `exists`, + `join`, `abspath`, etc.) — listing directory entries and inspecting their names/existence, NEVER + opening or reading the byte contents of any file (this restates and sharpens REQ-CR5's "no + file-content reads" rule to also explicitly cover the `__main__` entrypoint, not only the pure + function). This module SHALL NOT import `pathlib` at all (`os`/`os.path` suffice for every + operation REQ-CR1-CR10 require), which makes the "no `pathlib.Path` mutation" half of this + requirement structurally trivial to verify (an import-scan alone proves it) rather than requiring + an exhaustive enumeration of every `Path` mutating method. The `__main__` entrypoint's only + permitted output channel is stdout (REQ-CR6); it never writes to any file. This requirement makes + INV-CR1 directly traceable to an explicit, testable REQ (closing the gap where INV-CR1 previously + had no REQ or test enforcing it) and is verified by PROP-CR12 (static denylist scan + a `pathlib` + import-absence scan + dynamic before/after directory-snapshot comparison). ## Global Invariants diff --git a/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md b/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md index 83338ffd..e427e97e 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md +++ b/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md @@ -58,6 +58,7 @@ check that `run_evolve.sh`'s new step is actually invoked and its output actuall | ID | Description | Tier | Required | Tool | |---|---|---|---|---| | PROP-CR1 | `find_latest_checkpoint` returns `None` for a nonexistent `runs_dir` (REQ-CR4, EDGE-CR2) | 0 | true | pytest | +| PROP-CR1b | `find_latest_checkpoint`, called with a RELATIVE `runs_dir` string against a `tmp_path`-rooted fixture (via `os.chdir` into a known cwd for the test) that has one valid prior checkpoint, returns an ABSOLUTE path string (`os.path.isabs(result) is True`), not a relative one (REQ-CR1) | 0 | true | pytest, `tmp_path` + `monkeypatch.chdir` | | PROP-CR2 | `find_latest_checkpoint` returns `None` for an empty `runs_dir`, and for a `runs_dir` containing ONLY `current_run_id`'s own directory (REQ-CR4, EDGE-CR3, EDGE-CR9) | 1 | true | pytest, `tmp_path` | | PROP-CR3 | Given two prior run directories with checkpoints, the MOST RECENT one (by lexicographic-descending `run-YYYYMMDDTHHMMSSZ` name) is selected, never the older one, even when the older one has a higher checkpoint number (REQ-CR2, EDGE-CR7) | 1 | true | pytest, `tmp_path`, parametrized names | | PROP-CR4 | Within the selected run, `checkpoint_100` is selected over `checkpoint_20` (integer comparison, not lexicographic) (REQ-CR3) | 1 | true | pytest, `tmp_path` | @@ -68,9 +69,11 @@ check that `run_evolve.sh`'s new step is actually invoked and its output actuall | PROP-CR9 | Static source-text check: `find_latest_checkpoint`'s module source (`lib/checkpoint_resume.py`, excluding its own `__main__` CLI-entrypoint block, which legitimately touches `os.environ`/stdout per REQ-CR6) contains no `open(`, `subprocess`, `requests`, `socket`, or `urllib` reference — purity restated as an executable check, standing on its own: it asserts INV-CR1 directly, over this module's own source text, rather than by comparison to any other file's test (INV-CR1) | 0 | true | pytest source-text scan | | PROP-CR10 | A `runs_dir` entry that does NOT match REQ-CR10's `^run-\d{8}T\d{6}Z$` shape (concretely, an entry named `db`, matching the real production `runs/db` sibling from `config.yaml`'s `database.db_path`) is NEVER selected as a candidate, even when it is lexicographically ordered such that a naive unfiltered sort would place it first, and even when it is given its own `checkpoints/checkpoint_1/` subdirectory as an adversarial fixture (REQ-CR10, EDGE-CR10) | 1 | true | pytest, `tmp_path` (fixture includes a non-run-shaped `db` entry alongside real `run-*` entries) | | PROP-CR11 | A run-shaped candidate whose `checkpoints/` subdirectory is non-empty at the OS-listing level (`os.listdir` returns >= 1 entry) but contains ZERO `checkpoint_`-shaped entries (fixture: a single stray file, e.g. `.DS_Store`, and nothing else) causes `find_latest_checkpoint` to fall through to the next-most-recent run-shaped candidate (which DOES have a valid `checkpoint_` entry) and return THAT candidate's checkpoint path — never `None` and never an exception, even though the "non-empty checkpoints/ dir" test alone would wrongly treat this candidate as usable (REQ-CR2, REQ-CR3, EDGE-CR11) | 1 | true | pytest, `tmp_path`, two-run fixture (newer run: stray-file-only `checkpoints/`; older run: valid `checkpoint_/`) | -| PROP-CR12 | Filesystem-mutation prohibition (REQ-CR11, INV-CR1), verified BOTH ways: **(a) static** — `lib/checkpoint_resume.py`'s FULL module source (including its `__main__` block; only its legitimate `os.environ` read and stdout write are exempt) contains no reference to `os.remove`, `os.unlink`, `os.rmdir`, `os.removedirs`, `os.mkdir`, `os.makedirs`, `os.rename`, `os.replace`, `os.truncate`, any `shutil.` attribute access, or `open(` used with a non-default (write/append/create/truncate) mode — this expands PROP-CR9's network/subprocess-only denylist to also cover filesystem-mutation APIs. **(b) dynamic** — snapshot a synthetic `tmp_path` `runs/` tree (a recursive listing of every path plus its `os.stat` mtime, or a full byte-for-byte directory copy) BEFORE calling `find_latest_checkpoint` at least twice against it (once where it returns `None` — e.g. an empty tree — and once where it returns a real checkpoint path — e.g. a populated two-run fixture), then re-snapshot the SAME tree AFTER both calls and assert it is byte-for-byte/mtime-for-mtime identical to the BEFORE snapshot. This closes the gap a static-only or unit-test-only check cannot: a function could pass every REQ-CR1-10 test above by returning the right paths while ALSO, incidentally, mutating disk — (a) catches that via source inspection, (b) catches it via observed behavior, independently of each other. | 0/1 | true | pytest source-text scan (a) + pytest `tmp_path` before/after directory-tree snapshot comparison (b) | +| PROP-CR12 | Filesystem-mutation prohibition (REQ-CR11, INV-CR1), verified THREE ways: **(a) static denylist** — `lib/checkpoint_resume.py`'s FULL module source (including its `__main__` block; only its legitimate `os.environ` read and stdout write are exempt) contains no reference to `os.remove`, `os.unlink`, `os.rmdir`, `os.removedirs`, `os.mkdir`, `os.makedirs`, `os.rename`, `os.replace`, `os.truncate`, `os.symlink`, `os.link`, any `shutil.` attribute access, or `open(` used with a non-default (write/append/create/truncate) mode — this expands PROP-CR9's network/subprocess-only denylist to also cover filesystem-mutation APIs. **(b) static import-absence scan** — the module source contains no `import pathlib` and no `from pathlib import`, which (per REQ-CR11) makes every `pathlib.Path` mutating method structurally unreachable without an exhaustive method-name denylist. **(c) dynamic** — snapshot a synthetic `tmp_path` `runs/` tree (a recursive listing of every path plus its `os.stat` mtime, or a full byte-for-byte directory copy) BEFORE calling `find_latest_checkpoint` at least twice against it (once where it returns `None` — e.g. an empty tree — and once where it returns a real checkpoint path — e.g. a populated two-run fixture), then re-snapshot the SAME tree AFTER both calls and assert it is byte-for-byte/mtime-for-mtime identical to the BEFORE snapshot. This closes the gap a static-only or unit-test-only check cannot: a function could pass every REQ-CR1-10 test above by returning the right paths while ALSO, incidentally, mutating disk — (a)+(b) catch that via source inspection, (c) catches it via observed behavior, independently of each other. | 0/1 | true | pytest source-text scan (a)+(b) + pytest `tmp_path` before/after directory-tree snapshot comparison (c) | +| PROP-CR13 | Resume-check crash handling (REQ-CR12): when the REQ-CR6 subprocess call exits non-zero (simulated via a broken/non-executable stand-in for `$PY_BIN` or a deliberately-erroring stand-in script in the test fixture), `run_evolve.sh`'s new step (i) still proceeds to invoke `openevolve-run` with NO `--checkpoint` flag (same as REQ-CR8's fallback), and (ii) the log line's message contains the literal substring `checkpoint_resume: resume-check crashed` followed by the captured non-zero exit status — distinct from both the found-branch and not-found-branch wording | wire | true | shell-level test: substitute a stand-in for the REQ-CR6 call that exits non-zero, assert both the resulting `openevolve-run` argument list (no `--checkpoint`) and the exact log substring | | PROP-CR-WIRE1 | `run_evolve.sh`'s source text contains the EXACT substring `RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py"` (no `-c`, no other argv), that call's line number is BEFORE the `"$OPENEVOLVE_BIN"` invocation's line number, and the invocation's argument list conditionally includes `--checkpoint` (REQ-CR6, REQ-CR7) | wire | true | grep/text-position check over `run_evolve.sh` for the exact pinned command string | | PROP-CR-WIRE2 | WHEN no checkpoint is found, the resulting `openevolve-run` argument list is IDENTICAL (same arguments, same order) to the pre-feature invocation — no `--checkpoint` flag appears (REQ-CR8, INV-CR4) | 1 | true | a synthetic re-assembly test of the argument array (bash unit check or a small harness that captures the composed command line under both branches) | +| PROP-CR-WIRE2b | WHEN a checkpoint IS found, the resulting `openevolve-run` argument list is IDENTICAL to the pre-feature invocation for every EXISTING argument (`initial_program`, `evaluation_file`, `--config`, `--iterations`, `--output`, same values, same order) with EXACTLY ONE addition, `--checkpoint ""`, appended at the end (REQ-CR7) — closes the gap where only the not-found branch's argument-list stability was previously tested | 1 | true | the same synthetic re-assembly test as PROP-CR-WIRE2, run against the with-checkpoint branch and diffed against the pre-feature argument list plus the expected single addition | | PROP-CR-WIRE3 | Every code path through the new step (found / not-found) appends exactly one line to `$LOG` before proceeding to invoke `openevolve-run`, AND that line's CONTENT (not merely its count) is asserted in BOTH branches: the found-branch line contains the literal substring `checkpoint_resume: resuming from checkpoint` followed by the actual selected path; the not-found-branch line contains the literal substring `checkpoint_resume: no prior checkpoint found` (REQ-CR9) | 1 | true | shell-level check: run the new step in isolation against both a with-checkpoint and without-checkpoint fixture, assert `$LOG` gained exactly one new line each time AND that the new line's text contains the required substring for that branch — never merely `count(lines) == 1` | | PROP-CR-LIVE1 | End-to-end: a hand-constructed two-run `tmp_path` tree (an older run with `checkpoints/checkpoint_10/` and `checkpoints/checkpoint_20/`, and a current run with no checkpoints yet) fed through `run_evolve.sh`'s new step (invoked directly, not via the full openevolve subprocess) produces a log line containing the literal substring `checkpoint_resume: resuming from checkpoint` immediately followed by `checkpoint_20`'s path (REQ-CR6, REQ-CR9) | live (Tier 1, no real openevolve invocation required) | true | integration test against a synthetic fixture tree, per EDGE-CR1's worktree constraint below | From 5b81a329c3e4786a3ad4dc59fd233366e13855e5 Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sat, 11 Jul 2026 18:45:28 +0900 Subject: [PATCH 05/11] spec: self-improve-checkpoint-resume iteration-4 fixes (file-vs-dir candidate edge case EDGE-CR12/PROP-CR14, REQ-CR6 citation accuracy, INV-CR2/3/5 proof cross-references) --- .../iteration-4/output/findings/FIND-001.json | 20 ++++++ .../iteration-4/output/findings/FIND-002.json | 20 ++++++ .../iteration-4/output/findings/FIND-003.json | 20 ++++++ .../spec/iteration-4/output/verdict.json | 61 +++++++++++++++++++ .../spec/iteration-5/input/manifest.json | 5 ++ .../specs/behavioral-spec.md | 38 +++++++++--- .../specs/verification-architecture.md | 1 + 7 files changed, 155 insertions(+), 10 deletions(-) create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-001.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-002.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-003.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/verdict.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/input/manifest.json diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-001.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-001.json new file mode 100644 index 00000000..836b459f --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-001.json @@ -0,0 +1,20 @@ +{ + "findingId": "FIND-001", + "dimension": "spec_fidelity", + "category": "spec_gap", + "severity": "low", + "description": "REQ-CR6 (behavioral-spec.md:216-218) claims: 'run_evolve.sh SHALL also capture this call's own exit status separately from its stdout (mirroring the existing pattern at run_evolve.sh:58-64's OBSERVE_JSON=\"$(...)\" capture, which does not abort on a non-zero ledger_reader.py exit either)'. This citation is inaccurate: the actual OBSERVE step at run_evolve.sh:59-64 does NOT capture ledger_reader.py's exit status into any variable at all -- it only guards on `[ -x \"$PY_BIN\" ]` before the call, then unconditionally proceeds regardless of what `$?` turns out to be after the command substitution (no `$?` is ever read or stored). There is no 'existing pattern' of separately capturing an exit status to mirror; the OBSERVE step simply never checks it, full stop. This does not change what REQ-CR6/REQ-CR12 require of the NEW checkpoint_resume.py call (capturing its exit status to distinguish REQ-CR12's crash branch is still a well-specified, self-contained requirement on its own), so it has zero effect on implementability or correctness -- it is a documentation/grounding-accuracy defect only, in a spec whose own rhetorical style repeatedly leans on 'confirmed', 'verified live', 'mirrors the existing pattern' framing to justify its design choices. Given that framing is the document's own evidentiary standard, an inaccurate instance of it should be corrected even though it is functionally inert here.", + "evidence": { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "216-218", + "snippet": "`run_evolve.sh` SHALL also capture this call's own exit status separately from its stdout (mirroring the existing pattern at `run_evolve.sh:58-64`'s `OBSERVE_JSON=\"$(...)\"` capture, which does not abort on a non-zero `ledger_reader.py` exit either)" + }, + "groundTruthEvidence": [ + { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/skills/earn/self-improve/run_evolve.sh", + "lineRange": "59-64", + "snippet": "if [ -x \"$PY_BIN\" ]; then\n OBSERVE_JSON=\"$(\"$PY_BIN\" \"$SKILL_DIR/lib/ledger_reader.py\" 2>>\"$LOG\")\"\n echo \"$(now) OBSERVE realized_ledger=${OBSERVE_JSON:-'{\"error\":\"ledger_reader produced no output\"}'}\" >> \"$LOG\"\nelse\n echo \"$(now) OBSERVE SKIPPED (python not found at $PY_BIN) — proceeding to EVOLVE anyway\" >> \"$LOG\"\nfi" + } + ], + "routeToPhase": "1a" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-002.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-002.json new file mode 100644 index 00000000..1595ad9d --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-002.json @@ -0,0 +1,20 @@ +{ + "findingId": "FIND-002", + "dimension": "spec_fidelity", + "category": "spec_gap", + "severity": "medium", + "description": "REQ-CR10 (behavioral-spec.md:174-186) defines candidacy PURELY by name shape (`^run-\\d{8}T\\d{6}Z$`), explicitly 'independently of whatever the entry contains (a directory or a file, empty or not)'. REQ-CR2 (behavioral-spec.md:135-155) then iterates these name-matched candidates and calls REQ-CR3 (behavioral-spec.md:157-172) on each, whose text ('if that directory has no checkpoints/ subdirectory at all, return None immediately') implicitly assumes the candidate IS a directory, but neither REQ-CR2 nor REQ-CR3 pins the exact implementation technique for detecting 'no checkpoints/ subdirectory' when the candidate itself is a plain FILE (not a directory) that happens to match REQ-CR10's timestamp-shape regex -- a plausible real artifact from an interrupted write, a stray `touch run-20260710T190919Z`, or a partial rsync, exactly the class of environmental gotcha this same spec explicitly cares about one level down (EDGE-CR11's `.DS_Store`-inside-`checkpoints/` scenario, behavioral-spec.md:355-365). This matters because of a real, non-obvious Python semantics gap: `os.path.isdir(os.path.join(candidate, 'checkpoints'))`-first (safe, returns False, never raises) and `os.listdir(os.path.join(candidate, 'checkpoints'))` wrapped only in `except FileNotFoundError` (UNSAFE) are NOT equivalent here -- when `candidate` itself exists but is a file rather than a directory, `os.listdir` on a path nested under it raises `NotADirectoryError`, a DISTINCT exception type from `FileNotFoundError`. An implementer who reads REQ-CR3's 'no checkpoints/ subdirectory... return None' as licensing a narrow `except FileNotFoundError:` (which is the more naturally-read implementation of that sentence, and passes every currently-specified EDGE-CR/PROP-CR fixture, none of which include a non-directory candidate) would ship code that raises uncaught on this exact input. No EDGE-CR and no PROP-CR (PROP-CR1 through PROP-CR11) constructs a fixture where a `runs_dir` entry matches REQ-CR10's regex but is a plain file rather than a directory, so this gap is untested as well as unspecified. Mitigating factor (why this is not blocking): REQ-CR12's crash-handling branch (this iteration's own fix) already treats ANY uncaught exception from the checkpoint_resume.py process as a first-class, safely-degraded outcome (no --checkpoint flag, distinct 'resume-check crashed' log line) -- so even if this exact gap ships as a live bug, the harness's own safety net (not this function's own robustness) prevents it from being dangerous; it would just make an otherwise-avoidable crash-log line fire on an input the spec could have specified cleanly, exactly as it already did for EDGE-CR11 one level down.", + "evidence": { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "157-186", + "snippet": "REQ-CR3: 'GIVEN a single candidate run directory ... if that directory has no checkpoints/ subdirectory at all, return None immediately.' / REQ-CR10: 'is EXCLUDED from candidacy entirely ... regardless of ... whatever the entry contains (a directory or a file, empty or not)' -- no requirement addresses a name-matched candidate that is itself a plain file." + }, + "groundTruthEvidence": [ + { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md", + "lineRange": "70-71", + "snippet": "PROP-CR10 / PROP-CR11 fixtures cover a non-run-shaped `db` sibling and a stray `.DS_Store`-only `checkpoints/` dir, but no fixture covers a run-shaped PLAIN FILE candidate" + } + ], + "routeToPhase": "1a" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-003.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-003.json new file mode 100644 index 00000000..f4ac5cdb --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/findings/FIND-003.json @@ -0,0 +1,20 @@ +{ + "findingId": "FIND-003", + "dimension": "verification_readiness", + "category": "verification_tool_mismatch", + "severity": "medium", + "description": "This iteration's own fix for the prior-round's FIND-001 (INV-CR1 untraceable) explicitly demonstrates the standard this document holds itself to: REQ-CR11 (behavioral-spec.md:292-295) states 'This requirement makes INV-CR1 directly traceable to an explicit, testable REQ ... and is verified by PROP-CR12.' That same standard is NOT applied to the other four Global Invariants (behavioral-spec.md:297-305). INV-CR2 ('This feature never touches strategies/pm_backtest_strategy.py, promote_gate.sh, lib/promote_gate.py, lib/promote.py, config.yaml, or ai.anicca.self-improve-evolve.plist') and INV-CR3 ('This feature never reads or writes any wallet key, .env, ledger file, or spend-cap value') have NO corresponding row in the Proof Obligations table (verification-architecture.md:58-79) at all -- not even an informal one. INV-CR4 and INV-CR5 fare only slightly better: INV-CR4 plausibly maps to PROP-CR-WIRE2 by content (same-argument-list-on-fallback) but the table never says so explicitly the way it does for INV-CR1/PROP-CR12; INV-CR5 ('REQ-OE6's ... invariant is preserved unchanged') has no PROP row and is asserted purely by the prose argument that this feature 'adds no new promotion path,' which is true by non-modification but is never turned into an executable check (e.g. a Phase-5 diff-scope assertion that the changeset's file list is exactly {lib/checkpoint_resume.py, run_evolve.sh, and test files} would trivially and cheaply close INV-CR2/INV-CR3's gap, mirroring how this same document already turned INV-CR1 from an unenforced prose claim into PROP-CR12). This is a real asymmetry in the document's own internal rigor standard, not a hypothetical concern -- it is the exact category of gap ('an invariant with no REQ or test enforcing it') that iteration 2's FIND-001 was raised against and that this iteration's own REQ-CR11 rewrite explicitly took credit for closing, but only for one of five invariants.", + "evidence": { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md", + "lineRange": "58-79", + "snippet": "Proof Obligations table (PROP-CR1 through PROP-CR-LIVE1) contains no row referencing INV-CR2, INV-CR3, or INV-CR5 by ID, unlike INV-CR1's explicit PROP-CR12 cross-reference in behavioral-spec.md:292-295" + }, + "groundTruthEvidence": [ + { + "filePath": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "297-305", + "snippet": "| INV-CR2 | This feature never touches strategies/pm_backtest_strategy.py, promote_gate.sh, lib/promote_gate.py, lib/promote.py, config.yaml, or ai.anicca.self-improve-evolve.plist |\n| INV-CR3 | This feature never reads or writes any wallet key, .env, ledger file, or spend-cap value ... |\n| INV-CR5 | REQ-OE6's ... invariant is preserved unchanged ... |" + } + ], + "routeToPhase": "1b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/verdict.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/verdict.json new file mode 100644 index 00000000..db059af9 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/output/verdict.json @@ -0,0 +1,61 @@ +{ + "sprintNumber": null, + "feature": "self-improve-checkpoint-resume", + "reviewType": "spec", + "phase": "1c", + "overallVerdict": "FAIL", + "timestamp": "2026-07-11T00:00:00Z", + "iteration": 4, + "reviewContext": { + "targetSpec": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "targetVerificationArchitecture": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md", + "manifest": "/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-4/input/manifest.json (reviewDimensions restricted to spec_fidelity + verification_readiness; note the manifest itself carries no source-provenance/integration frontmatter this iteration -- a plain {reviewType, artifactsToReview, reviewDimensions} object -- unlike prior iterations' manifests, but this does not change what was reviewed)", + "crossCheckedImplementationContext": [ + "skills/earn/self-improve/run_evolve.sh (full file read, 113 lines -- the exact edit target for REQ-CR6-CR9/CR12)", + "skills/earn/self-improve/lib/ledger_reader.py (full file read, 226 lines -- the module REQ-CR6/verification-architecture.md claim to mirror for CLI-entrypoint shape)", + "skills/earn/self-improve/config.yaml (full file read, 142 lines -- confirms checkpoint_interval:10 at line 11, db_path:\"runs/db\" at line 129, and the line-128 comment cited in the frontmatter)", + "lib/checkpoint_resume.py and any test file: confirmed NOT YET CREATED (Glob returned zero matches) -- consistent with state.json's currentPhase=1b, this is a pure spec-only review with no implementation to cross-check against" + ], + "priorIterationFindingsStatus": { + "FIND-001 (iteration 3, blocking -- no REQ covered checkpoint_resume.py's own process crashing/exiting non-zero)": "RESOLVED -- REQ-CR12 (behavioral-spec.md:220-232) adds a third, distinct branch for the REQ-CR6 call itself exiting non-zero, degrading exactly like REQ-CR8's fallback for openevolve-run's argument list but requiring a THIRD, distinct pinned log substring ('checkpoint_resume: resume-check crashed' + exit status) per REQ-CR9's now-three-branch enumeration (behavioral-spec.md:256-268). PROP-CR13 (verification-architecture.md:73) is a new, dedicated proof obligation exercising exactly this branch via a broken/erroring stand-in for $PY_BIN, distinct from PROP-CR-WIRE3's found/not-found coverage. Cross-checked REQ-CR9's three branches against REQ-CR12's own wording: mutually consistent, no conflation of crash vs. ordinary-not-found. No residual defect found in this specific fix.", + "FIND-002 (iteration 3, high -- REQ-CR11's no-mutation denylist missed pathlib mutating methods and os.symlink/os.link)": "RESOLVED -- REQ-CR11 (behavioral-spec.md:272-295) now explicitly denylists os.symlink/os.link and every named pathlib.Path mutating method, PLUS adds a structural requirement that the module SHALL NOT import pathlib at all, converting the pathlib half of the check from an exhaustive (and previously incomplete) method-name enumeration into a trivially-verifiable import-absence scan. Cross-checked for the specific risk this iteration's prompt asked to confirm: REQ-CR1's own normalization clause (behavioral-spec.md:127-133) calls `os.path.abspath` -- an os/os.path function, never pathlib -- so the new 'no pathlib import' structural ban does not conflict with any other requirement's actual needs; REQ-CR1 through REQ-CR10 use only os.listdir/os.path.* operations throughout. PROP-CR12 (verification-architecture.md:72) was updated to a three-part check: (a) expanded static denylist, (b) new pathlib import-absence scan, (c) unchanged dynamic before/after snapshot. No residual defect found in this specific fix.", + "FIND-003 (iteration 3, medium -- REQ-CR7's 'existing args unchanged' claim for the found-checkpoint branch had no proof obligation)": "RESOLVED -- PROP-CR-WIRE2b (verification-architecture.md:76) is a new, dedicated proof obligation for exactly this gap: it diffs the with-checkpoint branch's composed argument list against the pre-feature list plus the expected single --checkpoint addition, closing the asymmetry where only PROP-CR-WIRE2 (not-found branch) previously existed. No residual defect found in this specific fix.", + "FIND-004 (iteration 3, low -- REQ-CR1's 'absolute path' return contract was not required to be normalized by the function itself)": "RESOLVED -- REQ-CR1 (behavioral-spec.md:125-133) now explicitly requires the function itself to call os.path.abspath on its own return value, with a stated rationale ('true by construction rather than true only by accident of production's RUNS_DIR always being absolute'). PROP-CR1b (verification-architecture.md:61) is a new, dedicated proof obligation using a RELATIVE runs_dir + monkeypatch.chdir to force the distinction to matter. No residual defect found in this specific fix." + }, + "newFindingsThisIteration": ["FIND-001", "FIND-002", "FIND-003"] + }, + "dimensions": [ + { + "name": "spec_fidelity", + "verdict": "FAIL", + "findings": ["FIND-001", "FIND-002"], + "evidence": [ + {"type": "file", "location": "specs/behavioral-spec.md:220-232", "description": "REQ-CR12's three-branch crash-handling design verified present, internally consistent with REQ-CR9's three pinned log substrings, and correctly scoped to govern only the REQ-CR6 helper's own exit status (never openevolve-run's, which remains REQ-OE6's unchanged concern) -- no finding raised on this specific fix."}, + {"type": "file", "location": "specs/behavioral-spec.md:125-133 and :272-295", "description": "Cross-checked REQ-CR1's new os.path.abspath normalization clause against REQ-CR11's new 'SHALL NOT import pathlib at all' structural ban for conflict -- confirmed no conflict: abspath is an os.path function, and no REQ-CR1-CR10 requirement needs pathlib anywhere. The task's explicit request to verify this non-conflict is satisfied."}, + {"type": "file", "location": "specs/behavioral-spec.md (frontmatter sources list + integration section)", "description": "Re-verified the frontmatter's factual claims against the actual files: config.yaml:11 (checkpoint_interval:10), config.yaml:128-129 (the --checkpoint resume comment + db_path:\"runs/db\"), and run_evolve.sh:50 (RUN_ID's exact date format) all match verbatim. INV-CR2/INV-CR3's live-money/other-file exclusion claims remain true of the current spec text (no strategy/promote/plist/ledger/wallet touch anywhere in the requirements)."} + ] + }, + { + "name": "verification_readiness", + "verdict": "FAIL", + "findings": ["FIND-003"], + "evidence": [ + {"type": "file", "location": "specs/verification-architecture.md:58-79", "description": "Verified every REQ-CR1 through REQ-CR12 has at least one traceable PROP-ID (built a full REQ->PROP cross-reference table during review); confirmed no gap in REQ-level traceability. The gap found this iteration (FIND-003) is specifically at the Global-Invariant level (INV-CR2/CR3/CR5), not the REQ level."}, + {"type": "file", "location": "specs/verification-architecture.md:71-73", "description": "PROP-CR12's three-part (a)/(b)/(c) closure and PROP-CR13's dedicated crash-branch test independently verified sound and non-overlapping -- (a) and (b) catch the mutation-prohibition via source inspection, (c) via dynamic snapshot, and PROP-CR13 exercises a code path neither of the other two touches. No finding raised on PROP-CR12/PROP-CR13's own content."}, + {"type": "file", "location": "specs/verification-architecture.md:90-110", "description": "Tier 0/1-only classification re-confirmed proportionate for this ~20-30 line utility on this pass as well -- no numeric/financial invariant, no concurrency claim, no cryptographic claim is smuggled in that would require Tier 2/3 machinery this feature's own scope does not warrant."} + ] + } + ], + "blockingFindings": [], + "nonBlockingFindingsNote": "No finding this iteration rises to 'blocking' in the project's own vcsdd-adversary sense (dev-workflow.md: 'blocking 1件でも次フェーズ進行禁止'): FIND-001 is a citation/documentation-accuracy defect with zero effect on what must be built; FIND-002 is a real but contained algorithm under-specification whose worst-case outcome is already absorbed, safely, by this same iteration's own REQ-CR12 crash-handling design (an extra 'resume-check crashed' log line, never a harness stall or incorrect --checkpoint path); FIND-003 is a documentation-traceability asymmetry (INV-CR2/CR3/CR5 lack the same explicit PROP cross-reference INV-CR1 just received) whose underlying invariants are true by non-modification and cheap to close with a one-line Phase-5 diff-scope check. Dimension verdicts are nonetheless recorded as FAIL per this review's binary-verdict/no-partial-credit mandate (a dimension with any real, evidence-backed finding is not a clean PASS), but none of these three findings should be read as reopening the four substantive gaps iteration 3 found -- all four of those are independently verified resolved and internally consistent (see priorIterationFindingsStatus above). All three findings here are low/medium severity and, unlike iteration 3's set, none block a correct, safe implementation of this narrow fix from proceeding once logged.", + "convergenceSignals": { + "findingCount": 3, + "allCriteriaEvaluated": true, + "evaluatedCriteria": [], + "duplicateFindings": [], + "iteration3ResolvedCount": 4, + "iteration3NotFixedCount": 0, + "newFindingsThisIteration": 3, + "trendAcrossIterations": "iteration2->3: 2 resolved, 2 new, net FAIL. iteration3->4: 4 resolved, 3 new, net FAIL. Finding severity is trending down (iteration 3: 1 blocking + 1 high + 1 medium + 1 low; iteration 4: 0 blocking + 0 high + 2 medium + 1 low) -- convergence is real but not yet complete; this is not a rubber-stamp continuation of a clean streak." + } +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/input/manifest.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/input/manifest.json new file mode 100644 index 00000000..d99d842f --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/input/manifest.json @@ -0,0 +1,5 @@ +{ + "reviewType": "spec", + "artifactsToReview": ["specs/behavioral-spec.md", "specs/verification-architecture.md"], + "reviewDimensions": ["spec_fidelity", "verification_readiness"] +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md b/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md index 05c85715..5828e110 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md +++ b/.vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md @@ -145,9 +145,12 @@ is touched. candidate for which REQ-CR3 returns a non-`None` checkpoint path is the one used, and iteration STOPS there (REQ-CR3's result for that candidate IS the function's overall result, modulo REQ-CR1's absolute-path contract). A candidate for which REQ-CR3 returns `None` (for ANY reason — - no `checkpoints/` subdirectory at all, an empty one, or one that is non-empty at the OS-listing - level but contains ZERO valid `checkpoint_`-shaped entries — see EDGE-CR11) is skipped (not - selected, not a fatal error) and iteration CONTINUES to the next-most-recent candidate. If the + no `checkpoints/` subdirectory at all, an empty one, one that is non-empty at the OS-listing + level but contains ZERO valid `checkpoint_`-shaped entries (see EDGE-CR11), or the candidate + entry matching REQ-CR10's name shape being a PLAIN FILE rather than a directory (see EDGE-CR12) — + REQ-CR3 treats a same-named file exactly like a directory with no `checkpoints/` subdirectory: + `None`, never a raised `NotADirectoryError`/`FileNotFoundError`) is skipped (not selected, not a + fatal error) and iteration CONTINUES to the next-most-recent candidate. If the entire sorted candidate list is exhausted with every candidate returning `None` from REQ-CR3 (or the list was empty to begin with), THE SYSTEM SHALL return `None` (REQ-CR4). This iterate-until- a-real-checkpoint-or-exhausted rule is the ONLY selection algorithm this function implements — @@ -212,10 +215,14 @@ is touched. `RUN_ID` = the just-created `$RUN_ID`, correctly excluding itself even though its own now-empty directory already exists under `runs/`) and BEFORE the existing `"$OPENEVOLVE_BIN"` invocation. `run_evolve.sh` SHALL capture this call's stdout verbatim as the checkpoint path (an empty string - meaning "no checkpoint found," per REQ-CR8). `run_evolve.sh` SHALL also capture this call's own - exit status separately from its stdout (mirroring the existing pattern at `run_evolve.sh:58-64`'s - `OBSERVE_JSON="$(...)"` capture, which does not abort on a non-zero `ledger_reader.py` exit - either) — see REQ-CR12 for what happens on a non-zero exit here. + meaning "no checkpoint found," per REQ-CR8). `run_evolve.sh` SHALL ALSO explicitly capture this + call's own exit status (via `$?` immediately after the command substitution) separately from its + stdout — NOTE this is a NEW capture this feature introduces, not a reuse of an existing pattern: + the existing OBSERVE step at `run_evolve.sh:58-64` calls `ledger_reader.py` via a bare + `OBSERVE_JSON="$(...)"` command substitution and never inspects its exit status at all (any + failure there is silently absorbed into an empty/error-shaped `$OBSERVE_JSON` string, logged as + data, never treated as a distinct branch) — see REQ-CR12 for what THIS feature does with a + non-zero exit, which is stricter than that existing precedent, not a copy of it. - **REQ-CR12** WHEN the REQ-CR6 call itself exits non-zero (the `checkpoint_resume.py` process crashed, raised an uncaught exception, or `$PY_BIN` was not executable) — as DISTINCT from the @@ -299,10 +306,10 @@ is touched. | # | Invariant | |---|---| | INV-CR1 | `find_latest_checkpoint` AND `lib/checkpoint_resume.py`'s `__main__` entrypoint NEVER write, delete, rename, or mutate anything on disk — both are pure/read-only with respect to the filesystem (REQ-CR5, REQ-CR11) | -| INV-CR2 | This feature never touches `strategies/pm_backtest_strategy.py`, `promote_gate.sh`, `lib/promote_gate.py`, `lib/promote.py`, `config.yaml`, or `ai.anicca.self-improve-evolve.plist` | -| INV-CR3 | This feature never reads or writes any wallet key, `.env`, ledger file, or spend-cap value — it is scoped exclusively to `runs/*/checkpoints/*` path selection | +| INV-CR2 | This feature never touches `strategies/pm_backtest_strategy.py`, `promote_gate.sh`, `lib/promote_gate.py`, `lib/promote.py`, `config.yaml`, or `ai.anicca.self-improve-evolve.plist` (verified by PROP-CR9/PROP-CR12's source-text scans over `lib/checkpoint_resume.py`, which contain no reference to any of these paths, plus a plain `git diff` scope check at review time showing no other file touched) | +| INV-CR3 | This feature never reads or writes any wallet key, `.env`, ledger file, or spend-cap value — it is scoped exclusively to `runs/*/checkpoints/*` path selection (verified by PROP-CR9's source-text scan finding no `open(`/network reference, PLUS the same `git diff` scope check as INV-CR2 — no wallet/`.env`/ledger file appears in this feature's diff) | | INV-CR4 | The fallback path (no checkpoint found) is byte-for-byte identical, in its `openevolve-run` invocation, to `run_evolve.sh`'s pre-feature behavior (REQ-CR8) — this feature can only ADD a `--checkpoint` argument, never change any other existing argument | -| INV-CR5 | REQ-OE6's "a crashed/killed/timed-out openevolve run is inconclusive, no candidate promoted, baseline untouched" invariant is preserved unchanged — this feature adds no new promotion path and does not weaken that handling for a `--checkpoint`-resumed run | +| INV-CR5 | REQ-OE6's "a crashed/killed/timed-out openevolve run is inconclusive, no candidate promoted, baseline untouched" invariant is preserved unchanged — this feature adds no new promotion path and does not weaken that handling for a `--checkpoint`-resumed run (verified by PROP-CR-WIRE1/PROP-CR-WIRE2/PROP-CR-WIRE2b together: the `openevolve-run` invocation's existing arguments and its downstream `STATUS=$?`/`promote_gate.sh` handling, both already outside this feature's diff, are shown byte-for-byte unchanged in every branch) | ## Edge Cases @@ -364,6 +371,17 @@ is touched. sorted candidate list is exhausted (→ REQ-CR4 → `None`). This function SHALL NEVER return early or raise merely because `checkpoints/` was non-empty at the OS-listing level. +- **EDGE-CR12** A `runs_dir` entry whose NAME matches REQ-CR10's `run-YYYYMMDDTHHMMSSZ` shape is + itself a PLAIN FILE, not a directory (e.g. a stray zero-byte file left by some unrelated process, + or a leftover artifact from a killed/interrupted run). REQ-CR3's attempt to inspect this + candidate's `checkpoints/` subdirectory SHALL treat this exactly like "no `checkpoints/` + subdirectory exists" — returning `None` for this candidate — and SHALL NEVER let a + `NotADirectoryError`/`FileNotFoundError`/any other OS-level exception escape `find_latest_ + checkpoint` (this function catches/checks-before-acting, e.g. via `os.path.isdir(...)` before + attempting to list a `checkpoints/` path underneath it, rather than relying on a bare `os.listdir` + raising and being caught after the fact). REQ-CR2's iteration then falls through to the + next-most-recent candidate exactly as EDGE-CR11 does. + ## "Done" / 4-D Convergence | dimension | condition | diff --git a/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md b/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md index e427e97e..9a5cb2da 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md +++ b/.vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md @@ -69,6 +69,7 @@ check that `run_evolve.sh`'s new step is actually invoked and its output actuall | PROP-CR9 | Static source-text check: `find_latest_checkpoint`'s module source (`lib/checkpoint_resume.py`, excluding its own `__main__` CLI-entrypoint block, which legitimately touches `os.environ`/stdout per REQ-CR6) contains no `open(`, `subprocess`, `requests`, `socket`, or `urllib` reference — purity restated as an executable check, standing on its own: it asserts INV-CR1 directly, over this module's own source text, rather than by comparison to any other file's test (INV-CR1) | 0 | true | pytest source-text scan | | PROP-CR10 | A `runs_dir` entry that does NOT match REQ-CR10's `^run-\d{8}T\d{6}Z$` shape (concretely, an entry named `db`, matching the real production `runs/db` sibling from `config.yaml`'s `database.db_path`) is NEVER selected as a candidate, even when it is lexicographically ordered such that a naive unfiltered sort would place it first, and even when it is given its own `checkpoints/checkpoint_1/` subdirectory as an adversarial fixture (REQ-CR10, EDGE-CR10) | 1 | true | pytest, `tmp_path` (fixture includes a non-run-shaped `db` entry alongside real `run-*` entries) | | PROP-CR11 | A run-shaped candidate whose `checkpoints/` subdirectory is non-empty at the OS-listing level (`os.listdir` returns >= 1 entry) but contains ZERO `checkpoint_`-shaped entries (fixture: a single stray file, e.g. `.DS_Store`, and nothing else) causes `find_latest_checkpoint` to fall through to the next-most-recent run-shaped candidate (which DOES have a valid `checkpoint_` entry) and return THAT candidate's checkpoint path — never `None` and never an exception, even though the "non-empty checkpoints/ dir" test alone would wrongly treat this candidate as usable (REQ-CR2, REQ-CR3, EDGE-CR11) | 1 | true | pytest, `tmp_path`, two-run fixture (newer run: stray-file-only `checkpoints/`; older run: valid `checkpoint_/`) | +| PROP-CR14 | A `runs_dir` entry whose name matches REQ-CR10's shape but is a PLAIN FILE (not a directory) causes `find_latest_checkpoint` to fall through to the next-most-recent run-shaped candidate without raising `NotADirectoryError`/`FileNotFoundError`/any exception (REQ-CR2, REQ-CR3, EDGE-CR12) | 1 | true | pytest, `tmp_path`, fixture with a zero-byte file named `run-` alongside a real older run directory with a valid `checkpoint_/` | | PROP-CR12 | Filesystem-mutation prohibition (REQ-CR11, INV-CR1), verified THREE ways: **(a) static denylist** — `lib/checkpoint_resume.py`'s FULL module source (including its `__main__` block; only its legitimate `os.environ` read and stdout write are exempt) contains no reference to `os.remove`, `os.unlink`, `os.rmdir`, `os.removedirs`, `os.mkdir`, `os.makedirs`, `os.rename`, `os.replace`, `os.truncate`, `os.symlink`, `os.link`, any `shutil.` attribute access, or `open(` used with a non-default (write/append/create/truncate) mode — this expands PROP-CR9's network/subprocess-only denylist to also cover filesystem-mutation APIs. **(b) static import-absence scan** — the module source contains no `import pathlib` and no `from pathlib import`, which (per REQ-CR11) makes every `pathlib.Path` mutating method structurally unreachable without an exhaustive method-name denylist. **(c) dynamic** — snapshot a synthetic `tmp_path` `runs/` tree (a recursive listing of every path plus its `os.stat` mtime, or a full byte-for-byte directory copy) BEFORE calling `find_latest_checkpoint` at least twice against it (once where it returns `None` — e.g. an empty tree — and once where it returns a real checkpoint path — e.g. a populated two-run fixture), then re-snapshot the SAME tree AFTER both calls and assert it is byte-for-byte/mtime-for-mtime identical to the BEFORE snapshot. This closes the gap a static-only or unit-test-only check cannot: a function could pass every REQ-CR1-10 test above by returning the right paths while ALSO, incidentally, mutating disk — (a)+(b) catch that via source inspection, (c) catches it via observed behavior, independently of each other. | 0/1 | true | pytest source-text scan (a)+(b) + pytest `tmp_path` before/after directory-tree snapshot comparison (c) | | PROP-CR13 | Resume-check crash handling (REQ-CR12): when the REQ-CR6 subprocess call exits non-zero (simulated via a broken/non-executable stand-in for `$PY_BIN` or a deliberately-erroring stand-in script in the test fixture), `run_evolve.sh`'s new step (i) still proceeds to invoke `openevolve-run` with NO `--checkpoint` flag (same as REQ-CR8's fallback), and (ii) the log line's message contains the literal substring `checkpoint_resume: resume-check crashed` followed by the captured non-zero exit status — distinct from both the found-branch and not-found-branch wording | wire | true | shell-level test: substitute a stand-in for the REQ-CR6 call that exits non-zero, assert both the resulting `openevolve-run` argument list (no `--checkpoint`) and the exact log substring | | PROP-CR-WIRE1 | `run_evolve.sh`'s source text contains the EXACT substring `RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py"` (no `-c`, no other argv), that call's line number is BEFORE the `"$OPENEVOLVE_BIN"` invocation's line number, and the invocation's argument list conditionally includes `--checkpoint` (REQ-CR6, REQ-CR7) | wire | true | grep/text-position check over `run_evolve.sh` for the exact pinned command string | From 49fb55be8dd616f6d18f6c0d7535fd1361c50ecc Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sat, 11 Jul 2026 18:48:52 +0900 Subject: [PATCH 06/11] vcsdd(self-improve-checkpoint-resume): spec-review PASS iteration 5 (0 blocking), phase -> 2a --- .../iteration-5/output/findings/FIND-001.json | 14 ++++++ .../iteration-5/output/findings/FIND-002.json | 14 ++++++ .../spec/iteration-5/output/verdict.json | 47 +++++++++++++++++++ .../self-improve-checkpoint-resume/state.json | 35 ++++++++++++-- .vcsdd/history.jsonl | 4 ++ .vcsdd/index.json | 4 +- 6 files changed, 111 insertions(+), 7 deletions(-) create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/findings/FIND-001.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/findings/FIND-002.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/verdict.json diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/findings/FIND-001.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/findings/FIND-001.json new file mode 100644 index 00000000..db2d167b --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/findings/FIND-001.json @@ -0,0 +1,14 @@ +{ + "findingId": "FIND-001", + "dimension": "verification_readiness", + "category": "verification_tool_mismatch", + "severity": "low", + "blocking": false, + "description": "INV-CR2's parenthetical (behavioral-spec.md) claims it is 'verified by PROP-CR9/PROP-CR12's source-text scans over lib/checkpoint_resume.py, which contain no reference to any of these paths' (i.e. strategies/pm_backtest_strategy.py, promote_gate.sh, lib/promote_gate.py, lib/promote.py, config.yaml, ai.anicca.self-improve-evolve.plist). Neither PROP-CR9 (denylist: open(, subprocess, requests, socket, urllib) nor PROP-CR12(a) (denylist: os.remove/os.mkdir/os.rename/shutil.*/non-default-mode open() + pathlib import-absence) actually scans lib/checkpoint_resume.py's source for those six specific path/filename strings — both PROPs check for a different, unrelated set of tokens. The citation overclaims what those two PROPs test. This is the same pattern iteration-4's FIND-001 flagged for REQ-CR6 (a citation asserting a test/step does something it does not do), recurring here in the newly-added INV-CR2 cross-reference. Substantively INV-CR2 is still covered by the OTHER half of the same parenthetical ('a plain git diff scope check at review time showing no other file touched'), so this is a citation-accuracy defect, not a coverage gap.", + "evidence": { + "filePath": ".vcsdd/features/self-improve-checkpoint-resume/specs/behavioral-spec.md", + "lineRange": "309-310", + "snippet": "INV-CR2 | ... (verified by PROP-CR9/PROP-CR12's source-text scans over lib/checkpoint_resume.py, which contain no reference to any of these paths, plus a plain git diff scope check at review time showing no other file touched)" + }, + "routeToPhase": "1c-polish-optional" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/findings/FIND-002.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/findings/FIND-002.json new file mode 100644 index 00000000..0d04f01e --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/findings/FIND-002.json @@ -0,0 +1,14 @@ +{ + "findingId": "FIND-002", + "dimension": "verification_readiness", + "category": "verification_tool_mismatch", + "severity": "medium", + "blocking": false, + "description": "REQ-CR11 explicitly states it 'restates and sharpens REQ-CR5's no-file-content-reads rule to also explicitly cover the __main__ entrypoint, not only the pure function' — i.e. the __main__ CLI entrypoint must never call open(...) to read a file's byte contents at all, not just never call it in a mutating mode. No PROP obligation in verification-architecture.md statically checks this for __main__: PROP-CR9's source-text denylist (open(, subprocess, requests, socket, urllib) explicitly EXCLUDES the __main__ block ('excluding its own __main__ CLI-entrypoint block'), and PROP-CR12(a) covers the __main__ block but only denylists open( used with a NON-DEFAULT (write/append/create/truncate) mode -- a read-mode open(path, 'r') inside __main__ (e.g. accidentally opening a checkpoint file, or a wallet/.env/ledger path) would pass both PROP-CR9 and PROP-CR12(a) undetected. PROP-CR12(c)'s dynamic before/after directory-snapshot only catches mutations (mtime changes), not read-only opens, so it would not catch this either. This leaves REQ-CR11's own explicit, sharpened 'never read byte contents, __main__ included' claim without a dedicated static or dynamic PROP that actually exercises it end-to-end.", + "evidence": { + "filePath": ".vcsdd/features/self-improve-checkpoint-resume/specs/verification-architecture.md", + "lineRange": "69,73", + "snippet": "PROP-CR9 | ... find_latest_checkpoint's module source (lib/checkpoint_resume.py, excluding its own __main__ CLI-entrypoint block ...) contains no open(, subprocess, requests, socket, or urllib reference ... / PROP-CR12 | ... (a) static denylist ... contains no reference to os.remove, ... or open( used with a non-default (write/append/create/truncate) mode" + }, + "routeToPhase": "1c-polish-optional" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/verdict.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/verdict.json new file mode 100644 index 00000000..e5ba07d7 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/spec/iteration-5/output/verdict.json @@ -0,0 +1,47 @@ +{ + "feature": "self-improve-checkpoint-resume", + "reviewType": "spec", + "overallVerdict": "PASS", + "timestamp": "2026-07-11T00:00:00Z", + "iteration": 5, + "dimensions": [ + { + "name": "spec_fidelity", + "verdict": "PASS", + "findings": [], + "evidence": [ + { + "type": "file", + "location": "specs/behavioral-spec.md:209-225 (REQ-CR6)", + "description": "Confirmed FIND-001 (iteration-4) fix is real and accurate: REQ-CR6 now explicitly states the exit-status capture is 'a NEW capture this feature introduces, not a reuse of an existing pattern', and correctly cites that run_evolve.sh's existing OBSERVE step (run_evolve.sh:60, read directly) never inspects $? after the OBSERVE_JSON=\"$(...)\" command substitution. The corrected citation matches the actual source exactly (no `$?` check appears anywhere near run_evolve.sh:59-64)." + }, + { + "type": "file", + "location": "specs/behavioral-spec.md:135-158,177-189,374-383 (REQ-CR2, REQ-CR10, EDGE-CR12) and verification-architecture.md:72 (PROP-CR14)", + "description": "Confirmed FIND-002 (iteration-4) fix is real and adequate: new EDGE-CR12 requires REQ-CR3 to treat a run-shaped-named PLAIN FILE exactly like 'no checkpoints/ subdirectory', explicitly mandates a check-before-acting approach (os.path.isdir before listing) rather than exception-catching, and REQ-CR2's own text (lines 150-152) folds this case into its enumerated 'REQ-CR3 returns None for ANY reason' list. New PROP-CR14 pins a concrete tmp_path fixture (zero-byte file named run- alongside a real older run with a valid checkpoint) that would fail today (module does not exist) and would pass once EDGE-CR12 is implemented as specified." + } + ] + }, + { + "name": "verification_readiness", + "verdict": "PASS", + "findings": ["FIND-001", "FIND-002"], + "evidence": [ + { + "type": "file", + "location": "specs/behavioral-spec.md:309-312 (INV-CR2, INV-CR3, INV-CR5)", + "description": "Confirmed FIND-003 (iteration-4) fix is real: INV-CR2, INV-CR3, and INV-CR5 each now carry an explicit PROP cross-reference parenthetical (PROP-CR9/PROP-CR12 for CR2, PROP-CR9+git-diff for CR3, PROP-CR-WIRE1/2/2b for CR5), closing the asymmetry with INV-CR1 that iteration-4 flagged. Non-blocking follow-on: FIND-001 notes the INV-CR2 citation of PROP-CR9/PROP-CR12 overclaims what those two PROPs literally scan for (see finding for detail); this does not remove INV-CR2's actual coverage, which the same parenthetical's git-diff-scope-check half already supplies." + } + ] + } + ], + "convergenceSignals": { + "findingCount": 2, + "allCriteriaEvaluated": false, + "evaluatedCriteria": [], + "duplicateFindings": [], + "priorIterationFindingsVerifiedFixed": ["FIND-001-iter4", "FIND-002-iter4", "FIND-003-iter4"], + "newFindingsThisIteration": ["FIND-001", "FIND-002"], + "blockingFindingCount": 0 + } +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/state.json b/.vcsdd/features/self-improve-checkpoint-resume/state.json index edc18dfe..54babd56 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/state.json +++ b/.vcsdd/features/self-improve-checkpoint-resume/state.json @@ -2,7 +2,7 @@ "featureName": "self-improve-checkpoint-resume", "mode": "lean", "language": "python", - "currentPhase": "1b", + "currentPhase": "2a", "phaseHistory": [ { "from": "init", @@ -13,19 +13,44 @@ "from": "1a", "to": "1b", "timestamp": "2026-07-11T09:01:42.933Z" + }, + { + "from": "1b", + "to": "1c", + "timestamp": "2026-07-11T09:48:45.128Z" + }, + { + "from": "1c", + "to": "2a", + "timestamp": "2026-07-11T09:48:45.135Z", + "sprint": 1 } ], "iterations": { "1a": 1, - "1b": 1 + "1b": 1, + "1c": 1, + "2a": 1 }, "proofObligations": [], "traceability": { "epicId": "VCSDD-self-improve-checkpoint-resume-1783760284512", "beads": [] }, - "gates": {}, - "sprintCount": 0, + "gates": { + "1c": { + "verdict": "PASS", + "timestamp": "2026-07-11T09:48:39.607Z", + "reviewedBy": "adversary", + "details": { + "iteration": 5, + "blockingFindings": 0 + }, + "adversaryVerdict": "PASS", + "adversaryReviewedAt": "2026-07-11T09:48:39.607Z" + } + }, + "sprintCount": 1, "createdAt": "2026-07-11T08:58:04.512Z", - "updatedAt": "2026-07-11T09:01:42.933Z" + "updatedAt": "2026-07-11T09:48:45.135Z" } diff --git a/.vcsdd/history.jsonl b/.vcsdd/history.jsonl index 0fba9977..32284ce9 100644 --- a/.vcsdd/history.jsonl +++ b/.vcsdd/history.jsonl @@ -348,3 +348,7 @@ {"timestamp":"2026-07-11T08:58:04.515Z","event":"feature_created","featureName":"self-improve-checkpoint-resume","mode":"lean","language":"python"} {"timestamp":"2026-07-11T09:01:42.933Z","event":"phase_transition","featureName":"self-improve-checkpoint-resume","from":"init","to":"1a","iteration":1,"phase":"1a"} {"timestamp":"2026-07-11T09:01:42.934Z","event":"phase_transition","featureName":"self-improve-checkpoint-resume","from":"1a","to":"1b","iteration":1,"phase":"1b"} +{"timestamp":"2026-07-11T09:48:39.608Z","event":"gate_recorded","featureName":"self-improve-checkpoint-resume","phase":"1c","verdict":"PASS","reviewedBy":"adversary"} +{"timestamp":"2026-07-11T09:48:45.130Z","event":"phase_transition","featureName":"self-improve-checkpoint-resume","from":"1b","to":"1c","iteration":1,"phase":"1c"} +{"timestamp":"2026-07-11T09:48:45.136Z","event":"phase_transition","featureName":"self-improve-checkpoint-resume","from":"1c","to":"2a","iteration":1,"phase":"2a"} +{"timestamp":"2026-07-11T09:48:45.136Z","event":"sprint_started","featureName":"self-improve-checkpoint-resume","sprintNumber":1,"phase":"2a"} diff --git a/.vcsdd/index.json b/.vcsdd/index.json index af8f9f97..30503d37 100644 --- a/.vcsdd/index.json +++ b/.vcsdd/index.json @@ -244,8 +244,8 @@ "createdAt": "2026-07-11T08:58:04.512Z", "mode": "lean", "language": "python", - "currentPhase": "1b", - "updatedAt": "2026-07-11T09:01:42.933Z" + "currentPhase": "2a", + "updatedAt": "2026-07-11T09:48:45.135Z" } }, "activeFeature": "self-improve-checkpoint-resume" From 515eb0895f73c1c14726d589bf867134b77946b6 Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sun, 12 Jul 2026 07:18:54 +0900 Subject: [PATCH 07/11] =?UTF-8?q?vcsdd(self-improve-checkpoint-resume):=20?= =?UTF-8?q?RED+GREEN=20phase=202a/2b=20=E2=80=94=20checkpoint=5Fresume.py?= =?UTF-8?q?=20+=20run=5Fevolve.sh=20wiring?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit lib/checkpoint_resume.py::find_latest_checkpoint (REQ-CR1-5,10,11): pure, read-only checkpoint discovery across runs/*/checkpoints/checkpoint_ dirs. run_evolve.sh gains one pre-invocation step (REQ-CR6-9,12) that resumes openevolve-run from the most recent prior cycle's checkpoint instead of cold-starting every ~6h cycle, degrading safely to today's exact fallback on any crash or missing prior checkpoint (INV-CR4). 31/31 new tests pass; 18 pre-existing test failures confirmed identical before/after (unrelated pm_backtest_strategy.py fixture drift + one worktree-path-only test, both present on main HEAD too — zero regressions introduced). bash 3.2 gotcha caught and fixed before shipping: "${CHECKPOINT_ARGS[@]}" on an empty array raises 'unbound variable' under set -u on bash 3.2 (this Mac's default /bin/bash) — used "${CHECKPOINT_ARGS[@]:-}" instead, verified live. --- .../evidence/green-phase.md | 582 ++++++++++++++++++ .../evidence/red-phase.md | 6 + .../self-improve/lib/checkpoint_resume.py | 81 +++ skills/earn/self-improve/run_evolve.sh | 22 + .../tests/test_checkpoint_resume.py | 417 +++++++++++++ .../tests/test_checkpoint_resume_wiring.py | 321 ++++++++++ 6 files changed, 1429 insertions(+) create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/evidence/green-phase.md create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/evidence/red-phase.md create mode 100644 skills/earn/self-improve/lib/checkpoint_resume.py create mode 100644 skills/earn/self-improve/tests/test_checkpoint_resume.py create mode 100644 skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py diff --git a/.vcsdd/features/self-improve-checkpoint-resume/evidence/green-phase.md b/.vcsdd/features/self-improve-checkpoint-resume/evidence/green-phase.md new file mode 100644 index 00000000..d9bfe4ad --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/evidence/green-phase.md @@ -0,0 +1,582 @@ +green-phase: GREEN — all 31 new tests pass, 0 regressions + +=== 31 new tests (test_checkpoint_resume.py + test_checkpoint_resume_wiring.py) === +============================= test session starts ============================== +platform darwin -- Python 3.14.6, pytest-9.1.1, pluggy-1.6.0 -- /Users/anicca/.anicca-venvs/self-improve/bin/python3 +cachedir: .pytest_cache +rootdir: /Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/skills/earn/self-improve +plugins: anyio-4.14.1 +collecting ... collected 31 items + +tests/test_checkpoint_resume.py::test_prop_cr1_returns_none_for_nonexistent_runs_dir PASSED [ 3%] +tests/test_checkpoint_resume.py::test_prop_cr1b_relative_runs_dir_returns_absolute_path PASSED [ 6%] +tests/test_checkpoint_resume.py::test_prop_cr2_empty_runs_dir_returns_none PASSED [ 9%] +tests/test_checkpoint_resume.py::test_prop_cr2_runs_dir_with_only_current_run_id_returns_none PASSED [ 12%] +tests/test_checkpoint_resume.py::test_prop_cr3_selects_most_recent_run_by_name_not_highest_checkpoint_number PASSED [ 16%] +tests/test_checkpoint_resume.py::test_prop_cr4_integer_comparison_not_lexicographic PASSED [ 19%] +tests/test_checkpoint_resume.py::test_prop_cr5_run_without_checkpoints_subdirectory_is_skipped PASSED [ 22%] +tests/test_checkpoint_resume.py::test_prop_cr5_run_with_literally_empty_checkpoints_subdirectory_is_skipped PASSED [ 25%] +tests/test_checkpoint_resume.py::test_prop_cr6_ignores_non_matching_entries_interleaved_with_a_valid_one PASSED [ 29%] +tests/test_checkpoint_resume.py::test_prop_cr7_current_run_id_excluded_even_with_its_own_checkpoints PASSED [ 32%] +tests/test_checkpoint_resume.py::test_prop_cr8_selected_purely_by_name_empty_checkpoint_dir_never_inspected PASSED [ 35%] +tests/test_checkpoint_resume.py::test_prop_cr8_selected_purely_by_name_garbage_content_never_inspected PASSED [ 38%] +tests/test_checkpoint_resume.py::test_prop_cr9_pure_function_source_has_no_effectful_references_outside_main PASSED [ 41%] +tests/test_checkpoint_resume.py::test_prop_cr10_non_run_shaped_db_sibling_alone_returns_none PASSED [ 45%] +tests/test_checkpoint_resume.py::test_prop_cr10_non_run_shaped_db_sibling_ignored_when_a_real_run_also_present PASSED [ 48%] +tests/test_checkpoint_resume.py::test_prop_cr11_stray_file_only_checkpoints_dir_falls_through_to_older_run PASSED [ 51%] +tests/test_checkpoint_resume.py::test_prop_cr14_run_shaped_plain_file_falls_through_without_raising PASSED [ 54%] +tests/test_checkpoint_resume.py::test_prop_cr14_run_shaped_plain_file_only_returns_none PASSED [ 58%] +tests/test_checkpoint_resume.py::test_prop_cr12a_full_module_source_has_no_filesystem_mutation_api_references PASSED [ 61%] +tests/test_checkpoint_resume.py::test_prop_cr12b_module_never_imports_pathlib PASSED [ 64%] +tests/test_checkpoint_resume.py::test_prop_cr12c_no_mutation_across_a_none_returning_call PASSED [ 67%] +tests/test_checkpoint_resume.py::test_prop_cr12c_no_mutation_across_a_real_path_returning_call PASSED [ 70%] +tests/test_checkpoint_resume.py::test_req_cr4_runs_dir_with_only_non_run_shaped_entries_returns_none PASSED [ 74%] +tests/test_checkpoint_resume_wiring.py::test_prop_cr_wire1_pinned_checkpoint_resume_call_present_and_before_openevolve_invocation PASSED [ 77%] +tests/test_checkpoint_resume_wiring.py::test_prop_cr_wire2_no_checkpoint_found_argument_list_unchanged_and_logged PASSED [ 80%] +tests/test_checkpoint_resume_wiring.py::test_prop_cr_wire2b_checkpoint_found_appends_single_checkpoint_flag PASSED [ 83%] +tests/test_checkpoint_resume_wiring.py::test_prop_cr_wire3_exactly_one_checkpoint_resume_log_line_not_found_branch PASSED [ 87%] +tests/test_checkpoint_resume_wiring.py::test_prop_cr_wire3_exactly_one_checkpoint_resume_log_line_found_branch PASSED [ 90%] +tests/test_checkpoint_resume_wiring.py::test_prop_cr13_resume_check_crash_falls_back_and_logs_distinct_message PASSED [ 93%] +tests/test_checkpoint_resume_wiring.py::test_prop_cr_live1_end_to_end_two_run_fixture_selects_and_logs_checkpoint_20 PASSED [ 96%] +tests/test_checkpoint_resume_wiring.py::test_regression_run_evolve_sh_remains_valid_bash_syntax PASSED [100%] + +============================== 31 passed in 1.55s ============================== + +=== full suite regression check (tests/) === +FFFFFFFFFF..................................F.......FF.............F.... [ 60%] +..................FF.....................FF..... [100%] +=================================== FAILURES =================================== +_____________ test_good_candidate_is_eligible_for_adversary_review _____________ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_good_candidate_is_eligibl0') + + def test_good_candidate_is_eligible_for_adversary_review(tmp_path): +> _, candidate_path = _good_candidate(tmp_path) + ^^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_adversary_disapprove.py:47: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_adversary_disapprove.py:26: in _good_candidate + code = patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +__ test_adversary_fail_blocks_promotion_even_though_deterministic_gates_pass ___ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_adversary_fail_blocks_pro0') + + def test_adversary_fail_blocks_promotion_even_though_deterministic_gates_pass(tmp_path): +> candidate_code, candidate_path = _good_candidate(tmp_path) + ^^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_adversary_disapprove.py:59: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_adversary_disapprove.py:26: in _good_candidate + code = patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +_ test_adversary_missing_or_erroring_verdict_also_blocks_promotion_fail_closed _ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_adversary_missing_or_erro0') + + def test_adversary_missing_or_erroring_verdict_also_blocks_promotion_fail_closed(tmp_path): + """EDGE-3: the adversary being unavailable/erroring must BLOCK promotion, never silently skip + the check (fail-closed), even though the deterministic gates passed.""" +> candidate_code, candidate_path = _good_candidate(tmp_path) + ^^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_adversary_disapprove.py:80: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_adversary_disapprove.py:26: in _good_candidate + code = patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +_ test_adversary_pass_on_eligible_candidate_promotes_and_calls_promote_exactly_once _ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_adversary_pass_on_eligibl0') + + def test_adversary_pass_on_eligible_candidate_promotes_and_calls_promote_exactly_once(tmp_path): +> candidate_code, candidate_path = _good_candidate(tmp_path) + ^^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_adversary_disapprove.py:94: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_adversary_disapprove.py:26: in _good_candidate + code = patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +_ test_ineligible_regressing_candidate_never_becomes_eligible_for_adversary_review _ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_ineligible_regressing_can0') + + def test_ineligible_regressing_candidate_never_becomes_eligible_for_adversary_review(tmp_path): + """A candidate that already fails a deterministic gate (the same loose-threshold regressing + candidate from test_heldout_regress.py) must never even be marked eligible for adversary + review — this is the flag `promote_gate.sh`'s real shell wrapper uses to skip the (costly) + LLM call entirely, not just skip promotion after asking.""" +> _, candidate_path = _regressing_candidate(tmp_path) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_adversary_disapprove.py:118: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_adversary_disapprove.py:36: in _regressing_candidate + code = patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("min_edge", 0.18)', 'config.get("min_edge", 0.01)'), ('config.get("min_confidence", 7.0)', 'config.get("...ce", 0.85)', 'config.get("max_price", 1.0)'), ('config.get("min_liquidity", 0.4)', 'config.get("min_liquidity", 0.0)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("min_edge", 0.18)' +E assert 'config.get("min_edge", 0.18)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +___________________ test_better_candidate_scope_guard_passes ___________________ + + def test_better_candidate_scope_guard_passes(): + baseline_code = read_baseline_code() +> candidate_code = _better_candidate_code() + ^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_baseline_beat.py:51: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_baseline_beat.py:43: in _better_candidate_code + return patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +_________ test_better_candidate_beats_baseline_on_stage2_walk_forward __________ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_better_candidate_beats_ba0') + + def test_better_candidate_beats_baseline_on_stage2_walk_forward(tmp_path): +> candidate_code = _better_candidate_code() + ^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_baseline_beat.py:60: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_baseline_beat.py:43: in _better_candidate_code + return patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +___________ test_better_candidate_has_non_negative_worst_window_oos ____________ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_better_candidate_has_non_0') + + def test_better_candidate_has_non_negative_worst_window_oos(tmp_path): + """Strengthens the beats-baseline claim above: the improvement must not be an aggregate-only + artifact (EDGE-5) — the WORST individual OOS window must itself be non-negative. Post-promotion + (commit a6f608c) the committed baseline ALREADY has a non-negative worst OOS window (the real + promoted candidate fixed that exact problem for the pre-promotion baseline) — this test now + asserts the candidate PRESERVES that non-negative-worst-window property while still improving + mean/std further, not that it fixes a still-losing baseline (there is no longer one).""" +> candidate_code = _better_candidate_code() + ^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_baseline_beat.py:77: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_baseline_beat.py:43: in _better_candidate_code + return patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +__________ test_full_evaluate_cascade_and_promotion_gate_return_true ___________ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_full_evaluate_cascade_and0') + + def test_full_evaluate_cascade_and_promotion_gate_return_true(tmp_path): +> candidate_code = _better_candidate_code() + ^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_baseline_beat.py:95: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_baseline_beat.py:43: in _better_candidate_code + return patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +____ test_promotion_gate_is_not_trivially_true_adversary_fail_still_blocks _____ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_promotion_gate_is_not_tri0') + + def test_promotion_gate_is_not_trivially_true_adversary_fail_still_blocks(tmp_path): + """Strengthens the "returns True" claim above: the SAME passing candidate must NOT be + promotable if the fresh adversary verdict is not PASS (REQ-RH4 step 3) or if the trip-wire is + not clear (REQ-RH4 step 2) — the gate is a real conjunction, not a constant.""" +> candidate_code = _better_candidate_code() + ^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_baseline_beat.py:120: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_baseline_beat.py:43: in _better_candidate_code + return patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +_____ test_evolve_block_only_edit_with_no_denylisted_reference_is_accepted _____ + + def test_evolve_block_only_edit_with_no_denylisted_reference_is_accepted(): + # Control case: an EVOLVE-BLOCK-only edit with clean content must PASS, proving the three + # rejections above fire for the RIGHT reason (scope/denylist), not because ANY edit at all is + # rejected. Literal updated 2026-07-08 (commit a6f608c promoted a new baseline whose + # min_confidence default is 7.0, not the pre-promotion 6.0) — this test only needs SOME clean + # in-scope edit, not any particular beats-baseline property. + baseline_code = read_baseline_code() +> candidate_code = patched_baseline_code( + ('config.get("min_confidence", 7.0)', 'config.get("min_confidence", 9.0)') + ) + +tests/test_denylist_reject.py:70: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("min_confidence", 7.0)', 'config.get("min_confidence", 9.0)'),) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("min_confidence", 7.0)' +E assert 'config.get("min_confidence", 7.0)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +_ test_regressing_candidate_passes_stage1_but_scores_worse_than_baseline_on_stage2_oos _ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_regressing_candidate_pass0') + + def test_regressing_candidate_passes_stage1_but_scores_worse_than_baseline_on_stage2_oos(tmp_path): + baseline_code = read_baseline_code() +> candidate_code = _regressing_candidate_code() + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_heldout_regress.py:47: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_heldout_regress.py:36: in _regressing_candidate_code + return patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("min_edge", 0.18)', 'config.get("min_edge", 0.01)'), ('config.get("min_confidence", 7.0)', 'config.get("...ce", 0.85)', 'config.get("max_price", 1.0)'), ('config.get("min_liquidity", 0.4)', 'config.get("min_liquidity", 0.0)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("min_edge", 0.18)' +E assert 'config.get("min_edge", 0.18)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +_____________ test_regressing_candidate_is_not_promoted_end_to_end _____________ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_regressing_candidate_is_n0') + + def test_regressing_candidate_is_not_promoted_end_to_end(tmp_path): + """Full evaluate() cascade + the promotion gate itself: even though the candidate "improved" + on the cheap stage1 filter, the ordered gate (REQ-RH4) must return False for promotion.""" +> candidate_code = _regressing_candidate_code() + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_heldout_regress.py:73: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_heldout_regress.py:36: in _regressing_candidate_code + return patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("min_edge", 0.18)', 'config.get("min_edge", 0.01)'), ('config.get("min_confidence", 7.0)', 'config.get("...ce", 0.85)', 'config.get("max_price", 1.0)'), ('config.get("min_liquidity", 0.4)', 'config.get("min_liquidity", 0.0)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("min_edge", 0.18)' +E assert 'config.get("min_edge", 0.18)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +__ test_realized_summary_default_path_points_at_the_real_earn_ledger_location __ + + def test_realized_summary_default_path_points_at_the_real_earn_ledger_location(): +> assert ledger_reader.DEFAULT_LEDGER_PATH.endswith("anicca/skills/earn/state/earn-ledger.jsonl") +E AssertionError: assert False +E + where False = ('anicca/skills/earn/state/earn-ledger.jsonl') +E + where = '/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/skills/earn/state/earn-ledger.jsonl'.endswith +E + where '/Users/anicca/anicca/.worktrees/self-improve-checkpoint-resume/skills/earn/state/earn-ledger.jsonl' = ledger_reader.DEFAULT_LEDGER_PATH + +tests/test_ledger_reader.py:192: AssertionError +_ test_resolved_false_blocks_promotion_unconditionally_even_with_adversary_pass _ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_resolved_false_blocks_pro0') + + def test_resolved_false_blocks_promotion_unconditionally_even_with_adversary_pass(tmp_path): + from lib import promote_gate + +> assessment = _eligible_assessment(tmp_path) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_realized_gate.py:176: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_realized_gate.py:145: in _eligible_assessment + code = patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +_______ test_gate_none_vacuous_pass_vs_unconditional_block_side_by_side ________ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_gate_none_vacuous_pass_vs0') + + def test_gate_none_vacuous_pass_vs_unconditional_block_side_by_side(tmp_path): + """spec-review F-4: the SAME assessment+adversary-PASS inputs must produce promote:True with + realized_gate=None (REQ-RL18 vacuous pass) and promote:False with realized_gate={"resolved": + False, ...} (REQ-RL7 unconditional block) — one test, two assertions, no ambiguity.""" + from lib import promote_gate + +> assessment = _eligible_assessment(tmp_path) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +tests/test_realized_gate.py:191: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_realized_gate.py:145: in _eligible_assessment + code = patched_baseline_code( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +_____ test_leverage_only_candidate_does_not_beat_baseline_on_real_fixture ______ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_leverage_only_candidate_d0') + + def test_leverage_only_candidate_does_not_beat_baseline_on_real_fixture(tmp_path): + """End-to-end (not just the pure-math proof above): a candidate whose ONLY EVOLVE-BLOCK change + is `base_stake: 5.0 -> 20.0` (identical min_edge/min_confidence thresholds -> identical trade + selection every window) must score IDENTICALLY to baseline's own risk-adjusted combined_score + on the real committed fixture, and therefore must NOT beat_baseline (a tie never beats, + EDGE-2).""" + leverage_code = patched_baseline_code(('config.get("base_stake", 5.0)', 'config.get("base_stake", 20.0)')) + leverage_path = write_candidate_with_fixtures(tmp_path, leverage_code) + + baseline_stage2 = evaluator.evaluate_stage2(evaluator.BASELINE_PATH) + leverage_stage2 = evaluator.evaluate_stage2(leverage_path) + +> assert math.isclose(leverage_stage2["combined_score"], baseline_stage2["combined_score"], rel_tol=1e-9) +E assert False +E + where False = (2.9014000695008613, 4.015488840463803, rel_tol=1e-09) +E + where = math.isclose + +tests/test_risk_adjusted_fitness.py:73: AssertionError +____ test_genuine_selection_change_beats_baseline_where_leverage_could_not _____ + +tmp_path = PosixPath('/private/var/folders/c1/11fd4j597vx4fb8tnv25y28r0000gn/T/pytest-of-anicca/pytest-18/test_genuine_selection_change_0') + + def test_genuine_selection_change_beats_baseline_where_leverage_could_not(tmp_path): + """Contrast case, same fixture: a genuine selection/sizing-emphasis change (real openevolve + promotion, commit a6f608c, 2026-07-08 — see test_baseline_beat.py's module docstring for the + full re-derivation against the NEW committed baseline) DOES beat baseline, unlike the + leverage-only candidate above — proving `risk_adjusted_score` rewards genuine edge-finding, + not just any code change.""" + baseline_code = read_baseline_code() +> selection_code = patched_baseline_code( + ('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), + ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)'), + ) + +tests/test_risk_adjusted_fitness.py:84: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +replacements = (('config.get("edge_weight", 0.25)', 'config.get("edge_weight", 0.4)'), ('config.get("conf_weight", 0.45)', 'config.get("conf_weight", 0.1)')) + + def patched_baseline_code(*replacements: tuple) -> str: + """Apply a sequence of (old, new) `str.replace(old, new, 1)` edits to the real baseline + program text. Asserts every replacement actually matched something in the current baseline + file, so a typo'd test literal fails loudly instead of silently producing a byte-identical + "candidate" that trivially passes scope_guard for the wrong reason.""" + code = read_baseline_code() + for old, new in replacements: +> assert old in code, f"expected baseline text not found (test fixture drifted?): {old!r}" +E AssertionError: expected baseline text not found (test fixture drifted?): 'config.get("edge_weight", 0.25)' +E assert 'config.get("edge_weight", 0.25)' in '"""pm_backtest_strategy.py — the self-contained, deterministic, backtestable openevolve program\nthis feature evolves...ost_usd": cost_usd, "net_usd": net, "n_trades": n_trades}\n\n\nif __name__ == "__main__":\n print(run_backtest())\n' + +tests/conftest.py:37: AssertionError +=========================== short test summary info ============================ +FAILED tests/test_adversary_disapprove.py::test_good_candidate_is_eligible_for_adversary_review +FAILED tests/test_adversary_disapprove.py::test_adversary_fail_blocks_promotion_even_though_deterministic_gates_pass +FAILED tests/test_adversary_disapprove.py::test_adversary_missing_or_erroring_verdict_also_blocks_promotion_fail_closed +FAILED tests/test_adversary_disapprove.py::test_adversary_pass_on_eligible_candidate_promotes_and_calls_promote_exactly_once +FAILED tests/test_adversary_disapprove.py::test_ineligible_regressing_candidate_never_becomes_eligible_for_adversary_review +FAILED tests/test_baseline_beat.py::test_better_candidate_scope_guard_passes +FAILED tests/test_baseline_beat.py::test_better_candidate_beats_baseline_on_stage2_walk_forward +FAILED tests/test_baseline_beat.py::test_better_candidate_has_non_negative_worst_window_oos +FAILED tests/test_baseline_beat.py::test_full_evaluate_cascade_and_promotion_gate_return_true +FAILED tests/test_baseline_beat.py::test_promotion_gate_is_not_trivially_true_adversary_fail_still_blocks +FAILED tests/test_denylist_reject.py::test_evolve_block_only_edit_with_no_denylisted_reference_is_accepted +FAILED tests/test_heldout_regress.py::test_regressing_candidate_passes_stage1_but_scores_worse_than_baseline_on_stage2_oos +FAILED tests/test_heldout_regress.py::test_regressing_candidate_is_not_promoted_end_to_end +FAILED tests/test_ledger_reader.py::test_realized_summary_default_path_points_at_the_real_earn_ledger_location +FAILED tests/test_realized_gate.py::test_resolved_false_blocks_promotion_unconditionally_even_with_adversary_pass +FAILED tests/test_realized_gate.py::test_gate_none_vacuous_pass_vs_unconditional_block_side_by_side +FAILED tests/test_risk_adjusted_fitness.py::test_leverage_only_candidate_does_not_beat_baseline_on_real_fixture +FAILED tests/test_risk_adjusted_fitness.py::test_genuine_selection_change_beats_baseline_where_leverage_could_not +18 failed, 102 passed, 1 skipped in 2.97s + +=== bash -n run_evolve.sh (syntax) === +OK diff --git a/.vcsdd/features/self-improve-checkpoint-resume/evidence/red-phase.md b/.vcsdd/features/self-improve-checkpoint-resume/evidence/red-phase.md new file mode 100644 index 00000000..b6b83c8b --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/evidence/red-phase.md @@ -0,0 +1,6 @@ +new-feature-tests: FAIL (pre-implementation, confirmed via git stash of run_evolve.sh + lib/checkpoint_resume.py) +regression-baseline: PASS + +=== RED (stashed impl, tests only) === +1 collection error (ImportError: cannot import name checkpoint_resume) blocking test_checkpoint_resume.py's 23 tests +test_checkpoint_resume_wiring.py: 7 failed, 1 passed (PROP-CR-WIRE1/2/2b/3x2/13/LIVE1 all failed as expected pre-wiring) diff --git a/skills/earn/self-improve/lib/checkpoint_resume.py b/skills/earn/self-improve/lib/checkpoint_resume.py new file mode 100644 index 00000000..866564a8 --- /dev/null +++ b/skills/earn/self-improve/lib/checkpoint_resume.py @@ -0,0 +1,81 @@ +"""checkpoint_resume.py — self-improve-checkpoint-resume (REQ-CR1..12). + +Pure checkpoint-discovery core for run_evolve.sh's recurring ~6h self-improve cycle (REQ-OE7). +Every cycle today starts openevolve-run from a cold population (initial_program only) and +discards its own MAP-Elites checkpoints at cycle end — this module lets a cycle resume from the +most recent PRIOR cycle's highest-numbered checkpoint instead, so iterations accumulate across +cycles rather than re-deriving the same already-known baseline every ~6h (the confirmed root +cause of 3 consecutive tied combined_score cycles / 18h zero forward progress — see +behavioral-spec.md sources). + +`find_latest_checkpoint` is PATH/EXISTENCE logic only (REQ-CR5, REQ-CR11): directory listing, +name-pattern matching, integer comparison. It never opens a file's byte contents, never mutates +the filesystem, and never imports pathlib (os/os.path suffice for every operation this needs). +""" +import os +import re +import sys + +_RUN_ID_RE = re.compile(r"^run-\d{8}T\d{6}Z$") +_CHECKPOINT_RE = re.compile(r"^checkpoint_(\d+)$") + + +def _latest_checkpoint_in_run(run_dir): + """REQ-CR3: the absolute path of the highest-numbered checkpoint_ under run_dir/checkpoints, + or None if run_dir has no checkpoints/ subdirectory, an empty one, or one with zero + checkpoint_-shaped entries. Never raises — a run-shaped candidate that is a plain FILE + (EDGE-CR12) is checked with os.path.isdir before any listdir is attempted.""" + checkpoints_dir = os.path.join(run_dir, "checkpoints") + if not os.path.isdir(checkpoints_dir): + return None + + best_n = None + best_name = None + for entry in os.listdir(checkpoints_dir): + match = _CHECKPOINT_RE.match(entry) + if match is None: + continue + n = int(match.group(1)) + if best_n is None or n > best_n: + best_n = n + best_name = entry + + if best_name is None: + return None + return os.path.abspath(os.path.join(checkpoints_dir, best_name)) + + +def find_latest_checkpoint(runs_dir, current_run_id): + """REQ-CR1: return the absolute path to a checkpoint directory to resume from, or None when + no prior run has any usable checkpoint. runs_dir may be relative -- the result is always + normalized to an absolute path (REQ-CR1).""" + runs_dir_abs = os.path.abspath(runs_dir) + if not os.path.isdir(runs_dir_abs): + return None + + candidates = [ + entry for entry in os.listdir(runs_dir_abs) + if _RUN_ID_RE.match(entry) and entry != current_run_id + ] + # REQ-CR2: run directory names (run-YYYYMMDDTHHMMSSZ) sort correctly as plain strings; + # descending lexicographic order == most-recent-first. + candidates.sort(reverse=True) + + for candidate in candidates: + checkpoint = _latest_checkpoint_in_run(os.path.join(runs_dir_abs, candidate)) + if checkpoint is not None: + return checkpoint + + return None + + +if __name__ == "__main__": + # Runnable as `RUNS_DIR=... RUN_ID=... python3 lib/checkpoint_resume.py` for run_evolve.sh's + # new pre-invocation step (REQ-CR6): no argv, env-var config only (mirrors lib/ledger_reader.py's + # own __main__ convention). Prints EXACTLY one line to stdout -- the resolved checkpoint path, + # or an empty string when None -- nothing else. Never writes to any file (REQ-CR11). + result = find_latest_checkpoint( + runs_dir=os.environ["RUNS_DIR"], + current_run_id=os.environ["RUN_ID"], + ) + sys.stdout.write(result if result is not None else "") diff --git a/skills/earn/self-improve/run_evolve.sh b/skills/earn/self-improve/run_evolve.sh index cc027f25..29dff196 100755 --- a/skills/earn/self-improve/run_evolve.sh +++ b/skills/earn/self-improve/run_evolve.sh @@ -63,6 +63,27 @@ else echo "$(now) OBSERVE SKIPPED (python not found at $PY_BIN) — proceeding to EVOLVE anyway" >> "$LOG" fi +# ---- checkpoint_resume (self-improve-checkpoint-resume, REQ-CR6-9,12): resume this cycle's +# openevolve-run from the most recent PRIOR cycle's checkpoint instead of cold-starting every +# ~6h cycle (root cause of 3 consecutive tied-score, zero-forward-progress cycles — see +# behavioral-spec.md sources). Read-only path selection only (lib/checkpoint_resume.py never +# mutates anything on disk); a crashed/missing-$PY_BIN resume-check degrades to the same +# no-checkpoint fallback as "no prior run exists yet" for the purposes of this invocation +# (REQ-CR12) — resuming is a pure optimization, never a precondition for this cycle to run. +# Positioned BEFORE the $OPENEVOLVE_BIN existence check (not just before its invocation) so +# resume-check evidence lands in $LOG even on the rare path where openevolve itself is missing. +CHECKPOINT_PATH="$(RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py" 2>>"$LOG")" +CHECKPOINT_STATUS=$? +CHECKPOINT_ARGS=() +if [ "$CHECKPOINT_STATUS" -ne 0 ]; then + echo "$(now) checkpoint_resume: resume-check crashed exit_status=$CHECKPOINT_STATUS" >> "$LOG" +elif [ -n "$CHECKPOINT_PATH" ]; then + echo "$(now) checkpoint_resume: resuming from checkpoint $CHECKPOINT_PATH" >> "$LOG" + CHECKPOINT_ARGS=(--checkpoint "$CHECKPOINT_PATH") +else + echo "$(now) checkpoint_resume: no prior checkpoint found" >> "$LOG" +fi + if [ ! -x "$OPENEVOLVE_BIN" ]; then # LOUD failure, never a silent "pretend success" exit 0: a missing openevolve binary means # REQ-OE7's recurring trigger cannot run at all, and that must be visible (non-zero exit + @@ -85,6 +106,7 @@ fi --config "$SKILL_DIR/config.yaml" \ --iterations "$ITERATIONS" \ --output "$RUN_DIR" \ + "${CHECKPOINT_ARGS[@]:-}" \ >> "$LOG" 2>&1 STATUS=$? diff --git a/skills/earn/self-improve/tests/test_checkpoint_resume.py b/skills/earn/self-improve/tests/test_checkpoint_resume.py new file mode 100644 index 00000000..eee343c1 --- /dev/null +++ b/skills/earn/self-improve/tests/test_checkpoint_resume.py @@ -0,0 +1,417 @@ +"""RED phase — self-improve-checkpoint-resume, Group CR-FIND (pure checkpoint discovery). + +Traces to behavioral-spec.md REQ-CR1-5, REQ-CR10, REQ-CR11, EDGE-CR1-12; verification- +architecture.md PROP-CR1, PROP-CR1b, PROP-CR2-12, PROP-CR14. + +`lib/checkpoint_resume.py` does not exist yet — every test below is EXPECTED TO FAIL at RED-phase +time (a collection-time `ModuleNotFoundError`/`ImportError` for `lib.checkpoint_resume`, or a +`FileNotFoundError` in the source-text-scan tests that open the not-yet-created file directly). + +EDGE-CR1: `runs/` is gitignored in this worktree (`skills/earn/self-improve/.gitignore` contains +`runs/`) — every fixture below builds its own synthetic `runs_dir` tree under pytest's `tmp_path`. +No test here ever reads, symlinks, or copies this worktree's own (nonexistent) `runs/` directory. +""" +import os +import re + +import pytest + +from lib import checkpoint_resume + +SELF_IMPROVE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +CHECKPOINT_RESUME_SOURCE_PATH = os.path.join(SELF_IMPROVE_DIR, "lib", "checkpoint_resume.py") + +# A fixed, always-valid current_run_id used by tests that don't care about its exact value (it +# just needs to match REQ-CR10's run-directory-name shape so it is a real candidate that gets +# correctly excluded per REQ-CR2/EDGE-CR9, not accidentally treated as non-run-shaped noise). +CURRENT_RUN_ID = "run-20260103T000000Z" + + +def _mkrun(runs_dir, run_id): + run_dir = runs_dir / run_id + run_dir.mkdir(parents=True, exist_ok=True) + return run_dir + + +def _mkcheckpoint(run_dir, n): + checkpoint_dir = run_dir / "checkpoints" / f"checkpoint_{n}" + checkpoint_dir.mkdir(parents=True, exist_ok=True) + return checkpoint_dir + + +# --- PROP-CR1: nonexistent runs_dir -> None (REQ-CR4, EDGE-CR2) --- + + +def test_prop_cr1_returns_none_for_nonexistent_runs_dir(tmp_path): + missing_runs_dir = tmp_path / "does-not-exist" + result = checkpoint_resume.find_latest_checkpoint(str(missing_runs_dir), CURRENT_RUN_ID) + assert result is None + + +# --- PROP-CR1b: relative runs_dir still returns an ABSOLUTE path (REQ-CR1) --- + + +def test_prop_cr1b_relative_runs_dir_returns_absolute_path(tmp_path, monkeypatch): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + older = _mkrun(runs_dir, "run-20260101T000000Z") + expected_checkpoint = _mkcheckpoint(older, 5) + monkeypatch.chdir(tmp_path) + + result = checkpoint_resume.find_latest_checkpoint("runs", CURRENT_RUN_ID) + + assert result is not None + assert os.path.isabs(result) is True + assert result == os.path.abspath(str(expected_checkpoint)) + + +# --- PROP-CR2: empty runs_dir, or runs_dir containing ONLY current_run_id -> None +# (REQ-CR4, EDGE-CR3, EDGE-CR9) --- + + +def test_prop_cr2_empty_runs_dir_returns_none(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + assert checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) is None + + +def test_prop_cr2_runs_dir_with_only_current_run_id_returns_none(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + _mkrun(runs_dir, CURRENT_RUN_ID) # its own freshly-created, empty directory + + assert checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) is None + + +# --- PROP-CR3: most recent run (by name) selected, never the older one, even when the older +# one has a higher checkpoint number (REQ-CR2, EDGE-CR7) --- + + +def test_prop_cr3_selects_most_recent_run_by_name_not_highest_checkpoint_number(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + older = _mkrun(runs_dir, "run-20260101T000000Z") + _mkcheckpoint(older, 100) + newer = _mkrun(runs_dir, "run-20260102T000000Z") + expected = _mkcheckpoint(newer, 10) + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(expected)) + + +# --- PROP-CR4: within the selected run, checkpoint_100 beats checkpoint_20 (integer, not +# lexicographic, comparison) (REQ-CR3) --- + + +def test_prop_cr4_integer_comparison_not_lexicographic(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + run = _mkrun(runs_dir, "run-20260101T000000Z") + _mkcheckpoint(run, 20) + expected = _mkcheckpoint(run, 100) + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(expected)) + + +# --- PROP-CR5: a prior run with no checkpoints/ subdir at all, or a literally empty one, is +# skipped in favor of the next-most-recent qualifying run (REQ-CR2, EDGE-CR4, EDGE-CR5) --- + + +def test_prop_cr5_run_without_checkpoints_subdirectory_is_skipped(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + _mkrun(runs_dir, "run-20260102T000000Z") # newer, but NO checkpoints/ dir at all + older = _mkrun(runs_dir, "run-20260101T000000Z") + expected = _mkcheckpoint(older, 5) + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(expected)) + + +def test_prop_cr5_run_with_literally_empty_checkpoints_subdirectory_is_skipped(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + newer = _mkrun(runs_dir, "run-20260102T000000Z") + (newer / "checkpoints").mkdir(parents=True) # exists, zero entries of any kind + older = _mkrun(runs_dir, "run-20260101T000000Z") + expected = _mkcheckpoint(older, 5) + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(expected)) + + +# --- PROP-CR6: non-checkpoint_-shaped entries interleaved with a valid one are ignored, +# never raise (REQ-CR3, EDGE-CR6) --- + + +def test_prop_cr6_ignores_non_matching_entries_interleaved_with_a_valid_one(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + run = _mkrun(runs_dir, "run-20260101T000000Z") + checkpoints_dir = run / "checkpoints" + checkpoints_dir.mkdir(parents=True) + (checkpoints_dir / ".DS_Store").write_text("junk") + (checkpoints_dir / "checkpoint_abc").mkdir() # non-numeric suffix: invalid shape + (checkpoints_dir / "not_a_checkpoint_dir").mkdir() + expected = _mkcheckpoint(run, 7) + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(expected)) + + +# --- PROP-CR7: current_run_id is always excluded, even when its own (empty, or adversarially +# non-empty) directory already exists under runs_dir at scan time (REQ-CR2, EDGE-CR9) --- + + +def test_prop_cr7_current_run_id_excluded_even_with_its_own_checkpoints(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + current_dir = _mkrun(runs_dir, CURRENT_RUN_ID) + _mkcheckpoint(current_dir, 999) # adversarial: current run "somehow" already has one + older = _mkrun(runs_dir, "run-20260101T000000Z") + expected = _mkcheckpoint(older, 1) + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(expected)) + + +# --- PROP-CR8: selection is purely name/existence-based — a checkpoint dir with zero internal +# files, or arbitrary garbage file contents, is still selected (REQ-CR5, EDGE-CR8) --- + + +def test_prop_cr8_selected_purely_by_name_empty_checkpoint_dir_never_inspected(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + run = _mkrun(runs_dir, "run-20260101T000000Z") + expected = _mkcheckpoint(run, 3) # completely empty — no internal openevolve state files + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(expected)) + + +def test_prop_cr8_selected_purely_by_name_garbage_content_never_inspected(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + run = _mkrun(runs_dir, "run-20260101T000000Z") + checkpoint_dir = _mkcheckpoint(run, 3) + (checkpoint_dir / "not_real_openevolve_state.bin").write_bytes(b"\x00\x01garbage\xff") + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(checkpoint_dir)) + + +# --- PROP-CR9: static source-text check — the pure function's own source (excluding its +# __main__ CLI-entrypoint block) references no open(/subprocess/requests/socket/urllib +# (INV-CR1) --- + + +def _read_checkpoint_resume_source(): + with open(CHECKPOINT_RESUME_SOURCE_PATH, "r", encoding="utf-8") as f: + return f.read() + + +def _source_before_main_block(source): + """Everything strictly BEFORE the `if __name__ == "__main__":` guard — i.e. the pure-function + portion of the module this PROP scopes to.""" + marker = re.search(r'^if __name__ == ["\']__main__["\']:', source, re.MULTILINE) + return source if marker is None else source[: marker.start()] + + +def test_prop_cr9_pure_function_source_has_no_effectful_references_outside_main(): + source = _read_checkpoint_resume_source() + pure_source = _source_before_main_block(source) + + for forbidden in ("open(", "subprocess", "requests", "socket", "urllib"): + assert forbidden not in pure_source, ( + f"forbidden effectful reference {forbidden!r} found in the pure-function portion of " + "lib/checkpoint_resume.py (PROP-CR9/INV-CR1)" + ) + + +# --- PROP-CR10: a runs_dir entry that does NOT match REQ-CR10's run-directory-name shape (e.g. +# the real production `runs/db` sibling) is NEVER a candidate, even with its own adversarial +# checkpoints/checkpoint_1/ subdirectory (REQ-CR10, EDGE-CR10) --- + + +def test_prop_cr10_non_run_shaped_db_sibling_alone_returns_none(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + db_dir = runs_dir / "db" + (db_dir / "checkpoints" / "checkpoint_1").mkdir(parents=True) # adversarial fixture + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result is None + + +def test_prop_cr10_non_run_shaped_db_sibling_ignored_when_a_real_run_also_present(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + db_dir = runs_dir / "db" + (db_dir / "checkpoints" / "checkpoint_1").mkdir(parents=True) + run = _mkrun(runs_dir, "run-20260101T000000Z") + expected = _mkcheckpoint(run, 2) + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(expected)) + + +# --- PROP-CR11: checkpoints/ non-empty at the OS-listing level but ZERO checkpoint_-shaped +# entries (e.g. a stray .DS_Store only) falls through to the next-most-recent candidate +# (REQ-CR2, REQ-CR3, EDGE-CR11) --- + + +def test_prop_cr11_stray_file_only_checkpoints_dir_falls_through_to_older_run(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + newer = _mkrun(runs_dir, "run-20260102T000000Z") + newer_checkpoints_dir = newer / "checkpoints" + newer_checkpoints_dir.mkdir(parents=True) + (newer_checkpoints_dir / ".DS_Store").write_text("junk") # non-empty listing, zero valid entries + older = _mkrun(runs_dir, "run-20260101T000000Z") + expected = _mkcheckpoint(older, 9) + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(expected)) + + +# --- PROP-CR14: a run-shaped entry that is a PLAIN FILE (not a directory) falls through without +# raising NotADirectoryError/FileNotFoundError/any exception (REQ-CR2, REQ-CR3, EDGE-CR12) --- + + +def test_prop_cr14_run_shaped_plain_file_falls_through_without_raising(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + stray_file = runs_dir / "run-20260102T000000Z" + stray_file.write_text("") # zero-byte plain FILE, matches the run-shape by NAME only + older = _mkrun(runs_dir, "run-20260101T000000Z") + expected = _mkcheckpoint(older, 4) + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result == os.path.abspath(str(expected)) + + +def test_prop_cr14_run_shaped_plain_file_only_returns_none(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + stray_file = runs_dir / "run-20260102T000000Z" + stray_file.write_text("") + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result is None + + +# --- PROP-CR12(a): static denylist — no filesystem-mutation API reference anywhere in the FULL +# module source (including __main__), except its legitimate os.environ read / stdout write +# (REQ-CR11, INV-CR1) --- + +FORBIDDEN_MUTATION_TOKENS = ( + "os.remove", + "os.unlink", + "os.rmdir", + "os.removedirs", + "os.mkdir", + "os.makedirs", + "os.rename", + "os.replace", + "os.truncate", + "os.symlink", + "os.link", + "shutil.", +) + + +def test_prop_cr12a_full_module_source_has_no_filesystem_mutation_api_references(): + source = _read_checkpoint_resume_source() + + found = [token for token in FORBIDDEN_MUTATION_TOKENS if token in source] + assert found == [], ( + f"forbidden filesystem-mutation API reference(s) {found} found in lib/checkpoint_resume.py " + "(PROP-CR12a/REQ-CR11)" + ) + + non_default_open_modes = re.findall( + r"""open\([^)]*['"](?:w\+?|a\+?|x\+?)['"]""", source + ) + assert non_default_open_modes == [], ( + "lib/checkpoint_resume.py contains an open() call with a non-default (write/append/" + "create) mode (PROP-CR12a/REQ-CR11)" + ) + + +# --- PROP-CR12(b): static import-absence scan — the module never imports pathlib (REQ-CR11) --- + + +def test_prop_cr12b_module_never_imports_pathlib(): + source = _read_checkpoint_resume_source() + + assert "import pathlib" not in source + assert "from pathlib" not in source + + +# --- PROP-CR12(c): dynamic before/after directory-tree snapshot is byte-for-byte/mtime-for-mtime +# identical across a None-returning call AND a real-path-returning call (REQ-CR11, INV-CR1) --- + + +def _snapshot(root_dir): + entries = [] + for current_dir, dirnames, filenames in os.walk(str(root_dir)): + for name in sorted(dirnames) + sorted(filenames): + full_path = os.path.join(current_dir, name) + entries.append( + (os.path.relpath(full_path, str(root_dir)), os.path.isdir(full_path), os.stat(full_path).st_mtime) + ) + return sorted(entries) + + +def test_prop_cr12c_no_mutation_across_a_none_returning_call(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + + before = _snapshot(runs_dir) + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + after = _snapshot(runs_dir) + + assert result is None + assert after == before + + +def test_prop_cr12c_no_mutation_across_a_real_path_returning_call(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + run = _mkrun(runs_dir, "run-20260101T000000Z") + _mkcheckpoint(run, 1) + + before = _snapshot(runs_dir) + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + after = _snapshot(runs_dir) + + assert result is not None + assert after == before + + +# --- REQ-CR4 (residual coverage): runs_dir containing only non-run-shaped entries (no run-* at +# all) returns None (EDGE-CR3/EDGE-CR10 combined) --- + + +def test_req_cr4_runs_dir_with_only_non_run_shaped_entries_returns_none(tmp_path): + runs_dir = tmp_path / "runs" + runs_dir.mkdir() + (runs_dir / "db").mkdir() + + result = checkpoint_resume.find_latest_checkpoint(str(runs_dir), CURRENT_RUN_ID) + + assert result is None diff --git a/skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py b/skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py new file mode 100644 index 00000000..5f5bf0be --- /dev/null +++ b/skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py @@ -0,0 +1,321 @@ +"""RED phase — self-improve-checkpoint-resume, Group CR-WIRE (wiring into run_evolve.sh). + +Traces to behavioral-spec.md REQ-CR6-9, REQ-CR12, EDGE-CR1; verification-architecture.md +PROP-CR-WIRE1, PROP-CR-WIRE2, PROP-CR-WIRE2b, PROP-CR-WIRE3, PROP-CR13, PROP-CR-LIVE1. + +Test-shape rationale (there is no pre-existing shell-level test convention in this suite to +mirror — `grep -rl "run_evolve.sh" tests/ lib/` finds only `lib/ledger_reader.py`'s own docstring +reference, no prior `test_run_evolve*.py`/bash-test-runner file): this file uses (1) a plain +source-text/position check against the REAL `run_evolve.sh` for PROP-CR-WIRE1's pinned command +string (no execution needed — it is a static shape claim), and (2) a `subprocess`-driven +integration harness for the argument-list/log-content properties (PROP-CR-WIRE2/2b/3, PROP-CR13, +PROP-CR-LIVE1) that actually RUNS `run_evolve.sh` end-to-end via `bash`, with `PY_BIN`/ +`OPENEVOLVE_BIN` swapped for tiny, fully deterministic stand-in scripts this file writes itself — +mirroring `tests/test_wiring.py`'s existing `subprocess.run([...], cwd=tmp_path, ...)` convention +in this same suite, extended here to drive a whole shell script rather than just `git`. This is +the `pytest`-based-inspection-plus-subprocess-integration shape the task description names as an +acceptable RED-phase choice when no shell-test convention already exists. + +EDGE-CR1 / execution-locus note: every sandboxed run below copies `run_evolve.sh` into its own +`tmp_path`-rooted fake `$SKILL_DIR` and builds its own synthetic `runs/` tree there — this file +NEVER executes `run_evolve.sh` against this worktree's own real `$SKILL_DIR` (which would create +real `runs/`/`state/` side effects outside a test sandbox), and NEVER reads/symlinks/copies a real +production `runs/` directory. + +At RED-phase time, `run_evolve.sh` has NOT been edited to add the REQ-CR6 pre-invocation step yet, +and `lib/checkpoint_resume.py` does not exist. Every test below is EXPECTED TO FAIL: PROP-CR-WIRE1 +because the pinned command substring is absent from `run_evolve.sh`'s current source; the +integration tests because the log NEVER gains a `checkpoint_resume:`-prefixed line and the +`openevolve-run` argument list NEVER gains a `--checkpoint` flag (the whole new step simply does +not run yet) — clean `AssertionError`s, not broken-test artifacts. +""" +import os +import re +import shutil +import subprocess +import sys + +SELF_IMPROVE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +RUN_EVOLVE_SH_PATH = os.path.join(SELF_IMPROVE_DIR, "run_evolve.sh") + +# REQ-CR6's ONE committed invocation shape, pinned verbatim so two independent implementers +# cannot diverge on it — see behavioral-spec.md REQ-CR6 / verification-architecture.md +# PROP-CR-WIRE1. +PINNED_CHECKPOINT_RESUME_CALL = ( + 'RUNS_DIR="$RUNS_DIR" RUN_ID="$RUN_ID" "$PY_BIN" "$SKILL_DIR/lib/checkpoint_resume.py"' +) + +FAKE_PY_BIN_SCRIPT = """#!/bin/bash +# Deterministic stand-in for $PY_BIN, dispatching on the requested script's basename so it can +# serve BOTH run_evolve.sh's existing OBSERVE call (lib/ledger_reader.py) and this feature's new +# REQ-CR6 call (lib/checkpoint_resume.py) without needing either file to actually exist on disk. +case "$1" in + */checkpoint_resume.py) + if [ -n "${FAKE_CHECKPOINT_CRASH:-}" ]; then + echo "simulated checkpoint_resume crash" >&2 + exit "${FAKE_CHECKPOINT_EXIT_CODE:-1}" + fi + printf '%s' "${FAKE_CHECKPOINT_PATH:-}" + ;; + *) + echo '{}' + ;; +esac +""" + +FAKE_OPENEVOLVE_BIN_SCRIPT = """#!/bin/bash +# Deterministic stand-in for $OPENEVOLVE_BIN: captures its full argv (one token per line) to +# $FAKE_OPENEVOLVE_CAPTURE_FILE and exits 0 — never actually runs openevolve. +printf '%s\\n' "$@" > "$FAKE_OPENEVOLVE_CAPTURE_FILE" +exit 0 +""" + + +def _write_executable(path, content): + with open(path, "w", encoding="utf-8") as f: + f.write(content) + os.chmod(path, 0o755) + return path + + +def _build_sandbox_skill_dir(tmp_path, copy_real_lib=False): + """A fresh $SKILL_DIR under tmp_path containing a copy of the REAL run_evolve.sh (so tests + exercise the actual production script text, not a hand-written stand-in), never the real + worktree's own $SKILL_DIR (EDGE-CR1: never touch the real runs//state/ trees).""" + skill_dir = tmp_path / "self-improve" + skill_dir.mkdir() + shutil.copy(RUN_EVOLVE_SH_PATH, skill_dir / "run_evolve.sh") + os.chmod(skill_dir / "run_evolve.sh", 0o755) + (skill_dir / "lib").mkdir() + if copy_real_lib: + real_lib_dir = os.path.join(SELF_IMPROVE_DIR, "lib") + for name in os.listdir(real_lib_dir): + src = os.path.join(real_lib_dir, name) + if os.path.isfile(src): + shutil.copy(src, skill_dir / "lib" / name) + return skill_dir + + +def _run_sandboxed_run_evolve(skill_dir, tmp_path, py_bin, extra_env=None): + capture_file = tmp_path / "openevolve_capture.txt" + fake_openevolve_bin = _write_executable( + tmp_path / "fake_openevolve_bin.sh", FAKE_OPENEVOLVE_BIN_SCRIPT + ) + + env = os.environ.copy() + env["PY_BIN"] = str(py_bin) + env["OPENEVOLVE_BIN"] = str(fake_openevolve_bin) + env["FAKE_OPENEVOLVE_CAPTURE_FILE"] = str(capture_file) + if extra_env: + env.update(extra_env) + + result = subprocess.run( + ["bash", str(skill_dir / "run_evolve.sh")], + env=env, + capture_output=True, + text=True, + timeout=60, + ) + + log_path = skill_dir.parent / "state" / "self-improve-evolve.log" + log_content = log_path.read_text(encoding="utf-8") if log_path.exists() else "" + capture_content = capture_file.read_text(encoding="utf-8") if capture_file.exists() else "" + return result, log_content, capture_content + + +# --- PROP-CR-WIRE1: run_evolve.sh's source text contains the EXACT pinned call, positioned +# BEFORE the "$OPENEVOLVE_BIN" invocation (REQ-CR6, REQ-CR7) --- + + +def test_prop_cr_wire1_pinned_checkpoint_resume_call_present_and_before_openevolve_invocation(): + with open(RUN_EVOLVE_SH_PATH, "r", encoding="utf-8") as f: + source = f.read() + + assert PINNED_CHECKPOINT_RESUME_CALL in source, ( + "run_evolve.sh does not yet contain the REQ-CR6 pinned checkpoint_resume.py invocation " + f"{PINNED_CHECKPOINT_RESUME_CALL!r}" + ) + + call_index = source.index(PINNED_CHECKPOINT_RESUME_CALL) + openevolve_index = source.index('"$OPENEVOLVE_BIN"') + assert call_index < openevolve_index, ( + "the REQ-CR6 checkpoint_resume.py call must appear BEFORE the existing " + '"$OPENEVOLVE_BIN" invocation in run_evolve.sh' + ) + + +# --- PROP-CR-WIRE2: no checkpoint found -> openevolve-run's argument list is byte-for-byte +# unchanged (no --checkpoint), AND the REQ-CR9 "no prior checkpoint found" log line is +# actually written (proves the new step really ran, not merely that the fallback is a no-op +# pre-feature) (REQ-CR8, REQ-CR9, INV-CR4) --- + + +def test_prop_cr_wire2_no_checkpoint_found_argument_list_unchanged_and_logged(tmp_path): + skill_dir = _build_sandbox_skill_dir(tmp_path) + fake_py_bin = _write_executable(tmp_path / "fake_py_bin.sh", FAKE_PY_BIN_SCRIPT) + + result, log_content, capture_content = _run_sandboxed_run_evolve( + skill_dir, tmp_path, fake_py_bin, extra_env={"FAKE_CHECKPOINT_PATH": ""} + ) + + assert result.returncode == 0, f"run_evolve.sh failed: stderr={result.stderr!r}" + assert "--checkpoint" not in capture_content, ( + "openevolve-run's argument list must NOT contain --checkpoint when no prior checkpoint " + "was found (REQ-CR8/INV-CR4)" + ) + assert "checkpoint_resume: no prior checkpoint found" in log_content, ( + "expected the REQ-CR9 'no prior checkpoint found' log line — proves the new REQ-CR6 step " + "actually ran, not merely that today's unmodified fallback happens to match" + ) + + +# --- PROP-CR-WIRE2b: checkpoint found -> the EXISTING argument list is preserved with EXACTLY +# ONE addition, --checkpoint "", appended at the end (REQ-CR7) --- + + +def test_prop_cr_wire2b_checkpoint_found_appends_single_checkpoint_flag(tmp_path): + skill_dir = _build_sandbox_skill_dir(tmp_path) + fake_py_bin = _write_executable(tmp_path / "fake_py_bin.sh", FAKE_PY_BIN_SCRIPT) + fake_checkpoint_path = str(tmp_path / "runs" / "run-20260101T000000Z" / "checkpoints" / "checkpoint_20") + + result, log_content, capture_content = _run_sandboxed_run_evolve( + skill_dir, tmp_path, fake_py_bin, extra_env={"FAKE_CHECKPOINT_PATH": fake_checkpoint_path} + ) + + assert result.returncode == 0, f"run_evolve.sh failed: stderr={result.stderr!r}" + capture_lines = capture_content.splitlines() + + assert "--checkpoint" in capture_lines, ( + "openevolve-run's argument list must gain --checkpoint when a prior checkpoint was found " + "(REQ-CR7)" + ) + checkpoint_flag_index = capture_lines.index("--checkpoint") + assert checkpoint_flag_index == len(capture_lines) - 2, ( + "--checkpoint must be appended at the END of the existing argument list (REQ-CR7), " + f"got args={capture_lines!r}" + ) + assert capture_lines[checkpoint_flag_index + 1] == fake_checkpoint_path + + # Every EXISTING argument (initial_program, evaluator, --config, --iterations, --output) + # must still be present, unchanged, before the new --checkpoint pair (INV-CR4). + existing_args = capture_lines[:checkpoint_flag_index] + assert existing_args.count("--config") == 1 + assert existing_args.count("--iterations") == 1 + assert existing_args.count("--output") == 1 + assert any(arg.endswith("pm_backtest_strategy.py") for arg in existing_args) + assert any(arg.endswith("evaluator.py") for arg in existing_args) + + assert "checkpoint_resume: resuming from checkpoint" in log_content + assert fake_checkpoint_path in log_content + + +# --- PROP-CR-WIRE3: exactly one checkpoint_resume:-prefixed log line per branch, with the +# REQ-CR9-pinned substring for that branch (never merely "some line got written") --- + + +def test_prop_cr_wire3_exactly_one_checkpoint_resume_log_line_not_found_branch(tmp_path): + skill_dir = _build_sandbox_skill_dir(tmp_path) + fake_py_bin = _write_executable(tmp_path / "fake_py_bin.sh", FAKE_PY_BIN_SCRIPT) + + _, log_content, _ = _run_sandboxed_run_evolve( + skill_dir, tmp_path, fake_py_bin, extra_env={"FAKE_CHECKPOINT_PATH": ""} + ) + + matching_lines = [line for line in log_content.splitlines() if "checkpoint_resume:" in line] + assert len(matching_lines) == 1, ( + f"expected exactly one checkpoint_resume: log line, got {matching_lines!r}" + ) + assert "checkpoint_resume: no prior checkpoint found" in matching_lines[0] + + +def test_prop_cr_wire3_exactly_one_checkpoint_resume_log_line_found_branch(tmp_path): + skill_dir = _build_sandbox_skill_dir(tmp_path) + fake_py_bin = _write_executable(tmp_path / "fake_py_bin.sh", FAKE_PY_BIN_SCRIPT) + fake_checkpoint_path = str(tmp_path / "runs" / "run-20260101T000000Z" / "checkpoints" / "checkpoint_20") + + _, log_content, _ = _run_sandboxed_run_evolve( + skill_dir, tmp_path, fake_py_bin, extra_env={"FAKE_CHECKPOINT_PATH": fake_checkpoint_path} + ) + + matching_lines = [line for line in log_content.splitlines() if "checkpoint_resume:" in line] + assert len(matching_lines) == 1, ( + f"expected exactly one checkpoint_resume: log line, got {matching_lines!r}" + ) + assert "checkpoint_resume: resuming from checkpoint" in matching_lines[0] + assert fake_checkpoint_path in matching_lines[0] + + +# --- PROP-CR13: the REQ-CR6 call itself crashing (non-zero exit) degrades to the REQ-CR8 +# fallback (no --checkpoint) AND logs the THIRD, distinct "resume-check crashed" wording with +# the captured exit status (REQ-CR12) --- + + +def test_prop_cr13_resume_check_crash_falls_back_and_logs_distinct_message(tmp_path): + skill_dir = _build_sandbox_skill_dir(tmp_path) + fake_py_bin = _write_executable(tmp_path / "fake_py_bin.sh", FAKE_PY_BIN_SCRIPT) + + result, log_content, capture_content = _run_sandboxed_run_evolve( + skill_dir, + tmp_path, + fake_py_bin, + extra_env={"FAKE_CHECKPOINT_CRASH": "1", "FAKE_CHECKPOINT_EXIT_CODE": "7"}, + ) + + assert result.returncode == 0, f"run_evolve.sh itself must not crash: stderr={result.stderr!r}" + assert "--checkpoint" not in capture_content, ( + "a crashed resume-check must degrade to the REQ-CR8 fallback (no --checkpoint), never " + "block the openevolve-run invocation (REQ-CR12)" + ) + assert "checkpoint_resume: resume-check crashed" in log_content, ( + "expected the REQ-CR12 'resume-check crashed' log line, distinct from both the " + "found/not-found wording" + ) + assert "7" in log_content.split("checkpoint_resume: resume-check crashed", 1)[1].splitlines()[0], ( + "expected the captured non-zero exit status (7) to appear immediately after the " + "'resume-check crashed' substring" + ) + assert "checkpoint_resume: no prior checkpoint found" not in log_content, ( + "a crash must NEVER be logged with the ordinary not-found wording — REQ-CR12 requires a " + "third, distinct branch" + ) + + +# --- PROP-CR-LIVE1: end-to-end — a hand-constructed two-run tmp_path tree fed through +# run_evolve.sh's REAL new step (via the real python interpreter and, once it exists, the +# real lib/checkpoint_resume.py — never a stand-in for the pure logic itself) selects and +# logs checkpoint_20's path (REQ-CR6, REQ-CR9) --- + + +def test_prop_cr_live1_end_to_end_two_run_fixture_selects_and_logs_checkpoint_20(tmp_path): + skill_dir = _build_sandbox_skill_dir(tmp_path, copy_real_lib=True) + + # Hand-constructed two-run fixture (per PROP-CR-LIVE1's own description): an older run with + # checkpoint_10 AND checkpoint_20 (REQ-CR3: the higher-numbered one must win), pre-existing + # BEFORE run_evolve.sh creates its own new (current) run directory. + runs_dir = skill_dir / "runs" + older_run = runs_dir / "run-20260101T000000Z" + (older_run / "checkpoints" / "checkpoint_10").mkdir(parents=True) + expected_checkpoint = older_run / "checkpoints" / "checkpoint_20" + expected_checkpoint.mkdir(parents=True) + + result, log_content, capture_content = _run_sandboxed_run_evolve( + skill_dir, tmp_path, py_bin=sys.executable + ) + + assert result.returncode == 0, f"run_evolve.sh failed: stderr={result.stderr!r}" + assert "checkpoint_resume: resuming from checkpoint" in log_content, ( + "PROP-CR-LIVE1: expected the real (not stand-in) checkpoint_resume.py, invoked via the " + "real python interpreter, to select and log the older run's checkpoint_20" + ) + assert str(expected_checkpoint) in log_content or os.path.abspath(str(expected_checkpoint)) in log_content + assert "--checkpoint" in capture_content + + +# --- Regression Table smoke check: run_evolve.sh remains valid bash after every edit --- + + +def test_regression_run_evolve_sh_remains_valid_bash_syntax(): + result = subprocess.run( + ["bash", "-n", RUN_EVOLVE_SH_PATH], capture_output=True, text=True, timeout=10 + ) + assert result.returncode == 0, f"run_evolve.sh has a bash syntax error: {result.stderr}" From 0288d3e03aaea660faf2af94e89bf74a451e4bd9 Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sun, 12 Jul 2026 07:19:21 +0900 Subject: [PATCH 08/11] vcsdd(self-improve-checkpoint-resume): phase 2a -> 3, RED+GREEN evidence recorded --- .../self-improve-checkpoint-resume/state.json | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/.vcsdd/features/self-improve-checkpoint-resume/state.json b/.vcsdd/features/self-improve-checkpoint-resume/state.json index 54babd56..cc482d8f 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/state.json +++ b/.vcsdd/features/self-improve-checkpoint-resume/state.json @@ -2,7 +2,7 @@ "featureName": "self-improve-checkpoint-resume", "mode": "lean", "language": "python", - "currentPhase": "2a", + "currentPhase": "3", "phaseHistory": [ { "from": "init", @@ -24,13 +24,22 @@ "to": "2a", "timestamp": "2026-07-11T09:48:45.135Z", "sprint": 1 + }, + { + "from": "2a", + "to": "3", + "timestamp": "2026-07-11T22:19:01.000Z", + "reason": "AGENT ECONOMY LOOP session: RED phase confirmed (git-stash-verified collection error / 7-of-8 wiring test failures before impl existed, evidence/red-phase.md), then GREEN phase 2b implemented: lib/checkpoint_resume.py::find_latest_checkpoint (REQ-CR1-5,10,11 pure read-only checkpoint discovery) + run_evolve.sh's REQ-CR6-9,12 pre-invocation step (positioned before the $OPENEVOLVE_BIN existence check so the pinned-string-position PROP-CR-WIRE1 test passes against the FIRST occurrence of \"$OPENEVOLVE_BIN\" in source). 31/31 new tests pass (evidence/green-phase.md); full suite shows the same 18 pre-existing failures confirmed present on main HEAD too (pm_backtest_strategy.py fixture drift + one worktree-path-only test) -- zero regressions. No separate 2c refactor needed (implementation was already minimal/clean on first GREEN pass). Live-verified and fixed a bash 3.2 gotcha before shipping: \"${CHECKPOINT_ARGS[@]}\" on an empty array raises \"unbound variable\" under this script's set -u on bash 3.2 (this Mac's default /bin/bash, confirmed via bash --version); used \"${CHECKPOINT_ARGS[@]:-}\" instead. commit 515eb089. Entering Phase 3 (impl-review, fresh adversary).", + "sprint": 1 } ], "iterations": { "1a": 1, "1b": 1, "1c": 1, - "2a": 1 + "2a": 1, + "2b": 1, + "3": 1 }, "proofObligations": [], "traceability": { From eba2270a9f9c4891092e0293012dbd3630fdd7c4 Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sun, 12 Jul 2026 07:29:31 +0900 Subject: [PATCH 09/11] vcsdd(self-improve-checkpoint-resume): fix FIND-001/FIND-002 from Phase 3 iteration 1 FIND-001 (critical, implementation_correctness): "${CHECKPOINT_ARGS[@]:-}" on an empty array under set -u on bash 3.2 (this Mac's default /bin/bash) expands to ONE stray empty-string argument, not zero arguments -- confirmed live via bash -c repro (argc=1, arg=[""]). This would have broken openevolve-run's argparse (no catch-all positional) on every 'no checkpoint found' cycle, i.e. nearly every run until a checkpoint accumulates -- the common case, not an edge case. Fixed to the canonical "${CHECKPOINT_ARGS[@]+"${CHECKPOINT_ARGS[@]}"}" pattern, confirmed live: argc=0 when empty, argc=2 when populated. FIND-002 (verification_readiness): PROP-CR-WIRE2's test only checked substring absence of '--checkpoint', which could not detect the FIND-001 stray-argument bug (no literal '--checkpoint' text present). Strengthened to assert exact argument count (8) and explicit absence of any empty-string argument -- verified this strengthened test FAILS against the pre-fix code (9 args, trailing '') and PASSES against the fix (8 args). Found by fresh Sonnet vcsdd-adversary Phase 3 iteration 1 review (mode: lean). 31/31 feature tests green, 18 pre-existing unrelated failures unchanged (0 regressions). --- skills/earn/self-improve/run_evolve.sh | 2 +- .../tests/test_checkpoint_resume_wiring.py | 21 +++++++++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/skills/earn/self-improve/run_evolve.sh b/skills/earn/self-improve/run_evolve.sh index 29dff196..11c6fe24 100755 --- a/skills/earn/self-improve/run_evolve.sh +++ b/skills/earn/self-improve/run_evolve.sh @@ -106,7 +106,7 @@ fi --config "$SKILL_DIR/config.yaml" \ --iterations "$ITERATIONS" \ --output "$RUN_DIR" \ - "${CHECKPOINT_ARGS[@]:-}" \ + "${CHECKPOINT_ARGS[@]+"${CHECKPOINT_ARGS[@]}"}" \ >> "$LOG" 2>&1 STATUS=$? diff --git a/skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py b/skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py index 5f5bf0be..01a2a489 100644 --- a/skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py +++ b/skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py @@ -163,6 +163,27 @@ def test_prop_cr_wire2_no_checkpoint_found_argument_list_unchanged_and_logged(tm "openevolve-run's argument list must NOT contain --checkpoint when no prior checkpoint " "was found (REQ-CR8/INV-CR4)" ) + # FIND-001/FIND-002 regression guard: REQ-CR8 requires the fallback argument list to be + # BYTE-FOR-BYTE identical to pre-feature behavior — not merely "no --checkpoint substring + # anywhere". A naive `"${CHECKPOINT_ARGS[@]:-}"` empty-array-expansion bug (bash <4.4/set -u) + # appends ONE stray empty-string argument even when CHECKPOINT_ARGS is empty, which the old + # substring-only assertion above could not detect (no literal "--checkpoint" text, but argv + # grows by one anyway). Assert the exact argument COUNT (8: 2 positionals + 3 flag pairs) and + # that no argument is an empty/stray string. + capture_lines = capture_content.splitlines() + assert len(capture_lines) == 8, ( + "openevolve-run's argument list must be BYTE-FOR-BYTE identical to pre-feature behavior " + f"when no checkpoint is found (REQ-CR8/INV-CR4) — expected exactly 8 arguments (2 " + f"positionals + 3 flag pairs), no stray trailing empty-string argument. got " + f"{len(capture_lines)}: {capture_lines!r}" + ) + assert "" not in capture_lines, ( + f"openevolve-run's argument list must never contain a stray empty-string argument " + f"(REQ-CR8/INV-CR4). got: {capture_lines!r}" + ) + assert capture_lines[2:7] == ["--config", str(skill_dir / "config.yaml"), "--iterations", "20", "--output"], ( + f"unexpected argument list shape: {capture_lines!r}" + ) assert "checkpoint_resume: no prior checkpoint found" in log_content, ( "expected the REQ-CR9 'no prior checkpoint found' log line — proves the new REQ-CR6 step " "actually ran, not merely that today's unmodified fallback happens to match" From d49903263dc491051f9c4eb1ec435435e87deb7c Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sun, 12 Jul 2026 07:38:12 +0900 Subject: [PATCH 10/11] vcsdd(self-improve-checkpoint-resume): Phase 5 formal hardening complete, phase -> 6 verification/{verification-report,security-report,purity-audit}.md. All 19 PROP-CR* proof obligations mapped to discharging tests, 31/31 re-run live and PASS. semgrep 0 findings. Purity boundary independently re-confirmed against shipped code. Zero regressions vs main HEAD's pre-existing 18-failure baseline. --- .../iteration-1/output/findings/FIND-001.json | 14 ++++ .../iteration-1/output/findings/FIND-002.json | 14 ++++ .../impl/iteration-1/output/verdict.json | 63 ++++++++++++++++ .../impl/iteration-2/output/verdict.json | 63 ++++++++++++++++ .../self-improve-checkpoint-resume/state.json | 36 +++++++++- .../verification/purity-audit.md | 52 ++++++++++++++ .../verification/security-report.md | 58 +++++++++++++++ .../verification/verification-report.md | 72 +++++++++++++++++++ 8 files changed, 369 insertions(+), 3 deletions(-) create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/findings/FIND-001.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/findings/FIND-002.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/verdict.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-2/output/verdict.json create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/verification/purity-audit.md create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/verification/security-report.md create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/verification/verification-report.md diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/findings/FIND-001.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/findings/FIND-001.json new file mode 100644 index 00000000..ec957943 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/findings/FIND-001.json @@ -0,0 +1,14 @@ +{ + "findingId": "FIND-001", + "dimension": "implementation_correctness", + "category": "requirement_mismatch", + "severity": "critical", + "blocking": true, + "description": "run_evolve.sh:109 appends the new checkpoint flag via `\"${CHECKPOINT_ARGS[@]:-}\"`. When no checkpoint is found (or the resume-check crashes), CHECKPOINT_ARGS stays the empty array declared at run_evolve.sh:77 (`CHECKPOINT_ARGS=()`). Under `set -u` (run_evolve.sh:26) on bash < 4.4 -- which is exactly the shell this script's own header comments (run_evolve.sh:17-25) say it targets, and which is macOS's stock, never-upgraded system `/bin/bash` (3.2, unchanged for licensing reasons) that a launchd-invoked script with a minimal PATH will resolve to -- expanding `${arr[@]}` on a declared-but-empty array is treated as 'unset' for nounset purposes (fixed only in bash 4.4's changelog item 'Expanding an empty array is no longer an error when nounset is enabled'). The `:-` operator's fallback, when triggered, substitutes its (empty) word literally as ONE quoted argument -- it does not re-apply `[@]` array-splitting to the fallback the way `${arr[@]+\"${arr[@]}\"}` (the canonical, fully-correct fix for this exact bash pitfall) does. The practical effect: in the 'no checkpoint found' branch (and the 'resume-check crashed' branch), `\"$OPENEVOLVE_BIN\" ... --output \"$RUN_DIR\" \"${CHECKPOINT_ARGS[@]:-}\"` likely passes ONE spurious trailing empty-string argument to openevolve-run, violating REQ-CR8/INV-CR4's explicit 'invoke openevolve-run with EXACTLY the same argument list it uses today ... byte-for-byte the pre-existing behavior'. Confirmed via direct read of the installed CLI's own arg parser (~/.anicca-venvs/self-improve/lib/python3.14/site-packages/openevolve/cli.py:22-26): openevolve-run's argparse defines EXACTLY two positional arguments (initial_program, evaluation_file) and no catch-all/nargs='*' positional -- a third stray positional token (even an empty string) is unrecognized and argparse raises 'error: unrecognized arguments' and calls sys.exit(2). Since the not-found branch is the MOST COMMON path (every first-ever run, and every run until a valid checkpoint accumulates -- which, if this bug fires, would never happen because every such cycle would crash openevolve-run immediately), this is a regression strictly worse than the tied-combined_score stagnation this feature was built to fix: instead of a wasted-but-completing cycle, every cycle in the fallback branch would fail outright. I could not execute a live `bash -c` repro or check this Mac's actual `/bin/bash --version` to settle this with 100% certainty -- this review session has no Bash/execute tool available, only Read/Grep/Glob/Write/Edit -- so this finding is reasoned from (a) documented bash nounset+empty-array semantics and the well-known `:-` vs `+` fallback distinction, (b) the directly-read openevolve argparse source, and (c) FIND-002's confirmation that no existing test actually checks full argument-list identity in this branch. Recommended fix: replace `\"${CHECKPOINT_ARGS[@]:-}\"` with `\"${CHECKPOINT_ARGS[@]+\"${CHECKPOINT_ARGS[@]}\"}\"`, or branch explicitly on `${#CHECKPOINT_ARGS[@]}` before invoking `$OPENEVOLVE_BIN`, then re-verify by actually running the composed command under this machine's real `/bin/bash` and inspecting the captured argv byte-for-byte.", + "evidence": { + "filePath": "skills/earn/self-improve/run_evolve.sh", + "lineRange": "77,103-110", + "snippet": "CHECKPOINT_ARGS=()\n...\n\"$OPENEVOLVE_BIN\" \\\n \"$SKILL_DIR/strategies/pm_backtest_strategy.py\" \\\n \"$SKILL_DIR/evaluator.py\" \\\n --config \"$SKILL_DIR/config.yaml\" \\\n --iterations \"$ITERATIONS\" \\\n --output \"$RUN_DIR\" \\\n \"${CHECKPOINT_ARGS[@]:-}\" \\\n >> \"$LOG\" 2>&1" + }, + "routeToPhase": "2b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/findings/FIND-002.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/findings/FIND-002.json new file mode 100644 index 00000000..df101043 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/findings/FIND-002.json @@ -0,0 +1,14 @@ +{ + "findingId": "FIND-002", + "dimension": "verification_readiness", + "category": "test_quality", + "severity": "high", + "blocking": true, + "description": "verification-architecture.md's PROP-CR-WIRE2 (line 76) claims to verify: 'WHEN no checkpoint is found, the resulting openevolve-run argument list is IDENTICAL (same arguments, same order) to the pre-feature invocation -- no --checkpoint flag appears (REQ-CR8, INV-CR4)'. The test that is supposed to discharge this proof obligation, test_prop_cr_wire2_no_checkpoint_found_argument_list_unchanged_and_logged (tests/test_checkpoint_resume_wiring.py:153-169), only asserts `result.returncode == 0`, `\"--checkpoint\" not in capture_content` (a substring check), and the presence of the REQ-CR9 log substring. It never asserts the captured argv (`capture_content`/`capture_lines`) has the same LENGTH or same CONTENT as the pre-feature argument list -- it does not even split capture_content into lines and count them, unlike its sibling test for the found-branch, test_prop_cr_wire2b_checkpoint_found_appends_single_checkpoint_flag (same file, lines 176-209), which DOES assert `checkpoint_flag_index == len(capture_lines) - 2` (i.e. explicitly checks nothing trails the appended pair). This asymmetry means a spurious extra token appended after `--output \"$RUN_DIR\"` in the not-found branch -- exactly what FIND-001 reasons the bash 3.2 `\"${CHECKPOINT_ARGS[@]:-}\"` idiom likely produces -- would silently pass this test suite. The proof obligation PROP-CR-WIRE2 is therefore not actually discharged by the test cited as discharging it; the 31/31-green evidence in evidence/green-phase.md provides false confidence specifically for the branch (the fallback / 'no checkpoint found' case) that is the MOST FREQUENTLY exercised branch of this feature in production. Fix: add an explicit assertion in test_prop_cr_wire2_no_checkpoint_found_argument_list_unchanged_and_logged that `capture_lines` (split of capture_content) has the exact same length/content as a captured pre-feature baseline argument list (or, at minimum, assert capture_content.strip() has no extra trailing token beyond '--output ').", + "evidence": { + "filePath": "skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py", + "lineRange": "153-169", + "snippet": "def test_prop_cr_wire2_no_checkpoint_found_argument_list_unchanged_and_logged(tmp_path):\n ...\n assert result.returncode == 0, ...\n assert \"--checkpoint\" not in capture_content, (\n \"openevolve-run's argument list must NOT contain --checkpoint when no prior checkpoint \"\n \"was found (REQ-CR8/INV-CR4)\"\n )\n assert \"checkpoint_resume: no prior checkpoint found\" in log_content, (...)" + }, + "routeToPhase": "2b" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/verdict.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/verdict.json new file mode 100644 index 00000000..5a919ae0 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-1/output/verdict.json @@ -0,0 +1,63 @@ +{ + "phase": "impl-review", + "feature": "self-improve-checkpoint-resume", + "sprintNumber": 1, + "iteration": 1, + "verdict": "FAIL", + "overallVerdict": "FAIL", + "timestamp": "2026-07-12T00:00:00Z", + "reviewedCommits": ["515eb089", "0288d3e0"], + "reviewerToolConstraints": "This review session had no Bash/execute tool available (only Read/Write/Edit/Grep/Glob). I could NOT independently run `python3 -m pytest`, `bash -n run_evolve.sh`, `bash --version`, `git diff`, or `git show`/`git log -p` as the review brief requested. Where the brief required dynamic execution, I substituted: (1) full direct reads of every changed/new file, (2) hand-tracing every test in tests/test_checkpoint_resume.py and tests/test_checkpoint_resume_wiring.py against the implementation logic line-by-line, (3) reading the installed openevolve CLI's own argparse source (~/.anicca-venvs/self-improve/lib/python3.14/site-packages/openevolve/cli.py) to determine real-world impact of a suspected bash bug, (4) treating evidence/green-phase.md's pytest transcript and evidence/red-phase.md as Builder-reported (not independently re-executed) evidence, and (5) a Glob-by-mtime scan of skills/earn/self-improve/** as a proxy for git-diff scope (config.yaml, promote_gate.sh, lib/promote_gate.py, lib/promote.py, strategies/pm_backtest_strategy.py, launchd/ai.anicca.self-improve-evolve.plist all sort well before run_evolve.sh/lib/checkpoint_resume.py in mtime order, consistent with them being untouched, but this is not as conclusive as an actual `git diff --stat`).", + "dimensions": { + "spec_fidelity": { + "verdict": "FAIL", + "findings": ["FIND-001"], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/lib/checkpoint_resume.py:15-69", "description": "find_latest_checkpoint's REQ-CR1-5/CR10 logic traced by hand against every PROP-CR test in tests/test_checkpoint_resume.py: absolute-path normalization (REQ-CR1, line 52), filter-then-exclude-then-sort-descending-then-iterate (REQ-CR2, lines 56-67), run-name regex `^run-\\d{8}T\\d{6}Z$` matches REQ-CR10 exactly (line 19), integer (not lexicographic) checkpoint comparison via int(match.group(1)) (REQ-CR3, lines 38-41), os.path.isdir guard before any listdir under a candidate (REQ-CR3/EDGE-CR12, line 29) -- all match the spec's literal requirement text."}, + {"type": "file", "location": "skills/earn/self-improve/run_evolve.sh:75,78-85", "description": "REQ-CR6's pinned call string is present verbatim and positioned before the first literal `\"$OPENEVOLVE_BIN\"` occurrence (line 87, the executable-check, not line 37's `${OPENEVOLVE_BIN:-...}` default-assignment which does not contain the literal substring). REQ-CR9's three pinned log substrings are each present verbatim in their correct branch."}, + {"type": "file", "location": "skills/earn/self-improve/run_evolve.sh:77,109 + ~/.anicca-venvs/self-improve/lib/python3.14/site-packages/openevolve/cli.py:22-26", "description": "FIND-001: REQ-CR8/INV-CR4's 'byte-for-byte identical' no-checkpoint fallback is reasoned to be violated on the script's own documented target shell (bash < 4.4 / macOS stock /bin/bash) because `\"${CHECKPOINT_ARGS[@]:-}\"` on the empty-array case likely injects one spurious empty-string positional argument, which openevolve-run's real 2-positional argparse (no catch-all) would reject as 'unrecognized arguments'."} + ] + }, + "edge_case_coverage": { + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume.py:45-417", "description": "Every EDGE-CR1 through EDGE-CR12 case in behavioral-spec.md has a dedicated, correctly-constructed tmp_path fixture and assertion: EDGE-CR2/3 (PROP-CR1/CR2, lines 45-83), EDGE-CR4/5 (PROP-CR5 x2, lines 123-145), EDGE-CR6 (PROP-CR6, lines 152-165), EDGE-CR7 (PROP-CR3, lines 90-100), EDGE-CR8 (PROP-CR8 x2, lines 189-209), EDGE-CR9 (PROP-CR7 + PROP-CR2's second test, lines 78-83,172-182), EDGE-CR10 (PROP-CR10 x2, lines 245-266), EDGE-CR11 (PROP-CR11, lines 274-286), EDGE-CR12 (PROP-CR14 x2, lines 293-314). Each fixture builds the exact adversarial tree the edge case names (e.g. a stray .DS_Store-only checkpoints/ dir for EDGE-CR11, a zero-byte plain file named run- for EDGE-CR12), not a simplified stand-in."}, + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py:253-280", "description": "REQ-CR12's resume-check-crash edge case (PROP-CR13) is exercised via a FAKE_PY_BIN_SCRIPT that actually exits non-zero and writes to stderr, asserting the fallback (no --checkpoint) AND the distinct third log-line wording, both present."} + ] + }, + "implementation_correctness": { + "verdict": "FAIL", + "findings": ["FIND-001"], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/lib/checkpoint_resume.py:1-82", "description": "The pure core itself (find_latest_checkpoint + _latest_checkpoint_in_run) is correct by hand-trace against every REQ-CR1-5/CR10/CR11: no file-content reads, no mutation, only os.listdir/os.path.isdir/os.path.abspath/os.path.join, regex-gated name filtering, integer comparison. This portion of the diff is sound."}, + {"type": "file", "location": "skills/earn/self-improve/run_evolve.sh:77,109", "description": "FIND-001 (blocking): the `CHECKPOINT_ARGS=()` / `\"${CHECKPOINT_ARGS[@]:-}\"` pairing is the known-flawed bash nounset+empty-array idiom (produces one spurious empty-string argument under bash < 4.4, rather than zero arguments), confirmed against openevolve-run's real argparse (exactly 2 required positionals, no catch-all) to be argparse-rejection-worthy if triggered -- this is the exact 'trickiest part' the reviewing brief asked to be dynamically verified, and I could not execute bash in this session to settle it with certainty (see reviewerToolConstraints)."} + ] + }, + "structural_integrity": { + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/lib/checkpoint_resume.py", "description": "82 lines total, single pure function + one small helper + a thin __main__ CLI wrapper; no duplication; names (find_latest_checkpoint, _latest_checkpoint_in_run) accurately describe behavior; module docstring explicitly cross-references its own purity contract. Follows lib/ledger_reader.py's established __main__ convention exactly as verification-architecture.md's Purity Boundary Map (lines 27-38) specifies -- confirmed by reading both files' __main__ blocks side by side."}, + {"type": "file", "location": "skills/earn/self-improve/run_evolve.sh:66-85", "description": "The new pre-invocation step is a single, well-commented ~20-line insertion at the pipeline position the spec pins (before the $OPENEVOLVE_BIN existence check, after $RUN_DIR creation); no dead code; no new abstraction introduced beyond what REQ-CR6-9 require."} + ] + }, + "verification_readiness": { + "verdict": "FAIL", + "findings": ["FIND-002"], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py:176-209", "description": "PROP-CR-WIRE2b (found-branch) correctly discharges its proof obligation: it asserts checkpoint_flag_index == len(capture_lines) - 2, i.e. explicit full-argument-list-position verification, proving the test-writing convention for doing this correctly EXISTS in this same file."}, + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py:153-169", "description": "FIND-002 (blocking): PROP-CR-WIRE2 (not-found branch) does NOT apply that same convention -- it only checks substring absence of '--checkpoint', never full argv identity/length -- so it cannot catch FIND-001's suspected spurious-empty-argument regression even though it is the test verification-architecture.md cites as proving REQ-CR8/INV-CR4 byte-for-byte identity."} + ] + } + }, + "blocking_count": 2, + "convergenceSignals": { + "findingCount": 2, + "allCriteriaEvaluated": true, + "evaluatedCriteria": [], + "duplicateFindings": [], + "blockingFindingCount": 2 + }, + "notes": "Non-blocking process observation (out of this review's scoped mandate, not filed as a code finding): evidence/green-phase.md's regression check (18 pre-existing failures) and evidence/red-phase.md were treated as Builder-reported, not independently re-executed by this adversary (no Bash tool in this review session). The 18 pre-existing failures shown are internally consistent with a pm_backtest_strategy.py fixture-drift explanation (repeated 'config.get(\"edge_weight\", 0.25)' literal not found in the current baseline file's docstring-truncated repr) plus one worktree-path-only ledger_reader test, both plausible as pre-existing/unrelated to this feature's diff, but this was reasoned from the transcript rather than independently reproduced via `git stash` + re-run on this reviewer's own execution." +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-2/output/verdict.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-2/output/verdict.json new file mode 100644 index 00000000..b09c27db --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/impl/iteration-2/output/verdict.json @@ -0,0 +1,63 @@ +{ + "phase": "impl-review", + "feature": "self-improve-checkpoint-resume", + "sprintNumber": 1, + "iteration": 2, + "verdict": "PASS", + "overallVerdict": "PASS", + "timestamp": "2026-07-12T00:00:00Z", + "reviewedCommits": ["eba2270a9f9c4891092e0293012dbd3630fdd7c4"], + "reviewerToolConstraints": "This review session had NO Bash/execute tool available (only Read/Write/Edit/Grep/Glob) -- identical constraint to iteration-1's reviewer. I could not myself invoke `bash --version`, a live `bash -c` repro of the old-vs-new array-expansion pattern, `bash -n run_evolve.sh`, `python3 -m pytest`, or `git diff --stat`/`git show`. Where the brief required dynamic execution I substituted: (1) direct reads of every file in the review brief plus git ref-log inspection (.git/worktrees/self-improve-checkpoint-resume/logs/HEAD) to independently confirm this worktree's HEAD is EXACTLY commit eba2270a9f9c4891092e0293012dbd3630fdd7c4 (the claimed fix commit) -- not a stale or different state; (2) hand-tracing bash `set -u`/nounset parameter-expansion semantics for `\"${CHECKPOINT_ARGS[@]+\"${CHECKPOINT_ARGS[@]}\"}\"` against the documented (and FIND-001's own cited) bash 3.2 empty-array quirk, confirming this is the canonical, widely-documented portable idiom (exactly FIND-001's own recommended fix, applied verbatim) that yields zero arguments when the array is empty and full array-splitting when populated; (3) hand-tracing the strengthened test assertions in tests/test_checkpoint_resume_wiring.py line-by-line against run_evolve.sh's actual argument-list construction (counted 8 arguments for the empty-CHECKPOINT_ARGS case: 2 positionals + 3 flag pairs, matching the test's `len(capture_lines) == 8` assertion exactly); (4) independently discovered and read on-disk pytest cache artifacts NOT authored by any narrative document -- skills/earn/self-improve/.pytest_cache/v/cache/lastfailed (18 failing node IDs, none from test_checkpoint_resume*.py) and .../nodeids (full 121-item collected list including all 31 checkpoint-resume tests) plus compiled lib/__pycache__/checkpoint_resume.cpython-314.pyc and tests/__pycache__/test_checkpoint_resume_wiring.cpython-314-pytest-9.1.1.pyc -- this is forensic evidence a real pytest run executed against this exact code and passed all 31 new tests while preserving exactly the same 18 pre-existing (unrelated, pm_backtest_strategy.py-fixture-drift + one worktree-path-only ledger_reader test) failures, corroborating (not merely trusting) evidence/green-phase.md's self-reported transcript; (5) a Grep scan across config.yaml/promote_gate.sh/lib/promote.py/lib/promote_gate.py/launchd/*.plist/strategies/pm_backtest_strategy.py for any checkpoint-resume fingerprint (found none) and a Grep scan of lib/checkpoint_resume.py for wallet/.env/ledger/spend-cap references (found none -- the one incidental regex hit was `os.environ` containing the substring '.env', not a real .env reference) as a substitute for `git diff --stat` scope confirmation.", + "dimensions": { + "spec_fidelity": { + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/run_evolve.sh:103-110", "description": "REQ-CR8/INV-CR4's fallback argument-list identity is now correctly implemented via the canonical `\"${CHECKPOINT_ARGS[@]+\"${CHECKPOINT_ARGS[@]}\"}\"` idiom (line 109), replacing iteration-1's flawed `\"${CHECKPOINT_ARGS[@]:-}\"`. Hand-traced: when CHECKPOINT_ARGS=() (empty), bash's `+` operator combined with bash<4.4's empty-array-treated-as-unset quirk yields ZERO expanded tokens (not one spurious empty string); when CHECKPOINT_ARGS=(--checkpoint path), the nested `\"${CHECKPOINT_ARGS[@]}\"` correctly array-splits into two quoted tokens. This is exactly FIND-001's own recommended fix, applied verbatim."}, + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py:166-186", "description": "REQ-CR8/INV-CR4's proof obligation (PROP-CR-WIRE2) is now genuinely discharged: the not-found-branch test asserts `len(capture_lines) == 8` (exact byte-for-byte argument count) and `\"\" not in capture_lines` (no stray empty-string token), not merely `\"--checkpoint\" not in capture_content` (the substring-only check FIND-002 identified as insufficient)."} + ] + }, + "edge_case_coverage": { + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume.py:45-417", "description": "EDGE-CR1 through EDGE-CR12 (all 12) remain fully covered by dedicated tmp_path fixtures, unchanged from and consistent with iteration-1's own PASS verdict on this dimension; lib/checkpoint_resume.py's pure-function logic (REQ-CR1-5,10,11) is byte-identical to what iteration-1 reviewed and found correct -- this iteration's fix is scoped entirely to run_evolve.sh's argument-array-expansion pattern, not the pure core."}, + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py:274-301", "description": "REQ-CR12's crash-branch edge case (PROP-CR13) still exercised via FAKE_CHECKPOINT_CRASH, and now implicitly also exercises the SAME fixed array-expansion line (CHECKPOINT_ARGS stays empty in the crash branch too, per run_evolve.sh:77-79), so FIND-001's bug class is closed for both the not-found AND crashed branches, not just the not-found one."} + ] + }, + "implementation_correctness": { + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/run_evolve.sh:75-110", "description": "Full pre-invocation step hand-traced end-to-end against REQ-CR6/7/8/9/12: command-substitution exit-status capture (line 75-76) correctly precedes CHECKPOINT_ARGS initialization (line 77); if/elif/else (lines 78-85) correctly prioritizes the crash branch before the found/not-found branches (so a nonzero exit is never misread even if stdout happened to be non-empty); all three REQ-CR9 pinned log substrings present verbatim in their correct branch; the fixed array-expansion at line 109 yields exactly 8 arguments in the fallback case (2 positionals + 3 flag pairs), confirmed both by hand-count against the literal command text and by the on-disk pytest lastfailed/nodeids cache showing test_prop_cr_wire2_no_checkpoint_found_argument_list_unchanged_and_logged (which asserts this exact count) is NOT among the 18 currently-failing tests."}, + {"type": "test-output", "location": "skills/earn/self-improve/.pytest_cache/v/cache/lastfailed + nodeids", "description": "Independently-discovered on-disk pytest cache (not a narrative claim): lastfailed lists exactly 18 tests, all in pre-existing files (test_adversary_disapprove.py, test_baseline_beat.py, test_denylist_reject.py, test_heldout_regress.py, test_ledger_reader.py, test_realized_gate.py, test_risk_adjusted_fitness.py) unrelated to this feature; nodeids confirms the full 121-test suite (including all 31 checkpoint-resume tests) was collected and run. Compiled lib/__pycache__/checkpoint_resume.cpython-314.pyc confirms the module was actually imported by a real interpreter."} + ] + }, + "structural_integrity": { + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/run_evolve.sh, lib/checkpoint_resume.py", "description": "No structural change from iteration-1's own PASS verdict on this dimension: single-purpose files, no duplication, no dead code, no new abstraction beyond REQ-CR6-9's requirements. The fix is a single-line change (run_evolve.sh:109) plus corresponding test strengthening -- no new complexity introduced."}, + {"type": "file", "location": "skills/earn/self-improve/config.yaml, promote_gate.sh, lib/promote.py, lib/promote_gate.py, launchd/ai.anicca.self-improve-evolve.plist, strategies/pm_backtest_strategy.py", "description": "Grep scan for any checkpoint-resume fingerprint across all six INV-CR2-protected files found zero matches -- these files show no sign of having been touched by this feature, consistent with INV-CR2. Grep scan of lib/checkpoint_resume.py for wallet/.env/ledger/spend-cap references found zero real matches (INV-CR3)."} + ] + }, + "verification_readiness": { + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py:153-190", "description": "FIND-002 resolved: test_prop_cr_wire2_no_checkpoint_found_argument_list_unchanged_and_logged now asserts full argument COUNT (`len(capture_lines) == 8`), absence of any empty-string token (`\"\" not in capture_lines`), AND the exact shape of the remaining arguments (`capture_lines[2:7] == [...]`) -- symmetric with its sibling found-branch test's existing full-position-verification convention (PROP-CR-WIRE2b, lines 213-218). PROP-CR-WIRE2's proof obligation (verification-architecture.md line 76) is now genuinely discharged, not merely claimed."}, + {"type": "test-output", "location": ".vcsdd/features/self-improve-checkpoint-resume/evidence/green-phase.md:1-582", "description": "Builder-reported transcript (corroborated, not solely trusted, per the pytest-cache cross-check above) shows all 31 new tests PASSED including the strengthened test, and the full-suite regression check shows the identical 18-failed/102-passed/1-skipped baseline as iteration-1 documented -- zero new regressions from this fix."} + ] + } + }, + "blocking_count": 0, + "convergenceSignals": { + "findingCount": 0, + "allCriteriaEvaluated": true, + "evaluatedCriteria": [], + "duplicateFindings": [], + "blockingFindingCount": 0, + "priorIterationFindingsResolved": ["FIND-001", "FIND-002"] + }, + "notes": "FIND-001 (critical, run_evolve.sh:109 empty-array-expansion argument-injection bug) and FIND-002 (high, test_checkpoint_resume_wiring.py's not-found-branch test only checked substring absence, not argument count) from iteration-1 are BOTH confirmed resolved on independent re-review: FIND-001 via the canonical `${arr[@]+\"${arr[@]}\"}` idiom now at run_evolve.sh:109 (exactly FIND-001's own recommended fix), and FIND-002 via the strengthened test at tests/test_checkpoint_resume_wiring.py:166-186 which now asserts exact argument count/content. This reviewer, like iteration-1's, had no Bash/execute tool in this session (see reviewerToolConstraints) -- confidence beyond hand-tracing is grounded in independently-discovered on-disk pytest cache artifacts (.pytest_cache/v/cache/lastfailed, nodeids, and compiled .pyc bytecode) that could not have been fabricated by a narrative document alone, plus git ref-log confirmation that this worktree's HEAD is exactly the claimed fix commit eba2270a9f9c4891092e0293012dbd3630fdd7c4. No new blocking findings identified in this fresh full pass (lean mode: mandate is BLOCKING findings only, none found)." +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/state.json b/.vcsdd/features/self-improve-checkpoint-resume/state.json index cc482d8f..ec5ce370 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/state.json +++ b/.vcsdd/features/self-improve-checkpoint-resume/state.json @@ -2,7 +2,7 @@ "featureName": "self-improve-checkpoint-resume", "mode": "lean", "language": "python", - "currentPhase": "3", + "currentPhase": "6", "phaseHistory": [ { "from": "init", @@ -31,6 +31,27 @@ "timestamp": "2026-07-11T22:19:01.000Z", "reason": "AGENT ECONOMY LOOP session: RED phase confirmed (git-stash-verified collection error / 7-of-8 wiring test failures before impl existed, evidence/red-phase.md), then GREEN phase 2b implemented: lib/checkpoint_resume.py::find_latest_checkpoint (REQ-CR1-5,10,11 pure read-only checkpoint discovery) + run_evolve.sh's REQ-CR6-9,12 pre-invocation step (positioned before the $OPENEVOLVE_BIN existence check so the pinned-string-position PROP-CR-WIRE1 test passes against the FIRST occurrence of \"$OPENEVOLVE_BIN\" in source). 31/31 new tests pass (evidence/green-phase.md); full suite shows the same 18 pre-existing failures confirmed present on main HEAD too (pm_backtest_strategy.py fixture drift + one worktree-path-only test) -- zero regressions. No separate 2c refactor needed (implementation was already minimal/clean on first GREEN pass). Live-verified and fixed a bash 3.2 gotcha before shipping: \"${CHECKPOINT_ARGS[@]}\" on an empty array raises \"unbound variable\" under this script's set -u on bash 3.2 (this Mac's default /bin/bash, confirmed via bash --version); used \"${CHECKPOINT_ARGS[@]:-}\" instead. commit 515eb089. Entering Phase 3 (impl-review, fresh adversary).", "sprint": 1 + }, + { + "from": "3", + "to": "3", + "timestamp": "2026-07-11T22:22:00.000Z", + "reason": "Phase 3 impl-review iteration 1 verdict: FAIL -- 2 blocking findings. FIND-001 (critical, implementation_correctness, run_evolve.sh:109): \"${CHECKPOINT_ARGS[@]:-}\" on an empty array under this script's set -u on bash 3.2 (this Mac's default /bin/bash) expands to ONE stray empty-string argument, not zero -- would break openevolve-run's argparse (no catch-all positional) on the common \"no checkpoint found\" path. FIND-002 (high, verification_readiness, tests/test_checkpoint_resume_wiring.py): PROP-CR-WIRE2's test only checked substring absence of '--checkpoint', not argument count, so it could not have caught FIND-001. reviews/impl/iteration-1/output/verdict.json. Gate 3 NOT yet passed.", + "sprint": 1 + }, + { + "from": "3", + "to": "5", + "timestamp": "2026-07-11T22:35:04.000Z", + "reason": "Iteration-1 findings fixed (commit eba2270a: FIND-001 -- live bash -c repro confirmed the bug (argc=1, arg=\"\") and the canonical fix (argc=0 empty / argc=2 populated), replaced with \"${CHECKPOINT_ARGS[@]+\"${CHECKPOINT_ARGS[@]}\"}\"; FIND-002 -- test_prop_cr_wire2 strengthened to assert exact argument count (8) + no empty-string argument, verified this strengthened test FAILS against the pre-fix code (9 args w/ trailing \"\") and PASSES against the fix). Phase 3 impl-review iteration 2 verdict: PASS -- 0 blocking findings across all 5 dimensions (spec_fidelity, edge_case_coverage, implementation_correctness, structural_integrity, verification_readiness); FIND-001/FIND-002 both independently confirmed resolved (forensic pytest-cache + .pyc + git HEAD corroboration). reviewed_commit eba2270a. 31/31 feature tests pass, same 18 pre-existing unrelated failures unchanged (0 regressions). reviews/impl/iteration-2/output/verdict.json. Gate 3 PASSED at commit eba2270a. Entering Phase 5 (formal hardening).", + "sprint": 1 + }, + { + "from": "5", + "to": "6", + "timestamp": "2026-07-11T22:37:46.000Z", + "reason": "Phase 5 formal hardening complete: verification/verification-report.md (all 19 PROP-CR* proof obligations from verification-architecture.md mapped to their discharging pytest test, 31/31 re-run live this session and PASS at commit eba2270a; full-suite regression baseline 18 pre-existing-failed/102 passed/1 skipped, same failure set confirmed present on main HEAD, zero new regressions), verification/security-report.md (semgrep --config auto over both new/modified files: 0 findings, 0 errors; threat-surface analysis: no network/subprocess/filesystem-mutation/file-content-read/wallet-or-ledger-or-live-money-path access anywhere in this feature), verification/purity-audit.md (PROP-CR9/12a/12b static scans + PROP-CR12c dynamic before/after directory-snapshot comparison independently re-confirmed against the actually-shipped code this session; zero deviations from the declared Purity Boundary Analysis). mode:lean -- no separate fuzz/mutation/proof-harness artifacts required beyond the pytest-discharged PROP table (verification/{fuzz-results,mutation-results,proof-harnesses,security-results}/ scaffold dirs left empty, consistent with lean mode). Entering Phase 6 (convergence).", + "sprint": 1 } ], "iterations": { @@ -39,7 +60,8 @@ "1c": 1, "2a": 1, "2b": 1, - "3": 1 + "3": 2, + "5": 1 }, "proofObligations": [], "traceability": { @@ -57,9 +79,17 @@ }, "adversaryVerdict": "PASS", "adversaryReviewedAt": "2026-07-11T09:48:39.607Z" + }, + "3": { + "verdict": "PASS", + "timestamp": "2026-07-11T22:35:04.000Z", + "reviewedBy": "adversary", + "details": "Phase 3 impl-review PASS at iteration 2 (reviews/impl/iteration-2/output/verdict.json): 0 blocking findings across all 5 dimensions. reviewed_commit eba2270a. Iteration 1 (reviews/impl/iteration-1/output/verdict.json) found 2 blocking findings (FIND-001 critical bash argv-injection bug, FIND-002 test-coverage gap), both fixed and independently re-verified resolved by iteration 2.", + "adversaryVerdict": "PASS", + "adversaryReviewedAt": "2026-07-11T22:35:04.000Z" } }, "sprintCount": 1, "createdAt": "2026-07-11T08:58:04.512Z", - "updatedAt": "2026-07-11T09:48:45.135Z" + "updatedAt": "2026-07-11T22:35:04.000Z" } diff --git a/.vcsdd/features/self-improve-checkpoint-resume/verification/purity-audit.md b/.vcsdd/features/self-improve-checkpoint-resume/verification/purity-audit.md new file mode 100644 index 00000000..5d0d2022 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/verification/purity-audit.md @@ -0,0 +1,52 @@ +# Purity Audit — self-improve-checkpoint-resume (Phase 5) + +## Declared purity boundary (behavioral-spec.md "Purity Boundary Analysis") + +- **Pure core**: `lib/checkpoint_resume.py::find_latest_checkpoint(runs_dir, current_run_id) -> + Optional[str]`. Declared: deterministic path/existence logic ONLY (`os.listdir`, + `os.path.isdir`, string parsing/sorting) — no subprocess, no network, no filesystem mutation, + no file-content reads. +- **Effectful shell**: `run_evolve.sh`'s new pre-invocation step (REQ-CR6-9,12) — one subprocess + call to `lib/checkpoint_resume.py`'s `__main__` CLI entrypoint (env-var config, single stdout + line, no argv, no file writes of its own) and, downstream, appending a conditional + `--checkpoint ` argument pair to the pre-existing `openevolve-run` invocation. + +## Verification against the actual shipped code (this session, commit `eba2270a`) + +1. **Static denylist scan** (`test_prop_cr9_...`, `test_prop_cr12a_...`): the FULL module source + of `lib/checkpoint_resume.py` (pure function AND `__main__` block) contains no `open(` in a + non-default mode, no `subprocess`, `requests`, `socket`, `urllib`, and no filesystem-mutation + API (`os.remove`/`unlink`/`rmdir`/`removedirs`/`mkdir`/`makedirs`/`rename`/`replace`/ + `truncate`/`symlink`/`link`, any `shutil.*`). Re-read the shipped file this session to confirm + directly: it contains exactly `import os`, `import re`, `import sys`, and no other import — + zero deviation from the declared boundary. +2. **Import-absence scan** (`test_prop_cr12b_...`): no `import pathlib` / `from pathlib` anywhere + in the module — confirmed by direct read. +3. **Dynamic before/after snapshot** (`test_prop_cr12c_...`, both variants): a full recursive + directory-tree snapshot (path, is-dir, mtime) taken immediately before and after calling + `find_latest_checkpoint` — for BOTH a `None`-returning call (empty `runs_dir`) and a real-path- + returning call (populated two-run fixture) — is byte-for-byte/mtime-for-mtime identical. This + is the strongest possible purity evidence: not "the source looks pure" but "observed behavior + against a real filesystem tree caused zero mutation," independently of (1)/(2). +4. **`__main__` entrypoint's only effect**: reads `os.environ["RUNS_DIR"]`/`os.environ["RUN_ID"]`, + calls the pure function, writes exactly one line to stdout (`sys.stdout.write(...)`, no + trailing newline, matching the spec's "prints EXACTLY one line" contract as interpreted by the + test fixtures' `printf '%s'`-shaped stand-ins). Never opens a file. Confirmed by direct read of + the `if __name__ == "__main__":` block (5 lines). +5. **Effectful-shell boundary in `run_evolve.sh`**: the new step's only side effects are (a) one + `2>>"$LOG"` stderr-append during the subprocess call (pre-existing convention, same as the + OBSERVE step's `ledger_reader.py` call) and (b) `echo ... >> "$LOG"` log lines (pre-existing + convention, REQ-CR9). It never writes to `runs/`, `state/` (other than the log), or anywhere + outside `$LOG` and the array variable it builds in-memory for the subsequent (pre-existing, + unmodified-except-for-the-one-appended-argument) `openevolve-run` invocation. + +## Deviations found + +**Zero.** Every claim in the declared Purity Boundary Analysis is true of the actually-shipped +code, verified by a combination of static source inspection (this audit + PROP-CR9/12a/12b) and +dynamic behavioral observation (PROP-CR12c) — not a "zero deviations" self-certification on trust +alone (the franklin-alwaysact-skill-router feature's own Phase 6 convergence history in this same +repo records that a purity-audit's bare "zero deviations" claim was found FALSE twice by later +adversary review on citation-accuracy grounds; this audit is written to withstand that same +scrutiny by pointing at the specific test/read that grounds each claim above, not asserting it +unsupported). diff --git a/.vcsdd/features/self-improve-checkpoint-resume/verification/security-report.md b/.vcsdd/features/self-improve-checkpoint-resume/verification/security-report.md new file mode 100644 index 00000000..1e6ce324 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/verification/security-report.md @@ -0,0 +1,58 @@ +# Security Report — self-improve-checkpoint-resume (Phase 5) + +## Static scan + +`semgrep --config auto skills/earn/self-improve/lib/checkpoint_resume.py +skills/earn/self-improve/run_evolve.sh` (run live this session): **0 findings, 0 errors**. + +## Threat surface + +This feature adds a PURE, read-only path-selection function +(`lib/checkpoint_resume.py::find_latest_checkpoint`) plus one new bash pre-invocation step in +`run_evolve.sh`. Neither component: +- opens a network connection (no `socket`/`requests`/`urllib` reference — PROP-CR9, statically + scanned by `test_prop_cr9_pure_function_source_has_no_effectful_references_outside_main` and + live-confirmed 0 hits this session) +- spawns a subprocess (no `subprocess` reference in the pure function — same PROP-CR9 scan; the + `__main__` CLI entrypoint's only effect is one stdout write, called BY `run_evolve.sh` as a + subprocess, never the other way) +- mutates the filesystem (PROP-CR12a/b/c — static denylist scan for every `os.*`/`shutil.*` + mutation API + `pathlib` import-absence scan + dynamic before/after directory-snapshot + comparison across both a `None`-returning and a real-path-returning call; all 4 checks pass) +- reads any file's byte contents, only directory-entry names/existence (`os.listdir`, + `os.path.isdir`) — a checkpoint directory's internal validity is never inspected by this + feature (REQ-CR5/PROP-CR8), delegated entirely to `openevolve-run` itself, unchanged +- reads or writes a wallet key, `.env`, or ledger file (grep-scanned this session: zero hits in + `lib/checkpoint_resume.py`; the sole `.env`-looking substring match was `os.environ`, a false + positive) +- touches the live-money path (`promote_gate.sh`, `lib/promote_gate.py`, `lib/promote.py`, + `strategies/pm_backtest_strategy.py`, `config.yaml`, or the launchd plist) — grep-scanned this + session, zero references found in the new/modified files + +## Injection surface (the one real risk class for a feature that builds a shell argument list) + +The new `run_evolve.sh` step builds `CHECKPOINT_ARGS` from `$CHECKPOINT_PATH`, itself the captured +stdout of `lib/checkpoint_resume.py`. That path is always constructed by +`os.path.abspath(os.path.join(...))` from directory-entry NAMES already present in `runs_dir` — +never from unsanitized external input — and is passed to `openevolve-run` as a single quoted +array element (`CHECKPOINT_ARGS=(--checkpoint "$CHECKPOINT_PATH")`), never interpolated into a +string that bash re-parses/re-splits, so a directory name containing spaces or shell metacharacters +cannot cause word-splitting or command injection (verified indirectly by +`test_prop_cr_wire2b_checkpoint_found_appends_single_checkpoint_flag`'s own fixture, which uses a +path containing no metacharacters but does exercise the array-element quoting mechanism the same +way a hostile name would). + +## Bash argument-list injection bug (found + fixed this session, not a security vuln but adjacent) + +See `verification-report.md`'s "Bug caught and fixed" section — FIND-001 was a functional argv +bug (`"${CHECKPOINT_ARGS[@]:-}"` on bash 3.2 injecting a stray empty-string argument), not an +attacker-controlled injection, but it lives in exactly the same code path a real injection would, +so it is recorded here for completeness. Fixed to the canonical +`"${CHECKPOINT_ARGS[@]+"${CHECKPOINT_ARGS[@]}"}"` idiom and independently re-verified by a second +fresh adversary iteration (PASS, 0 blocking findings). + +## Conclusion + +No security findings. This feature's entire runtime surface is read-only directory-name +inspection plus one already-existing, already-safe shell argument-array-append pattern (now +correctly bash-3.2-safe). diff --git a/.vcsdd/features/self-improve-checkpoint-resume/verification/verification-report.md b/.vcsdd/features/self-improve-checkpoint-resume/verification/verification-report.md new file mode 100644 index 00000000..61703151 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/verification/verification-report.md @@ -0,0 +1,72 @@ +# Verification Report — self-improve-checkpoint-resume (Phase 5) + +mode: lean. 19/19 proof obligations from `specs/verification-architecture.md` proved by the +listed test, all re-executed live this session (`cd skills/earn/self-improve && source +~/.anicca-venvs/self-improve/bin/activate && python3 -m pytest tests/test_checkpoint_resume.py +tests/test_checkpoint_resume_wiring.py -v`) at commit `eba2270a`: **31/31 pass**. + +| PROP | Discharged by | Result | +|---|---|---| +| PROP-CR1 | `test_prop_cr1_returns_none_for_nonexistent_runs_dir` | PROVED | +| PROP-CR1b | `test_prop_cr1b_relative_runs_dir_returns_absolute_path` | PROVED | +| PROP-CR2 | `test_prop_cr2_empty_runs_dir_returns_none`, `test_prop_cr2_runs_dir_with_only_current_run_id_returns_none` | PROVED | +| PROP-CR3 | `test_prop_cr3_selects_most_recent_run_by_name_not_highest_checkpoint_number` | PROVED | +| PROP-CR4 | `test_prop_cr4_integer_comparison_not_lexicographic` | PROVED | +| PROP-CR5 | `test_prop_cr5_run_without_checkpoints_subdirectory_is_skipped`, `test_prop_cr5_run_with_literally_empty_checkpoints_subdirectory_is_skipped` | PROVED | +| PROP-CR6 | `test_prop_cr6_ignores_non_matching_entries_interleaved_with_a_valid_one` | PROVED | +| PROP-CR7 | `test_prop_cr7_current_run_id_excluded_even_with_its_own_checkpoints` | PROVED | +| PROP-CR8 | `test_prop_cr8_selected_purely_by_name_empty_checkpoint_dir_never_inspected`, `test_prop_cr8_selected_purely_by_name_garbage_content_never_inspected` | PROVED | +| PROP-CR9 | `test_prop_cr9_pure_function_source_has_no_effectful_references_outside_main` | PROVED | +| PROP-CR10 | `test_prop_cr10_non_run_shaped_db_sibling_alone_returns_none`, `test_prop_cr10_non_run_shaped_db_sibling_ignored_when_a_real_run_also_present` | PROVED | +| PROP-CR11 | `test_prop_cr11_stray_file_only_checkpoints_dir_falls_through_to_older_run` | PROVED | +| PROP-CR12 | (a) `test_prop_cr12a_full_module_source_has_no_filesystem_mutation_api_references` (b) `test_prop_cr12b_module_never_imports_pathlib` (c) `test_prop_cr12c_no_mutation_across_a_none_returning_call`, `test_prop_cr12c_no_mutation_across_a_real_path_returning_call` | PROVED (all 3 sub-checks) | +| PROP-CR13 | `test_prop_cr13_resume_check_crash_falls_back_and_logs_distinct_message` | PROVED | +| PROP-CR14 | `test_prop_cr14_run_shaped_plain_file_falls_through_without_raising`, `test_prop_cr14_run_shaped_plain_file_only_returns_none` | PROVED | +| PROP-CR-WIRE1 | `test_prop_cr_wire1_pinned_checkpoint_resume_call_present_and_before_openevolve_invocation` | PROVED | +| PROP-CR-WIRE2 | `test_prop_cr_wire2_no_checkpoint_found_argument_list_unchanged_and_logged` (strengthened post-FIND-002 to assert exact argument count, not just substring absence) | PROVED | +| PROP-CR-WIRE2b | `test_prop_cr_wire2b_checkpoint_found_appends_single_checkpoint_flag` | PROVED | +| PROP-CR-WIRE3 | `test_prop_cr_wire3_exactly_one_checkpoint_resume_log_line_not_found_branch`, `test_prop_cr_wire3_exactly_one_checkpoint_resume_log_line_found_branch` | PROVED | +| PROP-CR-LIVE1 | `test_prop_cr_live1_end_to_end_two_run_fixture_selects_and_logs_checkpoint_20` (real `lib/checkpoint_resume.py`, real `sys.executable`, no stand-in for the pure logic) | PROVED | + +Residual REQ-CR4 coverage: `test_req_cr4_runs_dir_with_only_non_run_shaped_entries_returns_none`. +Regression smoke: `test_regression_run_evolve_sh_remains_valid_bash_syntax` (`bash -n`) PASS. + +## Full suite regression baseline + +`python3 -m pytest tests/ -q` at commit `eba2270a`: **18 failed, 102 passed, 1 skipped**. All 18 +failures confirmed pre-existing (unrelated `pm_backtest_strategy.py` test-fixture drift already +present on `main` HEAD, plus one worktree-path-only test that fails in ANY worktree checkout by +design — `test_ledger_reader.py::test_realized_summary_default_path_points_at_the_real_earn_ledger_location` +asserts a path ending in `anicca/skills/...` which is never true from `anicca/.worktrees// +skills/...`). Same 18 failure signatures confirmed on `main` HEAD via `cd ~/anicca/skills/earn/ +self-improve && python3 -m pytest tests/ --ignore=tests/test_ledger_wallet_filter.py -q` (17 of +the 18 — `test_ledger_wallet_filter.py` itself was excluded from that comparison run because it +needs the `hypothesis` package, not installed in `~/.anicca-venvs/self-improve`, an unrelated +pre-existing environment gap). Zero new regressions introduced by this feature. + +## Money-safety / scope discipline (adversary-verified, iteration 2) + +`git diff main...feature/self-improve-checkpoint-resume` touches exactly: `skills/earn/self-improve/ +lib/checkpoint_resume.py` (new), `skills/earn/self-improve/run_evolve.sh` (modified, one new +pre-invocation step), `skills/earn/self-improve/tests/test_checkpoint_resume.py` (new), +`skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py` (new), plus this feature's own +`.vcsdd/features/self-improve-checkpoint-resume/**` bookkeeping. No reference to `strategies/ +pm_backtest_strategy.py`, `promote_gate.sh`, `lib/promote_gate.py`, `lib/promote.py`, +`config.yaml`, `ai.anicca.self-improve-evolve.plist`, any wallet key, `.env`, or ledger file +anywhere in the diff (INV-CR2/INV-CR3 hold). This is a harness-only change to a paper/backtest +self-improvement loop; it never touches a live-money path. + +## Bug caught and fixed during this session (Phase 3, iteration 1 -> 2) + +Fresh Sonnet adversary FIND-001 (critical): `"${CHECKPOINT_ARGS[@]:-}"` on an empty bash array +under `set -u` expands to ONE stray empty-string argument on bash <4.4 (this Mac's default +`/bin/bash` is 3.2.57), not zero arguments — confirmed via a live `bash -c` repro +(`f() { echo "argc=$#"; }; arr=(); f "${arr[@]:-}"` -> `argc=1`). This would have broken +`openevolve-run`'s argparse (no catch-all positional, confirmed by reading the installed +`openevolve==0.3.0` CLI source) on the common "no prior checkpoint" path — i.e. nearly every +cycle until a checkpoint accumulates, worse than the stagnation bug this feature fixes. Corrected +to the canonical `"${CHECKPOINT_ARGS[@]+"${CHECKPOINT_ARGS[@]}"}"` idiom, confirmed live +(`argc=0` empty, `argc=2` populated). FIND-002 (test gap that let FIND-001 through) fixed by +strengthening `test_prop_cr_wire2_...` to assert exact argument count; confirmed this strengthened +test genuinely fails against the pre-fix code (9 args, trailing `""`) before confirming it passes +against the fix. From 32794d1e837883fad1ef7988916c57db2c5d9d20 Mon Sep 17 00:00:00 2001 From: Daisuke Sato Date: Sun, 12 Jul 2026 07:46:17 +0900 Subject: [PATCH 11/11] vcsdd(self-improve-checkpoint-resume): Phase 6 CONVERGED, phase -> complete 31/31 feature tests green, 0 regressions vs main HEAD baseline, semgrep 0 findings, purity boundary independently re-confirmed. Ready for merge. --- .../reviews/converge/output/verdict.json | 96 +++++++++++++++++++ .../self-improve-checkpoint-resume/state.json | 22 ++++- 2 files changed, 115 insertions(+), 3 deletions(-) create mode 100644 .vcsdd/features/self-improve-checkpoint-resume/reviews/converge/output/verdict.json diff --git a/.vcsdd/features/self-improve-checkpoint-resume/reviews/converge/output/verdict.json b/.vcsdd/features/self-improve-checkpoint-resume/reviews/converge/output/verdict.json new file mode 100644 index 00000000..d2dcea67 --- /dev/null +++ b/.vcsdd/features/self-improve-checkpoint-resume/reviews/converge/output/verdict.json @@ -0,0 +1,96 @@ +{ + "phase": "converge", + "feature": "self-improve-checkpoint-resume", + "verdict": "CONVERGED", + "iteration": 1, + "reviewerToolConstraints": "This convergence session had NO Bash/execute tool available (only Read/Write/Edit/Grep/Glob). I could not run `python3 -m pytest`, `semgrep`, `bash -n`, or `git diff`/`git log -p` myself. Per the task's own instructed fallback, I cross-referenced verification-report.md's '31/31 pass' and 'full suite 18 failed/102 passed/1 skipped' claims against the on-disk `.pytest_cache/v/cache/{lastfailed,nodeids}` artifacts (still present in this worktree) and against a byte-for-byte count of `def test_` occurrences in both actual test files, rather than re-executing pytest.", + "dimensions": [ + { + "name": "spec_fidelity", + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/lib/checkpoint_resume.py:15-69", "description": "Direct read this session: find_latest_checkpoint's absolute-path normalization (REQ-CR1, line 52 os.path.abspath), filter->exclude->sort-descending->iterate order (REQ-CR2, lines 56-67), REQ-CR10 run-name regex `^run-\\d{8}T\\d{6}Z$` (line 19), integer (not lexicographic) checkpoint comparison via int(match.group(1)) (REQ-CR3, lines 38-41), os.path.isdir guard before listdir (EDGE-CR12, line 29) all match behavioral-spec.md's literal requirement text."}, + {"type": "file", "location": "skills/earn/self-improve/run_evolve.sh:75-85,103-110", "description": "REQ-CR6's pinned call string `RUNS_DIR=\"$RUNS_DIR\" RUN_ID=\"$RUN_ID\" \"$PY_BIN\" \"$SKILL_DIR/lib/checkpoint_resume.py\"` is present verbatim at line 75, positioned after $RUN_DIR creation (line 52) and before the first `\"$OPENEVOLVE_BIN\"` occurrence (line 87). All three REQ-CR9 pinned log substrings ('checkpoint_resume: resuming from checkpoint', 'checkpoint_resume: no prior checkpoint found', 'checkpoint_resume: resume-check crashed') are present verbatim in their correct branch (lines 79/81/84). The FIND-001 fix idiom `\"${CHECKPOINT_ARGS[@]+\"${CHECKPOINT_ARGS[@]}\"}\"` is present at line 109, exactly as both impl-review verdicts describe -- independently re-confirmed by direct read this session, not merely trusted from the narrative."}, + {"type": "file", "location": "skills/earn/self-improve/config.yaml:11,129", "description": "behavioral-spec.md's sources section cites `checkpoint_interval: 10` at config.yaml line 11 and `database.db_path: \"runs/db\"` at config.yaml line 129 (EDGE-CR10). Both independently re-read this session and confirmed byte-accurate: line 11 is exactly `checkpoint_interval: 10 # top-level Config field...` and line 129 is exactly `db_path: \"runs/db\"`."} + ] + }, + { + "name": "edge_case_coverage", + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume.py:1-418", "description": "Counted 23 `def test_` functions in this file this session, matching verification-report.md's PROP-CR1/1b/2/3/4/5(x2)/6/7/8(x2)/9/10(x2)/11/12a/12b/12c(x2)/14(x2) plus the residual REQ-CR4 test -- every EDGE-CR1 through EDGE-CR12 case has a dedicated tmp_path fixture (EDGE-CR11's stray-.DS_Store-only checkpoints/ dir at lines 274-280, EDGE-CR12's zero-byte plain-file-named-like-a-run-dir at lines 293-303, both hand-verified this session)."}, + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py:1-343", "description": "Counted 8 `def test_` functions in this file this session (wire1/wire2/wire2b/wire3x2/cr13/live1/regression-syntax) = 23+8=31 total, matching every one of the 31 test node IDs actually present in `.pytest_cache/v/cache/nodeids` (grep-confirmed this session: exactly 31 `checkpoint_resume` occurrences, each an exact match to a real `def test_` name in these two files -- no phantom test name in either document)."} + ] + }, + { + "name": "implementation_correctness", + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/run_evolve.sh:75-113", "description": "Full pre-invocation step hand-traced this session against REQ-CR6/7/8/9/12: exit-status capture (line 76) precedes CHECKPOINT_ARGS initialization (line 77); if/elif/else (78-85) prioritizes the crash branch before found/not-found; the fixed array-expansion at line 109 (the FIND-001 fix) is present verbatim, matching what both impl-review verdicts describe as the reviewed state -- this session's direct read of the CURRENT worktree file (HEAD=d4990326, not a stale copy) shows no drift from what iteration-2's verdict (reviewed_commit eba2270a) described, since Phase 5's only subsequent commit (d4990326, 'Phase 5 formal hardening complete') is stated in state.json's own phaseHistory to have added only verification/*.md files, and this session's direct read of checkpoint_resume.py/run_evolve.sh/both test files independently confirms their content is unchanged from what iteration-2 reviewed."}, + {"type": "test-output", "location": "skills/earn/self-improve/.pytest_cache/v/cache/lastfailed", "description": "Read this session: exactly 18 entries, every one in a pre-existing, unrelated test file (test_adversary_disapprove.py, test_baseline_beat.py, test_denylist_reject.py, test_heldout_regress.py, test_ledger_reader.py, test_realized_gate.py, test_risk_adjusted_fitness.py) -- ZERO entries from test_checkpoint_resume.py or test_checkpoint_resume_wiring.py, corroborating (independently of the narrative) verification-report.md's '31/31 pass, 18 pre-existing unrelated failures' claim. This cache file is a durable, still-present artifact (unlike the specific .pyc bytecode files iteration-2's verdict also cited -- see structural_integrity finding below for that caveat)."} + ] + }, + { + "name": "structural_integrity", + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/{config.yaml,promote_gate.sh,lib/promote.py,lib/promote_gate.py,strategies/pm_backtest_strategy.py,launchd/ai.anicca.self-improve-evolve.plist}", "description": "Fresh Grep this session for the literal string 'checkpoint_resume' across all six INV-CR2-protected files: ZERO matches in any of them -- independently confirms INV-CR2 ('this feature never touches' these files) holds against the actually-current worktree state, not merely against a prior session's grep."}, + {"type": "file", "location": "skills/earn/self-improve/lib/checkpoint_resume.py:15-17", "description": "Direct read confirms the module's ONLY imports are `import os`, `import re`, `import sys` -- no subprocess/socket/requests/urllib/shutil/pathlib -- matching PROP-CR9/PROP-CR12a/PROP-CR12b's claims independently, not merely re-stating purity-audit.md's own self-report."} + ] + }, + { + "name": "verification_readiness", + "verdict": "PASS", + "findings": [], + "evidence": [ + {"type": "file", "location": "skills/earn/self-improve/tests/test_checkpoint_resume_wiring.py:166-186", "description": "Direct read confirms FIND-002's fix is real and present: `test_prop_cr_wire2_no_checkpoint_found_argument_list_unchanged_and_logged` asserts `len(capture_lines) == 8` and `\"\" not in capture_lines`, not merely substring absence of '--checkpoint' -- genuinely capable of catching a regression of FIND-001's empty-array-expansion bug, independently re-verified this session against the actual test source, not the verdict's own description of it."}, + {"type": "file", "location": ".vcsdd/features/self-improve-checkpoint-resume/verification/verification-architecture.md:60-79 vs verification-report.md:8-29", "description": "Every one of the 19 PROP-CR* rows in verification-architecture.md's Proof Obligations table has a corresponding 'PROVED' row in verification-report.md citing a real, exact test function name -- cross-checked this session against the actual test file contents (both files read in full) and against `.pytest_cache/v/cache/nodeids`: every cited test name exists verbatim in the real test files and was collected by a real pytest run. No PROP references a test name that does not exist."} + ] + } + ], + "criteria": { + "finding_diminishment": { + "verdict": "PASS", + "findings": [], + "evidence": "Phase 3 impl-review: 2 blocking findings (iteration 1) -> 0 blocking findings (iteration 2), both independently re-confirmed resolved this session by direct code read (run_evolve.sh:109's fixed array idiom; test_checkpoint_resume_wiring.py:166-186's strengthened assertion). This is this feature's first Phase 6 convergence pass; there is no prior converge iteration to diminish against, but the Phase 3 trend itself (2->0) is genuine, not a renamed/deferred finding." + }, + "finding_specificity": { + "verdict": "PASS", + "findings": [], + "evidence": "Both Phase 3 findings (FIND-001, FIND-002) carried exact file:line evidence and were independently re-derivable this session from the actual current file contents, not merely trusted from the fix narrative -- FIND-001's fix (run_evolve.sh:109) and FIND-002's fix (test_checkpoint_resume_wiring.py:166-186) were both located and read directly, not located via the verdict's own line-number claim alone (I independently re-read the surrounding lines to confirm the described content, not just that *a* line existed at that number)." + }, + "criteria_coverage": { + "verdict": "PASS", + "findings": [], + "evidence": "No contracts/sprint-N.md exists for this feature (Glob of .vcsdd/features/self-improve-checkpoint-resume/{contracts,reviews/contracts}/** returns nothing) -- this feature used lean-mode spec-review + impl-review only, no strict CRIT-XXX contract phase. All 19 PROP-CR* proof obligations in verification-architecture.md are accounted for in verification-report.md's discharge table (cross-checked this session, see verification_readiness dimension)." + }, + "four_dimensional_convergence": { + "verdict": "PASS", + "findings": [], + "evidence": "SPEC (behavioral-spec.md REQ-CR1-12/INV-CR1-5/EDGE-CR1-12, verification-architecture.md's 19 PROP-CR* rows), TEST (31/31 passing per .pytest_cache/nodeids cross-check against real test file contents), IMPL (lib/checkpoint_resume.py + run_evolve.sh's new step, directly re-read this session and matching every cited requirement), and VERIFICATION (verification-report.md/security-report.md/purity-audit.md, whose specific factual claims -- config.yaml:11/129 citations, import list, INV-CR2 file-touch scan, PROP discharge table -- were independently re-derived this session rather than rubber-stamped) are mutually consistent for this feature's entire surface." + }, + "duplicate_detection": { + "verdict": "PASS", + "findings": [], + "evidence": "FIND-001 and FIND-002 (Phase 3 iteration 1) are genuinely resolved, not renamed or deferred: the fix at run_evolve.sh:109 is the exact idiom both impl-review iteration-2 and this session independently identify as the canonical portable fix, and the strengthened test at test_checkpoint_resume_wiring.py:166-186 is demonstrably capable (by its own asserted argument-count/empty-string checks) of catching a regression of the exact bug class FIND-001 was. No new instance of either bug class was found this session in either file." + }, + "residuals_honest": { + "verdict": "PASS", + "findings": [], + "evidence": "Two non-blocking residuals are disclosed here rather than swept into a blanket 'zero deviations' claim: (1) the specific `.pyc` bytecode artifacts iteration-2's impl-review verdict cited as 'independently-discovered... could not have been fabricated' forensic evidence (lib/__pycache__/checkpoint_resume.cpython-314.pyc, tests/__pycache__/test_checkpoint_resume_wiring.cpython-314-pytest-9.1.1.pyc) do NOT exist in this worktree as of this session (confirmed via Glob of skills/earn/self-improve/**/__pycache__/** -- zero matches); this is most plausibly explained by __pycache__ being gitignored (skills/earn/self-improve/.gitignore:3-4) and transient/cleaned since that review session, and the claim is NOT falsified by this absence because the more durable `.pytest_cache/v/cache/{lastfailed,nodeids}` artifacts DO still exist and DO independently corroborate the same underlying '31/31 pass, 18 unrelated pre-existing failures' claim (cross-checked this session against the real test file contents, not merely re-read as narrative) -- so the underlying factual claim survives even though one specific corroborating artifact it cited no longer does. (2) Phase 1c's own spec-review iteration-5 verdict (reviews/spec/iteration-5/output/verdict.json) lists only 2 of the usual 5 review dimensions (spec_fidelity, verification_readiness) rather than a full 5-dimension re-evaluation -- a process-completeness observation about an already-gated (PASS, 0 blocking) prior phase, outside this session's primary Phase 3/5-cross-check mandate, noted here rather than omitted. Neither residual is blocking: no code-level claim in verification-report.md/security-report.md/purity-audit.md that this session checked (config.yaml line citations, import list, INV-CR2 grep scan, PROP-to-test-name mapping, argument-count assertions) was found false." + } + }, + "blocking_count": 0, + "reviewed_commit": "d4990326 (current worktree HEAD; Phase 3's own reviewed_commit was eba2270a -- the two intervening commits, per state.json's phaseHistory, added only verification/*.md hardening docs, and this session's direct read of the actual current lib/checkpoint_resume.py, run_evolve.sh, and both test files confirms their content is unchanged from what iteration-2 reviewed)", + "convergenceSignals": { + "findingCount": 0, + "allCriteriaEvaluated": true, + "evaluatedCriteria": [], + "duplicateFindings": [] + }, + "ts": "2026-07-12T00:00:00.000Z" +} diff --git a/.vcsdd/features/self-improve-checkpoint-resume/state.json b/.vcsdd/features/self-improve-checkpoint-resume/state.json index ec5ce370..caf9294c 100644 --- a/.vcsdd/features/self-improve-checkpoint-resume/state.json +++ b/.vcsdd/features/self-improve-checkpoint-resume/state.json @@ -2,7 +2,7 @@ "featureName": "self-improve-checkpoint-resume", "mode": "lean", "language": "python", - "currentPhase": "6", + "currentPhase": "complete", "phaseHistory": [ { "from": "init", @@ -52,6 +52,13 @@ "timestamp": "2026-07-11T22:37:46.000Z", "reason": "Phase 5 formal hardening complete: verification/verification-report.md (all 19 PROP-CR* proof obligations from verification-architecture.md mapped to their discharging pytest test, 31/31 re-run live this session and PASS at commit eba2270a; full-suite regression baseline 18 pre-existing-failed/102 passed/1 skipped, same failure set confirmed present on main HEAD, zero new regressions), verification/security-report.md (semgrep --config auto over both new/modified files: 0 findings, 0 errors; threat-surface analysis: no network/subprocess/filesystem-mutation/file-content-read/wallet-or-ledger-or-live-money-path access anywhere in this feature), verification/purity-audit.md (PROP-CR9/12a/12b static scans + PROP-CR12c dynamic before/after directory-snapshot comparison independently re-confirmed against the actually-shipped code this session; zero deviations from the declared Purity Boundary Analysis). mode:lean -- no separate fuzz/mutation/proof-harness artifacts required beyond the pytest-discharged PROP table (verification/{fuzz-results,mutation-results,proof-harnesses,security-results}/ scaffold dirs left empty, consistent with lean mode). Entering Phase 6 (convergence).", "sprint": 1 + }, + { + "from": "6", + "to": "complete", + "timestamp": "2026-07-11T22:45:30.000Z", + "reason": "Phase 6 convergence iteration 1 verdict: CONVERGED -- 0 blocking findings. Fresh Sonnet adversary cross-checked every factual claim in verification-report.md/security-report.md/purity-audit.md against the actual current file contents (line numbers, substrings, test counts) rather than rubber-stamping, per this repo's own precedent (franklin-alwaysact-skill-router's purity-audit was found false twice on citation-accuracy grounds by a prior feature's converge review) -- found zero citation drift, confirmed the 31-test count by direct def-test_ count in both test files matching .pytest_cache/v/cache/nodeids, confirmed INV-CR2 scope (zero references to checkpoint_resume in config.yaml/promote_gate.sh/lib/promote.py/lib/promote_gate.py/strategies/pm_backtest_strategy.py/the launchd plist), confirmed git reflog matches state.json's phaseHistory narrative. One non-blocking residual disclosed (a cited __pycache__/*.pyc forensic file from iteration-2's impl-review no longer exists in the worktree, gitignored/transient -- underlying claim independently corroborated by .pytest_cache instead, does not falsify anything). reviews/converge/output/verdict.json. Gate 6 PASSED (CONVERGED). Entering phase complete.", + "sprint": 1 } ], "iterations": { @@ -61,7 +68,8 @@ "2a": 1, "2b": 1, "3": 2, - "5": 1 + "5": 1, + "6": 1 }, "proofObligations": [], "traceability": { @@ -87,9 +95,17 @@ "details": "Phase 3 impl-review PASS at iteration 2 (reviews/impl/iteration-2/output/verdict.json): 0 blocking findings across all 5 dimensions. reviewed_commit eba2270a. Iteration 1 (reviews/impl/iteration-1/output/verdict.json) found 2 blocking findings (FIND-001 critical bash argv-injection bug, FIND-002 test-coverage gap), both fixed and independently re-verified resolved by iteration 2.", "adversaryVerdict": "PASS", "adversaryReviewedAt": "2026-07-11T22:35:04.000Z" + }, + "6": { + "verdict": "CONVERGED", + "timestamp": "2026-07-11T22:45:30.000Z", + "reviewedBy": "adversary", + "details": "Phase 6 convergence CONVERGED at iteration 1 (reviews/converge/output/verdict.json): 0 blocking findings, every factual claim in the three Phase 5 verification artifacts cross-checked against actual shipped-code contents (not rubber-stamped). One non-blocking residual disclosed (a transient __pycache__ forensic citation, independently corroborated another way).", + "adversaryVerdict": "CONVERGED", + "adversaryReviewedAt": "2026-07-11T22:45:30.000Z" } }, "sprintCount": 1, "createdAt": "2026-07-11T08:58:04.512Z", - "updatedAt": "2026-07-11T22:35:04.000Z" + "updatedAt": "2026-07-11T22:45:30.000Z" }