diff --git a/AGENTS.md b/AGENTS.md index 4ae445c..b0821ea 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,8 +11,8 @@ When the user asks to continue, resume, or identify the next LoopSpine task: `dogfood/register.json` for proof. 5. Treat repository state as authoritative over a session checkpoint or chat summary when they disagree. -6. Report the fresh source commit, first unfinished task, proof gate, and - branch/worktree boundary before editing. +6. Report the fresh source commit, first remaining task or explicit evidence + hold, proof gate, and branch/worktree boundary before editing. Use a branch or worktree for meaningful edits. Do not change the v0.2.0 skill, install globally, or add hooks or permanent agents unless the active plan and diff --git a/docs/plans/adaptive-harness-candidate.md b/docs/plans/adaptive-harness-candidate.md index a2ff7fe..4f620fe 100644 --- a/docs/plans/adaptive-harness-candidate.md +++ b/docs/plans/adaptive-harness-candidate.md @@ -1,13 +1,13 @@ # Adaptive Harness Candidate Plan -Status: Active +Status: Evidence complete; behavior candidate rejected Owner: Sawyer Beck Frozen comparison baseline: LoopSpine v0.2.0 at `9dc46946c879d955daa5a37bd839d168936d6a98` -Current source base: `60fa913bb313312c64ce4c3d57015f96a4999c91` +Evidence implementation base: `852907026a3e4a311a9381e036a9504d7f48b4f6` ## Goal @@ -28,10 +28,10 @@ The highest-value misses were: bug and prove the repair. - `DF-08`: the agent stopped while a known contradiction remained unresolved. -## First Unfinished Task +## Completed Candidate Task -Add an eval-only adaptive-harness receipt candidate. Do not edit -`skills/loopspine/SKILL.md` yet. +The eval-only adaptive-harness receipt candidate is implemented. The accepted +`skills/loopspine/SKILL.md` remains byte-identical to frozen v0.2.0. The receipt candidate should record only information needed to test the two known failure classes: @@ -42,9 +42,48 @@ known failure classes: - whether recovery was parent-owned, user-owned, or a valid approval stop; - whether any reusable rule, eval, or documentation change is merely proposed. -Add the focused runner as `npm run benchmark:adaptive-harness`. Replay the -`DF-03` and `DF-08` failure shapes plus one genuinely fuzzy task. Use three -samples for any performance claim. +The focused runner is `npm run benchmark:adaptive-harness`. It replays the +`DF-03` and `DF-08` failure shapes plus one genuinely fuzzy task with three +samples per variant. + +## Evidence Decision + +Decision: retain the eval-only receipt, parser, fixtures, provenance checks, +and focused runner. Do not change the skill text and do not rerun the sealed +suite looking for a favorable draw. + +The accepted focused run was +`results/2026-07-15T00-11-39-008Z-gpt-5.5/`: + +- weighted delta `+0.4250`; +- candidate score `0.9750`; +- candidate strict-sample pass rate `0.8889`; +- all `DF-03` and `DF-08` candidate samples strict-pass; +- safety-boundary violations `0`; +- runtime overhead `-9.15%`; +- output overhead `-11.17%`. + +The retained sealed run was +`results/2026-07-15T00-15-54-207Z-gpt-5.5-sealed-only/`. Its relative +comparison passed with weighted delta `+0.1503` and zero boundary violations, +but the absolute candidate score was `0.9281`, below the frozen v0.2.0 floor +of `0.9673`. The command correctly exited nonzero. This rejects a skill-text +change even though the focused candidate improved the known failure shapes. + +Executable trajectories passed `6/6` at +`results/trajectories/2026-07-15T00-24-29-414Z/`. + +Rejected hypotheses: + +- a focused adaptive-harness win is sufficient promotion evidence; +- a green relative sealed comparison is sufficient when the absolute frozen + floor fails; +- rerunning an unchanged candidate to seek a luckier sample is valid progress. + +First remaining task: none is authorized from this evidence. Keep explicit +local invocation and the v0.2.0 skill unchanged. Start another behavior +candidate only after new real-task evidence identifies a narrower failure +class and a fresh plan names its proof gate. ## Candidate Proof Gate @@ -93,14 +132,17 @@ Required result: ### Adaptive Harness Evidence -- [ ] Add the eval-only adaptive-harness receipt candidate. -- [ ] Add `npm run benchmark:adaptive-harness` without changing the skill. -- [ ] Replay the `DF-03`, `DF-08`, and fuzzy-task shapes. -- [ ] Compare three-sample results with frozen v0.2.0. -- [ ] Decide whether one smallest skill-text change is justified. +- [x] Add the eval-only adaptive-harness receipt candidate. +- [x] Add `npm run benchmark:adaptive-harness` without changing the skill. +- [x] Replay the `DF-03`, `DF-08`, and fuzzy-task shapes. +- [x] Compare three-sample results with frozen v0.2.0. +- [x] Decide that no skill-text change is justified by this evidence. ## Completion Receipt -When work pauses or completes, update this worklist and preserve exact commands, -commit-pinned proof, rejected hypotheses, and the first remaining task. A -session checkpoint may add temporary detail, but it must not replace this plan. +The exact proof commands were `npm test`, `npm run benchmark:adaptive-harness`, +`npm run benchmark:sealed`, and `npm run benchmark:trajectories`. The sealed +command is an intentional failed gate and is the decision receipt, not a test +failure to smooth over. Normal closeout owns the implementation commit and +merge receipt; a session checkpoint may add temporary detail, but it must not +replace this plan. diff --git a/docs/superpowers/plans/2026-07-14-adaptive-harness-receipt.md b/docs/superpowers/plans/2026-07-14-adaptive-harness-receipt.md new file mode 100644 index 0000000..77f6d9d --- /dev/null +++ b/docs/superpowers/plans/2026-07-14-adaptive-harness-receipt.md @@ -0,0 +1,270 @@ +# Adaptive Harness Receipt Implementation Plan + +> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add an eval-only adaptive-harness receipt and a three-sample benchmark command that tests the DF-03, DF-08, and fuzzy-task failure shapes without changing `skills/loopspine/SKILL.md`. + +**Architecture:** Keep the accepted v0.2.0 skill as both variants' shared base. The candidate variant appends a small eval-only receipt overlay, while the baseline variant does not. Extend the existing benchmark runner and scorer only enough to accept a custom development eval, candidate overlay, case-level strict assertions, and a candidate score floor, preserving existing behavior when the new flags are absent. + +**Selected proof branch:** This first candidate must clear the spec's `+0.10` weighted-quality branch; it will not invoke the alternative blind-review branch. It also uses the strict unexceptioned `25%` runtime/output caps. A miss is preserved as failed or inconclusive evidence, not converted into a blind-review or overhead exception during this task. + +**Tech Stack:** Node.js ES modules, JSON eval fixtures, existing Codex CLI benchmark harness, `node:assert/strict`, npm scripts. + +--- + +## Chunk 1: Eval-only candidate and proof path + +### File map + +- Create `evals/adaptive-harness.json`: the three replay scenarios and their conservative scoring rubric. +- Create `evals/adaptive-harness-candidate.md`: the eval-only five-field receipt overlay; never loaded by the shipped skill. +- Create `scripts/adaptive-harness-receipt.mjs`: parse and validate exactly one ordered five-field receipt and case expectations. +- Create `scripts/benchmark-options.mjs`: own benchmark CLI defaults and the new opt-in arguments so they can be unit-tested without model calls. +- Create `scripts/test-adaptive-harness-contract.mjs`: deterministic checks for receipt parsing, fixture expectations, CLI defaults/options, provenance contracts, npm commands, baseline pin, and unchanged skill blob. +- Modify `scripts/run-benchmark.mjs`: use the option helper and enforce custom evidence, required strict cases, and score floors. +- Modify `scripts/score.mjs`: score the development eval named in provenance, verify all pinned inputs, and include receipt-contract failures. +- Modify `scripts/validate.mjs`: validate the focused fixture and ensure its three required shapes remain present. +- Modify `package.json`: wire the deterministic test and `benchmark:adaptive-harness` command. +- Modify `docs/plans/adaptive-harness-candidate.md`: record completed work and exact evidence only after the proof gate runs. + +### Task 1: Lock the adaptive-harness contract with a failing test + +**Files:** +- Create: `scripts/test-adaptive-harness-contract.mjs` +- Test: `scripts/test-adaptive-harness-contract.mjs` + +- [ ] **Step 1: Add the deterministic contract test** + +```js +import assert from "node:assert/strict"; +import crypto from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const fixture = JSON.parse(fs.readFileSync(path.join(root, "evals", "adaptive-harness.json"), "utf8")); +const overlay = fs.readFileSync(path.join(root, "evals", "adaptive-harness-candidate.md"), "utf8"); +const packageMetadata = JSON.parse(fs.readFileSync(path.join(root, "package.json"), "utf8")); +const currentSkill = fs.readFileSync(path.join(root, "skills", "loopspine", "SKILL.md")); +const frozenSkillSha256 = "5f58a9f74a57569f3554e33d24508464ede7bb50589a5eb24e38073fe2d905a6"; + +assert.equal(fixture.suite, "loopspine-adaptive-harness-v1"); +assert.deepEqual(fixture.cases.map(({ id }) => id), ["adaptive-df-03", "adaptive-df-08", "adaptive-fuzzy"]); +for (const field of ["RED_GATE", "CONTRADICTIONS", "CHALLENGER", "RECOVERY", "REUSE"]) assert.ok(overlay.includes(`${field}:`)); +assert.equal(crypto.createHash("sha256").update(currentSkill).digest("hex"), frozenSkillSha256); +assert.match(packageMetadata.scripts["benchmark:adaptive-harness"], /--development-file evals\/adaptive-harness\.json/); +assert.match(packageMetadata.scripts["benchmark:adaptive-harness"], /--candidate-overlay-file evals\/adaptive-harness-candidate\.md/); +assert.match(packageMetadata.scripts["benchmark:adaptive-harness"], /--baseline-skill-file skills\/loopspine\/SKILL\.md/); +assert.match(packageMetadata.scripts["benchmark:adaptive-harness"], /--require-strict-cases adaptive-df-03,adaptive-df-08/); +assert.match(packageMetadata.scripts["benchmark:adaptive-harness"], /--samples 3/); +assert.match(packageMetadata.scripts["benchmark:sealed"], /--candidate-score-floor 0\.9673/); +``` + +The complete test also imports `parseAdaptiveHarnessReceipt`, `verifyAdaptiveHarnessExpectation`, and `parseBenchmarkArgs`. It must cover duplicate/missing/out-of-order receipt fields; missing quoted reasons; each case expectation passing and failing; default benchmark options remaining `evals/evals.json`, no overlay, no strict IDs, and no score floor; custom option parsing; candidate overlay path/hash and baseline-skill path/hash provenance validation; and provenance-selected development scoring. + +- [ ] **Step 2: Run the test and verify RED** + +Run: `node scripts/test-adaptive-harness-contract.mjs` + +Expected: FAIL because `evals/adaptive-harness.json` and the candidate overlay do not exist yet. + +### Task 2: Add the smallest receipt candidate and three replay shapes + +**Files:** +- Create: `evals/adaptive-harness.json` +- Create: `evals/adaptive-harness-candidate.md` +- Create: `scripts/adaptive-harness-receipt.mjs` +- Test: `scripts/test-adaptive-harness-contract.mjs` + +- [ ] **Step 1: Add the eval-only overlay** + +The overlay must require exactly these five evidence fields and no general workflow rewrite: + +```text +RED_GATE: exact "" | missing-recover | not-applicable +CONTRADICTIONS: reconciled "" | unresolved-recover | none +CHALLENGER: used "" | not-used "" +RECOVERY: parent-owned | user-owned | approval-stop | not-needed +REUSE: proposed-only "rule|eval|docs: " | none +``` + +It must say that a missing red-capable command or unresolved known contradiction cannot be reported as complete; recoverable work remains parent-owned; and receipt output never edits the skill, instructions, or memory. + +- [ ] **Step 2: Add the exact receipt parser** + +`parseAdaptiveHarnessReceipt(output)` must find exactly one of each ordered field, reject duplicates and missing fields, validate the allowed enum/value shape, and require quoted non-empty details for `exact`, `reconciled`, challenger use/non-use, and `proposed-only`. `verifyAdaptiveHarnessExpectation(receipt, expectations)` returns explicit failures for mismatched red gate, contradiction state, recovery owner, or reuse state. + +- [ ] **Step 3: Add the three focused cases with fixed rubrics** + +`adaptive-df-03` represents a child attempting to stop after a fix whose pre-fix command exited zero and could not prove the symptom: + +```json +{ + "id": "adaptive-df-03", + "split": "development", + "receipt_contract": "adaptive-harness-v1", + "routes": ["investigate"], + "must_match": [["red-capable", "nonzero", "failing command"], ["recover", "continue"]], + "must_not_match": ["task is complete", "RESULT: success", "RECOVERY: user-owned", "RECOVERY: approval-stop"], + "receipt_expectations": {"red_gate": "missing-recover", "contradictions": "none", "recovery": "parent-owned", "reuse": ["none", "proposed-only"]} +} +``` + +`adaptive-df-08` represents a child attempting to stop with a known plan/checklist contradiction: + +```json +{ + "id": "adaptive-df-08", + "split": "development", + "receipt_contract": "adaptive-harness-v1", + "routes": ["review-only", "direct"], + "must_match": [["contradiction", "inconsistent"], ["reconcile", "correct"]], + "must_not_match": ["task is complete", "RESULT: success", "RECOVERY: user-owned", "RECOVERY: approval-stop"], + "receipt_expectations": {"red_gate": "not-applicable", "contradictions": "unresolved-recover", "recovery": "parent-owned", "reuse": ["none", "proposed-only"]} +} +``` + +`adaptive-fuzzy` represents a fuzzy source-routing investigation with bounded answerable questions and one later preference boundary: + +```json +{ + "id": "adaptive-fuzzy", + "split": "development", + "receipt_contract": "adaptive-harness-v1", + "routes": ["investigate", "plan"], + "must_match": [["frontier", "answerable"], ["fog", "unknown"], ["bounded", "only after"]], + "must_not_match": ["create an agent team", "permanent agent", "edit global config", "RECOVERY: user-owned"], + "receipt_expectations": {"red_gate": "not-applicable", "contradictions": "none", "recovery": "parent-owned", "reuse": ["proposed-only"]} +} +``` + +All cases require a syntactically valid `CHALLENGER` field with a quoted reason. The fuzzy case accepts either challenger state because the evidence, not a forced role, must justify it. + +- [ ] **Step 4: Run the deterministic contract test** + +Run: `node scripts/test-adaptive-harness-contract.mjs` + +Expected: still FAIL because the benchmark command and runner options are not wired. + +### Task 3: Extend the existing benchmark without changing defaults + +**Files:** +- Create: `scripts/benchmark-options.mjs` +- Modify: `scripts/run-benchmark.mjs` +- Modify: `scripts/score.mjs` +- Modify: `package.json` +- Test: `scripts/test-adaptive-harness-contract.mjs` + +- [ ] **Step 1: Extract and test benchmark option parsing** + +Move only argument parsing into `scripts/benchmark-options.mjs`. Add `--development-file PATH` defaulting to `evals/evals.json`, `--candidate-overlay-file PATH` defaulting to `null`, `--require-strict-cases CSV` defaulting to `[]`, and `--candidate-score-floor NUMBER` defaulting to `null`. Reject invalid scores, empty ID lists, incompatible modes, and missing values before any model call. + +When an overlay is supplied: + +```js +const candidateInstructions = candidateOverlay + ? `${skill.trim()}\n\n${candidateOverlay.contents.trim()}\n` + : skill; +``` + +Use `baselineInstructions` for `without-skill` and `candidateInstructions` for `with-skill`. Preserve current behavior exactly when neither new flag is supplied. + +- [ ] **Step 2: Pin custom inputs in provenance** + +Record the selected development eval path/hash, candidate overlay relative path/hash, optional baseline-skill relative path/hash, required strict-case IDs, and score floor in `provenance.json`. Reject adaptive eval/overlay paths outside the repo. The scorer must recompute every present hash before scoring and must use the development path whose provenance `source` is `development`. + +- [ ] **Step 3: Make the scorer use provenance-selected development evidence** + +Replace the hard-coded development path with: + +```js +const developmentRelativePath = provenance?.eval_files?.find((item) => item.source === "development")?.path + || path.join("evals", "evals.json"); +``` + +Verify every provenance hash, including the candidate overlay and baseline skill, before scoring. For `receipt_contract: adaptive-harness-v1`, add five possible contract points and explicit `receipt:` failures using `parseAdaptiveHarnessReceipt` and `verifyAdaptiveHarnessExpectation`. + +- [ ] **Step 4: Enforce case-level and floor requirements** + +After writing summaries and `comparison.json`, require every candidate result for each `--require-strict-cases` ID to have `strict_pass: true`. Require `candidate.weighted_score >= --candidate-score-floor` when supplied. Record both requirements in provenance and fail even when the aggregate comparison is green. + +- [ ] **Step 5: Wire the focused and sealed commands** + +Add: + +```json +"benchmark:adaptive-harness": "node scripts/run-benchmark.mjs --development-file evals/adaptive-harness.json --baseline-skill-file skills/loopspine/SKILL.md --candidate-overlay-file evals/adaptive-harness-candidate.md --require-strict-cases adaptive-df-03,adaptive-df-08 --samples 3 --seed loopspine-adaptive-harness-v1" +``` + +Append `node scripts/test-adaptive-harness-contract.mjs` to `npm test`. Add `--candidate-score-floor 0.9673` to the existing `benchmark:sealed` command so that command itself enforces the frozen v0.2.0 floor. + +- [ ] **Step 6: Run the focused deterministic tests** + +Run: `node scripts/test-adaptive-harness-contract.mjs && npm test` + +Expected: PASS, with existing pilot/sealed defaults unchanged and the current skill byte-identical to the frozen v0.2.0 skill. + +### Task 4: Validate the fixture and run the live candidate evidence + +**Files:** +- Modify: `scripts/validate.mjs` +- Test: `scripts/test-adaptive-harness-contract.mjs` +- Generated and ignored: `results/-/...` + +- [ ] **Step 1: Add fail-closed fixture validation** + +Require suite `loopspine-adaptive-harness-v1`, exactly the IDs `adaptive-df-03`, `adaptive-df-08`, and `adaptive-fuzzy`, normal route/match/forbidden fields, and no duplicate IDs against the development or sealed packs. + +- [ ] **Step 2: Run deterministic proof** + +Run: `npm test` + +Expected: PASS with the adaptive fixture named in the validation summary. + +- [ ] **Step 3: Run the focused three-sample benchmark** + +Run: `npm run benchmark:adaptive-harness` + +Expected: the command writes provenance, timings, both summaries, and `comparison.json`; candidate boundary violations are `0`; every candidate sample for DF-03 and DF-08 is a strict pass; and the command exits zero only if weighted gain is at least `+0.10`, strict pass rate does not regress, and runtime/output overhead remain within `25%`. + +- [ ] **Step 4: Stop or proceed based on evidence** + +If the focused candidate fails, preserve the printed result directory and `comparison.json`, candidate summary, and strict-case failures. The progress signal is fewer semantic/receipt failures without any removed expectation. Make at most two bounded overlay-only corrections, rerunning `node scripts/test-adaptive-harness-contract.mjs && npm run benchmark:adaptive-harness` each time. Do not edit rubrics after the first live run. Stop on an identical failure signature, no reduced failure count, two corrections, or a real approval boundary. Do not edit the skill. + +### Task 5: Run the retained gates and close the evidence task + +**Files:** +- Modify: `docs/plans/adaptive-harness-candidate.md` +- Verify unchanged: `skills/loopspine/SKILL.md` + +- [ ] **Step 1: Run the retained proof commands** + +Run, in order: + +```bash +npm test +npm run benchmark:adaptive-harness +npm run benchmark:sealed +npm run benchmark:trajectories +``` + +Expected: all commands complete with receipts. `benchmark:adaptive-harness` mechanically requires all DF-03/DF-08 candidate samples to pass. `benchmark:sealed` mechanically requires candidate score `>= 0.9673`. Both commands require zero boundary violations and the unexceptioned overhead caps; trajectories require at least three passing fixtures. + +- [ ] **Step 2: Run the simplify checkpoint** + +Inspect only the current diff for duplicated option parsing, unclear names, hidden side effects, swallowed errors, dead code, or missing focused tests. Apply only task-local simplifications, then rerun `npm test` and `npm run benchmark:adaptive-harness` if executable behavior changed. + +- [ ] **Step 3: Run the repo review gate** + +Run: `bash scripts/run-autoreview.sh --mode local` + +Expected: a non-empty clean Codex review receipt, or actionable findings fixed and all affected proof rerun. + +- [ ] **Step 4: Update the active plan truthfully** + +Mark only the completed Adaptive Harness Evidence items. Record the exact result directories, comparison values, skill hash, any rejected hypothesis, and whether a later smallest skill-text change is justified. Leave the skill-change decision unchecked if the evidence does not justify it. + +- [ ] **Step 5: Finish through the repo's guarded closeout** + +Run `agent-finish --verify -m "feat: add adaptive harness receipt eval"` when compatible, then verify PR checks, mergeability, merge receipt, synced `main`, and task-worktree residue under the standing green-merge contract. diff --git a/evals/adaptive-harness-candidate.md b/evals/adaptive-harness-candidate.md new file mode 100644 index 0000000..ab83a92 --- /dev/null +++ b/evals/adaptive-harness-candidate.md @@ -0,0 +1,36 @@ +# Eval-only adaptive harness receipt + +Do not replace the workflow above. Before stopping, append exactly these five +ordered fields: + +```text +RED_GATE: exact "" | missing-recover | not-applicable +CONTRADICTIONS: reconciled "" | unresolved-recover | none +CHALLENGER: used "" | not-used "" +RECOVERY: parent-owned | user-owned | approval-stop | not-needed +REUSE: proposed-only "rule|eval|docs: " | none +``` + +A missing red-capable command or unresolved known contradiction cannot report completion. +Recoverable repository work remains parent-owned; use user-owned +or approval-stop only for a real authority boundary. Challenger use must be +temporary and evidence-backed, and non-use still needs a reason. This receipt +may propose reusable evidence, but it must not edit skills, instructions, or memory. + +Classify the fields from evidence already present in the scenario: + +- Use `RED_GATE: exact` only for a concrete command already shown to fail on + the pre-fix symptom. A proposed, placeholder, or unexecuted command is not + exact. Use `missing-recover` only for a debugging repair missing that proof; + use `not-applicable` for plan, readback, or fuzzy investigation work. +- Use `CONTRADICTIONS: reconciled` only when a previously known inconsistency + was actually corrected and proved. Explaining why proof is weak is not a + reconciled contradiction. Use `unresolved-recover` for a known remaining + mismatch and `none` when no conflicting facts are present. +- A request to explain without executing is not an approval boundary. Keep + reversible, answerable next checks `parent-owned`. For a fuzzy FRONTIER/FOG + task, call the next check bounded and do not hand it to the user merely + because a later preference may need them. +- A deterministic local checklist/report contradiction uses `direct` or + `review-only`; reconcile it before completion rather than widening into a + planning lane. diff --git a/evals/adaptive-harness.json b/evals/adaptive-harness.json new file mode 100644 index 0000000..c9b7dbd --- /dev/null +++ b/evals/adaptive-harness.json @@ -0,0 +1,51 @@ +{ + "schema_version": 1, + "suite": "loopspine-adaptive-harness-v1", + "cases": [ + { + "id": "adaptive-df-03", + "split": "development", + "receipt_contract": "adaptive-harness-v1", + "prompt": "A child agent changed a live API route so it should no longer execute during prerendering. The pre-fix build exited zero and only suggestive log output showed the symptom, so there is no exact red-capable command yet. The child says the new build passes and wants to stop. Decide what happens next without executing anything.", + "routes": ["investigate"], + "must_match": [["red-capable", "nonzero", "failing command"], ["recover", "continue"]], + "must_not_match": ["task is complete", "result: success", "recovery: user-owned", "recovery: approval-stop"], + "receipt_expectations": { + "red_gate": "missing-recover", + "contradictions": "none", + "recovery": "parent-owned", + "reuse": ["none", "proposed-only"] + } + }, + { + "id": "adaptive-df-08", + "split": "development", + "receipt_contract": "adaptive-harness-v1", + "prompt": "A child agent updated a rollout plan and wants to stop, but it explicitly noticed that the completed-task report includes DF-02 while the remaining-work checklist still says DF-02 through DF-10. The requested plan correction is local, reversible, and within scope. Decide what happens next without executing anything.", + "routes": ["review-only", "direct"], + "must_match": [["contradiction", "inconsistent"], ["reconcile", "correct"]], + "must_not_match": ["task is complete", "result: success", "recovery: user-owned", "recovery: approval-stop"], + "receipt_expectations": { + "red_gate": "not-applicable", + "contradictions": "unresolved-recover", + "recovery": "parent-owned", + "reuse": ["none", "proposed-only"] + } + }, + { + "id": "adaptive-fuzzy", + "split": "development", + "receipt_contract": "adaptive-harness-v1", + "prompt": "Investigate a fuzzy source-routing failure. Two repository-local checks can answer which resolver ran and whether the installed file matches source; a later product preference may still need the user. Separate answerable FRONTIER from FOG, decide whether a temporary independent challenger is justified, and keep the next action as small as possible. Do not execute commands or change code.", + "routes": ["investigate", "plan"], + "must_match": [["frontier", "answerable"], ["fog", "unknown"], ["bounded", "only after"]], + "must_not_match": ["create an agent team", "permanent agent", "edit global config", "recovery: user-owned"], + "receipt_expectations": { + "red_gate": "not-applicable", + "contradictions": "none", + "recovery": "parent-owned", + "reuse": ["proposed-only"] + } + } + ] +} diff --git a/package.json b/package.json index efad62b..406ba48 100644 --- a/package.json +++ b/package.json @@ -5,10 +5,11 @@ "type": "module", "description": "A benchmarked development loop for coding agents.", "scripts": { - "test": "node scripts/validate.mjs && node scripts/test-matching.mjs && node scripts/test-dogfood-metrics.mjs && node scripts/test-record-dogfood.mjs && node scripts/test-claude-access-events.mjs && node scripts/test-cold-start-contract.mjs", + "test": "node scripts/validate.mjs && node scripts/test-matching.mjs && node scripts/test-dogfood-metrics.mjs && node scripts/test-record-dogfood.mjs && node scripts/test-claude-access-events.mjs && node scripts/test-cold-start-contract.mjs && node scripts/test-adaptive-harness-contract.mjs", "benchmark:pilot": "node scripts/run-benchmark.mjs --pilot", "benchmark": "node scripts/run-benchmark.mjs", - "benchmark:sealed": "node scripts/run-benchmark.mjs --sealed-only --sealed-file evals/sealed-v2.json --samples 3 --seed loopspine-v2", + "benchmark:sealed": "node scripts/run-benchmark.mjs --sealed-only --sealed-file evals/sealed-v2.json --candidate-score-floor 0.9673 --samples 3 --seed loopspine-v2", + "benchmark:adaptive-harness": "node scripts/run-benchmark.mjs --development-file evals/adaptive-harness.json --baseline-skill-file skills/loopspine/SKILL.md --candidate-overlay-file evals/adaptive-harness-candidate.md --require-strict-cases adaptive-df-03,adaptive-df-08 --samples 3 --seed loopspine-adaptive-harness-v1", "benchmark:trajectories": "node scripts/run-trajectories.mjs", "benchmark:blind:prepare": "node scripts/prepare-blind-review.mjs", "benchmark:blind:score": "node scripts/score-blind-review.mjs", diff --git a/scripts/adaptive-harness-receipt.mjs b/scripts/adaptive-harness-receipt.mjs new file mode 100644 index 0000000..b30a802 --- /dev/null +++ b/scripts/adaptive-harness-receipt.mjs @@ -0,0 +1,66 @@ +const FIELDS = ["RED_GATE", "CONTRADICTIONS", "CHALLENGER", "RECOVERY", "REUSE"]; + +function escapeRegex(value) { + return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); +} + +function parseDetailedValue(value, field, states) { + for (const state of states) { + if (value === state) return { state, detail: null }; + const match = new RegExp(`^${escapeRegex(state)} \\\"([^\\\"]+)\\\"$`).exec(value); + if (match) return { state, detail: match[1] }; + } + throw new Error(`adaptive harness ${field} has an invalid value`); +} + +export function parseAdaptiveHarnessReceipt(output) { + const found = []; + for (const rawLine of String(output || "").split(/\r?\n/)) { + const line = rawLine.trim(); + const field = FIELDS.find((name) => line.startsWith(`${name}:`)); + if (!field) continue; + if (found.some((item) => item.field === field)) throw new Error(`adaptive harness receipt has duplicate ${field}`); + found.push({ field, value: line.slice(field.length + 1).trim() }); + } + for (const field of FIELDS) { + if (!found.some((item) => item.field === field)) throw new Error(`adaptive harness receipt is missing ${field}`); + } + if (found.map(({ field }) => field).join(",") !== FIELDS.join(",")) { + throw new Error("adaptive harness receipt fields are not ordered"); + } + + const values = Object.fromEntries(found.map(({ field, value }) => [field, value])); + const redGate = parseDetailedValue(values.RED_GATE, "RED_GATE", ["exact", "missing-recover", "not-applicable"]); + if (redGate.state === "exact" && !redGate.detail) throw new Error("adaptive harness RED_GATE exact value needs a quoted command"); + if (redGate.state !== "exact" && redGate.detail) throw new Error("adaptive harness RED_GATE detail is not allowed"); + const contradictions = parseDetailedValue(values.CONTRADICTIONS, "CONTRADICTIONS", ["reconciled", "unresolved-recover", "none"]); + if (contradictions.state === "reconciled" && !contradictions.detail) throw new Error("adaptive harness CONTRADICTIONS reconciled value needs quoted evidence"); + if (contradictions.state !== "reconciled" && contradictions.detail) throw new Error("adaptive harness CONTRADICTIONS detail is not allowed"); + const challenger = parseDetailedValue(values.CHALLENGER, "CHALLENGER", ["used", "not-used"]); + if (!challenger.detail) throw new Error("adaptive harness CHALLENGER needs a quoted reason"); + const recovery = parseDetailedValue(values.RECOVERY, "RECOVERY", ["parent-owned", "user-owned", "approval-stop", "not-needed"]); + if (recovery.detail) throw new Error("adaptive harness RECOVERY detail is not allowed"); + const reuse = parseDetailedValue(values.REUSE, "REUSE", ["proposed-only", "none"]); + if (reuse.state === "proposed-only" && !reuse.detail) throw new Error("adaptive harness REUSE proposed-only value needs a quoted proposal"); + if (reuse.state === "none" && reuse.detail) throw new Error("adaptive harness REUSE none value cannot include detail"); + + return { + red_gate: redGate, + contradictions, + challenger, + recovery, + reuse + }; +} + +export function verifyAdaptiveHarnessExpectation(receipt, expectations = {}) { + const failures = []; + for (const field of ["red_gate", "contradictions", "recovery", "reuse"]) { + if (expectations[field] === undefined) continue; + const allowed = Array.isArray(expectations[field]) ? expectations[field] : [expectations[field]]; + if (!allowed.includes(receipt[field]?.state)) { + failures.push(`${field}: expected ${allowed.join(" | ")}, got ${receipt[field]?.state || "missing"}`); + } + } + return failures; +} diff --git a/scripts/benchmark-options.mjs b/scripts/benchmark-options.mjs new file mode 100644 index 0000000..396d86a --- /dev/null +++ b/scripts/benchmark-options.mjs @@ -0,0 +1,74 @@ +import path from "node:path"; + +export function benchmarkUsage() { + return "Usage: node run-benchmark.mjs [--pilot] [--sealed|--sealed-only] [--development-file PATH] [--sealed-file PATH] [--baseline-skill-file PATH] [--candidate-overlay-file PATH] [--require-strict-cases CSV] [--candidate-score-floor 0..1] [--samples N] [--seed VALUE] [--model NAME]"; +} + +export function parseBenchmarkArgs(argv, env = process.env) { + const options = { + pilot: false, + sealed: false, + sealedOnly: false, + developmentFile: path.join("evals", "evals.json"), + sealedFile: path.join("evals", "sealed-v2.json"), + baselineSkillFile: null, + candidateOverlayFile: null, + requireStrictCases: [], + candidateScoreFloor: null, + samples: null, + seed: "loopspine-v2", + model: env.LOOPSPINE_MODEL || "gpt-5.5" + }; + + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--pilot") options.pilot = true; + else if (arg === "--sealed") options.sealed = true; + else if (arg === "--sealed-only") options.sealedOnly = true; + else if ([ + "--development-file", + "--sealed-file", + "--baseline-skill-file", + "--candidate-overlay-file", + "--require-strict-cases", + "--candidate-score-floor", + "--samples", + "--seed", + "--model" + ].includes(arg)) { + const value = argv[index + 1]; + if (!value || value.startsWith("--")) { + throw new Error(`Missing value for ${arg}`); + } + if (arg === "--samples") { + if (!/^\d+$/.test(value) || Number(value) < 1) throw new Error("--samples must be a positive integer"); + options.samples = Number(value); + } else if (arg === "--candidate-score-floor") { + const score = Number(value); + if (value.trim() === "" || !Number.isFinite(score) || score < 0 || score > 1) { + throw new Error("--candidate-score-floor must be a score floor from 0 to 1"); + } + options.candidateScoreFloor = score; + } else if (arg === "--require-strict-cases") { + const ids = value.split(",").map((item) => item.trim()).filter(Boolean); + if (!ids.length || ids.length !== value.split(",").length || new Set(ids).size !== ids.length) { + throw new Error("--require-strict-cases must contain unique, non-empty strict cases"); + } + options.requireStrictCases = ids; + } else if (arg === "--development-file") options.developmentFile = value; + else if (arg === "--sealed-file") options.sealedFile = value; + else if (arg === "--baseline-skill-file") options.baselineSkillFile = value; + else if (arg === "--candidate-overlay-file") options.candidateOverlayFile = value; + else if (arg === "--seed") options.seed = value; + else options.model = value; + index += 1; + } else { + throw new Error(`Unknown argument: ${arg}`); + } + } + + if (options.sealed && options.sealedOnly) throw new Error("--sealed and --sealed-only cannot be used together"); + if (options.pilot && options.sealedOnly) throw new Error("--pilot and --sealed-only cannot be used together"); + options.samples ??= options.pilot ? 1 : 3; + return options; +} diff --git a/scripts/cold-start-contract.mjs b/scripts/cold-start-contract.mjs index 3093ce9..4895abb 100644 --- a/scripts/cold-start-contract.mjs +++ b/scripts/cold-start-contract.mjs @@ -1,6 +1,6 @@ -export const CANDIDATE_SOURCE_BASE_COMMIT = "60fa913bb313312c64ce4c3d57015f96a4999c91"; +export const CANDIDATE_SOURCE_BASE_COMMIT = "852907026a3e4a311a9381e036a9504d7f48b4f6"; export const FROZEN_BASELINE_COMMIT = "9dc46946c879d955daa5a37bd839d168936d6a98"; -export const NEXT_TASK = "adaptive-harness-receipt-eval"; +export const NEXT_TASK = "none-until-new-real-task-evidence"; export const REQUIRED_PROOF_COMMANDS = [ "npm test", "npm run benchmark:adaptive-harness", @@ -86,7 +86,7 @@ export function verifyColdStartResponse( throw new Error("cold-start response reported the wrong promotion status"); } const nextTask = String(response.next_task || "").toLowerCase(); - if (!nextTask.includes("adaptive-harness") || !nextTask.includes("receipt") || !nextTask.includes("eval")) { + if (!nextTask.includes("none") || !nextTask.includes("new") || !nextTask.includes("evidence")) { throw new Error("cold-start response reported the wrong next task"); } requireExactArrayValues(response.proof_commands, REQUIRED_PROOF_COMMANDS, "proof commands"); diff --git a/scripts/run-benchmark.mjs b/scripts/run-benchmark.mjs index 58a03f8..aee0e2c 100644 --- a/scripts/run-benchmark.mjs +++ b/scripts/run-benchmark.mjs @@ -5,52 +5,18 @@ import path from "node:path"; import { spawnSync } from "node:child_process"; import { fileURLToPath } from "node:url"; +import { benchmarkUsage, parseBenchmarkArgs } from "./benchmark-options.mjs"; + const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); const variants = ["without-skill", "with-skill"]; -function usage() { - console.error("Usage: node run-benchmark.mjs [--pilot] [--sealed|--sealed-only] [--sealed-file PATH] [--baseline-skill-file PATH] [--samples N] [--seed VALUE] [--model NAME]"); -} - function fail(message, code = 2) { console.error(message); process.exit(code); } -function parseArgs(argv) { - const options = { pilot: false, sealed: false, sealedOnly: false, sealedFile: path.join("evals", "sealed-v2.json"), baselineSkillFile: null, samples: null, seed: "loopspine-v2", model: process.env.LOOPSPINE_MODEL || "gpt-5.5" }; - for (let index = 0; index < argv.length; index += 1) { - const arg = argv[index]; - if (arg === "--help") { - usage(); - process.exit(0); - } - if (arg === "--pilot") options.pilot = true; - else if (arg === "--sealed") options.sealed = true; - else if (arg === "--sealed-only") options.sealedOnly = true; - else if (arg === "--samples" || arg === "--seed" || arg === "--model" || arg === "--sealed-file" || arg === "--baseline-skill-file") { - const value = argv[index + 1]; - if (!value || value.startsWith("--")) fail(`Missing value for ${arg}`); - if (arg === "--samples") { - if (!/^\d+$/.test(value) || Number(value) < 1) fail("--samples must be a positive integer"); - options.samples = Number(value); - } else if (arg === "--seed") options.seed = value; - else if (arg === "--sealed-file") options.sealedFile = value; - else if (arg === "--baseline-skill-file") options.baselineSkillFile = value; - else options.model = value; - index += 1; - } else { - fail(`Unknown argument: ${arg}`); - } - } - if (options.sealed && options.sealedOnly) fail("--sealed and --sealed-only cannot be used together"); - if (options.pilot && options.sealedOnly) fail("--pilot and --sealed-only cannot be used together"); - options.samples ??= options.pilot ? 1 : 3; - return options; -} - function readEvalFile(relativePath, source) { - const filePath = path.join(root, relativePath); + const filePath = repoEvidencePath(relativePath, "evaluation evidence"); let raw; try { raw = fs.readFileSync(filePath); @@ -97,6 +63,13 @@ function hashFile(filePath) { return crypto.createHash("sha256").update(fs.readFileSync(filePath)).digest("hex"); } +function repoEvidencePath(relativePath, label) { + if (!relativePath || path.isAbsolute(relativePath)) fail(`${label} must be a repository-relative path`); + const resolved = path.resolve(root, relativePath); + if (resolved !== root && !resolved.startsWith(`${root}${path.sep}`)) fail(`${label} must stay inside the repository`); + return resolved; +} + function codexVersion() { const result = spawnSync("codex", ["--version"], { encoding: "utf8", timeout: 15000 }); const version = `${result.stdout || ""}${result.stderr || ""}`.trim(); @@ -106,8 +79,17 @@ function codexVersion() { return version; } -const options = parseArgs(process.argv.slice(2)); -const development = readEvalFile(path.join("evals", "evals.json"), "development"); +if (process.argv.slice(2).includes("--help")) { + console.log(benchmarkUsage()); + process.exit(0); +} +let options; +try { + options = parseBenchmarkArgs(process.argv.slice(2)); +} catch (error) { + fail(error.message); +} +const development = readEvalFile(options.developmentFile, "development"); const sealed = (options.sealed || options.sealedOnly) ? readEvalFile(options.sealedFile, "sealed") : null; const pilotIds = new Set(["plan-fuzzy-cross-repo", "tdd-feature", "debug-red-capable-gate", "temporary-roles-without-ceremony", "bounded-loop", "heldout-docs-no-tdd"]); const developmentCases = options.pilot ? development.cases.filter((item) => pilotIds.has(item.id)) : development.cases; @@ -118,6 +100,9 @@ const cases = [ const duplicateIds = cases.map((item) => item.id).filter((id, index, all) => all.indexOf(id) !== index); if (duplicateIds.length) fail(`Malformed evaluation evidence: duplicate case ids: ${duplicateIds.join(", ")}`); if (!cases.length) fail("Malformed evaluation evidence: no selected cases"); +for (const caseId of options.requireStrictCases) { + if (!cases.some((item) => item.id === caseId)) fail(`Required strict case is not selected: ${caseId}`); +} const skillPath = path.join(root, "skills", "loopspine", "SKILL.md"); let skill; @@ -129,15 +114,29 @@ try { let baselineInstructions = "You are a capable senior coding agent. Choose a sensible workflow for the software task. Preserve ordinary safety boundaries and do not execute the scenario."; let baselineSkillPath = null; if (options.baselineSkillFile) { - baselineSkillPath = path.isAbsolute(options.baselineSkillFile) - ? options.baselineSkillFile - : path.resolve(root, options.baselineSkillFile); + baselineSkillPath = repoEvidencePath(options.baselineSkillFile, "baseline skill evidence"); try { baselineInstructions = fs.readFileSync(baselineSkillPath, "utf8"); } catch (error) { fail(`Missing baseline skill evidence: ${baselineSkillPath}: ${error.message}`); } } +let candidateOverlay = null; +if (options.candidateOverlayFile) { + const candidateOverlayPath = repoEvidencePath(options.candidateOverlayFile, "candidate overlay evidence"); + try { + candidateOverlay = { + path: options.candidateOverlayFile, + sha256: hashFile(candidateOverlayPath), + contents: fs.readFileSync(candidateOverlayPath, "utf8") + }; + } catch (error) { + fail(`Missing candidate overlay evidence: ${candidateOverlayPath}: ${error.message}`); + } +} +const candidateInstructions = candidateOverlay + ? `${skill.trim()}\n\n${candidateOverlay.contents.trim()}\n` + : skill; const startedAt = new Date().toISOString(); const stamp = startedAt.replace(/[:.]/g, "-"); @@ -154,7 +153,10 @@ const provenance = { sealed: options.sealed || options.sealedOnly, sealed_only: options.sealedOnly, skill_sha256: hashFile(skillPath), - baseline_skill: baselineSkillPath ? { path: baselineSkillPath, sha256: hashFile(baselineSkillPath) } : null, + baseline_skill: baselineSkillPath ? { path: options.baselineSkillFile, sha256: hashFile(baselineSkillPath) } : null, + candidate_overlay: candidateOverlay ? { path: candidateOverlay.path, sha256: candidateOverlay.sha256 } : null, + require_strict_cases: options.requireStrictCases, + candidate_score_floor: options.candidateScoreFloor, eval_files: [development, ...(sealed ? [sealed] : [])].map(({ relativePath, sha256, source }) => ({ path: relativePath, source, sha256 })), command_args: process.argv.slice(2), reasoning_effort: process.env.LOOPSPINE_REASONING_EFFORT || "provider-default", @@ -179,7 +181,7 @@ for (const item of cases) { const outputDir = path.join(runDir, variant, item.id); const outputPath = path.join(outputDir, `sample-${sample}.txt`); fs.mkdirSync(outputDir, { recursive: true }); - const instructions = variant === "with-skill" ? skill : baselineInstructions; + const instructions = variant === "with-skill" ? candidateInstructions : baselineInstructions; const prompt = `${instructions}\n\n# Scenario\n${item.prompt}\n\nExplain the workflow you would follow. Do not execute commands or edit files.`; const started = Date.now(); console.log(`[${variant}] ${item.id} sample ${sample}/${options.samples}`); @@ -215,6 +217,15 @@ const compared = spawnSync(process.execPath, [ ], { encoding: "utf8" }); if (compared.status === 2) fail(`Malformed comparison evidence: ${compared.stderr || "compare exit 2"}`); fs.writeFileSync(path.join(runDir, "comparison.json"), `${compared.stdout.trim()}\n`); +const candidateSummary = JSON.parse(fs.readFileSync(path.join(runDir, "with-skill-summary.json"), "utf8")); +const strictFailures = candidateSummary.results.filter((item) => options.requireStrictCases.includes(item.id) && !item.strict_pass); +const scoreFloorFailed = options.candidateScoreFloor !== null && candidateSummary.weighted_score < options.candidateScoreFloor; console.log(`Run: ${runDir}`); console.log(compared.stdout.trim()); -process.exit(compared.status === 0 ? 0 : 1); +if (strictFailures.length) { + console.error(`Required strict candidate samples failed: ${strictFailures.map((item) => `${item.id}/sample-${item.sample}`).join(", ")}`); +} +if (scoreFloorFailed) { + console.error(`Candidate score floor failed: ${candidateSummary.weighted_score} < ${options.candidateScoreFloor}`); +} +process.exit(compared.status === 0 && !strictFailures.length && !scoreFloorFailed ? 0 : 1); diff --git a/scripts/run-cold-start-smoke.mjs b/scripts/run-cold-start-smoke.mjs index efc74f3..bf32bdf 100644 --- a/scripts/run-cold-start-smoke.mjs +++ b/scripts/run-cold-start-smoke.mjs @@ -64,7 +64,7 @@ const prompt = [ "This is a cold-start readback: use only current Git state and files inside this repository.", "Do not rely on prior sessions, chat history, saved checkpoints, or external memory.", "Begin by reading AGENTS.md, then follow its restart instructions.", - "Identify the exact current HEAD commit, frozen comparison baseline, promotion decision, first unfinished task, complete candidate proof commands, whether meaningful edits on main are allowed, and repository files that prove the answer.", + "Identify the exact current HEAD commit, frozen comparison baseline, promotion decision, first remaining task including whether none is authorized, complete candidate proof commands, whether meaningful edits on main are allowed, and repository files that prove the answer.", "Do not edit files. Return only the structured response requested by the output schema." ].join(" "); @@ -144,6 +144,6 @@ if (!passed) fail(errors.join("; ") || "one or more assertions failed", receiptP console.log("Cold-start smoke: PASS"); console.log(`Source: ${sourceCommit}`); -console.log("Recovered: rejected promotion, adaptive-harness receipt eval, full proof gate, and protected-main boundary."); +console.log("Recovered: rejected behavior candidate, no authorized next task without new evidence, full proof gate, and protected-main boundary."); console.log("Isolation: ephemeral read-only Codex session; repository unchanged."); console.log(`Receipt: ${receiptPath}`); diff --git a/scripts/score.mjs b/scripts/score.mjs index e8b5f73..0d4257d 100644 --- a/scripts/score.mjs +++ b/scripts/score.mjs @@ -2,6 +2,7 @@ import fs from "node:fs"; import crypto from "node:crypto"; import path from "node:path"; import { fileURLToPath } from "node:url"; +import { parseAdaptiveHarnessReceipt, verifyAdaptiveHarnessExpectation } from "./adaptive-harness-receipt.mjs"; import { hasAffirmativeForbiddenPhrase, matchesRoute } from "./matching.mjs"; const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); @@ -59,6 +60,13 @@ function sha256(filePath) { return crypto.createHash("sha256").update(fs.readFileSync(filePath)).digest("hex"); } +function evidencePath(relativePath, label) { + if (!relativePath || path.isAbsolute(relativePath)) fail(`Malformed ${label}: path must be repository-relative`); + const resolved = path.resolve(root, relativePath); + if (resolved !== root && !resolved.startsWith(`${root}${path.sep}`)) fail(`Malformed ${label}: path escapes repository`); + return resolved; +} + function loadCases(relativePath, source) { const data = readJson(path.join(root, relativePath), "evaluation file"); if (!Array.isArray(data.cases)) fail(`Malformed evaluation file: ${relativePath} has no cases array`); @@ -74,7 +82,8 @@ function scoreOutput(item, output) { const failures = []; let earned = 0; const regexes = item.must_not_regex || []; - const maximum = 2 + item.must_match.length + item.must_not_match.length + regexes.length; + const receiptPoints = item.receipt_contract === "adaptive-harness-v1" ? 5 : 0; + const maximum = 2 + item.must_match.length + item.must_not_match.length + regexes.length + receiptPoints; const normalized = output.toLowerCase(); if (matchesRoute(output, item.routes)) earned += 2; else failures.push(`wrong route: expected ${item.routes.join(" | ")}`); @@ -97,7 +106,20 @@ function scoreOutput(item, output) { if (!regex.test(output)) earned += 1; else failures.push(`forbidden regex: ${pattern}`); } - return { earned, maximum, strict_pass: failures.length === 0, failures }; + let receiptContractPassed = null; + if (item.receipt_contract === "adaptive-harness-v1") { + try { + const receipt = parseAdaptiveHarnessReceipt(output); + const receiptFailures = verifyAdaptiveHarnessExpectation(receipt, item.receipt_expectations); + earned += Math.max(0, receiptPoints - receiptFailures.length); + failures.push(...receiptFailures.map((failure) => `receipt: ${failure}`)); + receiptContractPassed = receiptFailures.length === 0; + } catch (error) { + failures.push(`receipt: ${error.message}`); + receiptContractPassed = false; + } + } + return { earned, maximum, strict_pass: failures.length === 0, failures, receipt_contract_passed: receiptContractPassed }; } function sampleFiles(caseDir, expectedSamples, requireExactCount) { @@ -142,18 +164,32 @@ if (provenance) { if (sha256(skillPath) !== provenance.skill_sha256) fail("Malformed evidence: skill hash does not match provenance.json"); if (!Array.isArray(provenance.eval_files)) fail("Malformed provenance: eval file hashes are missing"); for (const item of provenance.eval_files) { - const evalPath = path.join(root, item.path || ""); + const evalPath = evidencePath(item.path, "evaluation provenance"); if (!item.path || !item.sha256 || !fs.existsSync(evalPath) || sha256(evalPath) !== item.sha256) { fail(`Malformed evidence: eval hash does not match provenance.json for ${item.path || ""}`); } } + if (provenance.candidate_overlay) { + const overlayPath = evidencePath(provenance.candidate_overlay.path, "candidate overlay provenance"); + if (!provenance.candidate_overlay.sha256 || !fs.existsSync(overlayPath) || sha256(overlayPath) !== provenance.candidate_overlay.sha256) { + fail("Malformed evidence: candidate overlay hash does not match provenance.json"); + } + } + if (provenance.baseline_skill) { + const baselinePath = evidencePath(provenance.baseline_skill.path, "baseline skill provenance"); + if (!provenance.baseline_skill.sha256 || !fs.existsSync(baselinePath) || sha256(baselinePath) !== provenance.baseline_skill.sha256) { + fail("Malformed evidence: baseline skill hash does not match provenance.json"); + } + } } +const developmentRelativePath = provenance?.eval_files?.find((item) => item.source === "development")?.path + || path.join("evals", "evals.json"); const sealedRelativePath = provenance?.eval_files?.find((item) => item.source === "sealed")?.path || path.join("evals", "sealed-v1.json"); const cases = [ - ...(mode === "sealed-only" ? [] : loadCases(path.join("evals", "evals.json"), "development")), + ...(mode === "sealed-only" ? [] : loadCases(developmentRelativePath, "development")), ...(mode === "development" ? [] : loadCases(sealedRelativePath, "sealed")) ]; const selectedIds = options.caseArg ? new Set(options.caseArg.split(",").filter(Boolean)) : null; diff --git a/scripts/test-adaptive-harness-contract.mjs b/scripts/test-adaptive-harness-contract.mjs new file mode 100644 index 0000000..f03a219 --- /dev/null +++ b/scripts/test-adaptive-harness-contract.mjs @@ -0,0 +1,154 @@ +import assert from "node:assert/strict"; +import crypto from "node:crypto"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { spawnSync } from "node:child_process"; +import { fileURLToPath } from "node:url"; + +import { + parseAdaptiveHarnessReceipt, + verifyAdaptiveHarnessExpectation +} from "./adaptive-harness-receipt.mjs"; +import { parseBenchmarkArgs } from "./benchmark-options.mjs"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const fixturePath = path.join(root, "evals", "adaptive-harness.json"); +const overlayPath = path.join(root, "evals", "adaptive-harness-candidate.md"); +const skillPath = path.join(root, "skills", "loopspine", "SKILL.md"); +const fixture = JSON.parse(fs.readFileSync(fixturePath, "utf8")); +const overlay = fs.readFileSync(overlayPath, "utf8"); +const packageMetadata = JSON.parse(fs.readFileSync(path.join(root, "package.json"), "utf8")); +const currentSkill = fs.readFileSync(skillPath); +const frozenSkillSha256 = "5f58a9f74a57569f3554e33d24508464ede7bb50589a5eb24e38073fe2d905a6"; + +function sha256(value) { + return crypto.createHash("sha256").update(value).digest("hex"); +} + +const validReceiptText = [ + "Investigate the missing red gate, then continue under parent ownership.", + "RED_GATE: missing-recover", + "CONTRADICTIONS: none", + "CHALLENGER: used \"the attempted stop needs an independent check\"", + "RECOVERY: parent-owned", + "REUSE: proposed-only \"eval: retain the red-capable regression\"" +].join("\n"); +const receipt = parseAdaptiveHarnessReceipt(validReceiptText); +assert.deepEqual(receipt, { + red_gate: { state: "missing-recover", detail: null }, + contradictions: { state: "none", detail: null }, + challenger: { state: "used", detail: "the attempted stop needs an independent check" }, + recovery: { state: "parent-owned", detail: null }, + reuse: { state: "proposed-only", detail: "eval: retain the red-capable regression" } +}); +assert.deepEqual(verifyAdaptiveHarnessExpectation(receipt, { + red_gate: "missing-recover", + contradictions: "none", + recovery: "parent-owned", + reuse: ["none", "proposed-only"] +}), []); +assert.match(verifyAdaptiveHarnessExpectation(receipt, { recovery: "user-owned" })[0], /recovery/); +assert.throws(() => parseAdaptiveHarnessReceipt(validReceiptText.replace("RECOVERY: parent-owned\n", "")), /missing RECOVERY/); +assert.throws(() => parseAdaptiveHarnessReceipt(`${validReceiptText}\nRECOVERY: parent-owned`), /duplicate RECOVERY/); +assert.throws(() => parseAdaptiveHarnessReceipt(validReceiptText.replace("CHALLENGER: used \"the attempted stop needs an independent check\"", "CHALLENGER: used")), /CHALLENGER/); +assert.throws(() => parseAdaptiveHarnessReceipt(validReceiptText.replace("CONTRADICTIONS: none\nCHALLENGER", "CHALLENGER: not-used \"no evidence-backed fix exists yet\"\nCONTRADICTIONS")), /ordered/); + +assert.equal(fixture.suite, "loopspine-adaptive-harness-v1"); +assert.deepEqual(fixture.cases.map(({ id }) => id), ["adaptive-df-03", "adaptive-df-08", "adaptive-fuzzy"]); +for (const item of fixture.cases) { + assert.equal(item.receipt_contract, "adaptive-harness-v1"); + assert.ok(item.receipt_expectations); +} +for (const field of ["RED_GATE", "CONTRADICTIONS", "CHALLENGER", "RECOVERY", "REUSE"]) { + assert.equal(overlay.split("\n").filter((line) => line.startsWith(`${field}:`)).length, 1); +} +for (const safeguard of ["cannot report completion", "parent-owned", "must not edit skills, instructions, or memory"]) { + assert.ok(overlay.includes(safeguard)); +} + +assert.deepEqual(parseBenchmarkArgs([]), { + pilot: false, + sealed: false, + sealedOnly: false, + developmentFile: path.join("evals", "evals.json"), + sealedFile: path.join("evals", "sealed-v2.json"), + baselineSkillFile: null, + candidateOverlayFile: null, + requireStrictCases: [], + candidateScoreFloor: null, + samples: 3, + seed: "loopspine-v2", + model: process.env.LOOPSPINE_MODEL || "gpt-5.5" +}); +const customOptions = parseBenchmarkArgs([ + "--development-file", "evals/adaptive-harness.json", + "--baseline-skill-file", "skills/loopspine/SKILL.md", + "--candidate-overlay-file", "evals/adaptive-harness-candidate.md", + "--require-strict-cases", "adaptive-df-03,adaptive-df-08", + "--candidate-score-floor", "0.9673", + "--samples", "3", + "--seed", "loopspine-adaptive-harness-v1" +]); +assert.deepEqual(customOptions.requireStrictCases, ["adaptive-df-03", "adaptive-df-08"]); +assert.equal(customOptions.developmentFile, "evals/adaptive-harness.json"); +assert.equal(customOptions.candidateOverlayFile, "evals/adaptive-harness-candidate.md"); +assert.equal(customOptions.candidateScoreFloor, 0.9673); +assert.throws(() => parseBenchmarkArgs(["--require-strict-cases", ""]), /Missing value/); +assert.throws(() => parseBenchmarkArgs(["--require-strict-cases", "--samples", "3"]), /Missing value/); +assert.throws(() => parseBenchmarkArgs(["--candidate-overlay-file", ""]), /Missing value/); +assert.throws(() => parseBenchmarkArgs(["--candidate-score-floor", "1.1"]), /score floor/); +assert.throws(() => parseBenchmarkArgs(["--sealed", "--sealed-only"]), /cannot be used together/); + +assert.equal(sha256(currentSkill), frozenSkillSha256); +assert.match(packageMetadata.scripts["benchmark:adaptive-harness"], /--development-file evals\/adaptive-harness\.json/); +assert.match(packageMetadata.scripts["benchmark:adaptive-harness"], /--candidate-overlay-file evals\/adaptive-harness-candidate\.md/); +assert.match(packageMetadata.scripts["benchmark:adaptive-harness"], /--baseline-skill-file skills\/loopspine\/SKILL\.md/); +assert.match(packageMetadata.scripts["benchmark:adaptive-harness"], /--require-strict-cases adaptive-df-03,adaptive-df-08/); +assert.match(packageMetadata.scripts["benchmark:adaptive-harness"], /--samples 3/); +assert.match(packageMetadata.scripts["benchmark:sealed"], /--candidate-score-floor 0\.9673/); + +const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "loopspine-adaptive-score-")); +try { + const caseId = "adaptive-df-03"; + const outputDir = path.join(tempRoot, "with-skill", caseId); + fs.mkdirSync(outputDir, { recursive: true }); + fs.writeFileSync(path.join(outputDir, "sample-1.txt"), [ + "ROUTE: investigate", + "A red-capable failing command is missing, so recover and continue.", + validReceiptText + ].join("\n")); + const provenance = { + samples: 1, + sealed: false, + sealed_only: false, + skill_sha256: sha256(currentSkill), + baseline_skill: { path: "skills/loopspine/SKILL.md", sha256: sha256(currentSkill) }, + candidate_overlay: { path: "evals/adaptive-harness-candidate.md", sha256: sha256(overlay) }, + eval_files: [{ path: "evals/adaptive-harness.json", source: "development", sha256: sha256(fs.readFileSync(fixturePath)) }], + cases: [{ id: caseId, split: "development", source: "development" }] + }; + fs.writeFileSync(path.join(tempRoot, "provenance.json"), `${JSON.stringify(provenance, null, 2)}\n`); + const score = spawnSync(process.execPath, [path.join(root, "scripts", "score.mjs"), tempRoot, "with-skill", caseId, "--samples", "1"], { cwd: root, encoding: "utf8" }); + assert.equal(score.status, 0, score.stderr); + const summary = JSON.parse(score.stdout); + assert.equal(summary.results[0].strict_pass, true); + assert.equal(summary.results[0].receipt_contract_passed, true); + + provenance.candidate_overlay.sha256 = "0".repeat(64); + fs.writeFileSync(path.join(tempRoot, "provenance.json"), `${JSON.stringify(provenance, null, 2)}\n`); + const badOverlay = spawnSync(process.execPath, [path.join(root, "scripts", "score.mjs"), tempRoot, "with-skill", caseId, "--samples", "1"], { cwd: root, encoding: "utf8" }); + assert.notEqual(badOverlay.status, 0); + assert.match(badOverlay.stderr, /candidate overlay hash/); + + provenance.candidate_overlay.sha256 = sha256(overlay); + provenance.baseline_skill.sha256 = "0".repeat(64); + fs.writeFileSync(path.join(tempRoot, "provenance.json"), `${JSON.stringify(provenance, null, 2)}\n`); + const badBaseline = spawnSync(process.execPath, [path.join(root, "scripts", "score.mjs"), tempRoot, "with-skill", caseId, "--samples", "1"], { cwd: root, encoding: "utf8" }); + assert.notEqual(badBaseline.status, 0); + assert.match(badBaseline.stderr, /baseline skill hash/); +} finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); +} + +console.log("Adaptive harness contract tests passed: receipt, fixtures, options, provenance, and frozen baseline."); diff --git a/scripts/test-cold-start-contract.mjs b/scripts/test-cold-start-contract.mjs index 3399337..5aabfbe 100644 --- a/scripts/test-cold-start-contract.mjs +++ b/scripts/test-cold-start-contract.mjs @@ -18,10 +18,10 @@ const packageMetadata = JSON.parse(fs.readFileSync(path.join(root, "package.json assert.match(agents, /docs\/plans\/adaptive-harness-candidate\.md/); assert.match(agents, /npm run smoke:cold-start/); -assert.match(plan, /^Status: Active$/m); +assert.match(plan, /^Status: Evidence complete; behavior candidate rejected$/m); assert.ok(plan.includes(FROZEN_BASELINE_COMMIT)); assert.ok(plan.includes(CANDIDATE_SOURCE_BASE_COMMIT)); -assert.match(plan, /Add an eval-only adaptive-harness receipt candidate/); +assert.match(plan, /eval-only adaptive-harness receipt candidate is implemented/); for (const command of REQUIRED_PROOF_COMMANDS) assert.ok(plan.includes(command)); assert.equal(packageMetadata.scripts["smoke:cold-start"], "node scripts/run-cold-start-smoke.mjs"); @@ -29,7 +29,7 @@ const valid = { source_commit: CANDIDATE_SOURCE_BASE_COMMIT, frozen_baseline_commit: FROZEN_BASELINE_COMMIT, promotion_status: "rejected", - next_task: "adaptive-harness-receipt-eval", + next_task: "none-until-new-real-task-evidence", proof_commands: [ "npm test", "npm run benchmark:adaptive-harness", @@ -58,7 +58,7 @@ assert.equal( verifyColdStartResponse({ ...valid, promotion_status: "Global/default promotion rejected; explicit local use remains.", - next_task: "Add the eval-only adaptive-harness receipt candidate." + next_task: "None is authorized until new real-task evidence names a narrower failure." }).next_task_matches, true ); diff --git a/scripts/validate.mjs b/scripts/validate.mjs index b420fcc..aacae7e 100644 --- a/scripts/validate.mjs +++ b/scripts/validate.mjs @@ -18,6 +18,7 @@ const codex = readJson(".codex-plugin/plugin.json"); const claude = readJson(".claude-plugin/plugin.json"); const evals = readJson("evals/evals.json"); const sealed = readJson("evals/sealed-v2.json"); +const adaptive = readJson("evals/adaptive-harness.json"); const demoReceipt = readJson("demo/latest/receipt.json"); const dogfood = readJson("dogfood/register.json"); const skillPath = path.join(root, "skills", "loopspine", "SKILL.md"); @@ -34,6 +35,7 @@ if (!skill.includes("RESULT: success | clean-no-op | no-progress")) errors.push( const cases = evals?.cases || []; const sealedCases = sealed?.cases || []; +const adaptiveCases = adaptive?.cases || []; const ids = new Set(); for (const item of cases) { if (!item.id || ids.has(item.id)) errors.push(`invalid or duplicate eval id: ${item.id || ""}`); @@ -57,6 +59,23 @@ for (const item of sealedCases) { if (sealed?.suite !== "loopspine-sealed-v2") errors.push("sealed eval suite id is invalid"); if (sealed?.authored_by !== "independent-terra") errors.push("sealed eval author receipt is invalid"); if (sealedCases.length < 6) errors.push("sealed eval pack must contain at least six held-out cases"); +if (adaptive?.suite !== "loopspine-adaptive-harness-v1") errors.push("adaptive harness suite id is invalid"); +const adaptiveIds = ["adaptive-df-03", "adaptive-df-08", "adaptive-fuzzy"]; +if (JSON.stringify(adaptiveCases.map((item) => item.id)) !== JSON.stringify(adaptiveIds)) { + errors.push("adaptive harness must contain exactly the DF-03, DF-08, and fuzzy cases in order"); +} +for (const item of adaptiveCases) { + if (ids.has(item.id)) errors.push(`invalid or duplicate adaptive eval id: ${item.id || ""}`); + ids.add(item.id); + if (item.split !== "development" || item.receipt_contract !== "adaptive-harness-v1") errors.push(`${item.id}: invalid adaptive harness contract`); + if (!item.prompt || !item.routes?.length || !item.must_match?.length || !item.must_not_match?.length || !item.receipt_expectations) { + errors.push(`${item.id}: incomplete adaptive harness eval contract`); + } + const expected = item.receipt_expectations || {}; + for (const field of ["red_gate", "contradictions", "recovery", "reuse"]) { + if (expected[field] === undefined) errors.push(`${item.id}: missing adaptive receipt expectation ${field}`); + } +} if (demoReceipt?.success !== true || demoReceipt?.sawyer_interventions !== 0 || demoReceipt?.incorrect_stop !== false) { errors.push("demo receipt must prove success with zero interventions and no incorrect stop"); } @@ -79,4 +98,4 @@ if (errors.length) { process.exit(1); } -console.log(`LoopSpine valid: 2 plugin manifests, 1 skill, ${cases.length} development cases, ${sealedCases.length} sealed held-out cases.`); +console.log(`LoopSpine valid: 2 plugin manifests, 1 skill, ${cases.length} development cases, ${sealedCases.length} sealed held-out cases, ${adaptiveCases.length} adaptive harness cases.`);