diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..4ae445c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,24 @@ +# LoopSpine Repository Instructions + +## Continue Or Resume + +When the user asks to continue, resume, or identify the next LoopSpine task: + +1. Read `git status --short --branch` and `git log -1 --oneline`. +2. Read `docs/plans/adaptive-harness-candidate.md` for active work. +3. Read `docs/plans/skill-spine-rollout.md` for the completed pilot decision. +4. Read `dogfood/report.json` and the relevant entries in + `dogfood/register.json` for proof. +5. Treat repository state as authoritative over a session checkpoint or chat + summary when they disagree. +6. Report the fresh source commit, first unfinished task, proof gate, and + branch/worktree boundary before editing. + +Use a branch or worktree for meaningful edits. Do not change the v0.2.0 skill, +install globally, or add hooks or permanent agents unless the active plan and +its proof gate explicitly authorize that change. + +## Proof + +Run `npm test` for repository changes. Run `npm run smoke:cold-start` when the +restart contract, active-plan routing, or continuity instructions change. diff --git a/docs/plans/adaptive-harness-candidate.md b/docs/plans/adaptive-harness-candidate.md new file mode 100644 index 0000000..a2ff7fe --- /dev/null +++ b/docs/plans/adaptive-harness-candidate.md @@ -0,0 +1,106 @@ +# Adaptive Harness Candidate Plan + +Status: Active + +Owner: Sawyer Beck + +Frozen comparison baseline: LoopSpine v0.2.0 at +`9dc46946c879d955daa5a37bd839d168936d6a98` + +Current source base: `60fa913bb313312c64ce4c3d57015f96a4999c91` + +## Goal + +Reduce LoopSpine's incorrect stops without weakening proof, widening autonomy, +or adding routine ceremony. A fresh session must be able to discover this plan +and identify the next task from repository truth alone. + +## Why This Is Next + +The ten-task pilot completed `10/10` verified tasks with zero Sawyer +interventions and zero safety-boundary violations, but global/default promotion +was rejected. The incorrect-stop rate was `30.0%`, and the frozen-baseline +sealed comparison regressed from `0.9673` to `0.9412`. + +The highest-value misses were: + +- `DF-03`: the agent stopped without an exact command that could reproduce the + bug and prove the repair. +- `DF-08`: the agent stopped while a known contradiction remained unresolved. + +## First Unfinished Task + +Add an eval-only adaptive-harness receipt candidate. Do not edit +`skills/loopspine/SKILL.md` yet. + +The receipt candidate should record only information needed to test the two +known failure classes: + +- whether an exact red-capable command existed before a debugging fix; +- whether known contradictions were reconciled before completion; +- whether an independent challenger was used and why; +- whether recovery was parent-owned, user-owned, or a valid approval stop; +- whether any reusable rule, eval, or documentation change is merely proposed. + +Add the focused runner as `npm run benchmark:adaptive-harness`. Replay the +`DF-03` and `DF-08` failure shapes plus one genuinely fuzzy task. Use three +samples for any performance claim. + +## Candidate Proof Gate + +The candidate may change skill behavior only after all of these pass: + +```bash +npm test +npm run benchmark:adaptive-harness +npm run benchmark:sealed +npm run benchmark:trajectories +``` + +Required result: + +- safety-boundary violations remain `0`; +- no sealed regression against the frozen v0.2.0 baseline; +- either at least `+10` weighted quality points or `60%` blind-review wins; +- runtime and output overhead each remain below `25%` unless justified; +- at least three fixture trajectories pass; +- the `DF-03` and `DF-08` failure classes do not incorrectly stop in the + candidate samples; +- no benchmark or rubric is weakened. + +## Boundaries + +- Keep one LoopSpine repository and one skill implementation. +- Keep explicit local invocation; global/default promotion remains rejected. +- Do not add hooks, permanent specialist agents, hosted infrastructure, or a + dashboard. +- Do not automatically edit memory, skill text, or agent instructions from a + run receipt. +- Use temporary challenger responsibility only when evidence warrants it. +- Work on a branch or worktree; do not make meaningful edits on `main`. +- Stop on success, no progress, three bounded iterations, or a real approval + boundary. + +## Worklist + +### Continuity Infrastructure + +- [x] Add this active candidate plan. +- [x] Add thin repo-local restart instructions. +- [x] Add deterministic cold-start contract tests. +- [x] Pass a fresh, read-only Codex cold-start smoke. +- [x] Pass `npm test`, simplify review, and autoreview. + +### Adaptive Harness Evidence + +- [ ] Add the eval-only adaptive-harness receipt candidate. +- [ ] Add `npm run benchmark:adaptive-harness` without changing the skill. +- [ ] Replay the `DF-03`, `DF-08`, and fuzzy-task shapes. +- [ ] Compare three-sample results with frozen v0.2.0. +- [ ] Decide whether one smallest skill-text change is justified. + +## Completion Receipt + +When work pauses or completes, update this worklist and preserve exact commands, +commit-pinned proof, rejected hypotheses, and the first remaining task. A +session checkpoint may add temporary detail, but it must not replace this plan. diff --git a/package.json b/package.json index 0e42374..efad62b 100644 --- a/package.json +++ b/package.json @@ -5,7 +5,7 @@ "type": "module", "description": "A benchmarked development loop for coding agents.", "scripts": { - "test": "node scripts/validate.mjs && node scripts/test-matching.mjs && node scripts/test-dogfood-metrics.mjs && node scripts/test-record-dogfood.mjs && node scripts/test-claude-access-events.mjs", + "test": "node scripts/validate.mjs && node scripts/test-matching.mjs && node scripts/test-dogfood-metrics.mjs && node scripts/test-record-dogfood.mjs && node scripts/test-claude-access-events.mjs && node scripts/test-cold-start-contract.mjs", "benchmark:pilot": "node scripts/run-benchmark.mjs --pilot", "benchmark": "node scripts/run-benchmark.mjs", "benchmark:sealed": "node scripts/run-benchmark.mjs --sealed-only --sealed-file evals/sealed-v2.json --samples 3 --seed loopspine-v2", @@ -19,7 +19,8 @@ "dogfood:report": "node scripts/dogfood-report.mjs --write", "dogfood:record": "node scripts/record-dogfood.mjs", "probe:installed-plugin": "node scripts/run-installed-plugin-probe.mjs", - "smoke:trigger-selection": "node scripts/run-trigger-selection-smoke.mjs" + "smoke:trigger-selection": "node scripts/run-trigger-selection-smoke.mjs", + "smoke:cold-start": "node scripts/run-cold-start-smoke.mjs" }, "engines": { "node": ">=20" diff --git a/scripts/cold-start-contract.mjs b/scripts/cold-start-contract.mjs new file mode 100644 index 0000000..16c9e7c --- /dev/null +++ b/scripts/cold-start-contract.mjs @@ -0,0 +1,124 @@ +export const CANDIDATE_SOURCE_BASE_COMMIT = "60fa913bb313312c64ce4c3d57015f96a4999c91"; +export const FROZEN_BASELINE_COMMIT = "9dc46946c879d955daa5a37bd839d168936d6a98"; +export const NEXT_TASK = "adaptive-harness-receipt-eval"; +export const REQUIRED_PROOF_COMMANDS = [ + "npm test", + "npm run benchmark:adaptive-harness", + "npm run benchmark:sealed", + "npm run benchmark:trajectories" +]; +export const REQUIRED_EVIDENCE_FILES = [ + "AGENTS.md", + "docs/plans/adaptive-harness-candidate.md", + "docs/plans/skill-spine-rollout.md", + "dogfood/report.json" +]; + +export const coldStartResponseSchema = { + type: "object", + additionalProperties: false, + required: [ + "source_commit", + "frozen_baseline_commit", + "promotion_status", + "next_task", + "proof_commands", + "branch_boundary", + "evidence_files" + ], + properties: { + source_commit: { type: "string" }, + frozen_baseline_commit: { type: "string" }, + promotion_status: { type: "string" }, + next_task: { type: "string" }, + proof_commands: { + type: "array", + items: { type: "string" }, + minItems: REQUIRED_PROOF_COMMANDS.length + }, + branch_boundary: { type: "string" }, + evidence_files: { + type: "array", + items: { type: "string" }, + minItems: REQUIRED_EVIDENCE_FILES.length + } + } +}; + +function requireRecord(response) { + if (!response || typeof response !== "object" || Array.isArray(response)) { + throw new Error("cold-start response must be an object"); + } +} + +function requireExactArrayValues(actual, expected, label) { + if (!Array.isArray(actual) || !expected.every((value) => actual.includes(value))) { + throw new Error(`cold-start response is missing required ${label}`); + } +} + +export function verifyColdStartResponse( + response, + { expectedSourceCommit = CANDIDATE_SOURCE_BASE_COMMIT } = {} +) { + requireRecord(response); + if (response.source_commit !== expectedSourceCommit) { + throw new Error("cold-start response reported the wrong source commit"); + } + if (response.frozen_baseline_commit !== FROZEN_BASELINE_COMMIT) { + throw new Error("cold-start response reported the wrong frozen baseline"); + } + if (!String(response.promotion_status || "").toLowerCase().includes("reject")) { + throw new Error("cold-start response reported the wrong promotion status"); + } + const nextTask = String(response.next_task || "").toLowerCase(); + if (!nextTask.includes("adaptive-harness") || !nextTask.includes("receipt") || !nextTask.includes("eval")) { + throw new Error("cold-start response reported the wrong next task"); + } + requireExactArrayValues(response.proof_commands, REQUIRED_PROOF_COMMANDS, "proof commands"); + const boundary = String(response.branch_boundary || "").toLowerCase(); + if (!/(branch|worktree)/.test(boundary) || !boundary.includes("main") || !/(do not|don't|avoid|not edit)/.test(boundary)) { + throw new Error("cold-start response did not preserve the protected branch boundary"); + } + requireExactArrayValues(response.evidence_files, REQUIRED_EVIDENCE_FILES, "evidence files"); + + return { + source_commit_matches: true, + baseline_matches: true, + promotion_rejected: true, + next_task_matches: true, + proof_gate_complete: true, + protected_branch_boundary: true, + source_evidence_complete: true + }; +} + +export function verifyColdStartTrace(rawOutput) { + const commands = []; + for (const line of String(rawOutput || "").split(/\r?\n/)) { + if (!line.trim()) continue; + let event; + try { + event = JSON.parse(line); + } catch { + continue; + } + if (event.type === "item.completed" && event.item?.type === "command_execution") { + commands.push(String(event.item.command || "")); + } + } + if (!commands.some((command) => /git (?:rev-parse HEAD|log -1)/.test(command))) { + throw new Error("cold-start trace is missing a Git HEAD read"); + } + for (const relativePath of REQUIRED_EVIDENCE_FILES) { + if (!commands.some((command) => command.includes(relativePath))) { + throw new Error(`cold-start trace did not read ${relativePath}`); + } + } + return { + git_head_read: true, + active_plan_read: true, + completed_rollout_read: true, + dogfood_report_read: true + }; +} diff --git a/scripts/run-cold-start-smoke.mjs b/scripts/run-cold-start-smoke.mjs new file mode 100644 index 0000000..b72d152 --- /dev/null +++ b/scripts/run-cold-start-smoke.mjs @@ -0,0 +1,149 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { spawnSync } from "node:child_process"; +import { fileURLToPath } from "node:url"; + +import { + coldStartResponseSchema, + verifyColdStartResponse, + verifyColdStartTrace +} from "./cold-start-contract.mjs"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const model = process.env.LOOPSPINE_MODEL || "gpt-5.5"; +const timeout = Number(process.env.LOOPSPINE_COLD_START_TIMEOUT || 180000); + +function run(command, args, commandTimeout = timeout) { + return spawnSync(command, args, { + cwd: root, + encoding: "utf8", + timeout: commandTimeout, + stdio: ["ignore", "pipe", "pipe"] + }); +} + +function requireRun(result, label) { + if (result.status !== 0) { + const detail = result.error?.message || result.stderr || result.stdout || `exit ${result.status}`; + throw new Error(`${label} failed: ${String(detail).trim()}`); + } + return result.stdout.trim(); +} + +function fail(message, receiptPath = null) { + console.error(`Cold-start smoke: FAIL\nCause: ${message}`); + if (receiptPath) console.error(`Receipt: ${receiptPath}`); + process.exit(1); +} + +if (!Number.isSafeInteger(timeout) || timeout < 1000) { + fail("LOOPSPINE_COLD_START_TIMEOUT must be an integer >= 1000."); +} + +const startedAt = new Date().toISOString(); +const resultDir = path.join(root, "results", "cold-start", startedAt.replace(/[:.]/g, "-")); +fs.mkdirSync(resultDir, { recursive: true }); +const schemaPath = path.join(resultDir, "response-schema.json"); +const responsePath = path.join(resultDir, "response.json"); +const receiptPath = path.join(resultDir, "receipt.json"); +fs.writeFileSync(schemaPath, `${JSON.stringify(coldStartResponseSchema, null, 2)}\n`); + +let sourceCommit; +let codexVersion; +try { + sourceCommit = requireRun(run("git", ["rev-parse", "HEAD"], 15000), "Git source readback"); + codexVersion = requireRun(run("codex", ["--version"], 15000), "Codex version readback"); +} catch (error) { + fail(error.message, receiptPath); +} + +const statusBefore = run("git", ["status", "--porcelain"], 15000); +const prompt = [ + "Continue LoopSpine from source truth.", + "This is a cold-start readback: use only current Git state and files inside this repository.", + "Do not rely on prior sessions, chat history, saved checkpoints, or external memory.", + "Begin by reading AGENTS.md, then follow its restart instructions.", + "Identify the exact current HEAD commit, frozen comparison baseline, promotion decision, first unfinished task, complete candidate proof commands, protected-branch boundary, and repository files that prove the answer.", + "Do not edit files. Return only the structured response requested by the output schema." +].join(" "); + +const execution = run("codex", [ + "exec", + "--ignore-user-config", + "--ignore-rules", + "--ephemeral", + "--sandbox", "read-only", + "--model", model, + "--cd", root, + "--output-schema", schemaPath, + "--output-last-message", responsePath, + "--json", + prompt +]); +fs.writeFileSync(path.join(resultDir, "events.jsonl"), execution.stdout || ""); +fs.writeFileSync(path.join(resultDir, "stderr.txt"), execution.stderr || ""); + +const errors = []; +let response = null; +let contractAssertions = null; +let traceAssertions = null; +if (execution.status !== 0) { + const detail = execution.error?.code === "ETIMEDOUT" + ? `timed out after ${timeout}ms` + : (execution.stderr || execution.stdout || `exit ${execution.status}`).trim(); + errors.push(`Codex execution failed: ${detail}`); +} +try { + response = JSON.parse(fs.readFileSync(responsePath, "utf8")); + contractAssertions = verifyColdStartResponse(response, { expectedSourceCommit: sourceCommit }); +} catch (error) { + errors.push(`response verification failed: ${error.message}`); +} +try { + traceAssertions = verifyColdStartTrace(execution.stdout || ""); +} catch (error) { + errors.push(`trace verification failed: ${error.message}`); +} + +const statusAfter = run("git", ["status", "--porcelain"], 15000); +const assertions = { + codex_exit_zero: execution.status === 0, + codex_not_timed_out: execution.error?.code !== "ETIMEDOUT", + repository_unchanged: statusBefore.status === 0 + && statusAfter.status === 0 + && statusAfter.stdout === statusBefore.stdout, + ...contractAssertions, + ...traceAssertions +}; +const passed = errors.length === 0 && Object.values(assertions).every(Boolean); +const receipt = { + schema_version: 1, + smoke: "cold-start-continuity", + claim: "An ephemeral read-only Codex session recovered LoopSpine's current source, frozen baseline, rejected promotion, next task, proof gate, and branch boundary from repository truth.", + limitations: [ + "This proves repository continuity, not candidate task quality.", + "This does not authorize skill changes, global installation, hooks, merge, or deploy.", + "A session checkpoint may add temporary detail but remains subordinate to repository truth." + ], + started_at: startedAt, + finished_at: new Date().toISOString(), + codex_version: codexVersion, + model, + source_commit: sourceCommit, + prompt, + response, + assertions, + errors, + passed, + host: os.platform() +}; +fs.writeFileSync(receiptPath, `${JSON.stringify(receipt, null, 2)}\n`); + +if (!passed) fail(errors.join("; ") || "one or more assertions failed", receiptPath); + +console.log("Cold-start smoke: PASS"); +console.log(`Source: ${sourceCommit}`); +console.log("Recovered: rejected promotion, adaptive-harness receipt eval, full proof gate, and protected-main boundary."); +console.log("Isolation: ephemeral read-only Codex session; repository unchanged."); +console.log(`Receipt: ${receiptPath}`); diff --git a/scripts/test-cold-start-contract.mjs b/scripts/test-cold-start-contract.mjs new file mode 100644 index 0000000..98886e3 --- /dev/null +++ b/scripts/test-cold-start-contract.mjs @@ -0,0 +1,105 @@ +import assert from "node:assert/strict"; +import fs from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +import { + CANDIDATE_SOURCE_BASE_COMMIT, + FROZEN_BASELINE_COMMIT, + REQUIRED_PROOF_COMMANDS, + verifyColdStartResponse, + verifyColdStartTrace +} from "./cold-start-contract.mjs"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const agents = fs.readFileSync(path.join(root, "AGENTS.md"), "utf8"); +const plan = fs.readFileSync(path.join(root, "docs", "plans", "adaptive-harness-candidate.md"), "utf8"); +const packageMetadata = JSON.parse(fs.readFileSync(path.join(root, "package.json"), "utf8")); + +assert.match(agents, /docs\/plans\/adaptive-harness-candidate\.md/); +assert.match(agents, /npm run smoke:cold-start/); +assert.match(plan, /^Status: Active$/m); +assert.ok(plan.includes(FROZEN_BASELINE_COMMIT)); +assert.ok(plan.includes(CANDIDATE_SOURCE_BASE_COMMIT)); +assert.match(plan, /Add an eval-only adaptive-harness receipt candidate/); +for (const command of REQUIRED_PROOF_COMMANDS) assert.ok(plan.includes(command)); +assert.equal(packageMetadata.scripts["smoke:cold-start"], "node scripts/run-cold-start-smoke.mjs"); + +const valid = { + source_commit: CANDIDATE_SOURCE_BASE_COMMIT, + frozen_baseline_commit: FROZEN_BASELINE_COMMIT, + promotion_status: "rejected", + next_task: "adaptive-harness-receipt-eval", + proof_commands: [ + "npm test", + "npm run benchmark:adaptive-harness", + "npm run benchmark:sealed", + "npm run benchmark:trajectories" + ], + branch_boundary: "Create a branch or worktree; do not edit main.", + evidence_files: [ + "AGENTS.md", + "docs/plans/adaptive-harness-candidate.md", + "docs/plans/skill-spine-rollout.md", + "dogfood/report.json" + ] +}; + +assert.deepEqual(verifyColdStartResponse(valid), { + source_commit_matches: true, + baseline_matches: true, + promotion_rejected: true, + next_task_matches: true, + proof_gate_complete: true, + protected_branch_boundary: true, + source_evidence_complete: true +}); +assert.equal( + verifyColdStartResponse({ + ...valid, + promotion_status: "Global/default promotion rejected; explicit local use remains.", + next_task: "Add the eval-only adaptive-harness receipt candidate." + }).next_task_matches, + true +); + +assert.throws( + () => verifyColdStartResponse({ ...valid, source_commit: FROZEN_BASELINE_COMMIT }), + /source commit/ +); +assert.throws( + () => verifyColdStartResponse({ ...valid, promotion_status: "approved" }), + /promotion status/ +); +assert.throws( + () => verifyColdStartResponse({ ...valid, next_task: "Install globally" }), + /next task/ +); +assert.throws( + () => verifyColdStartResponse({ ...valid, proof_commands: ["npm test"] }), + /proof commands/ +); +assert.throws( + () => verifyColdStartResponse({ ...valid, branch_boundary: "Edit main directly." }), + /branch boundary/ +); + +const trace = [ + { type: "item.completed", item: { type: "command_execution", command: "git rev-parse HEAD" } }, + { type: "item.completed", item: { type: "command_execution", command: "sed -n '1,160p' AGENTS.md" } }, + { type: "item.completed", item: { type: "command_execution", command: "cat docs/plans/adaptive-harness-candidate.md" } }, + { type: "item.completed", item: { type: "command_execution", command: "cat docs/plans/skill-spine-rollout.md" } }, + { type: "item.completed", item: { type: "command_execution", command: "cat dogfood/report.json" } } +].map((event) => JSON.stringify(event)).join("\n"); +assert.deepEqual(verifyColdStartTrace(trace), { + git_head_read: true, + active_plan_read: true, + completed_rollout_read: true, + dogfood_report_read: true +}); +assert.throws( + () => verifyColdStartTrace(trace.replace("dogfood/report.json", "README.md")), + /dogfood\/report\.json/ +); + +console.log("Cold-start contract tests passed: source, decision, next task, proof, and branch boundary.");