Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# LoopSpine Repository Instructions

## Continue Or Resume

When the user asks to continue, resume, or identify the next LoopSpine task:

1. Read `git status --short --branch` and `git log -1 --oneline`.
2. Read `docs/plans/adaptive-harness-candidate.md` for active work.
3. Read `docs/plans/skill-spine-rollout.md` for the completed pilot decision.
4. Read `dogfood/report.json` and the relevant entries in
`dogfood/register.json` for proof.
5. Treat repository state as authoritative over a session checkpoint or chat
summary when they disagree.
6. Report the fresh source commit, first unfinished task, proof gate, and
branch/worktree boundary before editing.

Use a branch or worktree for meaningful edits. Do not change the v0.2.0 skill,
install globally, or add hooks or permanent agents unless the active plan and
its proof gate explicitly authorize that change.

## Proof

Run `npm test` for repository changes. Run `npm run smoke:cold-start` when the
restart contract, active-plan routing, or continuity instructions change.
106 changes: 106 additions & 0 deletions docs/plans/adaptive-harness-candidate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Adaptive Harness Candidate Plan

Status: Active

Owner: Sawyer Beck

Frozen comparison baseline: LoopSpine v0.2.0 at
`9dc46946c879d955daa5a37bd839d168936d6a98`

Current source base: `60fa913bb313312c64ce4c3d57015f96a4999c91`

## Goal

Reduce LoopSpine's incorrect stops without weakening proof, widening autonomy,
or adding routine ceremony. A fresh session must be able to discover this plan
and identify the next task from repository truth alone.

## Why This Is Next

The ten-task pilot completed `10/10` verified tasks with zero Sawyer
interventions and zero safety-boundary violations, but global/default promotion
was rejected. The incorrect-stop rate was `30.0%`, and the frozen-baseline
sealed comparison regressed from `0.9673` to `0.9412`.

The highest-value misses were:

- `DF-03`: the agent stopped without an exact command that could reproduce the
bug and prove the repair.
- `DF-08`: the agent stopped while a known contradiction remained unresolved.

## First Unfinished Task

Add an eval-only adaptive-harness receipt candidate. Do not edit
`skills/loopspine/SKILL.md` yet.

The receipt candidate should record only information needed to test the two
known failure classes:

- whether an exact red-capable command existed before a debugging fix;
- whether known contradictions were reconciled before completion;
- whether an independent challenger was used and why;
- whether recovery was parent-owned, user-owned, or a valid approval stop;
- whether any reusable rule, eval, or documentation change is merely proposed.

Add the focused runner as `npm run benchmark:adaptive-harness`. Replay the
`DF-03` and `DF-08` failure shapes plus one genuinely fuzzy task. Use three
samples for any performance claim.

## Candidate Proof Gate

The candidate may change skill behavior only after all of these pass:

```bash
npm test
npm run benchmark:adaptive-harness
npm run benchmark:sealed
npm run benchmark:trajectories
```

Required result:

- safety-boundary violations remain `0`;
- no sealed regression against the frozen v0.2.0 baseline;
- either at least `+10` weighted quality points or `60%` blind-review wins;
- runtime and output overhead each remain below `25%` unless justified;
- at least three fixture trajectories pass;
- the `DF-03` and `DF-08` failure classes do not incorrectly stop in the
candidate samples;
- no benchmark or rubric is weakened.

## Boundaries

- Keep one LoopSpine repository and one skill implementation.
- Keep explicit local invocation; global/default promotion remains rejected.
- Do not add hooks, permanent specialist agents, hosted infrastructure, or a
dashboard.
- Do not automatically edit memory, skill text, or agent instructions from a
run receipt.
- Use temporary challenger responsibility only when evidence warrants it.
- Work on a branch or worktree; do not make meaningful edits on `main`.
- Stop on success, no progress, three bounded iterations, or a real approval
boundary.

## Worklist

### Continuity Infrastructure

- [x] Add this active candidate plan.
- [x] Add thin repo-local restart instructions.
- [x] Add deterministic cold-start contract tests.
- [x] Pass a fresh, read-only Codex cold-start smoke.
- [x] Pass `npm test`, simplify review, and autoreview.

### Adaptive Harness Evidence

- [ ] Add the eval-only adaptive-harness receipt candidate.
- [ ] Add `npm run benchmark:adaptive-harness` without changing the skill.
- [ ] Replay the `DF-03`, `DF-08`, and fuzzy-task shapes.
- [ ] Compare three-sample results with frozen v0.2.0.
- [ ] Decide whether one smallest skill-text change is justified.

## Completion Receipt

When work pauses or completes, update this worklist and preserve exact commands,
commit-pinned proof, rejected hypotheses, and the first remaining task. A
session checkpoint may add temporary detail, but it must not replace this plan.
5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"type": "module",
"description": "A benchmarked development loop for coding agents.",
"scripts": {
"test": "node scripts/validate.mjs && node scripts/test-matching.mjs && node scripts/test-dogfood-metrics.mjs && node scripts/test-record-dogfood.mjs && node scripts/test-claude-access-events.mjs",
"test": "node scripts/validate.mjs && node scripts/test-matching.mjs && node scripts/test-dogfood-metrics.mjs && node scripts/test-record-dogfood.mjs && node scripts/test-claude-access-events.mjs && node scripts/test-cold-start-contract.mjs",
"benchmark:pilot": "node scripts/run-benchmark.mjs --pilot",
"benchmark": "node scripts/run-benchmark.mjs",
"benchmark:sealed": "node scripts/run-benchmark.mjs --sealed-only --sealed-file evals/sealed-v2.json --samples 3 --seed loopspine-v2",
Expand All @@ -19,7 +19,8 @@
"dogfood:report": "node scripts/dogfood-report.mjs --write",
"dogfood:record": "node scripts/record-dogfood.mjs",
"probe:installed-plugin": "node scripts/run-installed-plugin-probe.mjs",
"smoke:trigger-selection": "node scripts/run-trigger-selection-smoke.mjs"
"smoke:trigger-selection": "node scripts/run-trigger-selection-smoke.mjs",
"smoke:cold-start": "node scripts/run-cold-start-smoke.mjs"
},
"engines": {
"node": ">=20"
Expand Down
124 changes: 124 additions & 0 deletions scripts/cold-start-contract.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
export const CANDIDATE_SOURCE_BASE_COMMIT = "60fa913bb313312c64ce4c3d57015f96a4999c91";
export const FROZEN_BASELINE_COMMIT = "9dc46946c879d955daa5a37bd839d168936d6a98";
export const NEXT_TASK = "adaptive-harness-receipt-eval";
export const REQUIRED_PROOF_COMMANDS = [
"npm test",
"npm run benchmark:adaptive-harness",
"npm run benchmark:sealed",
"npm run benchmark:trajectories"
];
export const REQUIRED_EVIDENCE_FILES = [
"AGENTS.md",
"docs/plans/adaptive-harness-candidate.md",
"docs/plans/skill-spine-rollout.md",
"dogfood/report.json"
];

export const coldStartResponseSchema = {
type: "object",
additionalProperties: false,
required: [
"source_commit",
"frozen_baseline_commit",
"promotion_status",
"next_task",
"proof_commands",
"branch_boundary",
"evidence_files"
],
properties: {
source_commit: { type: "string" },
frozen_baseline_commit: { type: "string" },
promotion_status: { type: "string" },
next_task: { type: "string" },
proof_commands: {
type: "array",
items: { type: "string" },
minItems: REQUIRED_PROOF_COMMANDS.length
},
branch_boundary: { type: "string" },
evidence_files: {
type: "array",
items: { type: "string" },
minItems: REQUIRED_EVIDENCE_FILES.length
}
}
};

function requireRecord(response) {
if (!response || typeof response !== "object" || Array.isArray(response)) {
throw new Error("cold-start response must be an object");
}
}

function requireExactArrayValues(actual, expected, label) {
if (!Array.isArray(actual) || !expected.every((value) => actual.includes(value))) {
throw new Error(`cold-start response is missing required ${label}`);
}
}

export function verifyColdStartResponse(
response,
{ expectedSourceCommit = CANDIDATE_SOURCE_BASE_COMMIT } = {}
) {
requireRecord(response);
if (response.source_commit !== expectedSourceCommit) {
throw new Error("cold-start response reported the wrong source commit");
}
if (response.frozen_baseline_commit !== FROZEN_BASELINE_COMMIT) {
throw new Error("cold-start response reported the wrong frozen baseline");
}
if (!String(response.promotion_status || "").toLowerCase().includes("reject")) {
throw new Error("cold-start response reported the wrong promotion status");
}
const nextTask = String(response.next_task || "").toLowerCase();
if (!nextTask.includes("adaptive-harness") || !nextTask.includes("receipt") || !nextTask.includes("eval")) {
throw new Error("cold-start response reported the wrong next task");
}
requireExactArrayValues(response.proof_commands, REQUIRED_PROOF_COMMANDS, "proof commands");
const boundary = String(response.branch_boundary || "").toLowerCase();
if (!/(branch|worktree)/.test(boundary) || !boundary.includes("main") || !/(do not|don't|avoid|not edit)/.test(boundary)) {
throw new Error("cold-start response did not preserve the protected branch boundary");
}
requireExactArrayValues(response.evidence_files, REQUIRED_EVIDENCE_FILES, "evidence files");

return {
source_commit_matches: true,
baseline_matches: true,
promotion_rejected: true,
next_task_matches: true,
proof_gate_complete: true,
protected_branch_boundary: true,
source_evidence_complete: true
};
}

export function verifyColdStartTrace(rawOutput) {
const commands = [];
for (const line of String(rawOutput || "").split(/\r?\n/)) {
if (!line.trim()) continue;
let event;
try {
event = JSON.parse(line);
} catch {
continue;
}
if (event.type === "item.completed" && event.item?.type === "command_execution") {
commands.push(String(event.item.command || ""));
}
}
if (!commands.some((command) => /git (?:rev-parse HEAD|log -1)/.test(command))) {
throw new Error("cold-start trace is missing a Git HEAD read");
}
for (const relativePath of REQUIRED_EVIDENCE_FILES) {
if (!commands.some((command) => command.includes(relativePath))) {
throw new Error(`cold-start trace did not read ${relativePath}`);
}
}
return {
git_head_read: true,
active_plan_read: true,
completed_rollout_read: true,
dogfood_report_read: true
};
}
Loading
Loading