Skip to content

eval: add explicit design then implementation workflow - #46

Merged
iantocristian merged 8 commits into
mainfrom
codex/eval-two-phase
Aug 18, 2026
Merged

eval: add explicit design then implementation workflow#46
iantocristian merged 8 commits into
mainfrom
codex/eval-two-phase

Conversation

@iantocristian

Copy link
Copy Markdown
Owner

Summary

  • adds an eval-only --workflow design-implement mode to the PR replay runner
  • runs a read-only structured design phase, verifies it produced no patch, then injects the captured design verbatim into a fresh implementation call
  • records prompts, responses, event streams, token/command usage, duration, and jscout requests separately per phase and in aggregate
  • joins phase-specific telemetry correctly and refuses report-level pooling across single-phase and two-phase runs
  • records G15/G16 as parked product work; PR G15: add design-before-edit task memory #45 remains open with the blocked label

This adds no jscout command, MCP tool, semantic artifact, skill guidance, or runtime product behavior.

Failure behavior

A timed-out, nonzero, invalid, or source-mutating design phase does not unlock implementation and is not graded as an implementation attempt.

Validation

  • npm test (57 tests)
  • node --check scripts/eval-run-replay.mjs
  • git diff --check

@iantocristian

Copy link
Copy Markdown
Owner Author

Replay report added in commit 0a6588e: https://github.com/iantocristian/jscout/blob/codex/eval-two-phase/eval/results/next-root-layout-param-types-2026-08-18.md

The valid matrix completed 12 arms: 8/12 passed. The host-sleep Terra timeout is preserved but excluded and replaced by an unchanged retry that passed. Main findings: incremental bounded jscout usage worked; exact-identifier ranking and semantic-memory selection did not; memory-profile 4/4 is not attributable to retrieved artifacts; design-then-implement cost more and passed 3/6 versus 5/6 single-phase; dev self-verification still exposes EMFILE and sandboxed-install EPERM.

@iantocristian

Copy link
Copy Markdown
Owner Author

Follow-up pushed in d6c3014.

Correction: unconditional Playwright sidecar startup was introduced by 847148b and is already on main; it was not introduced by PR #46.

This PR now makes that existing behavior capability-aware without changing launcher contracts:

  • browser_server defaults to auto, starts only when the prepared workspace resolves Playwright;
  • required fails closed if Playwright is missing or launch fails; disabled skips explicitly;
  • per-arm browser-server.json and response rows record the decision;
  • Next replay task sets use required and WATCHPACK_POLLING=250;
  • the agent test command includes NEXT_SKIP_ISOLATE=1 to prevent nested installs in the sandbox.

Parallel runners need no invocation changes: ports are OS-assigned and registries/workspaces remain per arm/process. Existing output roots/trial IDs still need to be unique, as before.

Local verification: npm test — 62 passed, 1 platform skip, 0 failed.

@iantocristian

Copy link
Copy Markdown
Owner Author

Plan/eval follow-up pushed in 6d70ee7.

  • G15 remains parked after the root-layout two-phase arms preserved wrong contracts.
  • G16 is now explicitly independent of G15 and limited to G14 attached-memory false negatives; the root-layout run does not trigger it because the corpus lacked a useful causal artifact.
  • G17 is the immediate exact-identifier dominance milestone.
  • G18 covers scope-stratified/targeted scouting and support-aware compact semantic-memory retrieval without larger budgets.
  • Both committed Next task sets now inject NEXT_SKIP_ISOLATE=1 alongside WATCHPACK_POLLING=250.

Validated task JSON, diff whitespace, and replay/report tests: 15 passed.

@iantocristian
iantocristian merged commit 26d5303 into main Aug 18, 2026
4 checks passed
@iantocristian
iantocristian deleted the codex/eval-two-phase branch August 18, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant