eval: add explicit design then implementation workflow - #46
Conversation
# Conflicts: # PLAN.md
|
Replay report added in commit 0a6588e: https://github.com/iantocristian/jscout/blob/codex/eval-two-phase/eval/results/next-root-layout-param-types-2026-08-18.md The valid matrix completed 12 arms: 8/12 passed. The host-sleep Terra timeout is preserved but excluded and replaced by an unchanged retry that passed. Main findings: incremental bounded jscout usage worked; exact-identifier ranking and semantic-memory selection did not; memory-profile 4/4 is not attributable to retrieved artifacts; design-then-implement cost more and passed 3/6 versus 5/6 single-phase; dev self-verification still exposes EMFILE and sandboxed-install EPERM. |
|
Follow-up pushed in d6c3014. Correction: unconditional Playwright sidecar startup was introduced by 847148b and is already on main; it was not introduced by PR #46. This PR now makes that existing behavior capability-aware without changing launcher contracts:
Parallel runners need no invocation changes: ports are OS-assigned and registries/workspaces remain per arm/process. Existing output roots/trial IDs still need to be unique, as before. Local verification: npm test — 62 passed, 1 platform skip, 0 failed. |
|
Plan/eval follow-up pushed in 6d70ee7.
Validated task JSON, diff whitespace, and replay/report tests: 15 passed. |
Summary
--workflow design-implementmode to the PR replay runnerblockedlabelThis adds no jscout command, MCP tool, semantic artifact, skill guidance, or runtime product behavior.
Failure behavior
A timed-out, nonzero, invalid, or source-mutating design phase does not unlock implementation and is not graded as an implementation attempt.
Validation
npm test(57 tests)node --check scripts/eval-run-replay.mjsgit diff --check