[S1] Positioning + netscript-bench

[rickylabs]

Child of the Road-to-0.0.1-stable umbrella. Owns positioning measurement + netscript-bench.

Scope

Build netscript-bench as the measurement instrument that locks and defends NetScript's position (AI-agent build-efficiency). Fork Encore's ai-benchmark protocol: black-box vitest HTTP probes, composite score (test_pass_rate / production_rubric_pass_rate / turns_to_green / cost / lines_of_code), layered tasks from NetScript's tutorial spine (t1-storefront-api, t2-saga-queue-cron, t3-reactive-dashboard + t3b reactive Fresh-UI extended axis). Full design in validated netscript-bench-research.v2.md.

Sequencing

1. NetScript self-bench first (needs no user decisions): harness + scorer + t1 + t2 as a regression detector on every release. This is a beta gate.
2. Cross-framework batch + published leaderboard: BLOCKED ON user decisions D1-D7 (VM provider, framework list, task count, weights, model+cadence, repo location, license).

Acceptance criteria

• Self-bench runs t1+t2 green with test_pass_rate >= 0.90 median; captures all artifacts (diff, transcript, vitest.json); composite scorer with editable weights.
• (stable) Full core-5 batch published; NetScript composite top-2; reproduced across two model baselines; continuous per-release cadence.

Notes

Repo location is decision D6 (new rickylabs/netscript-bench vs subdir). Per-package benchmark port from netscript-start/benchmark/ is a separate later session. Corpus-overlap confound (agent training data includes NetScript docs) disclosed in methodology.

---

Parent epic: #301

Part of #301

[rickylabs]

S1 netscript-bench self-bench — locked implementation plan (design of record)

Opus-architected, grounded in the validated netscript-bench-research.v2.md. This is the POSITION-FIRST prime deliverable and the beta "measurement instrument exists" gate. Full plan below; supervisor rulings on the open questions follow.

Survey — reuse, don't rebuild

@netscript/cli-e2e (packages/cli/e2e/) already provides the mechanical backbone: CommandExecutor/HttpClient/Reporter/Clock ports, DenoCommandAdapter/FetchHttpAdapter/DockerResourceCleaner/SystemClock adapters, a GateDefinition/GateResult/StepResult/RunReport step-model, a phase-driven createSuiteRunner, and json/ndjson/pretty/report-file reporters. .llm/tools/scaffold-e2e-test.ts (+ deno task e2e:cli run scaffold.runtime) is the canonical scaffold→boot→HTTP-probe flow the green-detector reuses. .llm/tools/watch-run.ts is the poll-free supervise wake. The genuinely new code: the agent driver, turn/cost accounting, scorer/normalizer/rubric, and task specs.

1. Location

Instrument = new workspace package packages/bench/ → @netscript/bench (publish:false), added to the root deno.json workspace array (precedent: packages/cli/e2e). It must inherit check/lint/fmt/test gates (.llm/tools is lint/fmt-excluded → wrong home). Kept self-contained for a clean lift to a standalone public repo if D6 fires.
Run state, two tiers: raw traces → .llm/tmp/bench/<run-id>/ (gitignored, CI artifacts); scored summaries + regression history → packages/bench/results/history/*.json + results/latest.json + results/summary.md (committed — generated artifacts, never frozen prose).

Directory: packages/bench/{deno.json,README.md,mod.ts,bench.config.ts, src/{domain,ports,application/{runner,scoring,builders},adapters/{agent,sandbox,test-runner,reporting},presentation}, tasks/{t1-storefront-api,t2-saga-queue-cron}/{prompt.md,context/AGENTS.md,tests/,rubric.md,reference/netscript/}, results/}.

2. Tasks (both black-box; tests outside the agent's editable surface)

• t1 storefront-api (Encore analog t1-orders-api): typed products+orders CRUD, validation w/ typed errors, referential integrity, persistence across restart, typed client round-trip. NetScript reference = one @netscript/service + oRPC contract from @netscript/contracts + KV-backed persistence + generated @netscript/sdk client.
• t2 saga-queue-cron (Encore analog t2-events-and-schedule): durable saga (reserve→charge→confirm + compensation), background worker (exactly-once confirmation), scheduled expiry; idempotent + retry-safe. Seeds from the committed t1 golden reference. Adds plugins/sagas+@netscript/plugin-sagas-core, plugins/workers+@netscript/queue, @netscript/cron via netscript plugin add.

3. Runner

Four ports keep it model- & sandbox-agnostic: AgentDriver (headless Claude Code claude -p, pinned model/version/toolset, prompt-cached, yields turn stream + usage), SandboxProvider (local-workspace = throwaway temp dir outside the checkout w/ restricted Deno perms; VM seam for D1), TestRunner (deno-http, reuses cli-e2e ports; boots NetScript + frozen HTTP probe; used for in-loop green-detection AND scoring). turns_to_green = first turn index where the frozen suite is fully green (else null + best-turn test_pass_rate). Caps: 900s wall / 80 turns (also normalization anchors). Two modes: agent mode (bench:self, needs API key, on cadence) and conformance mode (bench:conformance, no agent — golden reference vs frozen tests; deterministic, key-free, cheap → the per-release CI gate).

4. Scorer + rubric

test_pass_rate (deterministic), turns_to_green, production_rubric_pass_rate (Run-3 hardening pass over rubric.md), cost (cache-adjusted token $, raw counts stored for recompute), lines_of_code (report-only, weight 0). Composite presets in bench.config.ts: default (v2) test 0.45 / rubric 0.20 / turns 0.15 / cost 0.10 / wall 0.10; --encore-parity test 0.50 / turns 0.20 / cost 0.15 / wall 0.15. Min-max normalization w/ Encore anchors (turns 80→5, cost $2.00→$0.05 recomputed at current pricing, wall 900→60s). Artifacts: raw trace.jsonl (manifest + per-turn), scored JSON (per-cell + median/IQR + variance verdict), summary.md.

5. CI + cadence

Tasks: bench:self, bench:self:t1, bench:self:t2, bench:conformance, bench:score (recompute), bench:report. Jobs: per-release conformance gate (deterministic, no key — ties to e2e-cli-prod precedent, satisfies "zero harness-level failures" + "continuous self-bench per release"); scheduled/dispatch agent measurement (ANTHROPIC_API_KEY secret, uploads traces, commits results history).

6. Deferred seam (STABLE, gated on D1–D7)

Extension point = FrameworkLane + SandboxProvider. Self-bench ships one lane (netscript) + local-workspace. Cross-framework = add lanes + VM adapter. D1→VM adapter, D2→lane list (+ the NestJS-median comparison), D3→t3/t3b/t4 task dirs, D4→weights (already editable), D5→model matrix, D6→repo lift, D7→license. Do NOT design lane guts/VM/leaderboard now.

7. Slices (each a reviewable PR)

1. Package skeleton + domain + 4 ports + claude-code-adapter + local-workspace + deno-http runner + single-task runner w/ turn accounting + scorer(test_pass_rate,turns_to_green,cost,LOC) + json/md reporters + t1 (prompt, frozen tests, golden reference, context) + bench:self:t1. Exit: run t1 once, emit scored JSON+summary.
2. Conformance mode + per-release CI gate.
3. N-repeats + variance controls (median/IQR, >2× swing guard, pinned manifest).
4. t2 task (layered from t1 golden reference).
5. Production rubric + composite + --encore-parity preset + Run-3 hardening.
6. Regression history (Mann-Whitney U) + land FrameworkLane/SandboxProvider interfaces (netscript lane only).

---

Supervisor rulings on the 7 open questions

ADOPTED defaults (non-product, unblock the build):

• OQ4 sandbox: throwaway temp dir OUTSIDE the repo checkout + restricted Deno perms for beta; container/VM deferred to D1.
• OQ5 turns_to_green: a "turn" = one assistant message boundary (tool-round inclusive). Pin Claude Code version + model ID + framework/Deno/lock versions in the run manifest (turn semantics shift across CC releases → comparability requires the pin).
• OQ6 t2 seeding: seed from the committed golden t1 reference (determinism over agent-state realism).
• OQ7 persistence: t1/t2 use @netscript/kv-backed persistence — avoids docker/mysql variance in a low-variance instrument AND cleanly sidesteps the epic: migrate NetScript DB layer to Prisma Next (Postgres-first pilot, deferred) #313 DB-layer carve-out (bench does not touch the DB layer). Disclosed as a confound in the README.

ESCALATED (genuine product decisions — non-blocking; instrument builds + validates via conformance without them):

• OQ1 production-rubric checklist: v2 forks Encore's weight (0.20) but not its items. Needs Encore's actual published rubric, or ratification of the proposed NetScript starter checklist (typed errors / input validation / structured logging / tracing wired / graceful shutdown / env-config / idempotency / no secrets). Only gates Slice 5.
• OQ2 agent-in-CI cost + key + cadence: live runs need ANTHROPIC_API_KEY (~$3–7/run). Proposed: conformance per-push (key-free), agent run per-release-candidate via workflow_dispatch. Confirm cadence + key ownership.
• OQ3 CRITERIA CONTRADICTION (touches the locked beta criteria): "t1 turns_to_green ≤ NestJS median" is cross-framework, but the self-bench is framework-list-independent and needs no user decisions. Proposed amendment: the self-bench beta gate records absolute turns_to_green + green + bounded variance; the "≤ NestJS median" comparison moves to the first cross-framework run (STABLE, gated on D2). This amends the beta milestone description — coordinator to update.

Build posture: Slice 1 lands the instrument + t1 + golden reference, validated by conformance mode + scorer unit tests — NO live paid agent run required to merge. The first live measurement run is a separate execution step after OQ2 is answered.