From 15465ff430f31e972acdf17339cc5b14a695b021 Mon Sep 17 00:00:00 2001 From: gogocat Date: Tue, 2 Jun 2026 15:57:40 +0300 Subject: [PATCH] =?UTF-8?q?docs(handoff):=202026-06-02=20=E2=80=94=20full-?= =?UTF-8?q?context=20session=20handoff?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Comprehensive resume doc: where the product is (~5/10; judges + site strong, real data is the gap), what shipped (judges operational, pass^k, leaderboard data, harness×model site), the two-half-sandbox architecture, the next build (stack executor / RFC-006) with phases, copy-paste tool usage (proxy, harness recipes, judges, smoke run, site, gates, forgeplan, git), the env landmines (uv-sync extras, moon fork-bomb, eslint, proxy key, be_01 inversion), a key-files map, and step-by-step instructions to continue in a fresh session. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/handoffs/HANDOFF-2026-06-02.md | 288 ++++++++++++++++++++++++++++ 1 file changed, 288 insertions(+) create mode 100644 docs/handoffs/HANDOFF-2026-06-02.md diff --git a/docs/handoffs/HANDOFF-2026-06-02.md b/docs/handoffs/HANDOFF-2026-06-02.md new file mode 100644 index 0000000..6563f5a --- /dev/null +++ b/docs/handoffs/HANDOFF-2026-06-02.md @@ -0,0 +1,288 @@ +# HANDOFF — 2026-06-02 + +> Full-context handoff after a large session. Read this to resume. Pairs with the +> auto-loaded `MEMORY.md` index and the forgeplan artifact graph (`forgeplan health`). +> For the *next build specifically* (the executor), the memory note +> `project_executor_build_plan` has the step-by-step. + +--- + +## 1. TL;DR — where we are + +POLLMEVALS is an **open evidence layer for choosing production LLM stacks**. It ranks +**stacks** = `model × harness × scaffolding (L0–L8)`, not bare models. Thesis: *a +cheap model with the right harness beats an expensive one without it.* + +**Product maturity ≈ 5/10.** Strong now: the methodology, the **judge panel +(operational, live-validated)**, the **public site** (a harness×model matrix +leaderboard). The one thing between us and "6/10" is **real data** — running real +agent-CLI harnesses on real tasks. That requires the **stack executor** (RFC-006), +which is designed but not built. Everything else it needs already exists. + +--- + +## 2. What is DONE (main) and what is IN FLIGHT (open PRs) + +### On `main` (shipped this session + before) +- **Judge panel — operational.** G1–G6 + proxy wiring + live-validated (Krippendorff + α=0.358 on real models). `apps/eval-core-py/src/orchestrator/judge_panel.py`. +- **Reliability metrics** — `pass@k` (ceiling) + `pass^k` (reliability) + flakiness: + `apps/eval-core-py/src/scoring/pass_k.py`. +- **Leaderboard data layer** — Run manifest → `leaderboard.json`: + `apps/eval-core-py/src/leaderboard/aggregate.py` + `scripts/build_leaderboard.py`. +- **Stack roster** — 12 `stacks/*/stack.yaml` (added OpenCode/Goose/OpenHands; + gemini-cli deprecated). Each carries its proven proxy recipe in comments. +- **`--with-judges`** flag wiring JudgePanel into GridRunner (`scripts/smoke_run.py`). +- **Atomic `requirements[]`** model (RFC-004), task packs migrated. +- **Forgeplan graph hygiene** — evidence activated, health `unhealthy → needs_attention`. +- Decisions/design: ADR-005 (α-gate), ADR-008 (requirements[]), ADR-013 (pass^k), + RFC-002 (judges), RFC-006 (executor design). + +### Open PRs (read on GitHub) +| PR | What | Notes | +|----|------|-------| +| **#36** | Public site: **harness×model leaderboard** (matrix hero + Pareto + ablation + per-task) | Renders **illustrative** data (flagged in-page). Real data needs the executor. | +| **#37** | RFC-006 implementation phases (executor design) | The plan for the next build. | +| **#24** | dynamic-eval: Linux node_modules for the sandbox (Half B / evaluator side) | Yours. Proved fe_01 runs e2e in Docker; surfaced 3 defects. | + +--- + +## 3. The architecture (full context) + +One **Run** = a frozen pass over many **Evals**, each = `(model, stack, task, seed) → output`, +then automatic metrics + judge scores → `final_score`. Immutability: ADR-0002 (a published +run is never edited; errors → new run + `supersedes`). + +The eval pipeline is a **two-half Docker sandbox**: + +``` + ┌─────────── Half A: CANDIDATE (the gap — RFC-006) ───────────┐ + task + │ StackExecutor: run the agent CLI (aider/codex/claude-code…) │ + repo ───► │ in a sandbox, model calls via the LiteLLM proxy │ ──► patch + snapshot │ → capture patch + trace + metered cost │ + trace + └─────────────────────────────────────────────────────────────┘ + │ patch + ▼ + ┌─────────── Half B: EVALUATOR (BUILT — #24 + evaluators) ────┐ + │ apply patch → run produced code in sandbox → correctness / │ ──► auto + │ coverage / lint / type_safety / complexity (Docker) │ metrics + └─────────────────────────────────────────────────────────────┘ + │ normalized output + ▼ + ┌─────────── JUDGES (operational — #28) ──────────────────────┐ + │ ≥3 judges, no self-judging, median, Krippendorff α gate │ ──► quality + └─────────────────────────────────────────────────────────────┘ + ▼ + aggregate → leaderboard.json / Board → apps/site (the public leaderboard) +``` + +The **LiteLLM proxy** (`localhost:4000`) is the linchpin: every CLI's model calls route +through it, so we (a) **choose which model** sits behind any harness and (b) **meter every +token/cost**. That's what makes "model × harness" measurable. + +--- + +## 4. What's NEXT — the stack executor (RFC-006) + +**Goal:** replace the site's illustrative data with REAL numbers by building Half A. + +**Why it's the only gap:** Docker is up, Half B evaluators + the sandbox runner exist, +judges are operational, the be_01 task pack has a Dockerfile + gold + rubric, and all 7 +CLI harnesses are *proven* to run headless via the proxy. Today GridRunner only runs +`raw-llm` (model completion via `InspectEvalCaller`). The executor adds the "run the real +harness" path. + +**First slice:** `aider × qwen-3-14b × be_01_jwt_auth` → real patch → Half B evaluators → +the **first real `model × harness × task` number**. Then widen to codex/opencode + models + tasks. + +**Phases** (see RFC-006 + `project_executor_build_plan` memory for detail): +1. `stack_executor.py` core — mocked, no Docker, no spend. +2. Sandbox image + first real single-harness run (HUMAN cost checkpoint, ~pennies). +3. Wire Half A → Half B → judges → first real score. +4. GridRunner dispatch-by-stack + emit the rich **Board** the site renders (drop illustrative). +5. Widen (more harnesses/models/tasks). + +**⚠️ Task-choice caveat:** `be_01` evaluators *score-invert* (broken 0.99 > perfect 0.76, +EVID-027). Either fix that first, or start the first **objective-scored** slice on `fe_01` +(post-#24 fixes) or `doc_01` (judge-only, no code-exec). Choose deliberately. + +--- + +## 5. HOW TO USE THE TOOLS (copy-paste) + +### 5.1 The LiteLLM proxy (the metered model gateway) +```bash +# bring it up (proxy + NATS/Redis/Postgres) +make docker-up +# auth = LITELLM_MASTER_KEY (sk-local…) from .env — NOT the OpenRouter upstream key +set -a && source .env && set +a +# health +curl -s -o /dev/null -w "%{http_code}\n" http://localhost:4000/health/liveliness +``` +Serves OpenAI `/v1/chat/completions` + `/v1/responses`, and Anthropic `/v1/messages`. +Model aliases (in `infra/litellm-config.yaml`): `claude-sonnet-4-6`, `gpt-5-mini`, +`gemini-3-flash`, `qwen-3-14b`, `llama-3-3-70b`, judge aliases `*-judge`. + +### 5.2 Run an agent-CLI harness headless via the proxy (proven recipes) +```bash +set -a && source .env && set +a +# aider (model-agnostic — easiest) +OPENAI_API_BASE=http://localhost:4000/v1 OPENAI_API_KEY=$LITELLM_MASTER_KEY \ + aider --model openai/qwen-3-14b --message "…task…" --yes-always --no-stream --map-tokens 0 + +# codex (needs an isolated CODEX_HOME with a config.toml provider; wire_api=responses) +# [model_providers.litellm] base_url=http://localhost:4000/v1 env_key=LITELLM_MASTER_KEY wire_api="responses" +CODEX_HOME=/tmp/ch codex exec --full-auto -m qwen-3-14b "…task…" + +# claude-code (Anthropic endpoint; use a native/compatible model) +ANTHROPIC_BASE_URL=http://localhost:4000 ANTHROPIC_API_KEY=$LITELLM_MASTER_KEY \ + claude -p "…task…" --model claude-sonnet-4-6 --dangerously-skip-permissions --mcp-config '{"mcpServers":{}}' +``` +Full recipe table for goose / openhands / opencode / hermes: memory +`research_cli_harness_execution`. **claude-code breaks on qwen** (use native pairing). + +### 5.3 Judges — live smoke +```bash +set -a && source .env && set +a +uv run --project apps/eval-core-py python apps/eval-core-py/scripts/judge_live_smoke.py +# ~$0.05; 3 judges on a doc_01 sample; prints per-judge cost/latency + Krippendorff α. +``` + +### 5.4 Smoke run (candidate evals) + leaderboard +```bash +# dry-run (no spend); add --confirm-spend for real +uv run --project apps/eval-core-py python apps/eval-core-py/scripts/smoke_run.py \ + --dry-run --tasks doc_01_cli_readme --models openrouter/qwen/qwen-2-5-14b --seeds 1 +# add --with-judges to run the judge panel on each eval (single candidate model only) + +# build leaderboard.json from a run manifest +uv run --project apps/eval-core-py python apps/eval-core-py/scripts/build_leaderboard.py --out /tmp/lb.json +``` + +### 5.5 The public site +```bash +pnpm install +pnpm --filter @pollmevals/site dev # http://localhost:3000 +pnpm --filter @pollmevals/site build # → apps/site/out/ (static export) +# preview the static build: +cd apps/site/out && python3 -m http.server 4400 # then open http://localhost:4400 +``` +Regenerate the illustrative board data: `node apps/site/scripts/gen-illustrative-board.mjs`. + +### 5.6 Gates (run BEFORE committing — CI runs these directly) +```bash +# Python (eval-core-py) +uv run --project apps/eval-core-py pytest apps/eval-core-py/tests/ -q +cd apps/eval-core-py && uv run mypy --strict src/ # run FROM the project dir +uv run --project apps/eval-core-py ruff check apps/eval-core-py/src apps/eval-core-py/tests +uv run --project apps/eval-core-py ruff format --check apps/eval-core-py/src apps/eval-core-py/tests +# TS (site) +cd apps/site && pnpm typecheck && pnpm test && pnpm build +# whole workspace (what the pre-commit hook runs) +pnpm exec moon run :typecheck +``` + +### 5.7 forgeplan (artifact graph — prefer MCP `mcp__forgeplan__*`, else CLI) +```bash +forgeplan health # blind spots, stale, anomalies +forgeplan list --kind rfc # browse +forgeplan get RFC-006 # read an artifact +forgeplan new evidence "…" --link RFC-006 # create + autolink +forgeplan validate # 0 MUST errors before activate +forgeplan activate # draft → active (do AFTER the final body update + links) +``` +**Rule:** never `Edit`/`Write` `.forgeplan/*` directly — go through forgeplan. +**Gotcha:** `forgeplan_update(body)` AFTER `forgeplan_link` wipes the `.md` frontmatter +`links:` projection (edges stay in the graph; cosmetic drift only) — link AFTER the +final body update. + +### 5.8 git workflow +- Branch per slice/wave (fewer-larger PRs). Never commit straight to `main`. +- Commit messages end with `Co-Authored-By: Claude Opus 4.8 (1M context) `. +- PR bodies end with the Claude Code generated-by line. +- **Red lines:** never `git commit --no-verify`; never force-push. +- OUTWARD actions (push, PR, merge) are human checkpoints — confirm. + +--- + +## 6. GOTCHAS / landmines (these cost real time — read before building) + +1. **NEVER run a bare `uv sync` at the repo root.** It prunes the dev *extras* + (mypy/ruff/pytest live in `apps/eval-core-py [project.optional-dependencies].dev`) + from the shared `.venv` → `uv run mypy` falls back to an ephemeral mypy with no + project deps → 59 spurious "Cannot find module pydantic/…" errors (runtime `import` + still works — that's the tell). **Restore:** `uv sync --all-packages --all-extras --all-groups`. + Canonical typecheck: `cd apps/eval-core-py && uv run mypy --strict src/`. +2. **Empty `apps/*` skeletons can fork-bomb moon.** A project with `typecheck: pnpm + typecheck` in `moon.yml` but *no* `package.json` resolves `pnpm typecheck` to the + ROOT script (`moon run :typecheck`) → infinite recursion → hundreds of procs, the + commit hangs. Dormant until a `pnpm-lock.yaml` change invalidates moon's cache. + Fixed for apps/api + apps/site (own package.json + local `tsc --noEmit`). If a + commit hangs: `pkill -9 -f "moon run"`, then `pnpm exec moon run :typecheck` + standalone — the spammed `:typecheck` name is the culprit. +3. **eslint 9 needs a flat-config at the lint CWD.** The pre-commit runs eslint from + the repo root → needs root `eslint.config.mjs` (committed). Legacy `.eslintrc.json` errors. +4. **Proxy auth = `LITELLM_MASTER_KEY`** (sk-local…), NOT `OPENROUTER_API_KEY*` + (the upstream sk-or-v1… key, rejected by the proxy with 401). +5. **be_01 evaluators score-invert** (EVID-027) — don't trust them for the first scored slice. +6. **Sub-agent coders run in isolated git worktrees without uv deps** → can't run tests + there → prefer building hard integration code in the main tree / a fresh focused context. + (memory `feedback_agent_worktree_gotcha`.) +7. **`dev/` + `NOTE-009/010`** = a background agent's work — DO NOT touch/commit. +8. **`zsh: command not found: _encode/_decode`** spam on every Bash call — harmless, ignore. +9. OpenRouter key budget is small (~$9 left); it 402s when exhausted (top up at + openrouter.ai/workspaces/pollmevals/keys/). Cap judge `max_tokens` (HTTP-402 hazard). + +--- + +## 7. KEY FILES / where things live + +| Area | Path | +|------|------| +| Judge panel | `apps/eval-core-py/src/orchestrator/judge_panel.py` | +| Grid runner (eval loop) | `apps/eval-core-py/src/orchestrator/grid_runner.py` | +| Candidate caller (raw-llm) | `apps/eval-core-py/src/orchestrator/eval_caller.py` | +| **Stack executor (TO BUILD)** | `apps/eval-core-py/src/orchestrator/stack_executor.py` (new) | +| Evaluators (Half B) | `apps/eval-core-py/src/evaluators/` + `evaluators/sandbox/runner.py` | +| Cost | `apps/eval-core-py/src/orchestrator/cost.py` | +| Scoring (requirements, pass^k) | `apps/eval-core-py/src/scoring/` | +| Leaderboard aggregate | `apps/eval-core-py/src/leaderboard/aggregate.py` | +| Contracts (Pydantic) | `apps/eval-core-py/src/contracts/` | +| Site (Next.js) | `apps/site/src/` (board.ts, components, app/page.tsx) | +| Stack adapters | `stacks//stack.yaml` (12, with proxy recipes) | +| Task packs | `evals/task-packs//` (prompt, gold, rubric, calibration, Dockerfile) | +| Proxy config | `infra/litellm-config.yaml` | +| Methodology (frozen v0.1) | `docs/02-methodology/` | +| Scoring contract | `docs/04-runbook/08-scoring-contract.md` | + +--- + +## 8. HOW TO CONTINUE (instructions for the next session) + +1. **Open a FRESH session** (this one's context is huge). `MEMORY.md` auto-loads; the + note `project_executor_build_plan` has the executor step-by-step. +2. Say something like: *"Build the stack executor per RFC-006, Phase 1"* — or run + `/smith` for a routed plan, or `/restore` to re-sync from git + memory. +3. **Phase 1 (no spend):** write `apps/eval-core-py/src/orchestrator/stack_executor.py` + with a Protocol mock-seam + unit tests. Run the gates (§5.6). Commit on a branch, PR. +4. **Phase 2 (cost checkpoint):** `make docker-up`, build the aider sandbox image, run + `aider × qwen × be_01` → capture a real patch. Confirm the spend first (~pennies). +5. **Phase 3:** patch → Half B evaluators + judges → the first real number. Mind the + be_01 inversion (§4 caveat). +6. Keep merging your open PRs (#36 site, #37 RFC, #24 dynamic-eval) when you've reviewed them. + +**Standing constraints:** fewer-larger PRs; methodology-first; OUTWARD actions are human +checkpoints; never `--no-verify`; don't touch `dev/`; design disputes → think it through +(decompose/evaluate/reason) before presenting choices. + +--- + +## 9. References + +- **Memory:** `project_executor_build_plan` (the build), `research_cli_harness_execution` + (harness recipes), `feedback_site_design_and_monorepo_gotchas` (env landmines), + `feedback_agent_worktree_gotcha`, `research_swe_rebench`, `research_afterquery_appbench`, + `project_state_2026-06-01` (working snapshot). +- **Forgeplan:** `forgeplan health`; RFC-006 (executor), RFC-002 (judges), ADR-005/008/013, + PRD-002/004/006, EPIC-002 (frozen v0.2 roadmap). +- **Prior handoff:** `docs/handoffs/HANDOFF-2026-05-27.md`.