Skip to content

docs(handoff): 2026-06-02 — full-context session handoff #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
explosivebit wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs(handoff): 2026-06-02 — full-context session handoff #38

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
288 changes: 288 additions & 0 deletions docs/handoffs/HANDOFF-2026-06-02.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,288 @@
# HANDOFF — 2026-06-02

> Full-context handoff after a large session. Read this to resume. Pairs with the
> auto-loaded `MEMORY.md` index and the forgeplan artifact graph (`forgeplan health`).
> For the *next build specifically* (the executor), the memory note
> `project_executor_build_plan` has the step-by-step.

---

## 1. TL;DR — where we are

POLLMEVALS is an **open evidence layer for choosing production LLM stacks**. It ranks
**stacks** = `model × harness × scaffolding (L0–L8)`, not bare models. Thesis: *a
cheap model with the right harness beats an expensive one without it.*

**Product maturity ≈ 5/10.** Strong now: the methodology, the **judge panel
(operational, live-validated)**, the **public site** (a harness×model matrix
leaderboard). The one thing between us and "6/10" is **real data** — running real
agent-CLI harnesses on real tasks. That requires the **stack executor** (RFC-006),
which is designed but not built. Everything else it needs already exists.

---

## 2. What is DONE (main) and what is IN FLIGHT (open PRs)

### On `main` (shipped this session + before)
- **Judge panel — operational.** G1–G6 + proxy wiring + live-validated (Krippendorff
α=0.358 on real models). `apps/eval-core-py/src/orchestrator/judge_panel.py`.
- **Reliability metrics** — `pass@k` (ceiling) + `pass^k` (reliability) + flakiness:
`apps/eval-core-py/src/scoring/pass_k.py`.
- **Leaderboard data layer** — Run manifest → `leaderboard.json`:
`apps/eval-core-py/src/leaderboard/aggregate.py` + `scripts/build_leaderboard.py`.
- **Stack roster** — 12 `stacks/*/stack.yaml` (added OpenCode/Goose/OpenHands;
gemini-cli deprecated). Each carries its proven proxy recipe in comments.
- **`--with-judges`** flag wiring JudgePanel into GridRunner (`scripts/smoke_run.py`).
- **Atomic `requirements[]`** model (RFC-004), task packs migrated.
- **Forgeplan graph hygiene** — evidence activated, health `unhealthy → needs_attention`.
- Decisions/design: ADR-005 (α-gate), ADR-008 (requirements[]), ADR-013 (pass^k),
RFC-002 (judges), RFC-006 (executor design).

### Open PRs (read on GitHub)
| PR | What | Notes |
|----|------|-------|
| **#36** | Public site: **harness×model leaderboard** (matrix hero + Pareto + ablation + per-task) | Renders **illustrative** data (flagged in-page). Real data needs the executor. |
| **#37** | RFC-006 implementation phases (executor design) | The plan for the next build. |
| **#24** | dynamic-eval: Linux node_modules for the sandbox (Half B / evaluator side) | Yours. Proved fe_01 runs e2e in Docker; surfaced 3 defects. |

---

## 3. The architecture (full context)

One **Run** = a frozen pass over many **Evals**, each = `(model, stack, task, seed) → output`,
then automatic metrics + judge scores → `final_score`. Immutability: ADR-0002 (a published
run is never edited; errors → new run + `supersedes`).

The eval pipeline is a **two-half Docker sandbox**:

```
┌─────────── Half A: CANDIDATE (the gap — RFC-006) ───────────┐
task + │ StackExecutor: run the agent CLI (aider/codex/claude-code…) │
repo ───► │ in a sandbox, model calls via the LiteLLM proxy │ ──► patch
snapshot │ → capture patch + trace + metered cost │ + trace
└─────────────────────────────────────────────────────────────┘
│ patch
▼
┌─────────── Half B: EVALUATOR (BUILT — #24 + evaluators) ────┐
│ apply patch → run produced code in sandbox → correctness / │ ──► auto
│ coverage / lint / type_safety / complexity (Docker) │ metrics
└─────────────────────────────────────────────────────────────┘
│ normalized output
▼
┌─────────── JUDGES (operational — #28) ──────────────────────┐
│ ≥3 judges, no self-judging, median, Krippendorff α gate │ ──► quality
└─────────────────────────────────────────────────────────────┘
▼
aggregate → leaderboard.json / Board → apps/site (the public leaderboard)
```

The **LiteLLM proxy** (`localhost:4000`) is the linchpin: every CLI's model calls route
through it, so we (a) **choose which model** sits behind any harness and (b) **meter every
token/cost**. That's what makes "model × harness" measurable.

---

## 4. What's NEXT — the stack executor (RFC-006)

**Goal:** replace the site's illustrative data with REAL numbers by building Half A.

**Why it's the only gap:** Docker is up, Half B evaluators + the sandbox runner exist,
judges are operational, the be_01 task pack has a Dockerfile + gold + rubric, and all 7
CLI harnesses are *proven* to run headless via the proxy. Today GridRunner only runs
`raw-llm` (model completion via `InspectEvalCaller`). The executor adds the "run the real
harness" path.

**First slice:** `aider × qwen-3-14b × be_01_jwt_auth` → real patch → Half B evaluators →
the **first real `model × harness × task` number**. Then widen to codex/opencode + models + tasks.

**Phases** (see RFC-006 + `project_executor_build_plan` memory for detail):
1. `stack_executor.py` core — mocked, no Docker, no spend.
2. Sandbox image + first real single-harness run (HUMAN cost checkpoint, ~pennies).
3. Wire Half A → Half B → judges → first real score.
4. GridRunner dispatch-by-stack + emit the rich **Board** the site renders (drop illustrative).
5. Widen (more harnesses/models/tasks).

**⚠️ Task-choice caveat:** `be_01` evaluators *score-invert* (broken 0.99 > perfect 0.76,
EVID-027). Either fix that first, or start the first **objective-scored** slice on `fe_01`
(post-#24 fixes) or `doc_01` (judge-only, no code-exec). Choose deliberately.

---

## 5. HOW TO USE THE TOOLS (copy-paste)

### 5.1 The LiteLLM proxy (the metered model gateway)
```bash
# bring it up (proxy + NATS/Redis/Postgres)
make docker-up
# auth = LITELLM_MASTER_KEY (sk-local…) from .env — NOT the OpenRouter upstream key
set -a && source .env && set +a
# health
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:4000/health/liveliness
```
Serves OpenAI `/v1/chat/completions` + `/v1/responses`, and Anthropic `/v1/messages`.
Model aliases (in `infra/litellm-config.yaml`): `claude-sonnet-4-6`, `gpt-5-mini`,
`gemini-3-flash`, `qwen-3-14b`, `llama-3-3-70b`, judge aliases `*-judge`.

### 5.2 Run an agent-CLI harness headless via the proxy (proven recipes)
```bash
set -a && source .env && set +a
# aider (model-agnostic — easiest)
OPENAI_API_BASE=http://localhost:4000/v1 OPENAI_API_KEY=$LITELLM_MASTER_KEY \
aider --model openai/qwen-3-14b --message "…task…" --yes-always --no-stream --map-tokens 0

# codex (needs an isolated CODEX_HOME with a config.toml provider; wire_api=responses)
# [model_providers.litellm] base_url=http://localhost:4000/v1 env_key=LITELLM_MASTER_KEY wire_api="responses"
CODEX_HOME=/tmp/ch codex exec --full-auto -m qwen-3-14b "…task…"

# claude-code (Anthropic endpoint; use a native/compatible model)
ANTHROPIC_BASE_URL=http://localhost:4000 ANTHROPIC_API_KEY=$LITELLM_MASTER_KEY \
claude -p "…task…" --model claude-sonnet-4-6 --dangerously-skip-permissions --mcp-config '{"mcpServers":{}}'
```
Full recipe table for goose / openhands / opencode / hermes: memory
`research_cli_harness_execution`. **claude-code breaks on qwen** (use native pairing).

### 5.3 Judges — live smoke
```bash
set -a && source .env && set +a
uv run --project apps/eval-core-py python apps/eval-core-py/scripts/judge_live_smoke.py
# ~$0.05; 3 judges on a doc_01 sample; prints per-judge cost/latency + Krippendorff α.
```

### 5.4 Smoke run (candidate evals) + leaderboard
```bash
# dry-run (no spend); add --confirm-spend for real
uv run --project apps/eval-core-py python apps/eval-core-py/scripts/smoke_run.py \
--dry-run --tasks doc_01_cli_readme --models openrouter/qwen/qwen-2-5-14b --seeds 1
# add --with-judges to run the judge panel on each eval (single candidate model only)

# build leaderboard.json from a run manifest
uv run --project apps/eval-core-py python apps/eval-core-py/scripts/build_leaderboard.py --out /tmp/lb.json
```

### 5.5 The public site
```bash
pnpm install
pnpm --filter @pollmevals/site dev # http://localhost:3000
pnpm --filter @pollmevals/site build # → apps/site/out/ (static export)
# preview the static build:
cd apps/site/out && python3 -m http.server 4400 # then open http://localhost:4400
```
Regenerate the illustrative board data: `node apps/site/scripts/gen-illustrative-board.mjs`.

### 5.6 Gates (run BEFORE committing — CI runs these directly)
```bash
# Python (eval-core-py)
uv run --project apps/eval-core-py pytest apps/eval-core-py/tests/ -q
cd apps/eval-core-py && uv run mypy --strict src/ # run FROM the project dir
uv run --project apps/eval-core-py ruff check apps/eval-core-py/src apps/eval-core-py/tests
uv run --project apps/eval-core-py ruff format --check apps/eval-core-py/src apps/eval-core-py/tests
# TS (site)
cd apps/site && pnpm typecheck && pnpm test && pnpm build
# whole workspace (what the pre-commit hook runs)
pnpm exec moon run :typecheck
```

### 5.7 forgeplan (artifact graph — prefer MCP `mcp__forgeplan__*`, else CLI)
```bash
forgeplan health # blind spots, stale, anomalies
forgeplan list --kind rfc # browse
forgeplan get RFC-006 # read an artifact
forgeplan new evidence "…" --link RFC-006 # create + autolink
forgeplan validate <ID> # 0 MUST errors before activate
forgeplan activate <ID> # draft → active (do AFTER the final body update + links)
```
**Rule:** never `Edit`/`Write` `.forgeplan/*` directly — go through forgeplan.
**Gotcha:** `forgeplan_update(body)` AFTER `forgeplan_link` wipes the `.md` frontmatter
`links:` projection (edges stay in the graph; cosmetic drift only) — link AFTER the
final body update.

### 5.8 git workflow
- Branch per slice/wave (fewer-larger PRs). Never commit straight to `main`.
- Commit messages end with `Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>`.
- PR bodies end with the Claude Code generated-by line.
- **Red lines:** never `git commit --no-verify`; never force-push.
- OUTWARD actions (push, PR, merge) are human checkpoints — confirm.

---

## 6. GOTCHAS / landmines (these cost real time — read before building)

1. **NEVER run a bare `uv sync` at the repo root.** It prunes the dev *extras*
(mypy/ruff/pytest live in `apps/eval-core-py [project.optional-dependencies].dev`)
from the shared `.venv` → `uv run mypy` falls back to an ephemeral mypy with no
project deps → 59 spurious "Cannot find module pydantic/…" errors (runtime `import`
still works — that's the tell). **Restore:** `uv sync --all-packages --all-extras --all-groups`.
Canonical typecheck: `cd apps/eval-core-py && uv run mypy --strict src/`.
2. **Empty `apps/*` skeletons can fork-bomb moon.** A project with `typecheck: pnpm
typecheck` in `moon.yml` but *no* `package.json` resolves `pnpm typecheck` to the
ROOT script (`moon run :typecheck`) → infinite recursion → hundreds of procs, the
commit hangs. Dormant until a `pnpm-lock.yaml` change invalidates moon's cache.
Fixed for apps/api + apps/site (own package.json + local `tsc --noEmit`). If a
commit hangs: `pkill -9 -f "moon run"`, then `pnpm exec moon run :typecheck`
standalone — the spammed `<proj>:typecheck` name is the culprit.
3. **eslint 9 needs a flat-config at the lint CWD.** The pre-commit runs eslint from
the repo root → needs root `eslint.config.mjs` (committed). Legacy `.eslintrc.json` errors.
4. **Proxy auth = `LITELLM_MASTER_KEY`** (sk-local…), NOT `OPENROUTER_API_KEY*`
(the upstream sk-or-v1… key, rejected by the proxy with 401).
5. **be_01 evaluators score-invert** (EVID-027) — don't trust them for the first scored slice.
6. **Sub-agent coders run in isolated git worktrees without uv deps** → can't run tests
there → prefer building hard integration code in the main tree / a fresh focused context.
(memory `feedback_agent_worktree_gotcha`.)
7. **`dev/` + `NOTE-009/010`** = a background agent's work — DO NOT touch/commit.
8. **`zsh: command not found: _encode/_decode`** spam on every Bash call — harmless, ignore.
9. OpenRouter key budget is small (~$9 left); it 402s when exhausted (top up at
openrouter.ai/workspaces/pollmevals/keys/). Cap judge `max_tokens` (HTTP-402 hazard).

---

## 7. KEY FILES / where things live

| Area | Path |
|------|------|
| Judge panel | `apps/eval-core-py/src/orchestrator/judge_panel.py` |
| Grid runner (eval loop) | `apps/eval-core-py/src/orchestrator/grid_runner.py` |
| Candidate caller (raw-llm) | `apps/eval-core-py/src/orchestrator/eval_caller.py` |
| **Stack executor (TO BUILD)** | `apps/eval-core-py/src/orchestrator/stack_executor.py` (new) |
| Evaluators (Half B) | `apps/eval-core-py/src/evaluators/` + `evaluators/sandbox/runner.py` |
| Cost | `apps/eval-core-py/src/orchestrator/cost.py` |
| Scoring (requirements, pass^k) | `apps/eval-core-py/src/scoring/` |
| Leaderboard aggregate | `apps/eval-core-py/src/leaderboard/aggregate.py` |
| Contracts (Pydantic) | `apps/eval-core-py/src/contracts/` |
| Site (Next.js) | `apps/site/src/` (board.ts, components, app/page.tsx) |
| Stack adapters | `stacks/<slug>/stack.yaml` (12, with proxy recipes) |
| Task packs | `evals/task-packs/<slug>/` (prompt, gold, rubric, calibration, Dockerfile) |
| Proxy config | `infra/litellm-config.yaml` |
| Methodology (frozen v0.1) | `docs/02-methodology/` |
| Scoring contract | `docs/04-runbook/08-scoring-contract.md` |

---

## 8. HOW TO CONTINUE (instructions for the next session)

1. **Open a FRESH session** (this one's context is huge). `MEMORY.md` auto-loads; the
note `project_executor_build_plan` has the executor step-by-step.
2. Say something like: *"Build the stack executor per RFC-006, Phase 1"* — or run
`/smith` for a routed plan, or `/restore` to re-sync from git + memory.
3. **Phase 1 (no spend):** write `apps/eval-core-py/src/orchestrator/stack_executor.py`
with a Protocol mock-seam + unit tests. Run the gates (§5.6). Commit on a branch, PR.
4. **Phase 2 (cost checkpoint):** `make docker-up`, build the aider sandbox image, run
`aider × qwen × be_01` → capture a real patch. Confirm the spend first (~pennies).
5. **Phase 3:** patch → Half B evaluators + judges → the first real number. Mind the
be_01 inversion (§4 caveat).
6. Keep merging your open PRs (#36 site, #37 RFC, #24 dynamic-eval) when you've reviewed them.

**Standing constraints:** fewer-larger PRs; methodology-first; OUTWARD actions are human
checkpoints; never `--no-verify`; don't touch `dev/`; design disputes → think it through
(decompose/evaluate/reason) before presenting choices.

---

## 9. References

- **Memory:** `project_executor_build_plan` (the build), `research_cli_harness_execution`
(harness recipes), `feedback_site_design_and_monorepo_gotchas` (env landmines),
`feedback_agent_worktree_gotcha`, `research_swe_rebench`, `research_afterquery_appbench`,
`project_state_2026-06-01` (working snapshot).
- **Forgeplan:** `forgeplan health`; RFC-006 (executor), RFC-002 (judges), ADR-005/008/013,
PRD-002/004/006, EPIC-002 (frozen v0.2 roadmap).
- **Prior handoff:** `docs/handoffs/HANDOFF-2026-05-27.md`.
Loading