evals: the state snapshot reads the active lens's Discover wishlist - #246
Conversation
snapshot() captured /users/me, /lenses and /monitor only, so a case that
declares Discover leads — their cities, their scores — had no record of them
in evidence of kind state-before, and `prepared-state` could never pass.
Seen on evals/cases/tour-plan-city-only, staging-us, 2026-09-15: the verifier
found the 30 wishlist leads in backend.ndjson and could not cite them, because
a citation of the wrong kind cannot substitute.
The snapshot now reads GET /lenses/{active}/leads/wishlist for the lens
leadbay_pull_leads would read, resolved the way the MCP resolves it:
/users/me.last_requested_lens, else is_last_active, is_default, first.
Fixes leadbay/product#4147
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Small, well-scoped fix — adds the active lens's Discover wishlist to the evidence snapshot so prepared-state can be verified. Infra-only change in .github/evals/journey.py, consistent with the eval-infra rules in CLAUDE.md. One logic note on active_lens's fallback resolution order below; otherwise looks good.
client.ts:resolveDefaultLens finds the first lens matching is_default OR default in list order. Two separate passes let an is_default lens later in the list beat a default lens earlier in it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
[Claude]: The live journey finished.
Citation: The same case returned The run started before b7152bf, so it exercised the first commit's |
34df7f0 to
bf3d2bb
Compare
Fixes leadbay/product#4147.
snapshot()in.github/evals/journey.pycaptured/users/me,/lensesand/monitor. A case that declares Discover leads had no record of them in evidenceof kind
state-before, soprepared-statecould only come backinsufficient-evidence. Onevals/cases/tour-plan-city-only, staging-us,2026-09-15, the verifier found the 30 wishlist leads in
backend.ndjsonand couldnot cite them, because a citation of the wrong kind cannot substitute.
The snapshot now also reads
GET /lenses/{active}/leads/wishlist?count=50&page=0&contacts=true.The active lens is resolved the way the MCP resolves it in
client.ts:resolveDefaultLens:/users/me.last_requested_lens, elseis_last_active,is_default, first lens. No new evidence kind, no schema change,no case edits.
Verified
snapshot()run live against four tenants, before and after:state-before.jsonbefore6186bfbd/lenses/4104/leads/wishlist, 50 leads, Austin 1, City of New York 11, Seattle, Miami, Los Angeles35a1731a/lenses/6201/…, 50 leads of 58c7523409/lenses/21580/…, 50 leads of 60f683f0f1/lenses/5885/…, 50 leadsA full live journey of
tour-plan-city-onlyon staging-us is running against thisbranch; its
prepared-stateverdict will be posted here.pnpm -r test959 passed,pnpm -r typecheckclean,pnpm test:evalvalidated 32 cases.Not done here
No version bump, at Milan's instruction. The release gate only starts journeys on
a bump, so CI runs no journey for this PR.
🤖 Generated with Claude Code