Skip to content

evals: the state snapshot reads the active lens's Discover wishlist - #246

Merged
milstan merged 3 commits into
mainfrom
milstan/fix-product-4147
Sep 16, 2026
Merged

milstan merged 3 commits into
mainfrom
milstan/fix-product-4147

Conversation

@milstan

@milstan milstan commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Fixes leadbay/product#4147.

snapshot() in .github/evals/journey.py captured /users/me, /lenses and
/monitor. A case that declares Discover leads had no record of them in evidence
of kind state-before, so prepared-state could only come back
insufficient-evidence. On evals/cases/tour-plan-city-only, staging-us,
2026-09-15, the verifier found the 30 wishlist leads in backend.ndjson and could
not cite them, because a citation of the wrong kind cannot substitute.

The snapshot now also reads GET /lenses/{active}/leads/wishlist?count=50&page=0&contacts=true.
The active lens is resolved the way the MCP resolves it in
client.ts:resolveDefaultLens: /users/me.last_requested_lens, else
is_last_active, is_default, first lens. No new evidence kind, no schema change,
no case edits.

Verified

snapshot() run live against four tenants, before and after:

Environment Tenant state-before.json before after
staging-us LeadBay 6186bfbd 3 keys, no city record + /lenses/4104/leads/wishlist, 50 leads, Austin 1, City of New York 11, Seattle, Miami, Los Angeles
staging-fr SOLMUR 35a1731a 3 keys + /lenses/6201/…, 50 leads of 58
prod-us LeadBay c7523409 3 keys + /lenses/21580/…, 50 leads of 60
prod-fr LEADBAY f683f0f1 3 keys + /lenses/5885/…, 50 leads

A full live journey of tour-plan-city-only on staging-us is running against this
branch; its prepared-state verdict will be posted here.

pnpm -r test 959 passed, pnpm -r typecheck clean, pnpm test:eval validated 32 cases.

Not done here

No version bump, at Milan's instruction. The release gate only starts journeys on
a bump, so CI runs no journey for this PR.

🤖 Generated with Claude Code

snapshot() captured /users/me, /lenses and /monitor only, so a case that
declares Discover leads — their cities, their scores — had no record of them
in evidence of kind state-before, and `prepared-state` could never pass.
Seen on evals/cases/tour-plan-city-only, staging-us, 2026-09-15: the verifier
found the 30 wishlist leads in backend.ndjson and could not cite them, because
a citation of the wrong kind cannot substitute.

The snapshot now reads GET /lenses/{active}/leads/wishlist for the lens
leadbay_pull_leads would read, resolved the way the MCP resolves it:
/users/me.last_requested_lens, else is_last_active, is_default, first.

Fixes leadbay/product#4147

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small, well-scoped fix — adds the active lens's Discover wishlist to the evidence snapshot so prepared-state can be verified. Infra-only change in .github/evals/journey.py, consistent with the eval-infra rules in CLAUDE.md. One logic note on active_lens's fallback resolution order below; otherwise looks good.

Comment thread .github/evals/journey.py Outdated
client.ts:resolveDefaultLens finds the first lens matching is_default OR
default in list order. Two separate passes let an is_default lens later in
the list beat a default lens earlier in it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@milstan

milstan commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

[Claude]: The live journey finished. tour-plan-city-only on staging-us, org 6186bfbd, against this branch: 9 of 9 criteria pass, gate exit 0.

prepared-state now cites the new record:

Before the product session started, the tenant's active lens (4104) held a populated Discover wishlist of 54 leads spanning many distinct cities (New York, Fairfield, Carson, San Jose, Bismarck, ... and one lead in Austin, Texas itself) ... This satisfies both declared starting conditions.

Citation: observed/state-before.json, /lenses/4104/leads/wishlist?count=50&page=0&contacts=true/body/pagination/total = 54, requirement before.

The same case returned insufficient-evidence on prepared-state on 2026-09-15, quoted in leadbay/product#4147.

The run started before b7152bf, so it exercised the first commit's active_lens. That tenant's /users/me.last_requested_lens is 4104, which short-circuits before the fallback b7152bf changed, so the wishlist read was identical.

@milstan
milstan force-pushed the milstan/fix-product-4147 branch from 34df7f0 to bf3d2bb Compare September 16, 2026 17:37
@milstan
milstan merged commit 7d23fbf into main Sep 16, 2026
7 checks passed
@milstan
milstan deleted the milstan/fix-product-4147 branch September 16, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant