Skip to content

test(facts-eval): reconcile the harness to the exact-by-name contract - #64

Merged
basicScandal merged 1 commit into
mainfrom
chore/facts-eval-reconcile
Jul 28, 2026
Merged

test(facts-eval): reconcile the harness to the exact-by-name contract#64
basicScandal merged 1 commit into
mainfrom
chore/facts-eval-reconcile

Conversation

@basicScandal

Copy link
Copy Markdown
Member

What & why

The facts matcher is unchanged by this PR. resolvePackage has been exact-by-name (normalized lowercase+trim) since 7f15673, which deliberately closed a fabrication class — fuzzy/token matching returned a different package's facts as authoritative ("express rate limit"express, @starloghq/facts-schemaq). That commit scoped harness reconciliation out; a later refactor rewired lookupFacts onto the exact path, silently changing what the scorer measured, and the baseline was never updated. So bench:facts had been reporting a phantom regression (recall 0.52 vs a 0.84 baseline left over from the retired substring matcher).

This reconciles the eval harness to the contract the matcher actually implements — no matcher change.

Changes

  • Split the old fuzzy-positive kind. The 5 case/whitespace variants that reduce to an exact key (lowercase+trim — the only tolerance the by-name path offers) → true-positive. The 15 NL/fuzzy/surface-form queries → a new search-scope kind (expected: null — silence is correct; resolution is starlog_search's job). Each names its intended package in the rationale, so nothing is hidden.
  • search-scope is scored in its own bucket and EXCLUDED from hit_recall / positive_precision / hard_negative_fp_rate. A new fabrication-regression gate fails the bench if the by-name path ever starts resolving one of them.
  • Fixed stale negative labels the corpus outgrew. axios/express/request-promise were added to L2, so their negative cases were factually broken (the bench flagged them as FPs). Swapped in genuinely out-of-corpus packages (react, webpack) and an equivalent substring trap (express-session), preserving the negative-set size. Re-kinded three NL queries about now-in-corpus packages (lodash, moment vs date-fns, moment or date-fns) → search-scope.
  • Re-baselined to the reconciled numbers.

On "100% recall"

This is a near-tautology of deterministic exact matching on exact-name inputs — not an achievement, and not "recall improved." The harness's real teeth now live in the negative (35 hard-negative + 7 out-of-corpus) and fabrication (18 search-scope) buckets.

What would this harness now fail on? Exactly three real regressions:

  1. a bare corpus name stops resolving,
  2. a hard-negative / out-of-corpus trap starts resolving (false positive),
  3. a search-scope query starts resolving (fabrication — the class 7f15673 closed).

Scorecard (reconciled)

hit_recall            : 100.0%
positive_precision    : 100.0%
hard_negative_fp_rate :   0.0%
search-scope (excluded): 18/18 correctly silent
per-kind: true-positive 16, hard-negative 35, out-of-corpus 7, robustness 29, search-scope 18  (=105)

Verification

Full suite green (606 passed), typecheck clean, npm run bench:facts PASS against the new baseline.

🤖 Generated with Claude Code

The facts matcher was NOT changed by this PR. `resolvePackage` has been
exact-by-name (normalized lowercase+trim) since 7f15673, which deliberately
closed a fabrication class (fuzzy/token matching returned a DIFFERENT package's
facts as authoritative — e.g. "express rate limit" -> express). That commit
scoped harness reconciliation out; a later refactor rewired lookupFacts onto the
exact path, silently changing what the scorer measured. The baseline was never
updated, so the bench had been reporting a phantom "regression" (recall 0.52 vs
a 0.84 baseline from the retired substring matcher).

This reconciles the eval harness to the contract the matcher actually implements:

- Split the old `fuzzy-positive` kind: the 5 case/whitespace variants that reduce
  to an exact key (lowercase+trim — the only tolerance the by-name path offers)
  become `true-positive`; the 15 NL/fuzzy/surface-form queries become a new
  `search-scope` kind (expected: null — silence IS correct; resolution is
  starlog_search's job). Each search-scope case names its intended package in the
  rationale so nothing is hidden.
- `search-scope` is scored in its OWN bucket and EXCLUDED from hit_recall /
  positive_precision / hard_negative_fp_rate; a new fabrication-regression gate
  fails the bench if the by-name path ever starts resolving one of them.
- Fixed stale negative labels the corpus outgrew: axios/express/request-promise
  were added to L2, so their negative cases were factually broken. Swapped in
  genuinely out-of-corpus packages (react, webpack) and an equivalent substring
  trap (express-session), preserving the negative-set size. Re-kinded three NL
  queries about now-in-corpus packages (lodash, moment-vs-date-fns,
  moment-or-date-fns) from out-of-corpus/hard-negative to search-scope.
- Re-baselined to the reconciled numbers.

On "100% recall": this is a near-tautology of deterministic exact matching on
exact-name inputs, NOT an achievement — do not read it as "recall improved." The
harness's real teeth now live in the negative (35 hard-negative + 7 out-of-corpus)
and fabrication (18 search-scope) buckets. It would fail on exactly three real
regressions: (1) a bare corpus name stops resolving, (2) a hard-negative /
out-of-corpus trap starts resolving (false positive), (3) a search-scope query
starts resolving (fabrication — the class 7f15673 closed).

Full suite green (606 passed); typecheck clean; bench PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@basicScandal
basicScandal merged commit e197eaf into main Jul 28, 2026
4 checks passed
@basicScandal
basicScandal deleted the chore/facts-eval-reconcile branch July 28, 2026 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant