test(facts-eval): reconcile the harness to the exact-by-name contract - #64
Merged
Conversation
The facts matcher was NOT changed by this PR. `resolvePackage` has been exact-by-name (normalized lowercase+trim) since 7f15673, which deliberately closed a fabrication class (fuzzy/token matching returned a DIFFERENT package's facts as authoritative — e.g. "express rate limit" -> express). That commit scoped harness reconciliation out; a later refactor rewired lookupFacts onto the exact path, silently changing what the scorer measured. The baseline was never updated, so the bench had been reporting a phantom "regression" (recall 0.52 vs a 0.84 baseline from the retired substring matcher). This reconciles the eval harness to the contract the matcher actually implements: - Split the old `fuzzy-positive` kind: the 5 case/whitespace variants that reduce to an exact key (lowercase+trim — the only tolerance the by-name path offers) become `true-positive`; the 15 NL/fuzzy/surface-form queries become a new `search-scope` kind (expected: null — silence IS correct; resolution is starlog_search's job). Each search-scope case names its intended package in the rationale so nothing is hidden. - `search-scope` is scored in its OWN bucket and EXCLUDED from hit_recall / positive_precision / hard_negative_fp_rate; a new fabrication-regression gate fails the bench if the by-name path ever starts resolving one of them. - Fixed stale negative labels the corpus outgrew: axios/express/request-promise were added to L2, so their negative cases were factually broken. Swapped in genuinely out-of-corpus packages (react, webpack) and an equivalent substring trap (express-session), preserving the negative-set size. Re-kinded three NL queries about now-in-corpus packages (lodash, moment-vs-date-fns, moment-or-date-fns) from out-of-corpus/hard-negative to search-scope. - Re-baselined to the reconciled numbers. On "100% recall": this is a near-tautology of deterministic exact matching on exact-name inputs, NOT an achievement — do not read it as "recall improved." The harness's real teeth now live in the negative (35 hard-negative + 7 out-of-corpus) and fabrication (18 search-scope) buckets. It would fail on exactly three real regressions: (1) a bare corpus name stops resolving, (2) a hard-negative / out-of-corpus trap starts resolving (false positive), (3) a search-scope query starts resolving (fabrication — the class 7f15673 closed). Full suite green (606 passed); typecheck clean; bench PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The facts matcher is unchanged by this PR.
resolvePackagehas been exact-by-name (normalized lowercase+trim) since7f15673, which deliberately closed a fabrication class — fuzzy/token matching returned a different package's facts as authoritative ("express rate limit"→express,@starloghq/facts-schema→q). That commit scoped harness reconciliation out; a later refactor rewiredlookupFactsonto the exact path, silently changing what the scorer measured, and the baseline was never updated. Sobench:factshad been reporting a phantom regression (recall 0.52 vs a 0.84 baseline left over from the retired substring matcher).This reconciles the eval harness to the contract the matcher actually implements — no matcher change.
Changes
fuzzy-positivekind. The 5 case/whitespace variants that reduce to an exact key (lowercase+trim — the only tolerance the by-name path offers) →true-positive. The 15 NL/fuzzy/surface-form queries → a newsearch-scopekind (expected: null— silence is correct; resolution isstarlog_search's job). Each names its intended package in the rationale, so nothing is hidden.search-scopeis scored in its own bucket and EXCLUDED fromhit_recall/positive_precision/hard_negative_fp_rate. A new fabrication-regression gate fails the bench if the by-name path ever starts resolving one of them.axios/express/request-promisewere added to L2, so their negative cases were factually broken (the bench flagged them as FPs). Swapped in genuinely out-of-corpus packages (react,webpack) and an equivalent substring trap (express-session), preserving the negative-set size. Re-kinded three NL queries about now-in-corpus packages (lodash,moment vs date-fns,moment or date-fns) →search-scope.On "100% recall"
This is a near-tautology of deterministic exact matching on exact-name inputs — not an achievement, and not "recall improved." The harness's real teeth now live in the negative (35 hard-negative + 7 out-of-corpus) and fabrication (18 search-scope) buckets.
What would this harness now fail on? Exactly three real regressions:
7f15673closed).Scorecard (reconciled)
Verification
Full suite green (606 passed),
typecheckclean,npm run bench:factsPASS against the new baseline.🤖 Generated with Claude Code