Fix eval catalogue listing missing evals/test_*.test.ts files - #12
Fix eval catalogue listing missing evals/test_*.test.ts files#12wyvern8 wants to merge 2 commits into
Conversation
…ic tests Discovery looked at the legacy plugins/*/evals/<kind>s/ tree, so the manifest kept empty coverage (or expected evals/test_*.test.ts) while JSON suites already lived next to each primitive. stampVitestPerPrimitive also prefixed kind twice onto kind-prefixed slugs, writing test_skill_skill_* files that regenerateVitest never looked for. Resolve eval_files from co-located JSON first, and emit evals/test_<kind>_<name>.test.ts from one shared basename helper. Co-authored-by: andrew.smartcreations <andrew.smartcreations@gmail.com>
Fill 38 empty eval_files rows with the JSON suites already on disk so the catalogue no longer implies missing generated evals/test_*.test.ts coverage for commands, agents, and hooks. Co-authored-by: andrew.smartcreations <andrew.smartcreations@gmail.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Approved. Cursor Bugbot was not running after the first check poll, so that signal was skipped; remaining checks that finished are clean and no approval policy requires human review. No reviewers were assigned.
Sent by Cursor Approval Agent: Pull Request Approver
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 12c70eb769
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const underCursor = | ||
| ownerPosix.endsWith("/.cursor") || ownerPosix.endsWith(".cursor"); |
There was a problem hiding this comment.
When a valid plugin directory name ends in .cursor (periods are accepted by pluginNamePattern) and only its legacy hook eval exists at plugins/<plugin>/evals/hooks/<plugin>.json, this suffix check misclassifies the plugin as the workspace .cursor directory and instead searches for hooks.json. Discovery then reports the hook as uncovered even though its eval file exists; compare the directory basename exactly with .cursor.
Useful? React with 👍 / 👎.
2 participants
Problem
The eval catalogue still treated generated
evals/test_*.test.tsfiles as coverage for primitives whose JSON suites already exist. Two bugs caused that:plugins/<plugin>/evals/<kind>s/<name>.json(and.cursor/evals/...). After the JSON-first migration, suites live at<kind-dir>/evals/<name>.json. Rediscovery therefore wrote emptyeval_filesand inventories reported hundreds of missing static tests.stampVitestPerPrimitivedouble-prefixed kind onto slugs frombuildPrimitiveMetaFromPayload(skill_nab-testing→test_skill_skill_nab-testing.test.ts).regenerateVitestcatalogued the single-kind name, so re-stamp never produced the file the catalogue listed.Fix
eval_filesfrom co-located JSON first, then the legacy tree. Never invent a path that is not on disk.evals/test_<kind>_<name>.test.tsfrom one sharedvitestStaticTestBasenamehelper used by the stamper andregenerateVitest.Verification
vitestcoverage for basename helper, co-located vs legacy discovery, and stamp path.eval:discovernow reportswith_coverage: 53and zeroevals/test_*.test.tscatalogue rows.eval:update --no-analyserdry-run: 0 critical deltas.Hosts that still use
static.framework: vitestshould re-stamp after this lands so files appear at the catalogued single-kind paths.