feat: add the missing tier-2 CI coverage gates (knowledge, docs, tests)#873
Conversation
The first of the missing tier-2 gates. Reconciles the live webjs check RULES against the troubleshooting page + gotcha docs (a new rule must be explained or exempted), and asserts the AGENTS.md headings the MCP init primer sources (derived from mcp-docs.js) still exist so a rename cannot silently empty the primer. Fills 4 real gaps with troubleshooting entries (use-server-needs-extension, use-server-exports-callable, one-action-per-configured-file, shell-in-non-root-layout); 5 basics/ sentinels exempted. Pure reconcile + synthetic counterfactuals.
The remaining missing gates. api-coverage.test.js reconciles every agent-facing @webjsdev/core + @webjsdev/server export (non-internal per the scaffold manifest, the single source of truth) against the docs corpus (AGENTS.md + agent-docs + docs site) and the test corpus, failing when a new public export ships undocumented or untested. Reuses the scaffold manifest's internal classification so there is no second 229- entry manifest; only aliases documented under a canonical name are exempted here. Fills the one real gap (createBrowserTestHandler in agent-docs/testing.md). Pure reconcile + staleExemptions + counterfactuals.
The gate test + exemption manifest for the docs/test coverage gate (the prior commit landed only the testing.md fill, since test/coverage/ is gitignored by the coverage/ pattern; renamed to test/api-coverage/).
Document the knowledge gate + api docs/test coverage gate next to the scaffold gate, and note the pre-existing family members (types, elision, llms.txt) so the whole set is discoverable.
- createBrowserTestHandler takes a positional appDir string, not { appDir }
(agent-docs/testing.md was wrong, the doc-gap fill this PR added).
- api-coverage failure messages pointed at test/coverage/ (the pre-rename
path); correct to test/api-coverage/.
- exclude the coverage gate's own files from the docs/test corpora so a
counterfactual-hardcoded export name cannot self-satisfy; sharpen the
comment on the word-match tradeoff (import-based was tried, too strict).
- use-server-exports-callable flags a file exporting no callable function,
not 'not async'; fix the troubleshooting description.
vivek7405
left a comment
There was a problem hiding this comment.
Reviewed the two gates against the framework source. Four real issues, fixed in 0a4c7cea.
The worst was in the one doc gap this PR was meant to close: I wrote createBrowserTestHandler({ appDir }) but the real signature is positional, createBrowserTestHandler(appDir). Fixed. The api-coverage failure messages still pointed at test/coverage/ from before I renamed the dir past the coverage/ gitignore, so a CI failure would have sent someone to a nonexistent path. And the use-server-exports-callable troubleshooting line said 'not async functions', but the rule actually flags a file that exports no callable at all.
On the word-match weakness the reviewer flagged: common-word export names (cache, html, route) trivially pass the corpus grep, and worse, the gate's own counterfactual literals were self-satisfying because the gate files sat in the tests corpus. Excluded the gate files from both corpora. I did NOT switch to an import-from-@webjsdev match: I measured it and ~55 agent-facing exports are exercised transitively or in the browser suite this corpus omits, so it would have forced a big false-positive exempt list. The word match stays a deliberate lightweight signal (catch a true orphan), with depth left to review, same as the scaffold gate's demo pointers. Documented that in the file.
…sage Round 2: the reconcileCoverage message was fixed in round 1 but the sibling staleExemptions message still pointed at the pre-rename test/coverage/ path. Correct it to test/api-coverage/.
vivek7405
left a comment
There was a problem hiding this comment.
Second pass. One residual from round 1: I corrected the reconcileCoverage failure-message path but left the sibling staleExemptions message pointing at the old test/coverage/ dir. Fixed in fe4c9850, and grepped to confirm no other stale test/coverage/ reference remains. Everything else verified against source: the createBrowserTestHandler signature, the corpus skip (matches only the two gate dirs, over-matches nothing), no export living only in a gate file, and all 5 knowledge exemptions honest.
vivek7405
left a comment
There was a problem hiding this comment.
Third pass to confirm convergence. No stale test/coverage paths remain anywhere, both manifests are valid JSON with no empty or stale exemptions, run-node-tests discovers both gate dirs, and I re-simulated both gates against the branch corpora: 0 doc gaps, 0 test gaps, 18 rules (13 explained / 5 exempt), both MCP anchors resolve. The added troubleshooting entries and the createBrowserTestHandler signature all check out against source. Clean.
Closes #872
Add the framework-surface tier-2 coverage gates that were missing, mirroring the scaffold gate (#861). Several surfaces are ALREADY gated (types #388, elision
lifecycle-coverage, llms.txt #261), so this PR fills only the genuine gaps.What lands
test/knowledge/knowledge-coverage.test.js): reconciles the livewebjs checkRULESagainst the troubleshooting page + gotcha docs (a new rule must be explained in a symptom-keyed surface or exempted), and asserts the AGENTS.md headings the MCPinitprimer sources (DERIVED frommcp-docs.js'ssectionByHeadingcalls) still exist so a rename cannot silently empty the primer. Fills 4 real gaps with troubleshooting entries (use-server-needs-extension,use-server-exports-callable,one-action-per-configured-file,shell-in-non-root-layout); 5 basics/sentinels exempted with reasons.test/api-coverage/api-coverage.test.js): every agent-facing@webjsdev/core+@webjsdev/serverexport (NON-internal per the scaffold manifest, reused as the single source of truth) must appear in the docs corpus (AGENTS.md + agent-docs + docs site) AND in a test. A new public export shipping undocumented or untested turns CI red. Fills the one real gap (createBrowserTestHandlerinagent-docs/testing.md); two aliases documented under their canonical name are exempted.agent-docs/framework-dev.mdnow describes the whole tier-2 coverage-gate FAMILY (scaffold, knowledge, api-coverage, types, elision, llms) so it is discoverable.Each gate has a pure
reconcile()core with synthetic counterfactuals proving every failure mode (new unclassified item, stale exemption, empty reason, renamed init anchor), plus the real-surface assertion. All live undertest/**, soscripts/run-node-tests.jsruns them in CI.Honest judgment (same as the scaffold
deferredbucket)I did NOT force contrived entries to zero out the counts. A check rule with no distinctive error signature (a basic like
tag-name-has-hyphen, a sentinel likeno-scaffold-placeholder) is exempt-with-reason, which is a valid classification, not a gap. An export markedinternal:in the scaffold manifest is exempt from docs/test coverage too, because it is framework plumbing an app author never writes.Verification
test/docs/llms.test.mjs(9/9) and carry no invariant release: bump core/server/cli versions, honest engines fields #11 punctuation.Definition of done
agent-docs/testing.md(createBrowserTestHandler),agent-docs/framework-dev.md(the gate family). Updated.test/**+ docs, notpackages/(core|server|cli)/srcor the served wire.