Skip to content

feat: add the missing tier-2 CI coverage gates (knowledge, docs, tests)#873

Merged
vivek7405 merged 6 commits into
mainfrom
feat/missing-coverage-gates
Jul 9, 2026
Merged

feat: add the missing tier-2 CI coverage gates (knowledge, docs, tests)#873
vivek7405 merged 6 commits into
mainfrom
feat/missing-coverage-gates

Conversation

@vivek7405

@vivek7405 vivek7405 commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Closes #872

Add the framework-surface tier-2 coverage gates that were missing, mirroring the scaffold gate (#861). Several surfaces are ALREADY gated (types #388, elision lifecycle-coverage, llms.txt #261), so this PR fills only the genuine gaps.

What lands

  1. Knowledge gate (test/knowledge/knowledge-coverage.test.js): reconciles the live webjs check RULES against the troubleshooting page + gotcha docs (a new rule must be explained in a symptom-keyed surface or exempted), and asserts the AGENTS.md headings the MCP init primer sources (DERIVED from mcp-docs.js's sectionByHeading calls) still exist so a rename cannot silently empty the primer. Fills 4 real gaps with troubleshooting entries (use-server-needs-extension, use-server-exports-callable, one-action-per-configured-file, shell-in-non-root-layout); 5 basics/sentinels exempted with reasons.
  2. API docs + test coverage gate (test/api-coverage/api-coverage.test.js): every agent-facing @webjsdev/core + @webjsdev/server export (NON-internal per the scaffold manifest, reused as the single source of truth) must appear in the docs corpus (AGENTS.md + agent-docs + docs site) AND in a test. A new public export shipping undocumented or untested turns CI red. Fills the one real gap (createBrowserTestHandler in agent-docs/testing.md); two aliases documented under their canonical name are exempted.
  3. Docs: agent-docs/framework-dev.md now describes the whole tier-2 coverage-gate FAMILY (scaffold, knowledge, api-coverage, types, elision, llms) so it is discoverable.

Each gate has a pure reconcile() core with synthetic counterfactuals proving every failure mode (new unclassified item, stale exemption, empty reason, renamed init anchor), plus the real-surface assertion. All live under test/**, so scripts/run-node-tests.js runs them in CI.

Honest judgment (same as the scaffold deferred bucket)

I did NOT force contrived entries to zero out the counts. A check rule with no distinctive error signature (a basic like tag-name-has-hyphen, a sentinel like no-scaffold-placeholder) is exempt-with-reason, which is a valid classification, not a gap. An export marked internal: in the scaffold manifest is exempt from docs/test coverage too, because it is framework plumbing an app author never writes.

Verification

  • Gate family: knowledge 8/8, api-coverage 7/7, scaffold 12/12, types 3/3 (30/30 together).
  • Current coverage: 18 check rules (13 explained, 5 exempt), 2 MCP init anchors verified; core 44 / server 58 agent-facing exports all documented + tested.
  • Doc edits do not break test/docs/llms.test.mjs (9/9) and carry no invariant release: bump core/server/cli versions, honest engines fields #11 punctuation.

Definition of done

  • Tests: the three gates + their counterfactuals. Updated.
  • Docs: troubleshooting page (+4 entries), agent-docs/testing.md (createBrowserTestHandler), agent-docs/framework-dev.md (the gate family). Updated.
  • Scaffold / four-app dogfood: N/A because this PR touches only test/** + docs, not packages/(core|server|cli)/src or the served wire.

The first of the missing tier-2 gates. Reconciles the live webjs check
RULES against the troubleshooting page + gotcha docs (a new rule must be
explained or exempted), and asserts the AGENTS.md headings the MCP init
primer sources (derived from mcp-docs.js) still exist so a rename cannot
silently empty the primer. Fills 4 real gaps with troubleshooting entries
(use-server-needs-extension, use-server-exports-callable,
one-action-per-configured-file, shell-in-non-root-layout); 5 basics/
sentinels exempted. Pure reconcile + synthetic counterfactuals.
@vivek7405 vivek7405 self-assigned this Jul 9, 2026
vivek7405 added 4 commits July 9, 2026 16:31
The remaining missing gates. api-coverage.test.js reconciles every
agent-facing @webjsdev/core + @webjsdev/server export (non-internal per
the scaffold manifest, the single source of truth) against the docs
corpus (AGENTS.md + agent-docs + docs site) and the test corpus, failing
when a new public export ships undocumented or untested. Reuses the
scaffold manifest's internal classification so there is no second 229-
entry manifest; only aliases documented under a canonical name are
exempted here. Fills the one real gap (createBrowserTestHandler in
agent-docs/testing.md). Pure reconcile + staleExemptions + counterfactuals.
The gate test + exemption manifest for the docs/test coverage gate
(the prior commit landed only the testing.md fill, since test/coverage/
is gitignored by the coverage/ pattern; renamed to test/api-coverage/).
Document the knowledge gate + api docs/test coverage gate next to the
scaffold gate, and note the pre-existing family members (types, elision,
llms.txt) so the whole set is discoverable.
- createBrowserTestHandler takes a positional appDir string, not { appDir }
  (agent-docs/testing.md was wrong, the doc-gap fill this PR added).
- api-coverage failure messages pointed at test/coverage/ (the pre-rename
  path); correct to test/api-coverage/.
- exclude the coverage gate's own files from the docs/test corpora so a
  counterfactual-hardcoded export name cannot self-satisfy; sharpen the
  comment on the word-match tradeoff (import-based was tried, too strict).
- use-server-exports-callable flags a file exporting no callable function,
  not 'not async'; fix the troubleshooting description.

@vivek7405 vivek7405 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the two gates against the framework source. Four real issues, fixed in 0a4c7cea.

The worst was in the one doc gap this PR was meant to close: I wrote createBrowserTestHandler({ appDir }) but the real signature is positional, createBrowserTestHandler(appDir). Fixed. The api-coverage failure messages still pointed at test/coverage/ from before I renamed the dir past the coverage/ gitignore, so a CI failure would have sent someone to a nonexistent path. And the use-server-exports-callable troubleshooting line said 'not async functions', but the rule actually flags a file that exports no callable at all.

On the word-match weakness the reviewer flagged: common-word export names (cache, html, route) trivially pass the corpus grep, and worse, the gate's own counterfactual literals were self-satisfying because the gate files sat in the tests corpus. Excluded the gate files from both corpora. I did NOT switch to an import-from-@webjsdev match: I measured it and ~55 agent-facing exports are exercised transitively or in the browser suite this corpus omits, so it would have forced a big false-positive exempt list. The word match stays a deliberate lightweight signal (catch a true orphan), with depth left to review, same as the scaffold gate's demo pointers. Documented that in the file.

…sage

Round 2: the reconcileCoverage message was fixed in round 1 but the
sibling staleExemptions message still pointed at the pre-rename
test/coverage/ path. Correct it to test/api-coverage/.

@vivek7405 vivek7405 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second pass. One residual from round 1: I corrected the reconcileCoverage failure-message path but left the sibling staleExemptions message pointing at the old test/coverage/ dir. Fixed in fe4c9850, and grepped to confirm no other stale test/coverage/ reference remains. Everything else verified against source: the createBrowserTestHandler signature, the corpus skip (matches only the two gate dirs, over-matches nothing), no export living only in a gate file, and all 5 knowledge exemptions honest.

@vivek7405 vivek7405 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Third pass to confirm convergence. No stale test/coverage paths remain anywhere, both manifests are valid JSON with no empty or stale exemptions, run-node-tests discovers both gate dirs, and I re-simulated both gates against the branch corpora: 0 doc gaps, 0 test gaps, 18 rules (13 explained / 5 exempt), both MCP anchors resolve. The added troubleshooting entries and the createBrowserTestHandler signature all check out against source. Clean.

@vivek7405 vivek7405 marked this pull request as ready for review July 9, 2026 11:23
@vivek7405 vivek7405 merged commit 22bfee0 into main Jul 9, 2026
10 checks passed
@vivek7405 vivek7405 deleted the feat/missing-coverage-gates branch July 9, 2026 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add the missing tier-2 CI coverage gates (knowledge, docs, tests)

1 participant