Skip to content

add the missing tier-2 CI coverage gates (knowledge, docs, tests) #872

Description

@vivek7405

Problem

The scaffold now has a tier-2 CI coverage gate (#861): a test that reconciles the LIVE framework surface against a manifest and fails when a new export/convention ships undemoed. The framework's OTHER primary agent-teaching surfaces have no such gate, so they can rot silently behind a new feature:

  • The MCP init primer sources two AGENTS.md sections by heading regex (packages/mcp/src/mcp-docs.js:181-182: /^##\s+Execution model/im and /^##\s+Invariants/im). A heading rename makes init silently emit an empty section, and nothing catches it.
  • The muscle-memory gotcha docs (agent-docs/nextjs-muscle-memory-gotchas.md, agent-docs/lit-muscle-memory-gotchas.md) and the troubleshooting page (docs/app/docs/troubleshooting/page.ts, keyed by symptom to an invariant + check rule) are the surfaces that counter agents' wrong priors, which is where agents actually fail. When a new webjs check correctness rule or a new numbered Invariant is added, nothing forces a matching "you will reach for X, do Y instead" / troubleshooting entry, so the counter-wrong-prior surface drifts behind the enforcement.

This is the same class of gap #848 was for scaffolds: the knowledge exists in one place (the rule, the invariant) but the teaching surface that an agent reads is not guaranteed to keep up.

Design / approach

Mirror the scaffold gate exactly (tier-2, CI-executed, un-skippable). Add a test (e.g. test/knowledge/knowledge-coverage.test.js) that:

  1. MCP init anchor integrity. Assert every AGENTS.md heading the MCP init primer sources still exists (derive the heading regexes from mcp-docs.js, or assert the known set ## Execution model, ## Invariants), so a rename fails CI instead of silently emptying the primer.
  2. Check-rule -> explanation coverage. Import the LIVE RULES from packages/server/src/check.js (export const RULES, ~19 rules, each { name }). Reconcile every rule name against a hand-curated test/knowledge/knowledge-coverage.json manifest: each rule is either { explained: <where> } (its name appears in the troubleshooting page or a gotcha doc) or { exempt: <reason> }. A new rule with no classification fails CI.
  3. Invariant -> gotcha coverage. Parse the numbered Invariants section of AGENTS.md (the ## Invariants block, currently 11 items) and reconcile each against the manifest the same way (explained in a gotcha doc / troubleshooting, or exempt).
  4. A pure reconcile() core with SYNTHETIC counterfactuals (new rule, new invariant, stale entry, missing explanation, renamed init anchor), plus the real-surface assertions.

Fill any gaps the gate surfaces (a check rule or invariant with no troubleshooting/gotcha entry gets one, or a justified exemption).

Implementation notes (for the implementing agent)

  • Check rules: packages/server/src/check.js export const RULES = [{ name: '...' }, ...]. Import it directly (or via @webjsdev/server/check). The MCP check-report projector (packages/mcp/src/check-report.js) also reads it if a stable projection is preferred.
  • MCP init: packages/mcp/src/mcp-docs.js sectionByHeading(md, headingRe) at :148, sourced anchors at :181-182. The gate should derive/assert those exact heading regexes.
  • Invariants: AGENTS.md ## Invariants section (numbered N. **...**, currently 11). Note AGENTS.md has OTHER numbered lists (workflow items), so anchor the parse to the ## Invariants heading, not any ^\d+\. line.
  • Gotcha docs: agent-docs/{nextjs,lit}-muscle-memory-gotchas.md. Troubleshooting: docs/app/docs/troubleshooting/page.ts.
  • Test discovery: scripts/run-node-tests.js walks test/**, so a test/knowledge/*.test.js file runs in CI automatically (no wiring).
  • Landmine: not every check rule maps 1:1 to a gotcha (some are pure lints with no muscle-memory prior). Those are legitimately { exempt } with a reason. Keep the exemption honest, do not force contrived gotcha entries (same lesson as the scaffold gate's deferred bucket).
  • Prose-punctuation invariant release: bump core/server/cli versions, honest engines fields #11 applies to all new content.
  • Docs surfaces to sync: agent-docs/framework-dev.md (add a section describing this gate next to the scaffold-gate section), and the webjs-doc-sync skill map if it should own this surface.

Acceptance criteria

  • A new webjs check rule fails CI until it is explained (troubleshooting/gotcha) or exempted
  • A new numbered Invariant fails CI until it is covered or exempted
  • Renaming an AGENTS.md heading the MCP init sources fails CI (counterfactual proves it)
  • Pure reconcile() with synthetic counterfactuals for every failure mode
  • Any gap the gate surfaces on first run is filled or honestly exempted
  • agent-docs/framework-dev.md documents the gate; runs under npm test in CI

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions