You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The scaffold now has a tier-2 CI coverage gate (#861): a test that reconciles the LIVE framework surface against a manifest and fails when a new export/convention ships undemoed. The framework's OTHER primary agent-teaching surfaces have no such gate, so they can rot silently behind a new feature:
The MCP init primer sources two AGENTS.md sections by heading regex (packages/mcp/src/mcp-docs.js:181-182: /^##\s+Execution model/im and /^##\s+Invariants/im). A heading rename makes init silently emit an empty section, and nothing catches it.
The muscle-memory gotcha docs (agent-docs/nextjs-muscle-memory-gotchas.md, agent-docs/lit-muscle-memory-gotchas.md) and the troubleshooting page (docs/app/docs/troubleshooting/page.ts, keyed by symptom to an invariant + check rule) are the surfaces that counter agents' wrong priors, which is where agents actually fail. When a new webjs check correctness rule or a new numbered Invariant is added, nothing forces a matching "you will reach for X, do Y instead" / troubleshooting entry, so the counter-wrong-prior surface drifts behind the enforcement.
This is the same class of gap #848 was for scaffolds: the knowledge exists in one place (the rule, the invariant) but the teaching surface that an agent reads is not guaranteed to keep up.
Design / approach
Mirror the scaffold gate exactly (tier-2, CI-executed, un-skippable). Add a test (e.g. test/knowledge/knowledge-coverage.test.js) that:
MCP init anchor integrity. Assert every AGENTS.md heading the MCP init primer sources still exists (derive the heading regexes from mcp-docs.js, or assert the known set ## Execution model, ## Invariants), so a rename fails CI instead of silently emptying the primer.
Check-rule -> explanation coverage. Import the LIVE RULES from packages/server/src/check.js (export const RULES, ~19 rules, each { name }). Reconcile every rule name against a hand-curated test/knowledge/knowledge-coverage.json manifest: each rule is either { explained: <where> } (its name appears in the troubleshooting page or a gotcha doc) or { exempt: <reason> }. A new rule with no classification fails CI.
Invariant -> gotcha coverage. Parse the numbered Invariants section of AGENTS.md (the ## Invariants block, currently 11 items) and reconcile each against the manifest the same way (explained in a gotcha doc / troubleshooting, or exempt).
A pure reconcile() core with SYNTHETIC counterfactuals (new rule, new invariant, stale entry, missing explanation, renamed init anchor), plus the real-surface assertions.
Fill any gaps the gate surfaces (a check rule or invariant with no troubleshooting/gotcha entry gets one, or a justified exemption).
Implementation notes (for the implementing agent)
Check rules: packages/server/src/check.jsexport const RULES = [{ name: '...' }, ...]. Import it directly (or via @webjsdev/server/check). The MCP check-report projector (packages/mcp/src/check-report.js) also reads it if a stable projection is preferred.
MCP init: packages/mcp/src/mcp-docs.jssectionByHeading(md, headingRe) at :148, sourced anchors at :181-182. The gate should derive/assert those exact heading regexes.
Invariants: AGENTS.md## Invariants section (numbered N. **...**, currently 11). Note AGENTS.md has OTHER numbered lists (workflow items), so anchor the parse to the ## Invariants heading, not any ^\d+\. line.
Test discovery: scripts/run-node-tests.js walks test/**, so a test/knowledge/*.test.js file runs in CI automatically (no wiring).
Landmine: not every check rule maps 1:1 to a gotcha (some are pure lints with no muscle-memory prior). Those are legitimately { exempt } with a reason. Keep the exemption honest, do not force contrived gotcha entries (same lesson as the scaffold gate's deferred bucket).
Docs surfaces to sync: agent-docs/framework-dev.md (add a section describing this gate next to the scaffold-gate section), and the webjs-doc-sync skill map if it should own this surface.
Acceptance criteria
A new webjs check rule fails CI until it is explained (troubleshooting/gotcha) or exempted
A new numbered Invariant fails CI until it is covered or exempted
Renaming an AGENTS.md heading the MCP init sources fails CI (counterfactual proves it)
Pure reconcile() with synthetic counterfactuals for every failure mode
Any gap the gate surfaces on first run is filled or honestly exempted
agent-docs/framework-dev.md documents the gate; runs under npm test in CI
Problem
The scaffold now has a tier-2 CI coverage gate (#861): a test that reconciles the LIVE framework surface against a manifest and fails when a new export/convention ships undemoed. The framework's OTHER primary agent-teaching surfaces have no such gate, so they can rot silently behind a new feature:
initprimer sources two AGENTS.md sections by heading regex (packages/mcp/src/mcp-docs.js:181-182:/^##\s+Execution model/imand/^##\s+Invariants/im). A heading rename makesinitsilently emit an empty section, and nothing catches it.agent-docs/nextjs-muscle-memory-gotchas.md,agent-docs/lit-muscle-memory-gotchas.md) and the troubleshooting page (docs/app/docs/troubleshooting/page.ts, keyed by symptom to an invariant + check rule) are the surfaces that counter agents' wrong priors, which is where agents actually fail. When a newwebjs checkcorrectness rule or a new numbered Invariant is added, nothing forces a matching "you will reach for X, do Y instead" / troubleshooting entry, so the counter-wrong-prior surface drifts behind the enforcement.This is the same class of gap #848 was for scaffolds: the knowledge exists in one place (the rule, the invariant) but the teaching surface that an agent reads is not guaranteed to keep up.
Design / approach
Mirror the scaffold gate exactly (tier-2, CI-executed, un-skippable). Add a test (e.g.
test/knowledge/knowledge-coverage.test.js) that:initprimer sources still exists (derive the heading regexes frommcp-docs.js, or assert the known set## Execution model,## Invariants), so a rename fails CI instead of silently emptying the primer.RULESfrompackages/server/src/check.js(export const RULES, ~19 rules, each{ name }). Reconcile every rule name against a hand-curatedtest/knowledge/knowledge-coverage.jsonmanifest: each rule is either{ explained: <where> }(its name appears in the troubleshooting page or a gotcha doc) or{ exempt: <reason> }. A new rule with no classification fails CI.## Invariantsblock, currently 11 items) and reconcile each against the manifest the same way (explainedin a gotcha doc / troubleshooting, orexempt).reconcile()core with SYNTHETIC counterfactuals (new rule, new invariant, stale entry, missing explanation, renamed init anchor), plus the real-surface assertions.Fill any gaps the gate surfaces (a check rule or invariant with no troubleshooting/gotcha entry gets one, or a justified exemption).
Implementation notes (for the implementing agent)
packages/server/src/check.jsexport const RULES = [{ name: '...' }, ...]. Import it directly (or via@webjsdev/server/check). The MCPcheck-reportprojector (packages/mcp/src/check-report.js) also reads it if a stable projection is preferred.packages/mcp/src/mcp-docs.jssectionByHeading(md, headingRe)at :148, sourced anchors at :181-182. The gate should derive/assert those exact heading regexes.AGENTS.md## Invariantssection (numberedN. **...**, currently 11). Note AGENTS.md has OTHER numbered lists (workflow items), so anchor the parse to the## Invariantsheading, not any^\d+\.line.agent-docs/{nextjs,lit}-muscle-memory-gotchas.md. Troubleshooting:docs/app/docs/troubleshooting/page.ts.scripts/run-node-tests.jswalkstest/**, so atest/knowledge/*.test.jsfile runs in CI automatically (no wiring).{ exempt }with a reason. Keep the exemption honest, do not force contrived gotcha entries (same lesson as the scaffold gate's deferred bucket).agent-docs/framework-dev.md(add a section describing this gate next to the scaffold-gate section), and thewebjs-doc-syncskill map if it should own this surface.Acceptance criteria
webjs checkrule fails CI until it is explained (troubleshooting/gotcha) or exemptedagent-docs/framework-dev.mddocuments the gate; runs undernpm testin CI