diff --git a/agent-docs/framework-dev.md b/agent-docs/framework-dev.md index ab018b86..b84d4314 100644 --- a/agent-docs/framework-dev.md +++ b/agent-docs/framework-dev.md @@ -48,6 +48,12 @@ The scaffold is webjs's primary teaching surface for AI agents, so a new framewo **When you add or rename a `@webjsdev/core` or `@webjsdev/server` export, or add a routing convention file the router parses, update the manifest** the same way you write a test: add a demo pointer / `{ demoed: true }`, or an honest exemption. All three surfaces are gated (the convention stems are derived from `packages/server/src/router.js`, so a new `stem === '...'` branch auto-appears and must be classified). +The scaffold gate is one of a FAMILY of tier-2 coverage gates that keep the framework's agent-facing surfaces from rotting behind a new feature. The others: + +- **Knowledge coverage** (`test/knowledge/knowledge-coverage.test.js`): reconciles the live `webjs check` `RULES` against the troubleshooting page + gotcha docs (a new rule must be explained in a symptom-keyed surface or exempted in `knowledge-coverage.json`), and asserts the AGENTS.md headings the MCP `init` primer sources (DERIVED from the `sectionByHeading(agents, /.../)` calls in `packages/mcp/src/mcp-docs.js`) still exist, so a heading rename cannot silently empty the primer. +- **API docs + test coverage** (`test/api-coverage/api-coverage.test.js`): every agent-facing `@webjsdev/core` + `@webjsdev/server` export (NON-internal per the scaffold manifest, the single source of truth) must be referenced in the docs corpus (AGENTS.md + agent-docs + docs site) AND in a test. A new public export that ships undocumented or untested turns CI red. +- **Types** (`test/types/dts-export-coverage.test.mjs`, `server-types.test.mjs`), **elision** (`packages/server/test/elision/lifecycle-coverage.test.js`), and **llms.txt** (`test/docs/llms.test.mjs`) round out the family. Each reads its live surface dynamically so it cannot go stale. + --- ### Changelog: per-package, per-version, auto-generated diff --git a/agent-docs/testing.md b/agent-docs/testing.md index 74a3bc81..89096529 100644 --- a/agent-docs/testing.md +++ b/agent-docs/testing.md @@ -299,6 +299,13 @@ not a test framework: each is a few lines over native `Request` / `Response`, and they reuse the REAL cookie / header names and the REAL wire serializer (so a test exercises the production contract, never a parallel fake). +For a real-browser test that needs the app served to a live page (rather than a +node `Response`), `createBrowserTestHandler(appDir)` from `@webjsdev/server` +is the browser-side counterpart (it takes the app dir as a positional string): +it stands up the same request pipeline behind a listener the web-test-runner +harness drives, so a component test loads the app's actual routes and modules in +Chromium. + ```js import { createRequestHandler } from '@webjsdev/server'; import { testRequest, invokeActionForTest, rawActionRequest, loginAndGetCookies, withSessionCookie } diff --git a/docs/app/docs/troubleshooting/page.ts b/docs/app/docs/troubleshooting/page.ts index caa82615..417341eb 100644 --- a/docs/app/docs/troubleshooting/page.ts +++ b/docs/app/docs/troubleshooting/page.ts @@ -81,6 +81,21 @@ export default function Troubleshooting() {

Cause: a webjs.dev.before / webjs.start.before step (the scaffold ships webjs db migrate) exited non-zero. As of #550, webjs dev / webjs start run these one-shot steps to completion BEFORE serving (replacing the old predev / prestart npm hooks), and they ABORT the boot on a failure so the app never serves a stale client or schema. A bare webjs dev and npm run dev run the same steps, so this is not a "wrong command" problem.

Fix: run the failing command directly to read its real error (for the scaffold's steps, webjs db generate or webjs db migrate), fix the cause (a malformed db/schema.server.ts, an unreachable DATABASE_URL), then re-run. A local binary in a step (drizzle-kit, tailwindcss) resolves under a bare webjs dev the same way npm run resolves it (the ancestor node_modules/.bin dirs are on the step's PATH).

+

A 'use server' file's exports are not callable from a component

+

Symptom: you added 'use server' to a plain .ts file and imported one of its functions into a component, but the call fails or webjs check flags use-server-needs-extension.

+

Cause: the RPC boundary keys on the FILE EXTENSION, not the directive alone. A 'use server' directive in a plain .ts file is a lint violation, because the file router only treats .server.{js,ts} files as the server boundary. Without the extension the source would also be served to the browser.

+

Fix: rename the file to *.server.ts. The extension makes it server-only (source-protected) and the 'use server' directive makes its exports RPC-callable. See Server Actions. The use-server-needs-extension check rule catches this, and use-server-exports-callable flags a 'use server' file that exports no callable function (it registers nothing, so the RPC 404s silently).

+ +

A configured server action file with more than one function is rejected

+

Symptom: webjs check flags one-action-per-configured-file on a *.server.ts that exports a config (method, cache, validate, middleware) alongside two callable functions.

+

Cause: the HTTP-verb and caching config exports (export const method, cache, tags, invalidates, validate, middleware) attach to the ONE action in the file, so a second callable function makes the config ambiguous.

+

Fix: keep one callable function per configured action file (the convention is one function per action file regardless). Split the second function into its own *.server.ts. See Server Actions. The one-action-per-configured-file check rule enforces this.

+ +

A nested layout or page's <html> shell is dropped

+

Symptom: you wrote <!doctype> / <html> / <head> / <body> in a non-root layout or a page and the tags vanish from the output, or webjs check flags shell-in-non-root-layout.

+

Cause: the framework auto-emits the document shell around the whole composition, so only the ROOT layout (app/layout.ts exactly) may write its own shell. A nested shell ends up inside the framework's <body> and the HTML parser drops the stray tags.

+

Fix: remove the shell tags from the nested layout or page and return just the content. Move any <html lang> / <body class> customization to the root layout, which the framework respects and splices its required tags into. See Routing. The shell-in-non-root-layout check rule flags this, framework invariant 8.

+

Still stuck

The framework source is in node_modules/@webjsdev/ with no build step, so what you read is what runs. Grep the relevant file (the SSR pipeline in @webjsdev/server/src/ssr.js, client hydration in @webjsdev/core/src/render-client.js, the convention rules in @webjsdev/server/src/check.js). Run webjs check to surface most of the issues above before they reach the browser, and run webjs check --rules to read what each rule enforces.

`; diff --git a/test/api-coverage/api-coverage.json b/test/api-coverage/api-coverage.json new file mode 100644 index 00000000..b867d755 --- /dev/null +++ b/test/api-coverage/api-coverage.json @@ -0,0 +1,11 @@ +{ + "$comment": "Exemptions for the API docs/test coverage gate (api-coverage.test.js). The gate reuses test/scaffolds/gallery-coverage.json for the INTERNAL classification (an internal export is exempt from docs + test coverage too), so this file only lists the few extra exemptions the scaffold manifest does not express. Keep each reason honest.", + "docsExempt": { + "cookieSession": "alias of cookieSessionStorage; documented under the canonical name (docs/app/docs/sessions)", + "storeSession": "alias of storeSessionStorage; documented under the canonical name (docs/app/docs/sessions)" + }, + "testExempt": { + "cookieSession": "alias of cookieSessionStorage; exercised through the canonical name", + "storeSession": "alias of storeSessionStorage; exercised through the canonical name" + } +} diff --git a/test/api-coverage/api-coverage.test.js b/test/api-coverage/api-coverage.test.js new file mode 100644 index 00000000..6caebb39 --- /dev/null +++ b/test/api-coverage/api-coverage.test.js @@ -0,0 +1,159 @@ +// API docs + test coverage gate (tier-2, un-skippable CI enforcement). +// +// Types are already gated (test/types/dts-export-coverage.test.mjs: every export +// is typed) and demos are gated (the scaffold gate). This adds the two remaining +// coverage dimensions for the public API surface: +// +// 1. DOCS coverage: every agent-facing @webjsdev/core + @webjsdev/server export +// is mentioned in a doc (AGENTS.md, agent-docs/*.md, or the docs site). +// 2. TEST coverage: every agent-facing export is referenced by a test file. +// +// "Agent-facing" = NOT classified `internal:` in the scaffold manifest +// (test/scaffolds/gallery-coverage.json), which is the single source of truth for +// what is framework plumbing vs an API an app author writes. So an export marked +// internal there is automatically exempt here too; this file only carries the few +// extra exemptions (aliases documented/tested under a canonical name). +// +// A new public export that ships undocumented or untested turns CI red. The +// corpus checks are word-boundary greps (a rare new name with zero doc/test +// mention is the signal; common names trivially pass because they are everywhere). +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync, readdirSync } from 'node:fs'; +import { join, dirname, sep } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import * as core from '@webjsdev/core'; +import * as server from '@webjsdev/server'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const REPO = join(__dirname, '..', '..'); +const SCAFFOLD_MANIFEST = JSON.parse(readFileSync(join(REPO, 'test', 'scaffolds', 'gallery-coverage.json'), 'utf8')); +const EXEMPT = JSON.parse(readFileSync(join(__dirname, 'api-coverage.json'), 'utf8')); + +function isInternal(name, section) { + const e = (SCAFFOLD_MANIFEST[section] || {})[name]; + return !!(e && typeof e.exempt === 'string' && e.exempt.startsWith('internal:')); +} + +function readAll(dir, exts, out) { + let entries; + try { entries = readdirSync(dir, { withFileTypes: true }); } catch { return out; } + for (const e of entries) { + if (e.name === 'node_modules' || e.name === '.git') continue; + const f = join(dir, e.name); + // Exclude the coverage gate's OWN files so an export name hardcoded in a + // counterfactual (e.g. 'html', 'signal') does not self-satisfy the corpus. + if (f.includes(`${sep}knowledge${sep}`) || f.includes(`${sep}api-coverage${sep}`)) continue; + if (e.isDirectory()) readAll(f, exts, out); + else if (exts.some((x) => e.name.endsWith(x))) { try { out.push(readFileSync(f, 'utf8')); } catch { /* skip */ } } + } + return out; +} + +// Corpora: built once. +const DOCS = [readFileSync(join(REPO, 'AGENTS.md'), 'utf8')] + .concat(readAll(join(REPO, 'agent-docs'), ['.md'], [])) + .concat(readAll(join(REPO, 'docs', 'app', 'docs'), ['.ts'], [])) + .join('\n'); +const TESTS = readAll(join(REPO, 'test'), ['.js', '.mjs', '.ts'], []) + .concat(readAll(join(REPO, 'packages', 'core', 'test'), ['.js', '.mjs'], [])) + .concat(readAll(join(REPO, 'packages', 'server', 'test'), ['.js', '.mjs'], [])) + .join('\n'); + +// A whole-word match on the export name. This is a deliberately LIGHTWEIGHT +// signal: it reliably catches a NEW export that appears nowhere in docs/tests +// (the case the gate exists for), but it does NOT prove depth of coverage, and a +// common-English-word export name (`cache`, `html`, `json`, `route`, `session`, +// `headers`, `stream`, `render`) trivially passes because the word saturates the +// corpora. Strengthening to an import-from-@webjsdev match was tried and rejected: +// ~55 agent-facing exports are exercised transitively, via subpath imports, or in +// the browser-test suite this corpus omits, so it produced a large false-positive +// exempt list with no real gain. Depth of coverage is a code-review concern, the +// same as with the scaffold gate's demo pointers. +const hasWord = (corpus, name) => new RegExp(`\\b${name}\\b`).test(corpus); + +/** + * Pure reconciliation. For each live export not internal and not exempt, require + * `present(name)` to be true. Returns error strings. + */ +export function reconcileCoverage(liveNames, internalOf, exemptSet, present, kind) { + const errors = []; + for (const name of liveNames) { + if (internalOf(name)) continue; + if (exemptSet.has(name)) continue; + if (!present(name)) { + errors.push(`${kind} gap: agent-facing export "${name}" is not referenced in the ${kind} corpus; document/test it or exempt it (with a reason) in test/api-coverage/api-coverage.json.`); + } + } + return errors; +} + +/** + * Exemptions that no longer name a live export. Validated against the UNION of + * all live export names (the exempt manifest is global, not per-surface), so a + * server-only alias is not flagged stale while auditing the core surface. + */ +export function staleExemptions(exemptNames, allLive) { + const live = new Set(allLive); + return [...exemptNames].filter((n) => !live.has(n)).map((n) => `stale exemption "${n}": no longer a live export, remove it from test/api-coverage/api-coverage.json.`); +} + +const SURFACES = [ + { label: 'core', names: Object.keys(core), section: 'exports' }, + { label: 'server', names: Object.keys(server), section: 'serverExports' }, +]; +const ALL_LIVE = [...Object.keys(core), ...Object.keys(server)]; + +test('every agent-facing export is documented (or exempt)', () => { + const exempt = new Set(Object.keys(EXEMPT.docsExempt || {})); + const errors = staleExemptions(exempt, ALL_LIVE); + for (const s of SURFACES) { + const e = reconcileCoverage(s.names, (n) => isInternal(n, s.section), exempt, (n) => hasWord(DOCS, n), 'docs'); + const agentFacing = s.names.filter((n) => !isInternal(n, s.section)); + console.log(`[api-coverage] ${s.label}: ${agentFacing.length} agent-facing exports, ${e.length} undocumented.`); + errors.push(...e); + } + assert.deepEqual(errors, [], `docs coverage gaps:\n - ${errors.join('\n - ')}`); +}); + +test('every agent-facing export is referenced by a test (or exempt)', () => { + const exempt = new Set(Object.keys(EXEMPT.testExempt || {})); + const errors = staleExemptions(exempt, ALL_LIVE); + for (const s of SURFACES) { + const e = reconcileCoverage(s.names, (n) => isInternal(n, s.section), exempt, (n) => hasWord(TESTS, n), 'test'); + const agentFacing = s.names.filter((n) => !isInternal(n, s.section)); + console.log(`[api-coverage] ${s.label}: ${agentFacing.length} agent-facing exports, ${e.length} untested.`); + errors.push(...e); + } + assert.deepEqual(errors, [], `test coverage gaps:\n - ${errors.join('\n - ')}`); +}); + +// --- counterfactuals --- + +test('reconcileCoverage FAILS on an undocumented agent-facing export', () => { + const errors = reconcileCoverage(['newThing'], () => false, new Set(), () => false, 'docs'); + assert.equal(errors.length, 1); + assert.match(errors[0], /agent-facing export "newThing" is not referenced/); +}); + +test('reconcileCoverage SKIPS an internal export', () => { + const errors = reconcileCoverage(['plumbing'], (n) => n === 'plumbing', new Set(), () => false, 'docs'); + assert.deepEqual(errors, []); +}); + +test('reconcileCoverage SKIPS an exempted export', () => { + const errors = reconcileCoverage(['aliasFn'], () => false, new Set(['aliasFn']), () => false, 'docs'); + assert.deepEqual(errors, []); +}); + +test('staleExemptions FAILS on an exemption that is not a live export', () => { + const errors = staleExemptions(new Set(['gone']), ['html', 'signal']); + assert.equal(errors.length, 1); + assert.match(errors[0], /stale exemption "gone"/); +}); + +test('staleExemptions passes a server-only alias while auditing across surfaces', () => { + // cookieSession is a live server export; it must NOT be flagged stale. + const errors = staleExemptions(new Set(['cookieSession']), ['html', 'cookieSession']); + assert.deepEqual(errors, []); +}); diff --git a/test/knowledge/knowledge-coverage.json b/test/knowledge/knowledge-coverage.json new file mode 100644 index 00000000..bbfa21b3 --- /dev/null +++ b/test/knowledge/knowledge-coverage.json @@ -0,0 +1,10 @@ +{ + "$comment": "Source of truth for agent-knowledge coverage. knowledge-coverage.test.js reconciles the LIVE webjs check RULES against the troubleshooting page + muscle-memory gotcha docs, and asserts the AGENTS.md headings the MCP init primer sources still exist. A new check rule that is neither explained in a symptom-keyed surface nor exempted here FAILS CI, and a renamed AGENTS.md anchor fails CI. Explained = the rule name appears in docs/app/docs/troubleshooting/page.ts or agent-docs/{nextjs,lit}-muscle-memory-gotchas.md. See agent-docs/framework-dev.md.", + "checkRulesExempt": { + "components-have-register": "basic setup requirement (call Class.register('tag')); framework invariant 3, not a distinctive error signature", + "tag-name-has-hyphen": "HTML-spec basic; framework invariant 3, no muscle-memory divergence to counter", + "no-duplicate-tag": "two components claim one tag name, a build-time collision surfaced by the check, not a runtime error signature", + "array-prop-uses-array-type": "shape hint (declare an array prop with Array not Object); the Array-vs-Object prop concept is taught in agent-docs/lit-muscle-memory-gotchas.md", + "no-scaffold-placeholder": "scaffold sentinel that fails check until example content is kept-or-pruned, not a code error signature" + } +} diff --git a/test/knowledge/knowledge-coverage.test.js b/test/knowledge/knowledge-coverage.test.js new file mode 100644 index 00000000..aec199c5 --- /dev/null +++ b/test/knowledge/knowledge-coverage.test.js @@ -0,0 +1,139 @@ +// Agent-knowledge coverage gate (tier-2, un-skippable CI enforcement). +// +// The scaffold gate (test/scaffolds/gallery-coverage.test.js) keeps the DEMO +// surface honest. This is its sibling for the surfaces that counter agents' +// WRONG PRIORS, which is where agents actually fail: the symptom-keyed +// troubleshooting page, the muscle-memory gotcha docs, and the MCP `init` +// primer. Two things are gated: +// +// 1. Every LIVE `webjs check` RULE is explained in an agent-facing surface +// (the troubleshooting page or a gotcha doc, matched by rule name) OR +// exempted with a reason in knowledge-coverage.json. A new rule that ships +// with no "here is the symptom and the fix" turns CI red. +// 2. Every AGENTS.md heading the MCP `init` primer sources (DERIVED from the +// `sectionByHeading(agents, /.../)` calls in packages/mcp/src/mcp-docs.js) +// still matches a heading in AGENTS.md, so a rename cannot silently empty +// the primer. +// +// The reconcile core is pure; failure modes are proven with synthetic inputs. +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync, existsSync } from 'node:fs'; +import { join, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { RULES } from '../../packages/server/src/check.js'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const REPO = join(__dirname, '..', '..'); +const MANIFEST = JSON.parse(readFileSync(join(__dirname, 'knowledge-coverage.json'), 'utf8')); + +const EXPLAIN_SURFACES = [ + 'docs/app/docs/troubleshooting/page.ts', + 'agent-docs/nextjs-muscle-memory-gotchas.md', + 'agent-docs/lit-muscle-memory-gotchas.md', +].map((p) => join(REPO, p)); +const AGENTS_MD = join(REPO, 'AGENTS.md'); +const MCP_DOCS = join(REPO, 'packages', 'mcp', 'src', 'mcp-docs.js'); + +/** True when the rule name appears as a whole token in any explanation surface. */ +function ruleIsExplained(name) { + const re = new RegExp(`\\b${name.replace(/[-]/g, '\\-')}\\b`); + return EXPLAIN_SURFACES.some((f) => { + try { return re.test(readFileSync(f, 'utf8')); } catch { return false; } + }); +} + +/** + * Pure reconciliation of check rules against the explanation surfaces + exempt + * map. Returns error strings; empty means covered. `explained` is injectable so + * synthetic cases need no fs. + */ +export function reconcileRules(liveRules, exempt, explained) { + const errors = []; + const exemptNames = new Set(Object.keys(exempt)); + const live = new Set(liveRules); + + for (const name of liveRules) { + const isExplained = explained(name); + const isExempt = exemptNames.has(name); + if (!isExplained && !isExempt) { + errors.push( + `check rule "${name}" is neither explained (in the troubleshooting page or a gotcha doc) ` + + `nor exempted in test/knowledge/knowledge-coverage.json. Add a symptom-keyed entry that ` + + `names the rule, or an exemption with a reason.`, + ); + } + } + // Stale exemption: a name exempted that is no longer a live rule. + for (const name of exemptNames) { + if (!live.has(name)) errors.push(`stale exemption "${name}": no longer a webjs check rule, remove it from knowledge-coverage.json.`); + else if (typeof exempt[name] !== 'string' || !exempt[name].trim()) errors.push(`exemption "${name}" needs a non-empty reason.`); + } + return errors; +} + +/** + * Derive the AGENTS.md heading regexes the MCP init primer sources, straight + * from the mcp-docs.js source, so a NEW sourced anchor is auto-checked. + */ +export function deriveInitAnchors(mcpDocsSrc) { + return [...mcpDocsSrc.matchAll(/sectionByHeading\(\s*agents\s*,\s*(\/[^/]+\/[a-z]*)\)/g)].map((m) => m[1]); +} +function regexFromLiteral(lit) { + const m = lit.match(/^\/(.*)\/([a-z]*)$/); + return new RegExp(m[1], m[2]); +} + +const liveRules = RULES.map((r) => r.name); + +test('every webjs check rule is explained in an agent-facing surface or exempted', () => { + const errors = reconcileRules(liveRules, MANIFEST.checkRulesExempt, ruleIsExplained); + const explained = liveRules.filter(ruleIsExplained).length; + console.log(`[knowledge-coverage] ${liveRules.length} check rules: ${explained} explained, ${Object.keys(MANIFEST.checkRulesExempt).length} exempted.`); + assert.deepEqual(errors, [], `check-rule knowledge gaps:\n - ${errors.join('\n - ')}`); +}); + +test('the AGENTS.md headings the MCP init primer sources still exist', () => { + const anchors = deriveInitAnchors(readFileSync(MCP_DOCS, 'utf8')); + assert.ok(anchors.length >= 2, `expected to derive the MCP init anchors from mcp-docs.js, got ${anchors.length}`); + const agents = readFileSync(AGENTS_MD, 'utf8'); + console.log(`[knowledge-coverage] MCP init sources ${anchors.length} AGENTS.md anchors: ${anchors.join(', ')}`); + for (const lit of anchors) { + assert.ok(regexFromLiteral(lit).test(agents), `MCP init sources ${lit} from AGENTS.md, but no heading matches it (a rename would silently empty the init primer).`); + } +}); + +// --- counterfactuals: prove each failure mode fires --- + +test('reconcileRules FAILS on a new unexplained, unexempted rule', () => { + const errors = reconcileRules(['brand-new-rule'], {}, () => false); + assert.equal(errors.length, 1); + assert.match(errors[0], /check rule "brand-new-rule" is neither explained/); +}); + +test('reconcileRules FAILS on a stale exemption', () => { + const errors = reconcileRules([], { 'gone-rule': 'reason' }, () => false); + assert.ok(errors.some((e) => /stale exemption "gone-rule"/.test(e)), errors.join('\n')); +}); + +test('reconcileRules FAILS on an empty exemption reason', () => { + const errors = reconcileRules(['x'], { x: ' ' }, () => false); + assert.ok(errors.some((e) => /needs a non-empty reason/.test(e)), errors.join('\n')); +}); + +test('reconcileRules passes an explained rule with no exemption', () => { + const errors = reconcileRules(['real-rule'], {}, (n) => n === 'real-rule'); + assert.deepEqual(errors, []); +}); + +test('deriveInitAnchors extracts the sectionByHeading regexes', () => { + const anchors = deriveInitAnchors("const a = sectionByHeading(agents, /^##\\s+Execution model/im);\nconst b = sectionByHeading(agents, /^##\\s+Invariants/im);"); + assert.deepEqual(anchors, ['/^##\\s+Execution model/im', '/^##\\s+Invariants/im']); +}); + +test('a renamed init anchor is caught (counterfactual)', () => { + const anchors = deriveInitAnchors(readFileSync(MCP_DOCS, 'utf8')); + const agentsWithRenamedHeading = readFileSync(AGENTS_MD, 'utf8').replace(/^##\s+Invariants.*$/m, '## Rules and guarantees'); + const stillMatches = anchors.every((lit) => regexFromLiteral(lit).test(agentsWithRenamedHeading)); + assert.equal(stillMatches, false, 'renaming the Invariants heading should break at least one derived anchor'); +});