From 31c221dc4024c1977d9edba43d42574dd8448915 Mon Sep 17 00:00:00 2001 From: Qwynn Marcelle Date: Sun, 26 Jul 2026 17:00:22 -0400 Subject: [PATCH 1/2] feat(conformance): add the executable four-path producer contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four stable read paths had a prose contract in docs/conformance.md and no executable one. This adds the standard-owned suite that measures a producer candidate against it, plus red tests proving each protected behavior is actually protected. check:conformance a candidate satisfies the contract check:conformance:test breaking each protected behavior makes that fail The suite runs against a candidate supplied by WORKSPACEJSON_CLI_CANDIDATE, because the producer lives in another repository. It does NOT skip when the candidate is absent — it exits non-zero with instructions. A conformance gate that goes green because it could not find the implementation reports conformance it never measured. Asserted per path: fileIndex populated from repository evidence with repository-root-relative POSIX keys that name real files, deterministically ordered; frameworkManifest publishing only dependency-corroborated frameworks at the documented confidence floor, and omitting uncorroborated tokens; manual.fragileFiles and manual.coChangePatterns preserved verbatim, and left absent rather than fabricated when a maintainer declared nothing. Beyond the paths: invalid artifacts refused rather than overwritten, --force moving them aside recoverably, producer identity not attributable to an invoker, output validating against the package-owned schema, a second run on an unchanged repository byte-identical, and mediated invocation producing the same artifact as direct after removing only generated.generatedAt. Deliberately NOT asserted: per-file values inside fileIndex. Every value field is optional in the published type, so an empty entry is conformant. Those values are behavioral and git-derived, and whether that source may enter the stable contract is an open determination made outside this repository. Requiring them would fail a producer that is behaving correctly. The scope boundary is stated in the suite so a later reader does not mistake it for an oversight. The red tests carry two honesty guards: each mutation asserts it changed bytes, so a stale anchor cannot leave a case silently testing nothing; and each names the check it expects to break, so reddening the suite for an unrelated reason does not count. A baseline case asserts a clean candidate is accepted, so the suite cannot pass by rejecting everything. Running this against the current producer found a real defect on the first attempt: the producer indexed its own artifact, so run 1 and run 2 disagreed and generate --check would fail on a repository's first CI run after adoption. Fixed in the producer repository; the case that catches it is retained here. --- .github/workflows/ci.yml | 65 ++++++ docs/conformance.md | 55 +++++ package.json | 2 + scripts/check-producer-conformance.mjs | 246 ++++++++++++++++++++ scripts/check-producer-conformance.test.mjs | 175 ++++++++++++++ scripts/producer-conformance-lib.mjs | 230 ++++++++++++++++++ 6 files changed, 773 insertions(+) create mode 100644 scripts/check-producer-conformance.mjs create mode 100644 scripts/check-producer-conformance.test.mjs create mode 100644 scripts/producer-conformance-lib.mjs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dd0aa8c..1fd74fc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -113,3 +113,68 @@ jobs: echo "spec binary should exit non-zero when invoked without a command" exit 1 fi + + # --------------------------------------------------------------------------- + # Four-path producer conformance. + # + # This repository owns the neutral contract; the producer under test lives in + # workspacejson/cli. The suite therefore needs a CLI candidate, which is why + # this is a separate job: it checks out a second repository and builds it, + # which the unit-test matrix should not pay for, and it does not need the + # Node matrix. + # + # The suite does NOT skip when the candidate is missing — it exits non-zero + # with instructions. A conformance gate that goes green because it could not + # find the implementation reports conformance it never measured. + # + # workspacejson/cli is a private repository, so this job requires a read + # token: WORKSPACEJSON_CLI_READ_TOKEN, a fine-grained PAT scoped to + # `contents: read` on workspacejson/cli and nothing else. + # --------------------------------------------------------------------------- + producer-conformance: + name: Four-path producer conformance + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Check out the standard + uses: actions/checkout@v7 + + - name: Check out the producer under test + uses: actions/checkout@v7 + with: + repository: workspacejson/cli + ref: main + path: .candidate + token: ${{ secrets.WORKSPACEJSON_CLI_READ_TOKEN }} + + - uses: pnpm/action-setup@v6 + + - uses: actions/setup-node@v7 + with: + node-version: 22 + cache: pnpm + + - name: Install and build the standard + run: | + pnpm install --frozen-lockfile + pnpm --filter @workspacejson/spec build + + - name: Build the producer candidate + working-directory: .candidate + run: | + pnpm install --no-frozen-lockfile + pnpm -r build + + # Green: the candidate satisfies the contract. + - name: Producer conformance + env: + WORKSPACEJSON_CLI_CANDIDATE: ${{ github.workspace }}/.candidate/packages/cli + run: pnpm run check:conformance + + # Red: each protected behavior, when broken, makes the suite fail. Runs + # after the green gate so the receipts appear in that order in the log. + - name: Producer conformance red tests + env: + WORKSPACEJSON_CLI_CANDIDATE: ${{ github.workspace }}/.candidate/packages/cli + run: pnpm run check:conformance:test diff --git a/docs/conformance.md b/docs/conformance.md index 979ad4e..3744c1a 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -140,12 +140,67 @@ CI runs on Node 20 and 22. In order: | Executable examples | `check:examples` | Every shipped example validates | | Export validation | inline in CI | Each declared export resolves and imports | | Binary behavior | inline in CI | `validate` succeeds on a valid document; a bare invocation exits non-zero | +| Producer conformance | `check:conformance` | A producer candidate satisfies the four-path contract | +| Producer conformance red tests | `check:conformance:test` | Breaking each protected behavior makes that suite fail | The guard red tests deserve emphasis. A guard that rejected everything would look identical to a working guard from a green build, so the suite includes a baseline case asserting that the unmodified repository is *accepted*. Coverage without that case is not evidence. +## The executable producer contract + +`check:conformance` is the standard's assertion about what any conforming +producer must do. It runs against a **candidate** — a built producer package — +because the producer lives in `workspacejson/cli`, not here: + +```bash +WORKSPACEJSON_CLI_CANDIDATE=/path/to/cli/packages/cli pnpm run check:conformance +WORKSPACEJSON_CLI_CANDIDATE=/path/to/cli/packages/cli pnpm run check:conformance:test +``` + +It **does not skip** when the candidate is absent; it exits non-zero with +instructions. A conformance gate that goes green because it could not find the +implementation reports conformance it never measured. + +What it asserts, by stable path: + +| Path | Asserted | +| -- | -- | +| `generated.fileIndex` | Non-empty from repository evidence; every key repository-root-relative POSIX; every key names a file that exists; the repository's real files are represented; keys deterministically ordered | +| `generated.frameworkManifest` | A framework corroborated by a declared dependency is published at the documented `>= 0.7` floor; an uncorroborated `AGENTS.md` token is **not**; entries deterministically ordered | +| `manual.fragileFiles` | Preserved verbatim across regeneration; absent evidence left absent, never fabricated | +| `manual.coChangePatterns` | Preserved verbatim across regeneration; absent evidence left absent, never fabricated | + +And beyond the four paths: an unparseable or schema-invalid artifact is refused +rather than overwritten, `--force` moves it aside recoverably instead of +destroying it, `generated.by.name` identifies the producer rather than an +invoker, output validates against the package-owned schema, a second run against +an unchanged repository is byte-identical, and mediated invocation produces the +same artifact as direct invocation after removing only `generated.generatedAt`. + +### What it deliberately does not assert + +**Per-file values inside `fileIndex`.** `FileIndexEntry` declares `fragility`, +`aiModificationCount` and `humanModificationCount` as optional, so `{}` is a +conformant entry. Those values are behavioral, their only available source is +git-derived, and whether that source may enter the stable contract is an open +question tracked outside this repository. A suite requiring them would fail a +producer that is behaving correctly, and would pre-empt a ruling the standard +does not own. + +Nor does it require non-empty human-owned fields, or add git-derived co-change +to the acceptance surface. + +### Vreko-mediated invocation + +The contract asserts that mediation does not change what a producer emits. It +verifies this against the **public** mediation surface — a host importing the +package and calling the exported producer. Vreko itself is private and outside +this repository's clean-room boundary, so it cannot be executed here; a +Vreko-specific regression belongs in that repository and does not replace this +contract. + ## Known gaps Stated plainly, because a conformance document that hides them is misleading: diff --git a/package.json b/package.json index d02ea6a..9e3ae8c 100644 --- a/package.json +++ b/package.json @@ -25,6 +25,8 @@ "check:architecture:test": "node scripts/check-architecture.test.mjs", "check:schema": "node scripts/verify-schema-provenance.mjs", "check:examples": "node scripts/validate-examples.mjs", + "check:conformance": "node scripts/check-producer-conformance.mjs", + "check:conformance:test": "node scripts/check-producer-conformance.test.mjs", "check:docs": "node scripts/check-docs.mjs", "release:verify-packs": "pnpm --filter @workspacejson/spec exec node ../../scripts/verify-package-tarball.mjs && pnpm --filter @workspacejson/rules exec node ../../scripts/verify-package-tarball.mjs", "release:verify-published": "node scripts/verify-published.mjs" diff --git a/scripts/check-producer-conformance.mjs b/scripts/check-producer-conformance.mjs new file mode 100644 index 0000000..696701d --- /dev/null +++ b/scripts/check-producer-conformance.mjs @@ -0,0 +1,246 @@ +#!/usr/bin/env node +// Executable four-path producer conformance contract. +// +// This repository owns the neutral standard. This suite is the standard's +// assertion about what any conforming producer must do; `workspacejson/cli` is +// the implementation currently measured against it. +// +// WORKSPACEJSON_CLI_CANDIDATE=/path/to/cli/packages/cli \ +// node scripts/check-producer-conformance.mjs +// +// WHAT THIS ASSERTS ABOUT fileIndex — and deliberately does not: +// +// `generated.fileIndex` must be POPULATED FROM REPOSITORY EVIDENCE: real +// keys, in the documented key format, deterministically ordered. +// +// It asserts NOTHING about per-file values. `FileIndexEntry` declares every +// value field optional, so `{}` is conformant. Those values are behavioral, +// their only source is git-derived, and whether that may enter the stable +// contract is an open determination made outside this repository. Requiring +// them here would pre-empt a ruling this repository does not own and would +// fail a producer that is behaving correctly. + +import { readFileSync, writeFileSync, existsSync, readdirSync } from 'node:fs'; +import { join } from 'node:path'; + +import { + resolveCandidate, makeFixture, cleanup, readArtifact, readArtifactRaw, + runDirect, runMediated, createReporter, listFiles, ARTIFACT, repoRoot, +} from './producer-conformance-lib.mjs'; + +const { check, section, state } = createReporter(); +const candidate = resolveCandidate(); + +// The standard's own validator is the reference implementation. Re-implementing +// it here would create a second source of truth, and they drift. +const { validate, validateV4 } = await import( + join(repoRoot, 'packages/spec/dist/index.js') +).catch(() => { + console.error('\n@workspacejson/spec is not built. Run `pnpm --filter @workspacejson/spec build` first.\n'); + process.exit(2); +}); + +console.log(`candidate: ${candidate.manifest.name}@${candidate.manifest.version}`); +console.log(` ${candidate.dir}`); + +// --------------------------------------------------------------------------- +section('1. generated.fileIndex — populated from repository evidence'); +// --------------------------------------------------------------------------- +{ + const repo = makeFixture(); + const run = runDirect(candidate, repo); + check('generate exits 0 on a clean repository', run.status === 0, + `status=${run.status}\n${run.stderr?.slice(0, 300) ?? ''}`); + + const doc = readArtifact(repo); + const fileIndex = doc.generated?.fileIndex ?? {}; + const keys = Object.keys(fileIndex); + const onDisk = listFiles(repo); + + check('fileIndex is non-empty — not an unpopulated stub', keys.length > 0, + `keys=${keys.length}`); + check('every key is repository-root-relative POSIX (spec fileIndex contract)', + keys.every((k) => !k.startsWith('/') && !k.startsWith('./') && !k.includes('\\') && !k.includes('..') && !/^[A-Za-z]:/.test(k)), + `offenders=${JSON.stringify(keys.filter((k) => k.startsWith('/') || k.startsWith('./') || k.includes('\\')).slice(0, 3))}`); + check('every key names a file that actually exists in the repository', + keys.every((k) => existsSync(join(repo, k))), + `missing=${JSON.stringify(keys.filter((k) => !existsSync(join(repo, k))).slice(0, 3))}`); + check('the repository\'s real files are represented', + onDisk.every((f) => keys.includes(f)), + `absent=${JSON.stringify(onDisk.filter((f) => !keys.includes(f)).slice(0, 5))}`); + check('keys are deterministically ordered (sorted, not filesystem order)', + JSON.stringify(keys) === JSON.stringify([...keys].sort())); + + // Explicitly recorded as out of scope, so a future reader does not mistake + // its absence for an oversight. + const allEmpty = keys.every((k) => Object.keys(fileIndex[k]).length === 0); + console.log(` NOTE per-file values are NOT asserted (out of scope). Observed: ${allEmpty ? 'all entries empty — conformant' : 'some entries carry values — also conformant'}`); + + cleanup(repo); +} + +// --------------------------------------------------------------------------- +section('2. generated.frameworkManifest — populated from repository evidence'); +// --------------------------------------------------------------------------- +{ + const repo = makeFixture(); + runDirect(candidate, repo); + const doc = readArtifact(repo); + const manifest = doc.generated?.frameworkManifest ?? []; + const names = manifest.map((e) => e.name.toLowerCase()); + + check('a framework corroborated by a declared dependency is published', + names.includes('express'), `manifest=${JSON.stringify(manifest)}`); + check('published entries meet the documented confidence floor (>= 0.7)', + manifest.every((e) => typeof e.confidence === 'number' && e.confidence >= 0.7), + `entries=${JSON.stringify(manifest)}`); + check('an AGENTS.md token corroborated by nothing is NOT published at that floor', + !names.includes('svelte'), + 'an uncorroborated guess at high confidence is worse than an omission — confidence tells the consumer to trust it'); + check('entries are deterministically ordered', + JSON.stringify(manifest.map((e) => e.name)) === JSON.stringify([...manifest.map((e) => e.name)].sort())); + + cleanup(repo); +} + +// --------------------------------------------------------------------------- +section('3. manual.* — preserved verbatim, never fabricated'); +// --------------------------------------------------------------------------- +{ + const manual = { + fragileFiles: [{ path: 'src/routes.js', reason: 'human-authored, must survive regeneration' }], + coChangePatterns: [{ files: ['src/index.js', 'src/routes.js'], note: 'human-authored' }], + }; + const repo = makeFixture({ manual }); + const before = readArtifact(repo).manual; + const run = runDirect(candidate, repo); + check('generate exits 0 over an existing artifact', run.status === 0, `status=${run.status}`); + + const after = readArtifact(repo).manual; + check('manual.fragileFiles preserved verbatim across regeneration', + JSON.stringify(after.fragileFiles) === JSON.stringify(before.fragileFiles), + `before=${JSON.stringify(before.fragileFiles)}\n after=${JSON.stringify(after.fragileFiles)}`); + check('manual.coChangePatterns preserved verbatim across regeneration', + JSON.stringify(after.coChangePatterns) === JSON.stringify(before.coChangePatterns), + `before=${JSON.stringify(before.coChangePatterns)}\n after=${JSON.stringify(after.coChangePatterns)}`); + cleanup(repo); + + // Safe absence: a producer must not invent human evidence where none exists. + // The boundary is explicit — do not require non-empty human-owned fields. + const bare = makeFixture(); + runDirect(candidate, bare); + const bareManual = readArtifact(bare).manual ?? {}; + check('absent manual evidence is left absent, never fabricated', + (bareManual.fragileFiles === undefined || bareManual.fragileFiles.length === 0) + && (bareManual.coChangePatterns === undefined || bareManual.coChangePatterns.length === 0), + `manual=${JSON.stringify(bareManual)}`); + cleanup(bare); +} + +// --------------------------------------------------------------------------- +section('4. Invalid artifacts fail safely without destroying human evidence'); +// --------------------------------------------------------------------------- +{ + // (a) unparseable JSON + const broken = makeFixture({ existingArtifact: '{ this is not json' }); + const original = readArtifactRaw(broken); + const run = runDirect(candidate, broken); + check('unparseable artifact: refuses (non-zero exit)', run.status !== 0, `status=${run.status}`); + check('unparseable artifact: the file on disk is untouched', + readArtifactRaw(broken) === original); + + // (b) --force preserves the original rather than destroying it + const forced = runDirect(candidate, broken, ['--force']); + const moved = readdirSync(join(broken, '.agents')).filter((f) => f.includes('invalid')); + check('--force moves the invalid artifact aside instead of destroying it', + forced.status === 0 && moved.length === 1, + `status=${forced.status} moved=${JSON.stringify(moved)}`); + check('--force: the original bytes remain recoverable', + moved.length === 1 && readFileSync(join(broken, '.agents', moved[0]), 'utf8') === original); + cleanup(broken); + + // (c) schema-invalid but parseable, carrying human evidence + const invalid = makeFixture({ + existingArtifact: { + manual: { fragileFiles: [{ path: 'src/index.js', reason: 'irreplaceable' }] }, + generated: { specVersion: 'not-a-version' }, + }, + }); + const invalidBefore = readArtifactRaw(invalid); + const invalidRun = runDirect(candidate, invalid); + check('schema-invalid artifact: refuses rather than overwriting', invalidRun.status !== 0, + `status=${invalidRun.status}`); + check('schema-invalid artifact: human evidence survives the refusal', + readArtifactRaw(invalid) === invalidBefore + && readArtifactRaw(invalid).includes('irreplaceable')); + cleanup(invalid); +} + +// --------------------------------------------------------------------------- +section('5. Producer identity, determinism, and mediation'); +// --------------------------------------------------------------------------- +{ + const repo = makeFixture(); + runDirect(candidate, repo); + const doc = readArtifact(repo); + + check('generated.by.name identifies @workspacejson/cli, not an invoker', + doc.generated?.by?.name === '@workspacejson/cli', + `by=${JSON.stringify(doc.generated?.by)}`); + check('output validates against the package-owned schema', + validate(doc) === true); + check('output validates as v0.4 when it declares specVersion 0.4', + doc.generated?.specVersion !== '0.4' || validateV4(doc) === true); + + // Determinism: same repository, same producer version, byte-identical. + const first = readArtifactRaw(repo); + const second = runDirect(candidate, repo); + const afterSecond = readArtifactRaw(repo); + check('a second run against an unchanged repository is byte-identical', + first === afterSecond, 'material projection changed with no repository change'); + check('a second run reports no material drift (usable as a CI gate)', + second.status === 0 && /skip|no change|unchanged|up to date/i.test(`${second.stdout}${second.stderr}`) || first === afterSecond, + `stdout=${second.stdout?.slice(0, 200)}`); + + cleanup(repo); +} +{ + // Direct vs mediated, on the SAME repository in sequence. + // + // Comparing two separately-created fixtures cannot work: `hygiene.scannedAt` + // is a scan timestamp, so two independent runs legitimately differ by more + // than `generatedAt` and the assertion would be measuring wall-clock, not + // mediation. Running both against one repository isolates the variable — + // the second invocation sees an unchanged material projection, so a + // conforming producer carries the prior values forward and writes nothing. + const repo = makeFixture(); + const direct = runDirect(candidate, repo); + const afterDirect = readArtifactRaw(repo); + const mediated = runMediated(candidate, repo); + const afterMediated = readArtifactRaw(repo); + + check('mediated invocation succeeds', mediated.status === 0, + `status=${mediated.status}\n${mediated.stderr?.slice(0, 300) ?? ''}`); + + if (direct.status === 0 && mediated.status === 0) { + const strip = (raw) => { + const clone = JSON.parse(raw); + delete clone.generated.generatedAt; + return JSON.stringify(clone); + }; + check('direct and mediated output are identical after removing only generated.generatedAt', + strip(afterDirect) === strip(afterMediated), + 'mediation must not change what the producer emits'); + check('mediated invocation over a current artifact writes nothing', + afterDirect === afterMediated, + 'a mediator must not cause a rewrite the direct path would not'); + } + cleanup(repo); +} + +// --------------------------------------------------------------------------- +console.log(`\n${'='.repeat(70)}`); +console.log(` RESULT: ${state.pass} passed, ${state.fail} failed (total ${state.pass + state.fail})`); +if (state.fail) console.log(` FAILED: ${state.failures.join(', ')}`); +console.log('='.repeat(70)); +process.exit(state.fail ? 1 : 0); diff --git a/scripts/check-producer-conformance.test.mjs b/scripts/check-producer-conformance.test.mjs new file mode 100644 index 0000000..a089488 --- /dev/null +++ b/scripts/check-producer-conformance.test.mjs @@ -0,0 +1,175 @@ +#!/usr/bin/env node +// Red tests for the four-path producer conformance contract (see the +// suite itself): mutating any protected behavior must make it fail. +// +// A conformance suite nobody has watched fail is an unverified claim. Each case +// below copies the candidate, breaks exactly one protected behavior in the +// built output, and asserts the suite goes RED. +// +// Two guards make this honest: +// +// 1. Each mutation asserts it actually changed bytes. A stale anchor that no +// longer matches would otherwise leave the candidate pristine, the suite +// green, and this file reporting "the guard works" having tested nothing. +// 2. Each case names the check it expects to break, and the run must contain +// that failure — not merely any failure. A mutation that reds the suite +// for an unrelated reason is not evidence about the behavior it targeted. +// +// A baseline case asserts the unmutated candidate is accepted, so the suite +// cannot pass these tests by rejecting everything. +// +// WORKSPACEJSON_CLI_CANDIDATE=/path/to/cli/packages/cli \ +// node scripts/check-producer-conformance.test.mjs + +import { cpSync, mkdtempSync, readFileSync, writeFileSync, rmSync, readdirSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { spawnSync } from 'node:child_process'; + +const here = dirname(fileURLToPath(import.meta.url)); +const SUITE = join(here, 'check-producer-conformance.mjs'); + +const source = process.env.WORKSPACEJSON_CLI_CANDIDATE; +if (!source) { + console.error('\nWORKSPACEJSON_CLI_CANDIDATE is required. See check-producer-conformance.mjs.\n'); + process.exit(2); +} + +/** + * Mutations are applied to the BUILT output, not to source, so a case does not + * depend on the candidate's toolchain being available. tsup emits shared code + * into content-hashed chunks, so every `dist/*.js` is patched rather than a + * named file — a chunk name is not a contract. + */ +const CASES = [ + { + name: 'fileIndex emptied — an unpopulated stub', + expect: 'fileIndex is non-empty', + find: 'for (const key of keys) index[key] = {};', + replace: 'for (const key of []) index[key] = {};', + }, + { + name: 'fileIndex key ordering removed', + expect: 'deterministically ordered', + find: '.filter((key) => !isProducerOutput(key)).sort();', + replace: '.filter((key) => !isProducerOutput(key)).reverse();', + }, + { + name: 'producer-output exclusion removed — the non-convergence defect returns', + expect: 'byte-identical', + find: '.filter((key) => !isProducerOutput(key)).sort();', + replace: '.sort();', + }, + { + name: 'framework confidence dropped below the documented floor', + expect: 'confidence floor', + find: 'confidence: 0.9', + replace: 'confidence: 0.5', + }, + { + name: 'manual evidence discarded instead of preserved', + expect: 'manual.fragileFiles preserved verbatim', + find: 'manual: existing?.manual ?? {},', + replace: 'manual: {},', + }, + { + name: 'producer identity misattributed to an invoker', + expect: 'generated.by.name identifies @workspacejson/cli', + find: 'by: { name: producer.name, version: producer.version }', + replace: 'by: { name: "vreko", version: producer.version }', + }, + { + name: 'refusal on an invalid artifact removed', + expect: 'refuses', + find: 'if (!options.force || options.dryRun || options.check) throw new GenerateRefusalError(message);', + replace: 'if (false) throw new GenerateRefusalError(message);', + }, +]; + +function patchDist(dir, find, replace) { + const distDir = join(dir, 'dist'); + let replacements = 0; + for (const entry of readdirSync(distDir)) { + if (!entry.endsWith('.js')) continue; + const path = join(distDir, entry); + const before = readFileSync(path, 'utf8'); + if (!before.includes(find)) continue; + const after = before.split(find).join(replace); + if (after === before) continue; + writeFileSync(path, after); + replacements += before.split(find).length - 1; + } + return replacements; +} + +function runSuite(candidateDir) { + return spawnSync(process.execPath, [SUITE], { + encoding: 'utf8', + env: { ...process.env, WORKSPACEJSON_CLI_CANDIDATE: candidateDir, NO_COLOR: '1' }, + }); +} + +let pass = 0; +let fail = 0; +const failures = []; + +function record(label, condition, detail = '') { + if (condition) { + console.log(`PASS ${label}`); + pass += 1; + } else { + console.log(`FAIL ${label}${detail ? `\n ${detail}` : ''}`); + fail += 1; + failures.push(label); + } +} + +// --------------------------------------------------------------------------- +// Baseline — the suite must ACCEPT a clean candidate. Without this, a suite +// that rejected everything would pass every red test below. +// --------------------------------------------------------------------------- +{ + const dir = mkdtempSync(join(tmpdir(), 'conformance-baseline-')); + const candidate = join(dir, 'candidate'); + cpSync(resolve(source), candidate, { recursive: true }); + const run = runSuite(candidate); + record('baseline: an unmutated candidate is accepted', + run.status === 0, + `status=${run.status}\n ${(run.stdout ?? '').split('\n').filter((l) => l.includes('FAIL')).slice(0, 3).join('\n ')}`); + rmSync(dir, { recursive: true, force: true }); +} + +// --------------------------------------------------------------------------- +for (const testCase of CASES) { + const dir = mkdtempSync(join(tmpdir(), 'conformance-mutant-')); + const candidate = join(dir, 'candidate'); + cpSync(resolve(source), candidate, { recursive: true }); + + const replacements = patchDist(candidate, testCase.find, testCase.replace); + if (replacements === 0) { + record(`${testCase.name}`, false, + `MUTATION DID NOT APPLY — anchor not found in the built candidate:\n ${testCase.find}\n The anchor is stale. This case tested nothing; fix the anchor rather than deleting the case.`); + rmSync(dir, { recursive: true, force: true }); + continue; + } + + const run = runSuite(candidate); + const output = `${run.stdout ?? ''}${run.stderr ?? ''}`; + const wentRed = run.status !== 0; + const brokeExpected = output.split('\n').some((line) => line.includes('FAIL') && line.includes(testCase.expect)); + + record(`rejected — ${testCase.name}`, + wentRed && brokeExpected, + wentRed + ? `suite went red but not on the expected check "${testCase.expect}"\n observed: ${output.split('\n').filter((l) => l.includes('FAIL')).slice(0, 3).join(' | ')}` + : `suite stayed GREEN with ${replacements} mutation(s) applied — the behavior is unprotected`); + + rmSync(dir, { recursive: true, force: true }); +} + +console.log(`\nProducer conformance red tests: ${pass} passed, ${fail} failed.`); +if (fail) { + console.log(`FAILED: ${failures.join(', ')}`); + process.exit(1); +} diff --git a/scripts/producer-conformance-lib.mjs b/scripts/producer-conformance-lib.mjs new file mode 100644 index 0000000..f9c26d8 --- /dev/null +++ b/scripts/producer-conformance-lib.mjs @@ -0,0 +1,230 @@ +// Shared machinery for the four-path producer conformance contract. +// +// This repository owns the neutral contract; `workspacejson/cli` is the +// implementation under test. The suite therefore needs a CLI *candidate* — a +// built producer package — supplied from outside this repository. +// +// SCOPE BOUNDARY, read before extending: +// +// This contract asserts that `generated.fileIndex` is POPULATED FROM +// REPOSITORY EVIDENCE. It asserts nothing about per-file *values*. +// +// `FileIndexEntry` declares `fragility`, `aiModificationCount` and +// `humanModificationCount` as OPTIONAL, so `{}` is a conformant entry. Those +// values are behavioral, their only available source is git-derived, and +// whether that source may enter the stable contract is an open determination +// made outside this repository. A conformance suite that required them would +// pre-empt a ruling this repository does not own, and would fail a producer +// that is behaving correctly. +// +// Do not add value assertions here. Do not add git-derived co-change. + +import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, existsSync, readdirSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { spawnSync } from 'node:child_process'; + +export const repoRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..'); + +// --------------------------------------------------------------------------- +// Candidate resolution +// --------------------------------------------------------------------------- + +/** + * Resolve the producer under test. + * + * `WORKSPACEJSON_CLI_CANDIDATE` points at a BUILT `@workspacejson/cli` package + * directory — one containing `package.json` and `dist/`. + * + * There is deliberately no fallback that skips. A conformance gate that goes + * green because it could not find the implementation is worse than no gate: + * it reports conformance it never measured. Absent a candidate this exits + * non-zero with instructions. + */ +export function resolveCandidate() { + const raw = process.env.WORKSPACEJSON_CLI_CANDIDATE; + if (!raw) { + console.error(` +The producer conformance suite needs a built @workspacejson/cli candidate. + + WORKSPACEJSON_CLI_CANDIDATE=/path/to/cli/packages/cli node scripts/check-producer-conformance.mjs + +To produce one from a checkout of workspacejson/cli: + + pnpm install && pnpm -r build # emits packages/cli/dist/ + +This suite does not skip when the candidate is missing. A conformance gate that +passes without measuring anything is worse than no gate. +`); + process.exit(2); + } + const dir = resolve(raw); + const manifestPath = join(dir, 'package.json'); + const cliEntry = join(dir, 'dist', 'cli.js'); + const libEntry = join(dir, 'dist', 'index.js'); + + for (const [label, path] of [['package.json', manifestPath], ['dist/cli.js', cliEntry], ['dist/index.js', libEntry]]) { + if (!existsSync(path)) { + console.error(`\nCandidate at ${dir} is missing ${label}.`); + console.error(existsSync(manifestPath) + ? 'The package exists but is not built. Run `pnpm -r build` in the candidate repository.\n' + : 'That path does not look like a @workspacejson/cli package directory.\n'); + process.exit(2); + } + } + + const manifest = JSON.parse(readFileSync(manifestPath, 'utf8')); + return { dir, manifest, cliEntry, libEntry }; +} + +// --------------------------------------------------------------------------- +// Fixture repositories +// --------------------------------------------------------------------------- + +/** + * A repository shaped to exercise every stable path at once. + * + * `express` is declared as a dependency AND named in AGENTS.md, so it is + * corroborated. `svelte` is named in AGENTS.md and declared nowhere, so a + * conformant producer must NOT publish it at the documented >= 0.7 confidence + * floor — an uncorroborated guess at high confidence is worse than an omission, + * because the confidence tells the consumer to trust it. + */ +export function makeFixture({ manual = null, existingArtifact = null } = {}) { + const dir = mkdtempSync(join(tmpdir(), 'wsjson-conformance-')); + + writeFileSync(join(dir, 'package.json'), `${JSON.stringify({ + name: 'conformance-fixture', + version: '1.0.0', + private: true, + dependencies: { express: '^4.18.0' }, + }, null, 2)}\n`); + + writeFileSync(join(dir, 'AGENTS.md'), [ + '# Conformance fixture', + '', + '## Stack', + '', + 'This service uses express for routing and svelte on the front end.', + '', + '## Layout', + '', + '- `src/` — application source', + '- `test/` — tests', + '', + ].join('\n')); + + mkdirSync(join(dir, 'src'), { recursive: true }); + mkdirSync(join(dir, 'test'), { recursive: true }); + writeFileSync(join(dir, 'src', 'index.js'), "export const answer = 42;\n"); + writeFileSync(join(dir, 'src', 'routes.js'), "export const routes = [];\n"); + writeFileSync(join(dir, 'test', 'index.test.js'), "// intentionally empty\n"); + writeFileSync(join(dir, 'README.md'), '# fixture\n'); + + if (manual || existingArtifact) { + mkdirSync(join(dir, '.agents'), { recursive: true }); + const artifact = existingArtifact ?? { + manual, + generated: { + specVersion: '0.4', + generatedAt: '2020-01-01T00:00:00.000Z', + by: { name: '@workspacejson/cli', version: '0.0.0' }, + frameworkManifest: [], + fileIndex: {}, + }, + agents: {}, + health: { intelligenceState: 'INSUFFICIENT_DATA', observationCount: 0, confidence: 0 }, + }; + writeFileSync( + join(dir, '.agents', 'workspace.json'), + typeof artifact === 'string' ? artifact : `${JSON.stringify(artifact, null, 2)}\n`, + ); + } + + return dir; +} + +export function cleanup(dir) { + rmSync(dir, { recursive: true, force: true }); +} + +export const ARTIFACT = join('.agents', 'workspace.json'); + +export function readArtifact(repo) { + return JSON.parse(readFileSync(join(repo, ARTIFACT), 'utf8')); +} + +export function readArtifactRaw(repo) { + return readFileSync(join(repo, ARTIFACT), 'utf8'); +} + +// --------------------------------------------------------------------------- +// Invocation +// --------------------------------------------------------------------------- + +/** Direct invocation — the published binary entry point. */ +export function runDirect(candidate, repo, args = []) { + return spawnSync(process.execPath, [candidate.cliEntry, 'generate', repo, ...args], { + encoding: 'utf8', + cwd: repo, + env: { ...process.env, NO_COLOR: '1' }, + }); +} + +/** + * Mediated invocation — a host tool importing the package and calling the + * exported producer, which is how an integrating tool drives generation. + * + * The issue names Vreko as the mediator. Vreko is private and outside this + * repository's clean-room boundary, so it cannot be executed here. What is + * asserted instead is the property that mediation must preserve, against the + * public mediation surface the package exports. A Vreko-specific regression + * belongs in that repository, which this contract does not replace. + */ +export function runMediated(candidate, repo) { + const script = ` + import { generateWorkspaceJson } from ${JSON.stringify(candidate.libEntry)}; + const result = await generateWorkspaceJson(${JSON.stringify(repo)}); + process.stdout.write(JSON.stringify({ written: result.written, skipped: result.skipped })); + `; + return spawnSync(process.execPath, ['--input-type=module', '-e', script], { + encoding: 'utf8', + cwd: repo, + env: { ...process.env, NO_COLOR: '1' }, + }); +} + +// --------------------------------------------------------------------------- +// Reporting +// --------------------------------------------------------------------------- + +export function createReporter() { + const state = { pass: 0, fail: 0, failures: [] }; + return { + state, + check(label, condition, detail = '') { + if (condition) { + console.log(` PASS ${label}`); + state.pass += 1; + } else { + console.log(` FAIL ${label}${detail ? `\n ${detail}` : ''}`); + state.fail += 1; + state.failures.push(label); + } + }, + section(title) { + console.log(`\n${'='.repeat(70)}\n ${title}\n${'='.repeat(70)}`); + }, + }; +} + +export function listFiles(dir, base = dir, acc = []) { + for (const entry of readdirSync(dir, { withFileTypes: true })) { + if (entry.name === '.agents' || entry.name === 'node_modules' || entry.name === '.git') continue; + const full = join(dir, entry.name); + if (entry.isDirectory()) listFiles(full, base, acc); + else acc.push(full.slice(base.length + 1).split('\\').join('/')); + } + return acc; +} From dd9dcf2b89cb43f6e09b61e51ca4c1e3d7bf1a1c Mon Sep 17 00:00:00 2001 From: Qwynn Marcelle Date: Sun, 26 Jul 2026 17:28:11 -0400 Subject: [PATCH 2/2] ci(conformance): drop the read token now that the producer repo is public MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The conformance job checked out the producer with a fine-grained PAT because that repository was private. It is public, so the checkout needs no token and no secret. This is strictly better than the token arrangement it replaces: - the gate runs on forks and on first-time contributors' pull requests, which a secret-dependent job cannot; - there is no credential to provision, rotate or leak; - nothing has to be configured before the gate can enforce anything, so the job does not sit red waiting on an out-of-band step. Verified with an unauthenticated clone against a disabled global git config, so the result does not depend on a credential helper that happens to be present locally. Also ignores .candidate/, the checkout path. A conformance run leaves no untracked noise, and a vendored copy of the producer cannot be committed by accident — `packages/cli` under a tracked path would trip the repository-boundary guard. The job continues to track the producer's default branch rather than a pinned SHA. Noticing when the implementation drifts from the contract is the point; a pin would only re-prove a known-good pairing. --- .github/workflows/ci.yml | 11 +++++++---- .gitignore | 6 ++++++ 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1fd74fc..f2b84cb 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -127,9 +127,13 @@ jobs: # with instructions. A conformance gate that goes green because it could not # find the implementation reports conformance it never measured. # - # workspacejson/cli is a private repository, so this job requires a read - # token: WORKSPACEJSON_CLI_READ_TOKEN, a fine-grained PAT scoped to - # `contents: read` on workspacejson/cli and nothing else. + # workspacejson/cli is public, so the checkout needs no token and no secret. + # The job therefore runs on forks and on first-time contributors' pull + # requests, which a secret-dependent gate could not. + # + # It tracks the producer's `main` rather than a pinned SHA on purpose: the + # point of a conformance gate is to notice when the implementation drifts + # from the contract. A pin would only ever re-prove a known-good pairing. # --------------------------------------------------------------------------- producer-conformance: name: Four-path producer conformance @@ -146,7 +150,6 @@ jobs: repository: workspacejson/cli ref: main path: .candidate - token: ${{ secrets.WORKSPACEJSON_CLI_READ_TOKEN }} - uses: pnpm/action-setup@v6 diff --git a/.gitignore b/.gitignore index 36d88fd..42ae3aa 100644 --- a/.gitignore +++ b/.gitignore @@ -18,6 +18,12 @@ audit-output/ # Local swarm system — not for public repo .vreko-swarm/ +# Producer candidate for the conformance suite — a checkout of another +# repository, never content this one owns. Ignored so a local run leaves no +# untracked noise, and so a vendored copy cannot be committed by accident: +# `packages/cli` under a tracked path would trip the repository-boundary guard. +.candidate/ + # Local artifacts agents-workspace-cannon-repo/ audit-findings/