Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -113,3 +113,71 @@ jobs:
echo "spec binary should exit non-zero when invoked without a command"
exit 1
fi

# ---------------------------------------------------------------------------
# Four-path producer conformance.
#
# This repository owns the neutral contract; the producer under test lives in
# workspacejson/cli. The suite therefore needs a CLI candidate, which is why
# this is a separate job: it checks out a second repository and builds it,
# which the unit-test matrix should not pay for, and it does not need the
# Node matrix.
#
# The suite does NOT skip when the candidate is missing — it exits non-zero
# with instructions. A conformance gate that goes green because it could not
# find the implementation reports conformance it never measured.
#
# workspacejson/cli is public, so the checkout needs no token and no secret.
# The job therefore runs on forks and on first-time contributors' pull
# requests, which a secret-dependent gate could not.
#
# It tracks the producer's `main` rather than a pinned SHA on purpose: the
# point of a conformance gate is to notice when the implementation drifts
# from the contract. A pin would only ever re-prove a known-good pairing.
# ---------------------------------------------------------------------------
producer-conformance:
name: Four-path producer conformance
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: Check out the standard
uses: actions/checkout@v7

- name: Check out the producer under test
uses: actions/checkout@v7
with:
repository: workspacejson/cli
ref: main
path: .candidate

- uses: pnpm/action-setup@v6

- uses: actions/setup-node@v7
with:
node-version: 22
cache: pnpm

- name: Install and build the standard
run: |
pnpm install --frozen-lockfile
pnpm --filter @workspacejson/spec build

- name: Build the producer candidate
working-directory: .candidate
run: |
pnpm install --no-frozen-lockfile
pnpm -r build

# Green: the candidate satisfies the contract.
- name: Producer conformance
env:
WORKSPACEJSON_CLI_CANDIDATE: ${{ github.workspace }}/.candidate/packages/cli
run: pnpm run check:conformance

# Red: each protected behavior, when broken, makes the suite fail. Runs
# after the green gate so the receipts appear in that order in the log.
- name: Producer conformance red tests
env:
WORKSPACEJSON_CLI_CANDIDATE: ${{ github.workspace }}/.candidate/packages/cli
run: pnpm run check:conformance:test
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@ audit-output/
# Local swarm system — not for public repo
.vreko-swarm/

# Producer candidate for the conformance suite — a checkout of another
# repository, never content this one owns. Ignored so a local run leaves no
# untracked noise, and so a vendored copy cannot be committed by accident:
# `packages/cli` under a tracked path would trip the repository-boundary guard.
.candidate/

# Local artifacts
agents-workspace-cannon-repo/
audit-findings/
Expand Down
55 changes: 55 additions & 0 deletions docs/conformance.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,12 +140,67 @@ CI runs on Node 20 and 22. In order:
| Executable examples | `check:examples` | Every shipped example validates |
| Export validation | inline in CI | Each declared export resolves and imports |
| Binary behavior | inline in CI | `validate` succeeds on a valid document; a bare invocation exits non-zero |
| Producer conformance | `check:conformance` | A producer candidate satisfies the four-path contract |
| Producer conformance red tests | `check:conformance:test` | Breaking each protected behavior makes that suite fail |

The guard red tests deserve emphasis. A guard that rejected everything would look
identical to a working guard from a green build, so the suite includes a baseline
case asserting that the unmodified repository is *accepted*. Coverage without
that case is not evidence.

## The executable producer contract

`check:conformance` is the standard's assertion about what any conforming
producer must do. It runs against a **candidate** — a built producer package —
because the producer lives in `workspacejson/cli`, not here:

```bash
WORKSPACEJSON_CLI_CANDIDATE=/path/to/cli/packages/cli pnpm run check:conformance
WORKSPACEJSON_CLI_CANDIDATE=/path/to/cli/packages/cli pnpm run check:conformance:test
```

It **does not skip** when the candidate is absent; it exits non-zero with
instructions. A conformance gate that goes green because it could not find the
implementation reports conformance it never measured.

What it asserts, by stable path:

| Path | Asserted |
| -- | -- |
| `generated.fileIndex` | Non-empty from repository evidence; every key repository-root-relative POSIX; every key names a file that exists; the repository's real files are represented; keys deterministically ordered |
| `generated.frameworkManifest` | A framework corroborated by a declared dependency is published at the documented `>= 0.7` floor; an uncorroborated `AGENTS.md` token is **not**; entries deterministically ordered |
| `manual.fragileFiles` | Preserved verbatim across regeneration; absent evidence left absent, never fabricated |
| `manual.coChangePatterns` | Preserved verbatim across regeneration; absent evidence left absent, never fabricated |

And beyond the four paths: an unparseable or schema-invalid artifact is refused
rather than overwritten, `--force` moves it aside recoverably instead of
destroying it, `generated.by.name` identifies the producer rather than an
invoker, output validates against the package-owned schema, a second run against
an unchanged repository is byte-identical, and mediated invocation produces the
same artifact as direct invocation after removing only `generated.generatedAt`.

### What it deliberately does not assert

**Per-file values inside `fileIndex`.** `FileIndexEntry` declares `fragility`,
`aiModificationCount` and `humanModificationCount` as optional, so `{}` is a
conformant entry. Those values are behavioral, their only available source is
git-derived, and whether that source may enter the stable contract is an open
question tracked outside this repository. A suite requiring them would fail a
producer that is behaving correctly, and would pre-empt a ruling the standard
does not own.

Nor does it require non-empty human-owned fields, or add git-derived co-change
to the acceptance surface.

### Vreko-mediated invocation

The contract asserts that mediation does not change what a producer emits. It
verifies this against the **public** mediation surface — a host importing the
package and calling the exported producer. Vreko itself is private and outside
this repository's clean-room boundary, so it cannot be executed here; a
Vreko-specific regression belongs in that repository and does not replace this
contract.

## Known gaps

Stated plainly, because a conformance document that hides them is misleading:
Expand Down
2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@
"check:architecture:test": "node scripts/check-architecture.test.mjs",
"check:schema": "node scripts/verify-schema-provenance.mjs",
"check:examples": "node scripts/validate-examples.mjs",
"check:conformance": "node scripts/check-producer-conformance.mjs",
"check:conformance:test": "node scripts/check-producer-conformance.test.mjs",
"check:docs": "node scripts/check-docs.mjs",
"release:verify-packs": "pnpm --filter @workspacejson/spec exec node ../../scripts/verify-package-tarball.mjs && pnpm --filter @workspacejson/rules exec node ../../scripts/verify-package-tarball.mjs",
"release:verify-published": "node scripts/verify-published.mjs"
Expand Down
Loading