Skip to content

test(organization): require rejection fixtures to prove their intended diagnostic - #122

Merged
bketelsen merged 1 commit into
mainfrom
cockpit/worker-610ea0ea59d41b6e
Aug 25, 2026
Merged

test(organization): require rejection fixtures to prove their intended diagnostic#122
bketelsen merged 1 commit into
mainfrom
cockpit/worker-610ea0ea59d41b6e

Conversation

@bketelsen

Copy link
Copy Markdown
Contributor

Summary

  • organization/fixtures/v1/invalid/expected-rejections.json names the stable diagnostic each invalid organization fixture is meant to exercise (constraint label + a substring that must appear in the thrown OrganizationValidationError).
  • scripts/lib/organization-validation.mjs: validateOrganization now loads that manifest, fails closed if its keys drift from the invalid-fixture corpus (missing or stale entries), and calls the new exported assertExpectedRejection on every invalid-fixture rejection so a fixture rejected for the wrong reason fails the check instead of silently passing because any OrganizationValidationError was thrown.
  • test/organization-validation.test.mjs: focused coverage that (a) an unrelated OrganizationValidationError does not satisfy a fixture's expectation while a matching one does, (b) a fixture with no expectation fails closed, and (c) every fixture file under organization/fixtures/v1/invalid has a non-empty expected diagnostic in the manifest.
  • No schema semantics or product/runtime behavior changed — this is rejection-fixture signal quality only, per the item's scope.

Risk tier

Risk tier: 2 — quality-gate/test-only change (scripts/lib/organization-validation.mjs, test/**, and a new fixture-expectations manifest). No schema semantics, product runtime behavior, or CI wiring changed; docs/skills/templates untouched so tier 1 doesn't apply.

Docs housekeeping

Not applicable — no docs touched.

Verification

npm run check
# check:docs   -> ok (58 docs, 428 links, 10 symlinks)
# check:organization -> organization authority valid: 7 declarations, 0 verification profiles,
#                        0 Goals, 7 valid fixtures, 18 rejection fixtures
# check:scripts -> ok bash_syntax: 4/4
# test          -> 27 pass, 2 fail (pre-existing: sync-skills-containment.test.mjs requires the
#                  `rsync` binary, which is unavailable in this sandbox; confirmed by stashing
#                  this change and re-running on the clean tree — same 2 failures, unrelated to
#                  this diff)

Also verified by hand that a deliberately mismatched manifest entry makes validateOrganization throw rejected for a different diagnostic than its expected "..." constraint instead of silently accepting the fixture.

Test plan

  • npm run check:organization — 18/18 invalid fixtures match their expected diagnostic
  • node --test test/organization-validation.test.mjs — new tests pass
  • npm run check — only the pre-existing, unrelated rsync-dependent failures remain (verified present on the clean tree too)

…d diagnostic

Any OrganizationValidationError previously satisfied an invalid fixture, so a
fixture could keep "passing" after drifting to reject for the wrong reason.
Add an expected-rejections.json manifest naming the stable diagnostic each
invalid fixture must produce, enforce it during corpus validation, and cover
the mismatch and missing-expectation paths with focused tests.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@bketelsen
bketelsen marked this pull request as ready for review August 25, 2026 01:01
@bketelsen
bketelsen added this pull request to the merge queue Aug 25, 2026
Merged via the queue into main with commit 26f3df8 Aug 25, 2026
3 checks passed
@bketelsen
bketelsen deleted the cockpit/worker-610ea0ea59d41b6e branch August 25, 2026 01:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant