Shadow Score Spec v2.0.0 — Level 4: Adversarial Independence

[DUBSOpenHub]

The hole this closes

Levels 1–3 progressively harden information isolation. They answer "can the Implementer see the sealed tests?" — and they answer it well.

None of them answer "does the Implementer think like the Seal Author?"

All three are silently defeated by a single configuration choice: pointing the Seal Author and the Implementer at the same model. Same-family agents share training data, reasoning priors, and therefore failure modes. If the family doesn't think to test a scenario, it also doesn't think to handle it. The sealed test is never written, the implementation is never hardened, the score reads 0%, and the defect ships green.

The bias is directional. It always pushes toward 0%. A perfectly isolated, fully Level 3 conformant pipeline built on one model family systematically overclaims quality — and does so most confidently on exactly the requirements neither side considered.

This was not hypothetical. The reference implementation had qa_sealed and lead_eng both set to claude-sonnet-4.6: same family on both sides of a test designed to measure independence.

What's new

Section	Adds
§3.6	Authorship Independence — strong/weak classes + bias-direction table
§4.2.1	Independence enforced pre-dispatch, not audited afterwards
§4.4.1	Progressive disclosure — bounded escalation that never reveals test source
§4.4.2	Escalation may not move the Implementer into a Seal Author's family
§4.6	Seal Plurality — contradiction vs divergence, and Spec Ambiguity
§5.4	Advisory Reports — when a score MUST be marked non-authoritative
§6	Level 4 — Adversarial Independence

The bias-direction table is the interesting part

A blanket "all roles must differ" rule is wrong — it imposes cost where there's no risk. What matters is which way the bias moves:

Role pair	Effect	Verdict
Seal Author == Implementer	Too low — overclaims quality	❌ MUST NOT
Specifier == Implementer	Too low — infers unstated assumptions	❌ MUST NOT
Specifier == Seal Author	Too high — conservative	⚠️ MAY, must disclose
Validator == anyone	None — it executes, doesn't author	✅ MAY

Upward bias penalises the Implementer for requirements never stated: visible and correctable. Downward bias hides defects. Only the second kind is forbidden.

Seal plurality grades your spec, not your code

One sealed suite tells you whether the implementation is right.
Two independent sealed suites tell you whether the specification is right.

Two authors testing different things is divergence — the expected coverage dividend. Two authors asserting incompatibly on the same behaviour is a contradiction: proof the spec supports two different systems. spec_ambiguity measures that, and it's catchable before any code exists.

Corrections to v1.0.0

§4.3 — sealed tests no longer enter the implementation workspace. v1.0.0 step 1 said to copy them in. That's safe only when validation is terminal. Hardening (§4.4) re-invokes the Implementer after validation, so by cycle 2 the sealed tests sit in a directory it can grep. Agents with unscoped shell tools are bounded by what they can do, not by what the prompt asked them not to. v2.0 validates in a disposable workspace built from the Implementer's commit. Single-shot validators keep the old behaviour via workspace_isolation: "legacy".

§3.5 / §4.4 — Hardening Velocity was uncomputable. Level 3 required tracking it, but nothing required storing initial_shadow_score, and only the final score was persisted. Now has a formula and a recording requirement.

§4.2 — isolation ranking. Topological isolation now ranks above context isolation, with an explicit note that context isolation is insufficient for agents holding unscoped filesystem tools.

Repair: the conformance suite was 0/18

The gap-score → shadow-score rename had been applied to filenames but to nothing that referenced them. run_conformance.py defaulted to validators/gap-score.py, which doesn't exist — every fixture died with exit code 2. fixtures.json still expected gap_score fields the validators stopped emitting. CONTRIBUTING.md and SECURITY.md pointed at files that aren't there.

CI never ran the suite, so nobody found out.

before:  Results: 0/18 passed (18 failed)
after:   Results: 18/18 passed ✅ All conformant

CI additions, so it can't happen again

• run the conformance suite
• assert SPEC.md version matches all three validators
• fail the build on stale gap-score naming
• verify every relative markdown link resolves
• check the schema is well-formed Draft-07

Verification

✅ conformance suite      18/18
✅ Python unit tests      39/39
✅ Go validator           builds + smoke tests pass
✅ JSON schema            valid Draft-07; conditional rules verified
✅ section cross-refs     no dangling § references
✅ markdown links         all resolve
✅ version consistency    SPEC.md == py == sh == go == 2.0.0

Schema conditionals were tested directly — Level 4 without provenance is rejected, advisory: true without advisory_reason is rejected, Level 2 without provenance is allowed.

Reference implementation

DUBSOpenHub/dark-factory#27 implements Level 4 and ships green: implementer anthropic, seal authors openai + google, independence strong, enforced at Phase 0 with abort-on-violation and re-checked in CI.

---

Breaking: major version bump. Level 3 implementations remain conformant at Level 3; §4.3 changes behaviour for anyone who hardens, with legacy as the opt-out.