v1.0.0 — Shadow Score Spec Level 4 (Adversarial Independence)

.github/DISCUSSION_TEMPLATE/share-your-build.yml

@@ -23,7 +23,9 @@ body:
       description: "Which mode did you use?"
       options:
         - Full Pipeline
+        - Dark (autonomous)
         - Express
+        - Tournament
         - Premium
     validations:
       required: true
@@ -37,6 +39,17 @@ body:
     validations:
       required: false
 
+  - type: input
+    id: independence
+    attributes:
+      label: "Model independence"
+      description: >
+        Which families authored the sealed tests, and which built the code? A score
+        from same-family pairing is advisory only.
+      placeholder: "e.g., seals: openai + google | implementer: anthropic"
+    validations:
+      required: false
+
   - type: textarea
     id: highlights
     attributes:

.github/PULL_REQUEST_TEMPLATE.md

@@ -13,28 +13,44 @@ Fixes #
 
 ## Factory impact
 
-- Affects phases: 0 / 1 / 2 / 3 / 4 / 5 / 6 / 7
+- Affects phases: 0 / 1 / 2 / 2.5 / 3 / 4 / 4.5 / 5 / 6 / 7
 - Agent prompts touched (if any):
+- Does this change `config.yml` invariants, roles, or families? **yes / no**
 
 ## QA checklist (from docs/TESTING.md)
 
-- [ ] 🏭 Factory banner displays on start (run ID, mode, agent lineup)
-- [ ] 📋 PRD.md produced with user stories and acceptance criteria
-- [ ] 🏗️ ARCH.md produced with components, file structure, tech choices
+**Independence (Level 4)**
+
+- [ ] 🧬 Phase 0 prints the invariant table and the resolved family of every role
+- [ ] 🧬 Seal author and implementer are in **different** families
+- [ ] 🧬 A same-family config aborts the run (or warns, per `invariants.on_violation`)
+- [ ] 🧬 Every dispatch omits `reasoning_effort` / `context_tier` for models that lack them
+- [ ] 🧬 `python3 .github/scripts/validate_conformance.py` passes
+
+**Sealed envelope**
+
 - [ ] 🔒 Sealed tests generated from PRD only (never from code/arch)
-- [ ] 🔒 SHA-256 hash of sealed directory displayed at Phase 2
-- [ ] 👩‍💻 Engineer never accesses `.factory/sealed/` during build
-- [ ] ✅ Both test suites run in Phase 4
-- [ ] 📊 GAP-REPORT.md produced with gap score
-- [ ] 🔧 Hardening sends failure messages only (no test code)
-- [ ] 🔧 Hardening caps at configured max cycles, then escalates
+- [ ] 🔒 SHA-256 hash of the sealed envelope displayed at Phase 2
+- [ ] 🔒 Vault is outside the build worktree; validation uses a disposable verify worktree
+- [ ] 🔒 Hardening sends failure messages only (test source never revealed)
+
+**Pipeline**
+
+- [ ] 🏭 Factory banner displays on start (run ID, mode, agent lineup)
+- [ ] 📋 PRD.md, 🏗️ ARCH.md, 🧐 ARCH-CRITIQUE.md, ❓ AMBIGUITY.md produced
+- [ ] 📊 SHADOW-REPORT.md **and** SHADOW-REPORT.json produced
+- [ ] 🗡️ RED-TEAM.md classifies findings spec-gap vs implementation-bug
+- [ ] 📈 `initial_shadow_score` recorded before cycle 1; `hardening_velocity` reported
+
+**Control**
+
+- [ ] 🚦 Gates print measured value, threshold, and evidence on breach
 - [ ] 📋 Checkpoints use `ask_user` with exactly 4 choices
-- [ ] ⏭️ Skip-all still shows final delivery checkpoint
-- [ ] 💾 state.json written on every phase transition
-- [ ] 🔄 Resume works from state.json
-- [ ] 🛑 Abort cleans up worktree and branch
-- [ ] ⚡ Express mode skips PRD/arch, one checkpoint at delivery
-- [ ] 🗄️ SQL tables track runs and phase results
+- [ ] ⏭️ Skip-all still enforces gates and still shows final delivery checkpoint
+- [ ] 💾 state.json written on every phase transition, including gate results
+- [ ] 🔄 Resume works; 🛑 Abort cleans up worktrees, branches, and the vault
+- [ ] ⚡ Express mode skips PRD/arch — but never skips gates
+- [ ] 🗄️ SQL tables track runs, phases, gates, costs, and learning memory
 
 ## Notes for reviewers
 

.github/copilot-instructions.md

@@ -1,39 +1,48 @@
 # Copilot Instructions — Dark Factory
 
-This repository contains **Dark Factory**, a GitHub Copilot CLI skill that orchestrates an agent pipeline with **sealed-envelope testing**.
+This repository contains **Dark Factory**, a GitHub Copilot CLI skill that orchestrates an agent pipeline with **sealed-envelope testing** and **cross-family model independence** (Shadow Score Spec v2.0, Level 4 — Adversarial Independence).
 
 ## File map
 
 | File/Dir | Purpose |
 |---|---|
 | `SKILL.md` | Factory Manager orchestrator prompt (the brain) |
-| `agents/*.md` | Specialist agent prompts |
+| `agents/*.md` | 8 specialist agent prompts |
 | `templates/*.md` | Artifact output formats |
-| `protocols/*.md` | Protocol invariants (sealed envelope, checkpoints) |
-| `config.yml` | Tunables (models, thresholds, timeouts) |
+| `protocols/sealed-envelope.md` | Isolation invariants |
+| `protocols/model-independence.md` | Why cross-family separation is required |
+| `protocols/checkpoint-gate.md` | Autonomy modes + quality gates |
+| `config.yml` | Tunables (invariants, families, capabilities, roles, gates, thresholds) |
 | `catalog.yml` | Skill metadata + file references |
+| `.github/scripts/validate_conformance.py` | Level 4 conformance check (CI) |
 | `docs/TESTING.md` | Playbooks + QA checklist |
 | `docs/ADR.md` | Why these decisions |
 
 ## Non-negotiables
 
-1. **Sealed envelope is sacred.** The Lead Engineer must never see sealed tests; the QA Sealed agent must never see code.
-2. **Config is the source of truth.** Never hardcode model names or tunables inside prompts.
-3. **Agent prompts <= 200 lines.** If a prompt grows, split responsibilities instead.
-4. **Just a skill.** Do not add runtime code, package managers, telemetry, dashboards, or plugin systems.
-5. **Worktree isolation.** All build work happens under `.factory/` until delivery approval.
+1. **Sealed envelope is sacred.** The Lead Engineer must never see sealed tests; the QA Sealed agent must never see code. Isolation is filesystem topology (vault outside the repo, disposable verify worktree), not prompt wording.
+2. **Model independence is sacred.** Seal author and implementer must be in different families. Same-family pairing biases every Shadow Score toward 0% — it overclaims quality. Phase 0 aborts on violation.
+3. **Config is the source of truth.** Never hardcode model names or tunables inside prompts.
+4. **Agent prompts <= 200 lines.** If a prompt grows, split responsibilities instead.
+5. **Just a skill.** Do not add runtime code, package managers, telemetry, dashboards, or plugin systems. The one exception is `.github/scripts/validate_conformance.py`, which runs in CI only and must stay dependency-light.
+6. **Worktree isolation.** All build work happens under `.factory/` until delivery approval.
+7. **Delivery is always human-approved**, in every autonomy mode including `dark`.
 
 ## Prohibited actions
 
 - Exposing sealed test contents to the user before Phase 4.
 - Passing sealed test source code into any builder/hardening prompt.
+- Leaking test source, function names, or fixtures through `AMBIGUITY.md`.
+- Reporting a Shadow Score without `independence` and `seal_author_families` provenance.
+- Passing `reasoning_effort` or `context_tier` to a model that does not support it — check `config.model_capabilities` and omit.
 - Editing user files outside the factory worktree.
 
 ## PR requirements
 
 Before opening a PR:
 
 - Run Playbook 1 (Full) and Playbook 2 (Express) from `docs/TESTING.md`.
-- Ensure `catalog.yml` references are valid.
+- Run `python3 .github/scripts/validate_conformance.py`.
+- Ensure `catalog.yml` references are valid and every `agents/*.md` is listed.
 - Ensure YAML parses (`config.yml`, `catalog.yml`).
 - Ensure CI validate workflow passes.

.github/scripts/validate_conformance.py

@@ -0,0 +1,244 @@
+#!/usr/bin/env python3
+"""Validate Dark Factory config against Shadow Score Spec v2.0 Level 4 invariants.
+
+Checks:
+  1. Every dispatched model exists in families and model_capabilities.
+  2. Every reasoning_effort / context_tier is supported by its model.
+  3. cross_family_required: seal authors share no family with the implementer.
+  4. seal_plurality_min families authoring sealed suites.
+  5. tournament.enforce_judge_independence: no competitor shares a seal family.
+  6. Hardening ladder rungs avoid seal-author families.
+  7. product_mgr does not share a family with the implementer (bias direction).
+  8. arch_critic differs in family from architect.
+  9. red_team differs in family from the implementer.
+ 10. premium overrides preserve every independence invariant.
+ 11. documented report contract matches Spec section 5.2 field paths.
+"""
+import sys
+import pathlib
+import yaml
+
+ERRORS: list[str] = []
+WARNINGS: list[str] = []
+
+
+def err(msg: str) -> None:
+    ERRORS.append(msg)
+
+
+def warn(msg: str) -> None:
+    WARNINGS.append(msg)
+
+
+def main() -> int:
+    root = pathlib.Path(__file__).resolve().parents[2]
+    cfg = yaml.safe_load((root / "config.yml").read_text(encoding="utf-8"))
+
+    families = cfg["families"]
+    caps = cfg["model_capabilities"]
+    family_of = {m: f for f, models in families.items() for m in models}
+
+    def check_dispatch(label: str, spec: dict) -> None:
+        """Validate one model + effort + context triple."""
+        model = spec.get("model")
+        if model not in family_of:
+            err(f"{label}: model '{model}' missing from families map")
+            return
+        if model not in caps:
+            err(f"{label}: model '{model}' missing from model_capabilities")
+            return
+        effort = spec.get("reasoning_effort")
+        allowed_effort = caps[model]["effort"]
+        if effort is not None and effort not in allowed_effort:
+            err(
+                f"{label}: model '{model}' does not support reasoning_effort "
+                f"'{effort}' (supports: {allowed_effort or 'none — omit the parameter'})"
+            )
+        tier = spec.get("context_tier")
+        allowed_tier = caps[model]["context"]
+        if tier is not None and tier not in allowed_tier:
+            err(
+                f"{label}: model '{model}' does not support context_tier "
+                f"'{tier}' (supports: {allowed_tier})"
+            )
+
+    # --- 1 & 2: every dispatch is capability-valid -------------------------
+    for role, spec in cfg["roles"].items():
+        check_dispatch(f"roles.{role}", spec)
+    for i, spec in enumerate(cfg["seal_plurality"]["authors"]):
+        check_dispatch(f"seal_plurality.authors[{i}]", spec)
+    for i, spec in enumerate(cfg["tournament"]["competitors"]):
+        check_dispatch(f"tournament.competitors[{i}]", spec)
+    for i, spec in enumerate(cfg["hardening"]["ladder"]):
+        check_dispatch(f"hardening.ladder[{i}]", spec)
+    for role, spec in (cfg.get("premium") or {}).items():
+        check_dispatch(f"premium.{role}", spec)
+
+    # --- family resolution -------------------------------------------------
+    def fam(spec: dict) -> str | None:
+        return family_of.get(spec.get("model"))
+
+    impl_family = fam(cfg["roles"]["lead_eng"])
+    seal_families = {fam(a) for a in cfg["seal_plurality"]["authors"]}
+    seal_families.add(fam(cfg["roles"]["qa_sealed"]))
+    seal_families.discard(None)
+
+    inv = cfg["invariants"]
+
+    # --- 3: cross-family requirement (the core Level 4 invariant) ---------
+    if inv.get("cross_family_required"):
+        if impl_family in seal_families:
+            err(
+                f"invariants.cross_family_required: implementer family "
+                f"'{impl_family}' also authors sealed tests. Correlated blind "
+                f"spots make the Shadow Score optimistically biased."
+            )
+
+    # --- 4: seal plurality -------------------------------------------------
+    need = inv.get("seal_plurality_min", 1)
+    if cfg["seal_plurality"].get("enabled") and len(seal_families) < need:
+        err(
+            f"invariants.seal_plurality_min={need} but only "
+            f"{len(seal_families)} seal family/families configured: {sorted(seal_families)}"
+        )
+
+    # --- 5: tournament judge independence ---------------------------------
+    trn = cfg["tournament"]
+    if trn.get("enforce_judge_independence"):
+        for i, comp in enumerate(trn["competitors"]):
+            if fam(comp) in seal_families:
+                err(
+                    f"tournament.competitors[{i}] ({comp.get('model')}) shares family "
+                    f"'{fam(comp)}' with a seal author. It would score artificially "
+                    f"well, biasing model selection."
+                )
+
+    # --- 6: hardening rungs stay independent of sealers -------------------
+    for i, rung in enumerate(cfg["hardening"]["ladder"]):
+        if fam(rung) in seal_families:
+            err(
+                f"hardening.ladder[{i}] ({rung.get('model')}) shares family "
+                f"'{fam(rung)}' with a seal author — the engineer would share priors "
+                f"with the tests judging it."
+            )
+
+    # --- 7: PM bias direction ---------------------------------------------
+    pm_family = fam(cfg["roles"]["product_mgr"])
+    if pm_family == impl_family:
+        err(
+            f"roles.product_mgr family '{pm_family}' matches the implementer. "
+            f"The builder would infer unstated PM assumptions, deflating the "
+            f"Shadow Score and overclaiming quality."
+        )
+    elif pm_family in seal_families:
+        warn(
+            f"roles.product_mgr family '{pm_family}' matches a seal author. "
+            f"This biases the Shadow Score conservatively (upward) — acceptable, "
+            f"but should be disclosed in the report."
+        )
+
+    # --- 8: arch critic independence --------------------------------------
+    if fam(cfg["roles"]["arch_critic"]) == fam(cfg["roles"]["architect"]):
+        err(
+            "roles.arch_critic shares a family with roles.architect — a critic "
+            "with the architect's priors approves the architect's blind spots."
+        )
+
+    # --- 9: red team independence from builder ----------------------------
+    if fam(cfg["roles"]["red_team"]) == impl_family:
+        err(
+            "roles.red_team shares a family with the implementer — it will miss "
+            "the same attack surfaces the builder missed."
+        )
+
+    # --- 10: premium overrides preserve every independence invariant ------
+    prem = cfg.get("premium") or {}
+    if prem:
+        p_impl = fam(prem.get("lead_eng", cfg["roles"]["lead_eng"]))
+        p_seal = {fam(prem.get("qa_sealed", cfg["roles"]["qa_sealed"]))}
+        p_seal.discard(None)
+        if inv.get("cross_family_required") and p_impl in p_seal:
+            err(
+                f"premium: implementer family '{p_impl}' also authors sealed tests. "
+                f"A premium run must not collapse the pipeline into one family."
+            )
+        if fam(prem.get("arch_critic", cfg["roles"]["arch_critic"])) == fam(
+            prem.get("architect", cfg["roles"]["architect"])
+        ):
+            err("premium: arch_critic shares a family with architect")
+        if fam(prem.get("red_team", cfg["roles"]["red_team"])) == p_impl:
+            err("premium: red_team shares a family with the implementer")
+        for role in cfg["roles"]:
+            if role in prem and fam(prem[role]) != fam(cfg["roles"][role]):
+                warn(
+                    f"premium.{role} changes family "
+                    f"'{fam(cfg['roles'][role])}' -> '{fam(prem[role])}'. "
+                    f"Premium should raise capability within a role's family, not move it."
+                )
+
+    # --- 11: report contract matches Spec section 5.2 field paths ---------
+    # A provenance field at the wrong depth is not present as far as a
+    # validator is concerned, so the Level 4 check silently never runs and the
+    # report claims a level nobody verified. Guard the documented contract.
+    import json
+    import re
+
+    root_only = {"shadow_score_spec_version", "report", "sealed_tests",
+                 "open_tests", "failures", "coverage_comparison", "hardening"}
+    must_nest = {"shadow_score", "level", "conformance_level", "independence",
+                 "implementer_family", "seal_author_families", "advisory"}
+    l4_required = {"independence", "implementer_family", "seal_author_families"}
+    renamed = {"seal_families": "seal_author_families"}
+
+    contracts = [
+        root / "agents" / "qa-validator.md",
+        root / "templates" / "shadow-report-template.md",
+    ]
+    for path in contracts:
+        if not path.exists():
+            err(f"missing report contract file: {path.relative_to(root)}")
+            continue
+        rel = path.relative_to(root)
+        for block in re.findall(r"```json\n(.*?)```", path.read_text(encoding="utf-8"), re.S):
+            if '"shadow_score_spec_version"' not in block:
+                continue
+            try:
+                doc = json.loads(block)
+            except json.JSONDecodeError as exc:
+                err(f"{rel}: report contract is not valid JSON: {exc}")
+                continue
+            report = doc.get("report", {})
+            for old, new in renamed.items():
+                if old in report or old in doc:
+                    err(f"{rel}: '{old}' was renamed to '{new}' in Spec v2.0 section 5.2")
+            for field in sorted(must_nest & set(doc)):
+                err(
+                    f"{rel}: '{field}' must live at report.{field}, not the document root "
+                    f"(Spec section 5.2) — at the root it is invisible to the Level 4 check"
+                )
+            for field in sorted(root_only & set(report)):
+                err(f"{rel}: '{field}' belongs at the document root, not under report")
+            missing = sorted(l4_required - set(report))
+            if missing:
+                err(f"{rel}: report contract omits Level 4 provenance: {missing}")
+            if report.get("independence") == "weak" and not report.get("advisory"):
+                err(f"{rel}: weak independence must be stamped advisory (Spec section 5.4)")
+
+    for w in WARNINGS:
+        print(f"::warning::{w}")
+    for e in ERRORS:
+        print(f"::error::{e}")
+
+    if ERRORS:
+        print(f"\n❌ {len(ERRORS)} Level 4 conformance violation(s)")
+        return 1
+
+    print("✅ Shadow Score Spec v2.0 Level 4 conformance checks passed")
+    print(f"   implementer family : {impl_family}")
+    print(f"   seal families      : {sorted(seal_families)}")
+    print(f"   independence       : strong")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())