Skip to content

Council vNext: challenge-derived seats and proposal-only findings #52

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Kitahl wants to merge 1 commit into
base: upgrade/challenge-vnext-mind
Choose a base branch
from
Open

Council vNext: challenge-derived seats and proposal-only findings #52

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 151 additions & 42 deletions docs/specs/COUNCIL_ENGINEERING_SPEC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,54 +2,163 @@

## Obligation

Provide selective independent review of a concrete artifact after a strong direct pass, while measuring rather than assuming diversity/independence.
Provide selective, controlled review of a concrete artifact after a strong direct pass while measuring rather than assuming review diversity or independence.

## Current workflow
Council owns only the `REVIEW` obligation. It may produce supported findings and proposal-only neutral challenges for target modules, but it cannot clear a `PROOF`, `DISCOVERY`, `SYNTHESIS`, `ENGINEERING`, `EVALUATION`, `ASSURANCE`, `PREFLIGHT`, `ADAPTATION`, or `ADVERSARY` obligation.

Specification: 3–6 artifact-derived seats, skeptic, frozen independent first passes, disjoint evidence where feasible, reveal/cross-critique, evidence-ranked synthesis, matched direct control.
## Runtime and schemas

Closest executable helper currently uses generic paired red-team prompts and synthesis; it does not enforce the full Council contract.
- Runtime: `tools/council_runtime.py`
- Domain receipt schema: existing `egrt.runtime.v1`
- Challenge schema: additive `egrt.challenge.v1`
- Neutral challenge implementation: `tools/egrt_challenge_types.py` and `tools/egrt_challenge.py`
- State machine: `COMMIT -> REVEAL -> CROSS_CRITIQUE -> CLOSED`

## vNext state machine
Council never imports `foil_*`. FOIL remains an optional later controller and is not part of the native Council path.

`COMMIT -> REVEAL -> CROSS_CRITIQUE -> CLOSED`
## Frozen Council state

A Council has:

- artifact hash and frozen total-budget hash;
- 3–6 distinct seat questions;
- role/method/evidence partition per seat;
- an optional bound task ID;
- a frozen artifact hash and frozen total-budget hash;
- 3–6 seats;
- first-pass commitment hashes;
- sealed submissions;
- revealed submissions;
- provenance/evidence/method/finding overlap diagnostics;
- structured cross-critique records, with every seat contributing after reveal;
- DIRECT and optional VOTE control receipt IDs.

## vNext workflow

1. Confirm concrete artifact and existing direct analysis.
2. Derive 3–6 seats from distinct failure/verification obligations.
3. Freeze seat role/question/method/evidence partition.
4. Each seat commits first-pass submission hash before reveal.
5. Reveal only when all commits are frozen; tampered reveals fail hash verification.
6. Compute evidence/provenance/method/finding overlap.
7. Cross-critique only after reveal.
8. Require each seat to contribute a structured cross-critique against another revealed seat.
9. Synthesize by claims/evidence, preserving disagreement.
10. Compare against a DIRECT control on the same frozen artifact and matching total-budget hash; optionally compare VOTE.
11. Council's *marginal-value* claim remains UNKNOWN without same-artifact/same-budget direct-control evidence.
12. Record confidence as uncalibrated unless prospective Brier/log-score calibration exists.

## Runtime

`tools/council_runtime.py`

## Mechanical tests

- reject <3 or >6 seats;
- reject duplicate seat questions;
- double commit forbidden;
- tampered reveal rejected;
- overlap matrix correct;
- missing direct control keeps marginal-value verdict UNKNOWN.
- sealed and revealed submissions;
- structured cross-critiques;
- method, evidence, provenance, reviewer-provenance, and finding-overlap diagnostics;
- structured Council findings;
- supported finding IDs selected at synthesis;
- a mandatory same-artifact, same-total-budget `DIRECT` control receipt;
- an optional `VOTE` control receipt;
- one Council receipt that can clear only the bound `REVIEW` obligation.

## Challenge-derived seat contract

A vNext `CouncilSeat` extends the legacy seat fields with:

- `challenge_kind`;
- `discriminator`;
- `required_capability`;
- `target_obligation_id`;
- `refuter`;
- `reviewer_provenance`;
- `challenge_contract="vnext"`.

Legacy four-field seats remain readable and executable for backward compatibility. A Council cannot mix legacy and vNext seats in one run.

### Residual-gap seat classes

Derive seats from unresolved neutral challenge classes rather than generic personalities:

| Residual class | Typical role | Decisive partition or discriminator |
|---|---|---|
| Formal correctness | proof/formalization | exact derivation, contradiction, alternate formalization, counterexample |
| Evidence and provenance | source assessment | first-party evidence, source conflict, citation/provenance lineage |
| Executable behavior | implementation/integration | real entrypoint, failure class, output contract, metamorphic relation |
| Measurement validity | evaluation design | baseline/estimand, contamination, scorer, stopping or budget boundary |
| Novelty or costume | prior-art/ablation | strongest known costume, assumption knockout, transfer or ablation |
| Operational feasibility | state/decision/operations | state drift, resource boundary, reversal condition |

`derive_challenge_seats()` deterministically converts open `ChallengeRequest` objects into vNext seats. Resolved and dismissed challenges are ignored. Load-bearing, higher-risk, higher-information, and lower-cost challenges are prioritized within the frozen maximum of six seats.

Every seat must own a distinct question. Duplicate challenge kinds or duplicate discriminators are rejected unless both seats declare genuinely distinct evidence partitions. A skeptic/adversarial seat is mandatory and must name a concrete refuter; a generic instruction such as “be critical” does not satisfy the contract.

## Commit–reveal and cross-critique

1. Freeze the artifact, total budget, seats, questions, methods, evidence partitions, challenge bindings, and reviewer provenance.
2. Each seat commits a salted hash of its structured first-pass submission before any reveal.
3. Reveal only after all commitments are frozen. A wrong nonce or altered submission fails verification.
4. Compute evidence, provenance, method, reviewer-provenance, and finding overlap.
5. Cross-critique only after every reveal.
6. Require every seat to contribute a structured critique against another seat.
7. Synthesize by supported finding IDs and preserve unresolved disagreement.

Nominal seat count never establishes statistical independence. Every pair reports `independence_status="NOT_ESTABLISHED"`, and the Council-level diagnostic is `NOT_ESTABLISHED_BY_SEAT_COUNT`. Same-model or same-reviewer provenance remains explicit.

## Structured findings and neutral challenge output

A supported vNext finding is recorded as `CouncilFinding` with:

- seat ID;
- target module and target obligation ID;
- challenge kind;
- hypothesis, alternative, concrete refuter, and consequence if true;
- load-bearing flag and required capability;
- evidence partition;
- candidate, scope, and obligation-set SHA-256 bindings;
- provenance and typed metadata.

`record_finding()` is permitted only after all seats reveal. For vNext seats, the finding must match the seat’s challenge kind, target obligation, required capability, concrete refuter, and evidence partition, and its finding ID must have appeared in that seat’s frozen revealed submission.

A vNext Council cannot clear its `REVIEW` obligation with a supported finding that lacks a structured `CouncilFinding` record.

After Council finalization, `propose_supported_finding_challenge()` may convert one supported structured finding into a neutral `ChallengeRequest` with `origin=COUNCIL`. The emitted object is:

- bound to the same task, target obligation, candidate, scope, and obligation set;
- stored initially as `PROPOSED`;
- marked `authority=PROPOSAL_ONLY`;
- unable to resolve itself;
- unable to create a domain receipt;
- unable to clear the target module’s obligation;
- subject to the shared challenge budget, deduplication, selection, and release rules.

The target module or host must select a discriminator and the claim-native module must produce the resolving evidence.

## Control and release semantics

A Council review requires a real `DIRECT` control receipt with the same:

- review obligation ID;
- frozen artifact hash;
- frozen total-budget hash.

The optional `VOTE` control is diagnostic only.

Council finalization returns:

- `UNKNOWN` when commit/reveal is incomplete;
- `UNKNOWN` when cross-critique participation is incomplete;
- `UNKNOWN` when the matched `DIRECT` control is absent, invalid, or mismatched;
- `UNKNOWN` when a vNext supported finding lacks a structured record;
- `UNKNOWN` when synthesis supports no findings;
- `CLEARED` only when the frozen Council review protocol completed and at least one supported finding remains.

`CLEARED` means only that the controlled Council protocol completed for the `REVIEW` obligation. It is not a verdict on the target domain claim and not evidence that Council outperforms `DIRECT` or `VOTE`.

When a bound task exists, attempting to finalize Council against a non-`REVIEW` obligation raises `CouncilAuthorityError`. A Council receipt records `authority=REVIEW_ONLY` and `target_domain_clearance_authorized=false`.

## Backward compatibility

- Existing `CouncilSeat(seat_id, role, question, method, evidence_partition=None)` construction remains valid.
- Existing commit/reveal, overlap, cross-critique, control, and finalization behavior remains valid for legacy Council state.
- Historical Council state loads through inferred legacy challenge fields; historical receipts are not rewritten.
- Native Council challenge behavior is additive and uses the shared challenge configuration.
- No FOIL profile, FOIL receipt, or seat count changes evidence authority.

## Mechanical acceptance tests

`tests/test_council_challenge_seats.py` proves:

- distinct challenge questions and duplicate-kind/discriminator rejection;
- deterministic derivation from open challenge classes;
- concrete skeptic/refuter behavior;
- structured supported finding to neutral `COUNCIL` challenge conversion;
- target-domain authority denial;
- mandatory same-artifact/same-budget `DIRECT` control;
- same-model seats are never labeled independent solely by seat count.

Existing Council tests continue to prove:

- 3–6 seat bounds;
- skeptic requirement;
- commitment hiding and binding;
- tampered reveal rejection;
- overlap diagnostics;
- complete cross-critique participation;
- matching control semantics;
- non-recallable finalization.

## Non-goals and efficacy boundary

This work does not establish that Council improves task outcomes. It does not treat agreement as truth, seat count as independence, or a Council finding as claim-native evidence. Marginal-value claims still require prospective, matched-budget comparison against `DIRECT`; null and negative results must be retained.
81 changes: 56 additions & 25 deletions skills/council-of-elders/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,53 +1,84 @@
---
name: council-of-elders
description: Evidence Review Panel. Trigger: /council, /forum, "independent review", or equivalent. Selectively convenes independent, evidence-grounded review roles around a concrete artifact after a strong direct pass exists. Uses commit-reveal, disjoint evidence where possible, a skeptic, and a matched direct control. Agreement is not truth.
description: Evidence Review Panel. Trigger: /council, /forum, "independent review", or equivalent. Selectively convenes challenge-derived, evidence-grounded review seats around a concrete artifact after a strong direct pass exists. Uses commit-reveal, distinct discriminators or evidence partitions, a concrete skeptic refuter, cross-critique, overlap diagnostics, and a matched direct control. Agreement is not truth.
---

# Evidence Review Panel

The public panel uses **artifact-derived review roles**, not simulated authority from named personalities.
Use artifact-derived review seats, not named personalities or simulated authority.

## When to convene

Default: **off**.

Convene only when:

- there is a concrete artifact/claim/design to review;
- there is a concrete artifact, claim, design, or candidate to review;
- a strong direct analysis already exists;
- independent evidence/methods are plausibly available;
- the marginal value can be compared against a direct control.
- at least three genuinely different challenge questions, discriminators, or evidence partitions are available;
- the total artifact and review budget can be frozen;
- marginal value can be compared against a same-artifact, same-budget direct control.

Do not convene merely because the problem is hard or more opinions sound reassuring.
Do not convene merely because a problem is difficult or additional opinions sound reassuring.

## Seat construction
## Construct challenge-derived seats

Prefer 3–6 roles derived from the artifact, for example:
Prefer 3–6 seats derived from unresolved load-bearing gaps:

- formal correctness;
- empirical/measurement validity;
- implementation/integration;
- adversarial failure modes;
- prior art/novelty;
- cost/operational feasibility.
- evidence and provenance;
- executable behavior or integration;
- measurement validity;
- novelty or strongest costume;
- operational feasibility.

Include a skeptic role. Add a role only when it has a distinct question, evidence pack, verifier, or method.
Every seat must own:

## Commit-reveal
- one distinct challenge question;
- one concrete discriminator or refuter;
- one required capability when applicable;
- one target obligation;
- one method or explicit evidence partition.

1. Each role independently freezes its first-pass conclusion before seeing other roles.
2. Use disjoint searches/evidence where feasible.
3. Reveal and cross-critique after independent commits.
4. Normalize claims to the same scope.
5. Rank by evidence, not identity or vote count.
6. Preserve unresolved disagreement.
Duplicate challenge kinds or discriminators are allowed only when the evidence partitions are explicitly different. Include a skeptic/adversarial seat with a concrete refuter; “be critical” is not a discriminator.

## Control
## Commit–reveal and cross-critique

Run one strong direct analysis with access to the same combined evidence budget when feasible. Attribute panel value only to findings the control did not produce or to stronger verification/coverage.
1. Freeze the artifact hash, total-budget hash, seats, methods, challenge bindings, evidence partitions, and reviewer provenance.
2. Each seat independently commits its first-pass structured submission before seeing another seat’s result.
3. Reveal only after all commitments are frozen.
4. Reject altered submissions or wrong nonces.
5. Compute method, evidence, provenance, reviewer-provenance, and finding overlap.
6. Require every seat to cross-critique another revealed seat.
7. Synthesize by claim and evidence, preserving disagreement.

Panel output remains a claim until independently checked when the conclusion is load-bearing.
Do not call seats independent because there are several of them or because they produced separate messages. Same-model and shared-evidence common causes remain explicit.

## Supported findings

A load-bearing supported finding remains a review finding until the target module checks it.

The Council runtime may convert a supported structured finding into an additive neutral challenge with `origin=COUNCIL`. That challenge is proposal-only. It cannot resolve itself, replace a domain receipt, clear the target obligation, apply a repair, or authorize a write.

## Control and authority

Run a strong direct analysis against the same frozen artifact and matching total budget. Without that `DIRECT` control, Council’s review verdict remains `UNKNOWN`.

A Council receipt may clear only a `REVIEW` obligation. It never clears proof, discovery, synthesis, engineering, evaluation, assurance, preflight, adaptation, or adversary obligations. User or host authority remains required for adoption or writes.

## Typed runtime contract

`tools/council_runtime.py` enforces 3–6 distinct seat questions, a skeptic/adversarial seat, frozen commitment hashes before reveal, reveal integrity, and evidence/provenance overlap diagnostics. Council's REVIEW verdict remains `UNKNOWN` until commit/reveal is complete, every seat contributes cross-critique, and a real DIRECT control matches the same frozen artifact and total-budget hashes. Confidence is recorded as uncalibrated unless later scored prospectively. See `docs/specs/COUNCIL_ENGINEERING_SPEC.md`.
`tools/council_runtime.py` enforces:

- 3–6 seats and a skeptic/adversarial seat;
- challenge kind, discriminator, capability, target obligation, refuter, and evidence partition for vNext seats;
- rejection of duplicate challenge/discriminator seats without distinct partitions;
- commitment hiding and reveal integrity;
- complete cross-critique participation;
- overlap diagnostics with `NOT_ESTABLISHED_BY_SEAT_COUNT`;
- structured `CouncilFinding` binding;
- proposal-only `COUNCIL` challenge emission;
- `REVIEW_ONLY` receipt authority;
- a same-artifact, same-total-budget `DIRECT` control.

Confidence remains uncalibrated unless scored prospectively. See `docs/specs/COUNCIL_ENGINEERING_SPEC.md`.
Loading