Milestone: full ACS-Core conformance (requirement ledger + behavioral tests, beyond adapter emission)

[bar-capsule]

Note: Scope proposed during PR #22 review; item selection and sequencing are for the workstream leads to decide, not settled here.

Context

PR #22 (reference adapters) now proves emission conformance: the reference adapters, driven through their real production entry points, emit schema-valid, signed ACS v0.1.0 Core traffic and honor the decisions the suite tests, validated against the canonical schemas by an independent oracle. Three concrete Guardian bugs found by direct probing (version forward-compat, malformed-ClientHello acceptance, non-object/batch JSON-RPC crash) were also fixed.

This is not the same as a full ACS-Core deployment conformance claim, which spans the adapter, the Guardian, the framework wiring, and production configuration together. This issue tracks that larger program so the emission work can ship with an honestly-scoped claim and the rest is visible, not silently implied.

Candidate scope (unordered — for lead prioritization)

• Normative-requirement ledger. A machine-readable ledger over every MUST/REQUIRED in docs/spec/conformance.md and the sections it cites (~58 occurrences, some conditional), each row:

  • stable requirement ID + exact normative text
  • role: adapter / Guardian / framework / deployment
  • profile + applicability condition
  • implementation location
  • positive, negative, integration tests
  • status: conformant / partial / deferred / not-applicable

  CI fails when an applicable MUST has no falsifying test. Resolve the normative inconsistency the review flagged: conformance.md lists all five dispositions and Wrapped MCP as Core, while the suite/README treat some as conditional / "Core minus full Wrapped MCP" (see the Wrapped MCP item, and PR Slim ACS-Core: relax MODIFY, system/ping, and wrapped MCP to SHOULD #21 which relaxes MODIFY/ping/MCP).

• Handshake & negotiation semantics. Beyond the three fixed bugs: PROVENANCE_REQUIRED, unknown-field handling, startup proceed/refuse, session-refusal. Adapters currently consume only on_decision_failure; prove they honor timeout_config (incl. per-method), methods_evaluated, selected_transport, negotiated signature algorithms, skew_window_ms, accepted profiles. Reconcile the stdio advertisement in acs_common with HTTP-only transport.

• Two-sided live payload validation. Guardian must validate request envelopes for handshake/ping and hook payloads against method-specific schemas on the live path (not just locally). Prove: malformed ClientHello/ping/every hook payload rejected on the wire; valid unknown fields ignored; unsupported batches → -32600; adapters validate the full ServerHello + response-envelope schema, not only signature/binding; contradictory responses follow the failure posture.

• Complete decision semantics. Negotiated timeouts applied; invalid MODIFY composition (wholesale+structured, overlapping paths) fails closed; live redaction / modified_content application; DEFER suspends until resolution or resolution_timeout_ms (not immediate deny/ask); cascading-DEFER bounds; ASK approver auth, recursive-ASK rejection, approver-incapable substitution, session-scoped Intent extension; side-effect-negative DENY tests on every platform.

• Hook completeness (black-box). Real-framework tests attempting shell/process exec, file read/write/delete, network egress, MCP calls, subagent spawn, memory/knowledge access — assert the Guardian receives the hook BEFORE execution and DENY leaves a canary untouched. Cursor stays manual until a GUI/product-level harness exists.

• Wrapped MCP: implement or formally remove from Core. Either implement + test full protocols/MCP/* (intact wrapping, version negotiation, decision application, error mapping, request/result correlation), or change the normative Core definition so generic steps/toolCallRequest/Result collapsing is explicitly conformant. Spec-owner decision — being taken to the ACS leads.

Profiles (do not implement until intending to claim)

• ACS-Trace — OTel/OCSF per supported step, decisions, provenance forwarding, non-blocking sink failure
• ACS-Inspect — pre-content agbom/snapshot, Guardian acceptance, audit-chain inclusion, CycloneDX/SPDX/SWID
• Inspect-Dynamic — agbom/changed on every mutation
• ACS-Provenance — strict schemas + deterministic provenance on every data-bearing field
• ACS-Crypto — ML-DSA-65 + verification/negotiation
• ACS-Audit — every ContextEntry carries request_hash

Deployment attestation

A preflight that verifies a running deployment: signing enabled / no ACS_DEV_MODE; ACS_DISABLED unset; HTTPS for non-loopback; secret-file perms + rotation; durable audit sink + fail-open alerts; approved framework/adapter versions + config hashes; no unexpected skips without recorded evidence.

[astrogilda]

"CI fails when an applicable MUST has no falsifying test" is the load-bearing line in this issue, and it is the one with no instrument behind it. The ledger you describe cannot check that on its own. I have been running the thing that can for months, on a public conformance corpus for an in-toto attestation predicate, and the numbers are all published.

A ledger row proves a requirement is named. It cannot prove the named test would fail if the requirement were violated, and those are different properties that look identical in a table. The measurement is mutation: switch off exactly one rule in the reference implementation, replay the entire corpus against the weakened build, and see whether the corpus notices. My published baseline at the current head of the corpus is seven hundred and fifty-four single-site weakenings of the reference rail, recorded as four hundred and twenty-four killed, twenty-seven silent, two hundred and ninety-eight dead, and five inconclusive, where the file's own definitions are that killed means some vector goes from pass to fail, silent means the corpus sees the rail change and stays green, dead means nothing in the corpus depends on that rule at all, and inconclusive means the mutant did not build and terminate cleanly so nothing is asserted about it. The corpus is two hundred and fifty vectors, and it cites ninety-three normative conditions with at least one vector each. So on the ledger axis it looks complete. Twenty-seven weakenings still went through unnoticed, and two hundred and ninety-eight rules have nothing in the corpus depending on them. A conformance ledger built without this instrument will report those as covered, because they are covered. They are simply not forced, and forced is what your CI requirement actually means.

ACS should have a companion document to that baseline from day one of the program, because it is the artifact that buys the standard credibility with a buyer. A contributor will already take your word for it; a buyer will not. Mine is called "Where this suite is weak", it is generated from the mutation baseline on every run and never hand-written, and it states that of the ninety-three conditions the corpus cites, sixty-six are forced by a vector no other condition's vectors duplicate, twenty-seven are covered only redundantly, and zero are not forced at all. It also names the sites where "unforced" would be the wrong word, either a true equivalent mutant no vector can ever kill or a real rule an earlier check reaches first, and each of those annotations is itself falsifiable, because a site annotated as unkillable that is ever killed fails the gate. A suite that publishes only what it covers is asking to be taken on trust, which is the posture the conformance label is supposed to replace.

Every killed row is a tighten-only ratchet: a rule that has stopped being forced fails the build, and the baseline never quietly updates itself, which is the difference between a measurement and a status page. And the generated document is re-derived by a second implementation sharing no code with the generator, because drift and correctness are different properties and a --check flag only ever sees the first.

On your requirement-ledger scope specifically, I would add a column and a companion file. The column is the mutation result per requirement, so a row reads conformant only when a weakening of that requirement is killed by a test the row names. The companion file is the one I did not know I needed until an independent implementer worked from my text alone: a registry of every interpretation decision a from-spec implementer must make, with the specification lines it turns on, the rival reading it was chosen over, and the vector that locks it. Mine carries twenty-one decisions, all classified forced, with no open corners, and it also carries an explicit list of nineteen of those twenty-one for which no differential witness has yet been produced, with the gate set to fail any id left on that list once it gains one. I mention the unfinished part because that list is the most useful thing in the file. Agreement between two implementations is not evidence that the text forces a reading; it is evidence that both made the same choice. For ACS, methods_evaluated semantics, the §6.4 refusal-versus-failure question in the issue next door, and the Wrapped MCP disposition you flag here as needing a spec-owner decision are all exactly this shape: places where two conformant implementations can already diverge and nobody would find out until an incident.

That is also the honest answer to your Wrapped MCP item. The normative inconsistency you name, conformance.md listing all five dispositions and Wrapped MCP as Core while the suite and README treat some as conditional, is not resolvable by testing, because a test cannot tell you which of two documents the working group meant. What testing does is put a bound on how long it can stay unresolved: a requirement ledger with a mutation column reports it immediately as a Core requirement with no forcing test, and it stays on the report until somebody decides, which is a good deal cheaper than finding out from a vendor's marketing page.

One governance rule I would want written down at the start, while it is still cheap, since this issue is about a claim as much as it is about tests: the party that scores an implementation must not be the party that shipped it, and every recorded run states its independence relation. The only reason my own numbers mean anything is that an unrelated author wrote a from-spec verifier with its own parser, canonicalizer, Merkle root and signature tier, with no sight of my reference code, and scored one hundred and seventy-nine out of two hundred and thirty-two blind. The corpus records that blind figure as the headline and the later two-hundred-and-thirty-two-out-of-two-hundred-and-thirty-two directed run underneath it, with a note that the directed pass followed a specification diff already read and therefore says nothing about whether the text determines the reading. Fifty-three mismatches published as the headline number is worth more to a buyer than a clean sheet, because it is the only version of the number that could have come out differently.

Everything above is public and Apache-2.0 at github.com/astrogilda/aee-conformance: the mutation baseline, the generated weakness page, the interpretation registry and the gates that hold all three. I have put the longer version of this, covering the registry row shape and the steward question, on #19. Concretely, the first cycle I would take on here: extract the ledger you describe, one row per MUST in conformance.md and the sections it cites, with the role, profile and applicability columns you list plus the mutation column; run the first forcing campaign against the reference Guardian and publish the baseline and the generated weakness document alongside it; and wire the falsifying-test gate so it fails on an applicable MUST whose weakening nothing catches, rather than on an applicable MUST with no test named. Tell me which piece is most useful first and I will start there.

Sankalp Gilda

[ryjen]

One additional behavioral conformance case may fit #33: valid authorization for action A should not count as successful execution if the downstream effect materially corresponds to B instead.

A simple way to think about it is: permission to do one thing is not proof that the system actually did that thing.

Proposed paired case:

authorize A
-> dispatch A
-> downstream effect is B
-> independently observed execution or terminal state confirms B
-> correspondence FAIL

Positive control:

authorize A
-> dispatch A
-> downstream effect is A
-> independently observed terminal state confirms A
-> correspondence PASS

This is intentionally different from #29/#30. Those cases primarily test whether an action should be admitted before execution. This case starts after a valid authorization and asks whether the effect that actually occurred still corresponds to the action that was authorized.

The property under test is:

authorization evidence
  != dispatch evidence
  != provider result
  != independently observed terminal state

A valid authorization for A establishes that A was permitted under the modeled policy inputs. It does not establish that the framework, tool adapter, or downstream provider actually performed A, and it should not retroactively authorize a materially different B.

ACS already has useful request/result correlation through toolCallRequest, toolCallResult, and request_id_ref. I would not treat the result hook alone as authoritative proof of what physically happened, though. The stronger behavioral assertion is to correlate the authorized request with independently observed execution evidence or deterministic terminal-state evidence.

For example, if an authorized operation is:

write "approved-content" to workspace/allowed.txt

but the downstream effect instead changes:

workspace/protected.txt

the authorization record can still remain valid and immutable while the execution-correspondence check fails. That distinction seems useful because it separates “was this action allowed?” from “is this the action that actually happened?”

I would keep this at the behavioral/integration conformance layer initially rather than proposing a schema change. If the #33 workstream finds that current request/result correlation is insufficient to express the test cleanly, that would be the point to discuss whether additional representation is needed.

Scope is intentionally narrow. This would not claim production complete mediation, honesty of arbitrary provider receipts, absence of unobserved side effects, correctness of every provider-specific semantic mapping, or universal TOCTOU protection. It would test only the authorized-action versus observed-effect correspondence boundary.

[narko4u]

Three strong contributions are already on the table: the ledger itself, the mutation-forcing instrument that makes "has a falsifying test" measurable, and the authorized-action vs observed-effect correspondence case. One dimension is missing from all three, and it is the one that decides what the conformance label is worth to a buyer: who can check the claim from outside the deployment, and against what.

Everything in the current scope is producer-side. Mutation forcing proves the reference implementation's rules are testable, which is a property of the test suite; it does not prove a specific vendor deployment's Guardian enforced them, which is a property of that deployment. The independence rule governs who does the scoring. The remaining question is what the scored artifact can say about a different deployment, and the ledger has no vocabulary for it today. "Independently observed execution or terminal state" is the right direction, but it needs the same question answered: independently by whom, observed against what basis, on a path the authorization cannot influence? If the terminal-state evidence is produced by the same framework that executed the action, the result is correlation, not correspondence.

Proposal, additive to the ledger:

1. Witness-scope column. Each requirement row names who can observe its satisfaction: SELF (the deployment's own reports), PEER (another party within an agreement), EXTERNAL (a party outside the trust domain). Most enforcement rows will be SELF-only under the current wire format. That is a finding, not a failure, and it should be visible in the ledger.

2. Verification-basis column. What artifact the witness checks against: wire transcript, signed decision records, policy state, terminal-state evidence, anchor commitments. Where the only basis is the deployment's own report, the row is SELF-only.

3. Claim-scope rule. A conformance claim must state its witness scope. "Emission-conformant" and "operationally conformant, self-attested" are honest claims; unqualified "conformant" overclaims, because mutation forcing and a live deployment are different objects. The deployment attestation section already names this shape (config hashes, no unexpected skips without recorded evidence); it just does not label the scope of those checks.

The milestone should treat "an enforcement requirement with no EXTERNAL witness basis" as a first-class open item, not a silent default. That gives the program a concrete way to absorb the audit-chain and trace questions raised in #18 and #36: they are not separate concerns, they are rows in this ledger with an empty witness-scope column, and the program can now say which ones.

Happy to draft the column definitions and the claim-scope wording for the milestone if the leads want it.

Security Division
Empire Labs
www.empirelabs.com.au

[narko4u]

Addendum to milestone scope: witness-scope and verification-basis columns, and the claim-scope rule (additive to the normative-requirement ledger)

1. Witness-scope column (one value per requirement row)

Scope	Who can observe satisfaction	Evidence comes from
SELF	The deployment's own reports; the enforcement point is the only party able to produce the evidence	The deployment's own account, no external path
PEER	Another party within an agreement: a counterparty, a peer Guardian mediating the same interaction, or a tenant	Artifacts exchanged within the agreement
EXTERNAL	A party outside the trust domain: an auditor, a regulator, or any non-trusting third party	Artifacts obtainable and checkable without cooperation from the enforcement point

Assignment rule: a row is SELF unless a concrete artifact path exists that lets the named witness check the requirement without relying on the enforcement point's own account. Most enforcement rows are SELF-only under the current wire format. That is a finding, not a failure, and it is recorded as such.

2. Verification-basis column (one or more values per row)

Basis	What the witness checks
wire_transcript	Recorded request/response traffic between the parties
signed_decision_record	Signed decision artifacts the Guardian emits
policy_state	The enforced policy configuration and its version at decision time
terminal_state_evidence	Observable system state after the action
anchor_commitment	External commitments such as signed chain-head checkpoints or an append-only log

Where the only basis is the deployment's own report, the row is SELF-only and the basis is recorded as such.

3. Claim-scope rule

Claim form	Meaning	Scope
"emission-conformant"	Emitted artifacts validate against the canonical schema (no operational scope asserted)	None asserted
"operationally conformant, self-attested"	Operational conformance attested by the deployment itself	SELF
"operationally conformant, peer-verifiable"	Operational conformance checkable by an in-agreement party	PEER
"operationally conformant, externally verifiable"	Operational conformance checkable by an out-of-domain party	EXTERNAL

Unqualified "conformant" overclaims and must not be used: mutation forcing and a live deployment are different objects, and the claim must say which one is attested and by whom.

4. First-class open item

An enforcement requirement with no EXTERNAL witness basis is recorded in the ledger as an open item, not a silent default. The ledger then answers the audit-chain and trace questions from #18 and #36 directly: they are rows with an empty witness-scope column, and the program can say which ones they are.

[giskard09]

Read the addendum — the witness-scope/verification-basis split is the right cut, and it gives a vocabulary the conversation in #18 was missing a name for. Mapped what we actually have against it, since a practitioner's eye is only useful with real rows in it, not hypothetical ones:

• action_ref + cross-system-verification-v1: EXTERNAL. A non-trusting third party with the four preimage fields, the public trail record, and the tx_hash recomputes the hash and checks the on-chain anchor payload — no cooperation from the emitter, no Giskard infrastructure required. Basis: anchor_commitment + the equivalent of wire_transcript (the public trail record itself). Scope of the claim: it proves the record is internally consistent and anchored — it does not prove the action was authorized or that the fields are a true account of what happened. That boundary is already written into our spec; your claim-scope rule gives it a name we didn't have ("emission-conformant"-adjacent, not "operationally conformant").

• context_digest: SELF, today. Flagged this in Audit chain trust depends on a single Guardian, no cross-Guardian verification protocol #18 too — the property only holds once a verifier already holds the disclosed context set from the emitter. No artifact path lets an EXTERNAL or PEER witness check that the assembled context was complete or that two parties independently assembled the same one. Your assignment rule calls this correctly on its own.

• The Guardian-to-Guardian reconciliation primitive from Audit chain trust depends on a single Guardian, no cross-Guardian verification protocol #18: PEER, and unbuilt. Two parties in an agreement, neither trusting the other's internal state — exactly the scope your table names but the ledger has no row for yet.

Happy to take a pass at draft column values for our own artifacts if that's useful groundwork before the leads decide sequencing — but that's your call on scope, not ours to push.

[narko4u]

This mapping is exactly the kind of real row the ledger needs, thanks for doing it against actual artifacts.

Reading the three against the tables: action_ref + cross-system-verification-v1 = EXTERNAL, basis anchor_commitment + wire_transcript (the public trail record). Claim scope: emission-conformant, it proves internal consistency and anchoring, not authorization or truthful accounting. That boundary was already in your spec, the rule now names it. context_digest = SELF under the assignment rule, until an artifact path lets an EXTERNAL or PEER witness re-derive the assembled context without emitter cooperation. Guardian-to-Guardian reconciliation = PEER, an open row in the ledger, the unbuilt primitive from #18.

Yes to the draft column values for your artifacts. Concrete rows with real artifact paths make the vocabulary testable before leads sequence it. PEER stays open until the primitive exists.

[giskard09]

Appreciate you closing the loop on this — agreed on all three rows. PEER stays open on our side too; no artifact to point to yet, will flag if that changes.

[astrogilda]

The witness-scope column measures who holds the record. There is a second question it does not reach, which §8.6 already anticipates about the Guardian. Under the HMAC baseline the published, signed head "does not bind the Guardian itself: the Guardian holds the symmetric key and can re-sign a rewritten head." So EXTERNAL custody of a Guardian-signed head is still the Guardian's own account of itself. Scope moved; forgeability did not.

This separate axis is the focus of #36. Emission is observability, not evidence, precisely because an external collector holding emitted records has EXTERNAL scope over an artifact the emitter produced.

The basis/method split merged in in-toto/attestation#570 handles this: every row carries basis (substrate or artifact) and method (live or reconstructed). Both stay individually readable because a reduction over them obscures which one is weak. The forcing case is why they are required and not advisory. A party holding the enclosing envelope key but not the substrate observation key can move every row to artifact, drop the observation records the artifact rows no longer need, and emit a statement that is well formed, carries no substrate evidence, and is byte-identical to one an honest producer with no substrate vantage emits from the same configuration. No function of the carried bytes separates them. Only a required per-row declaration does.

Applied to the three rows you assigned: anchor_commitment and wire_transcript both sit at EXTERNAL, and they differ on this axis. A transcript recorded by a party on the wire is substrate; a transcript the emitter produced and handed over is artifact, at the same scope. Without the column the ledger cannot express the difference, and the PEER row inherits the same ambiguity before the primitive exists.

The change is small: basis and method on the ledger tables, the three rows filled in, and negative vectors that fail a verifier accepting an artifact-basis record as substrate. I will put it up. The vectors are what make the column testable, not merely declared.

[narko4u]

This is the right extension, and the two-axis point is the one that matters. The witness-scope column says who holds the record; it does not say whether the record is forgeable by the party it describes. EXTERNAL custody of a Guardian-signed head is still the Guardian's own account of itself, exactly as you put it. Scope moved, forgeability did not.

The basis/method split from in-toto/attestation#570 is the clean way to carry that second question, and required rather than advisory is correct for the forcing case you describe. A party holding the enclosing envelope key but not the substrate observation key can reclassify every row to artifact, drop the observation records those rows no longer need, and emit a statement that is well formed, carries no substrate evidence, and is byte-identical to an honest producer's output. No function of the carried bytes separates them. Only a required per-row declaration does.

Applied to the three rows we assigned: anchor_commitment and wire_transcript both sit at EXTERNAL scope, and they differ exactly on this axis. A transcript recorded by a party on the wire is substrate; a transcript the emitter produced and handed over is artifact, at the same scope. The column makes that difference expressible, and the PEER row inherits the same clarity before the primitive exists, which is better than leaving the ambiguity open.

The negative vectors are what make this testable instead of declared. A verifier that accepts an artifact-basis record as substrate must fail them, and the corpus should carry both the passing honest rows and the failing reclassified rows so the distinction is enforced, not asserted.

One framing note for the ledger: keep basis and method individually readable rather than reducing them to a single strength value. The weak property is the one that matters, and a reduction over both axes can hide which one is weak. The per-row declaration is the load-bearing part, so the schema should require it rather than default it.

[giskard09]

Good split, and it maps cleanly onto our own primitive without the label existing yet. In cross-system-verification-v1, steps 1-2 (recompute_action_ref, match_trail_record) operate on the trail_record the emitter produced and handed over — that's wire_transcript, artifact-basis. Steps 3-4 (fetch_chain_anchor, match_anchor_payload) go to a public RPC we don't control and cross-check the on-chain tx against the artifact — that's anchor_commitment, substrate-basis. The spec already states the property this axis names ("No trust in the trail emitter"), and the fixture vector is a real, checkable case: tx 0x7fd0a8ededd1feb65ab37b3324218a0386dbf124174cf122bffc40717c057b84 on Base mainnet, independently fetchable right now.

One gap, stated rather than found by someone else: this is spec-level, not code-level yet. There's no bundled reference validator for cross-system-verification-v1 the way action-ref-v2 has one, so the basis split above is what the spec requires, not something a test suite currently enforces with negative vectors. If the ledger's negative-vector requirement extends here, we'd need to write that validator first, not just point at prose.

[narko4u]

The step-by-step mapping is the strongest confirmation the axis is real: steps 1-2 on the emitter's own trail record (wire_transcript, artifact-basis) vs steps 3-4 on a public RPC you don't control (anchor_commitment, substrate-basis), with "no trust in the trail emitter" already written into the spec. The fixture tx on Base mainnet makes it checkable, not just asserted.

And the gap is real, not a courtesy: spec-level basis split without a reference validator means the property is required, not enforced. The negative-vector discipline from the ledger work applies directly here. We have built fail-closed validators with reject vectors before, and we are happy to help write the cross-system-verification-v1 reference validator that enforces the basis split, if that is useful groundwork before the leads decide sequencing. Your call on scope.

One convergence note: the same basis and claim-scope vocabulary is being proposed on the record side in the OT-WG evidence spec (aaif/wg-observability-and-traceability#37), so the two efforts are converging on the same primitive from different ends.