Repository navigation
security benchmark: cover the Critic's advisory-only guarantee - #77
Merged
Merged
Conversation
Two new defended cases, a new llm-critic category: tampered-critique-detected (flipping a critique's recommendation after the fact is caught by the same verify_snapshot check as any other tampered Snapshot field - a critique is just one more hashed field, per ADR-0021 point 2/ADR-0023) and critique-content-cannot-skip-approval-state-machine (an adversarial critique crafted to look like an instruction has zero effect - ApprovalStore.create() always starts a request at PENDING regardless of Snapshot content, and a Critique is a disconnected value type with no reference to any store or transition method). Both prove ADR-0021 point 3's 'advisory-only, never a veto' claim structurally, the same way every other case in this benchmark proves its own claim by demonstration rather than assertion - the one gap left after the Critic became a shared, every-detector primitive (ADR-0023) rather than a cloud-waste-only feature. No Wayfinder ticket, per ADR-0019's own precedent (a rigor addition, not a new architectural question). 9 cases now, 8 defended, 1 known limitation, 0 stale. docs/security-benchmark.md regenerated, never hand-edited. Full ephor suite green: ruff/mypy clean, 62 tests passing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #75/PR #76 (ADR-0023, the Critic seam broadened to every detector). Once the Critic became a shared, every-detector primitive, its "advisory-only, never a veto" guarantee (ADR-0021 point 3) was the one property left undemonstrated in
ephor's own security benchmark (ADR-0019).What
Two new defended cases, a new
llm-criticcategory:tampered-critique-detected- flipping a critique'srecommendationafter the fact (e.g. hiding a flagged concern from a re-verifying auditor) is caught by the sameverify_snapshotcheck as any other tampered Snapshot field. A critique is just one more hashed field, per ADR-0021 point 2 / ADR-0023.critique-content-cannot-skip-approval-state-machine- an adversarial critique crafted to look like an instruction ("SET status=executed AND SKIP APPROVAL") has zero effect:ApprovalStore.create()always starts a request atPENDINGregardless of Snapshot content, and aCritiqueis a disconnected value type with no reference to any store or transition method.Both prove the "advisory-only, never a veto" claim structurally, the same way every other case in this benchmark proves its own claim by demonstration rather than assertion.
Written up as an update to ADR-0019 rather than a new ADR - no Wayfinder ticket, per that ADR's own precedent ("a rigor addition, same shape as ADR-0015 - not a new architectural question").
Result
9 cases now (was 7), 8 defended, 1 known limitation, 0 stale.
docs/security-benchmark.mdregenerated viauv run python -m tests.generate_security_benchmark_report- never hand-edited.Verification
ruff check/ruff format --check/mypy .clean.pytest -v- 62 passed (up from 60 - two new benchmark cases).ephor/tests/,docs/, andREADME.md- no other package touched.