feat(codex): add local review evidence checker by t3chn · Pull Request #110 · heurema/signum

t3chn · 2026-05-16T10:09:49Z

Linked intent

Link the Issue or Discussion this PR implements.

Issue / Discussion: maintainer-directed follow-up from Codex local agent-review protocol work
Closes/Fixes/Resolves: N/A

For non-trivial changes, open an Issue or Discussion before code review. Direct PRs are intended only for typo/docs fixes, small test-only changes, clearly scoped bug fixes, or maintainer-approved work.

Problem

The Codex prompt now requires an internal local agent-review artifact for non-trivial medium/high-risk work, but that rule was still easy to miss manually. A deterministic check is needed before a Codex run can safely claim AUTO_OK.

Why now

PR #109 made the Codex review artifact contract explicit. This PR adds the smallest deterministic guard for that contract so future runs can verify the evidence exists instead of relying on prompt memory alone.

Existing options checked

The existing Codex prompt eval catches simulated invariant failures, but it does not validate a real contract artifact root after a run. The proofpack validator checks general proofpack shape, but not the Codex-local review evidence requirement.

Alternatives considered

Wire the checker into CI/runtime immediately: deferred to keep this PR narrow and avoid changing global CI behavior.
Extend scanner or policy rules: not relevant; this is an audit artifact evidence check.
Depend only on prompt text: insufficient because the missing artifact case should be machine-checkable.

No-code alternative

The prompt text already documents the rule. That helps but does not prevent missing local review evidence from slipping through.

Why code is needed

A small checker can validate the actual .signum/contracts/<contractId>/ artifact root and fail deterministically when medium/high AUTO_OK lacks Codex local review evidence.

Summary

Adds scripts/check_codex_agent_review.py.
For medium/high AUTO_OK, validates ready Codex coverage, reviews/codex.json, audit artifact refs, and proofpack evidence.
Allows low-risk and non-AUTO_OK runs to bypass this gate.
Adds shell coverage for happy path, bypasses, and missing/degraded evidence failures.
Updates the Codex skill to run the checker before accepting medium/high AUTO_OK.

Type

Mark all that apply.

Scope

In:

Codex local review evidence checker
Codex prompt reference to the checker
Checker and metadata tests

Out:

Scanner/catalog behavior
Claude overlay runtime files
Commands/runtime pipeline wiring
CI wiring
Eval baseline updates
signum-evolve behavior

Risk areas

Mark anything touched in this PR.

Docs impact

no docs update needed
README.md or QUICKSTART.md updated
AGENTS.md updated
docs/how-it-works.md or docs/reference.md updated
docs/SECURITY.md updated
follow-up docs work is needed

Docs / rationale:

The Codex skill prompt documents the checker. Broader user docs can follow if this becomes part of canonical runtime wiring.

Validation / proof

Commands run:

python3 -m py_compile scripts/check_codex_agent_review.py
bash tests/test-codex-agent-review-check.sh
bash tests/test-codex-plugin-metadata.sh
bash tests/test-codex-prompt-evals.sh
bash tests/test-codex-prompt-eval-compare.sh
bash scripts/run-deterministic-tests.sh

Observed:

tests/test-codex-agent-review-check.sh: 12 passed, 0 failed
tests/test-codex-plugin-metadata.sh: 62 passed, 0 failed
scripts/run-deterministic-tests.sh: Deterministic tests passed.

DCO / authorship

Every commit in this PR is signed off (git commit -s) and complies with DCO.md

Reviewer notes

This PR intentionally does not wire the checker into global CI or the root command runtime. It adds the deterministic checker and Codex prompt call site first, keeping behavior changes bounded.

Why: - Medium/high-risk Codex runs should not claim AUTO_OK unless the Signum AUDIT phase produced concrete local agent-review evidence. - The prompt now defines the artifact contract, but a deterministic checker is needed so missing evidence is caught before packaging or review. What changed: - Add a stdlib-only checker for Codex local review evidence under the active contract root. - Require ready Codex review coverage, a concrete reviews/codex.json artifact, audit artifact references, and proofpack evidence for medium/high AUTO_OK runs. - Document the checker in the Codex skill and cover pass, bypass, and failure cases with shell tests. Testing: - python3 -m py_compile scripts/check_codex_agent_review.py - bash tests/test-codex-agent-review-check.sh - bash tests/test-codex-plugin-metadata.sh - bash tests/test-codex-prompt-evals.sh - bash tests/test-codex-prompt-eval-compare.sh - bash scripts/run-deterministic-tests.sh Risk: - narrow - this adds an opt-in deterministic Codex review evidence gate and prompt reference without changing scanner/catalog/CI behavior. Signed-off-by: Vitaly D. <netmin@pm.me>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0954dad106

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-16T10:20:42Z

+
+
+def _determine_risk(contract: dict[str, Any], audit: dict[str, Any], proofpack: dict[str, Any]) -> str | None:
+    for source in (proofpack, audit, contract):


Use contract risk as source of truth

_determine_risk currently prefers proofpack/audit_summary over contract.json, so a medium/high contract can be downgraded to low by those mutable artifacts and the checker will skip the hard gate (required: false) even when AUTO_OK is claimed without Codex review evidence. Repro: contract riskLevel=high, audit/proofpack riskLevel=low returns exit 0 and no violations. This defeats the stated purpose of enforcing medium/high AUTO_OK review evidence.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-05-16T10:20:42Z

+SCHEMA_VERSION = "1.0"
+REQUIRED_RISKS = {"medium", "high"}
+READY_STATE = "ready"
+APPROVING_VERDICTS = {"APPROVE", "CONDITIONAL"}


Exclude CONDITIONAL from approving verdicts

The gate treats CONDITIONAL as an approving verdict, so medium/high AUTO_OK can pass even when the Codex review explicitly reports a conditional outcome (typically unresolved major issues). Because codex_review.verdict_not_approving is the only verdict check, including CONDITIONAL here weakens the gate and allows inconsistent AUTO_OK decisions to be marked as hard-gate passed.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-05-16T10:20:42Z

+        if review.get("reviewerType") != "local_agent":
+            violations.append("codex_review.reviewer_type_missing")


Validate reviewer ID before passing AUTO_OK gate

The skill contract says medium/high AUTO_OK requires a ready reviewer with a non-empty reviewer ID, but this checker never validates any reviewer identifier field in reviews/codex.json. As a result, an anonymous or blank-identity local review still passes the hard gate, so the deterministic check does not enforce a stated acceptance condition.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-05-16T10:20:42Z

+    if contract_error and contract_error != "missing":
+        violations.append("contract.invalid_json")


Fail when contract.json is missing

Missing contract.json is not recorded as a violation (only invalid JSON is), so the gate can return hardGatePassed: true using only audit_summary.json/proofpack.json values. That makes it possible to bypass medium/high enforcement via stale or downgraded risk metadata when the canonical contract source is absent.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-05-16T10:20:42Z

+    if codex_review.get("status") == "present":
+        return True


Require codex identity in proofpack review evidence

_proofpack_has_codex_review returns true as soon as checks.reviews.codex.status is present, without verifying that the referenced content/path is actually the Codex review artifact. If packing mistakenly points checks.reviews.codex to a non-Codex file, the checker still reports proofpackIncludesCodexReview: true and passes medium/high AUTO_OK, so the proofpack evidence requirement can be satisfied by the wrong artifact.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-05-16T10:20:43Z

+
+    decision = _determine_decision(audit, proofpack)
+    risk_level = _determine_risk(contract, audit, proofpack)
+    required = decision == "AUTO_OK" and risk_level in REQUIRED_RISKS


Reject malformed AUTO_OK decision values

The hard-gate condition is an exact string match (decision == "AUTO_OK"), so malformed variants like "AUTO_OK " or casing differences skip enforcement (required: false) even on medium/high risk runs. In that case the checker can exit successfully without any Codex review evidence, so invalid decision tokens should fail closed instead of bypassing the gate.

Useful? React with 👍 / 👎.

github-actions Bot added the intake/pass PR intake passed label May 16, 2026

t3chn marked this pull request as ready for review May 16, 2026 10:13

t3chn merged commit 70ad1a5 into main May 16, 2026
4 checks passed

t3chn deleted the codex/agent-review-proofpack-validator branch May 16, 2026 10:14

chatgpt-codex-connector Bot reviewed May 16, 2026

View reviewed changes

t3chn mentioned this pull request May 16, 2026

Track Codex review parity for Signum audit flow #107

Closed

Navigation Menu

Uh oh!

feat(codex): add local review evidence checker - #110

feat(codex): add local review evidence checker#110
t3chn merged 1 commit into
mainfrom
codex/agent-review-proofpack-validator

t3chn commented May 16, 2026

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant



		def _determine_risk(contract: dict[str, Any], audit: dict[str, Any], proofpack: dict[str, Any]) -> str \| None:
		for source in (proofpack, audit, contract):

		if review.get("reviewerType") != "local_agent":
		violations.append("codex_review.reviewer_type_missing")

		if contract_error and contract_error != "missing":
		violations.append("contract.invalid_json")

Uh oh!

Conversation

t3chn commented May 16, 2026

Linked intent

Problem

Why now

Existing options checked

Alternatives considered

No-code alternative

Why code is needed

Summary

Type

Scope

Risk areas

Docs impact

Validation / proof

DCO / authorship

Reviewer notes

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot May 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant