test(codex): require agent review coverage before AUTO_OK - #108
Merged
Conversation
Why: - Codex Signum could previously treat medium/high-risk audit output as safe even when agent review evidence was absent or materially reduced. - Final human PR review should not substitute for review artifacts produced inside the Signum AUDIT phase. What changed: - Adds Codex prompt eval invariants and fixtures for agentReviewCoverage and agentReviewArtifacts before medium/high-risk AUTO_OK. - Updates Codex skill guidance to require recorded agent review evidence and to avoid AUTO_OK when coverage is missing or reduced. - Refreshes the Codex prompt eval baseline and smoke checks for the new metric. Testing: - python3 -m py_compile evals/codex_prompt/checks_codex_prompt.py evals/codex_prompt/run_codex_prompt_eval.py evals/codex_prompt/compare_codex_prompt_eval.py - bash tests/test-codex-prompt-evals.sh - bash tests/test-codex-prompt-eval-compare.sh - bash tests/test-codex-plugin-metadata.sh - bash tests/test-policy-scanner-evals.sh - bash tests/test-policy-scanner-eval-compare.sh - bash tests/test-test-plan-check.sh - bash scripts/run-deterministic-tests.sh Risk: - narrow - this changes Codex prompt guidance and offline eval expectations, not scanner/runtime implementation or CI wiring.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 139dba52a9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Why: - Codex Review found that the new agent review AUTO_OK gate could be bypassed with degraded provider states, placeholder artifact paths, or empty reviewer IDs. - These are offline eval invariant gaps, not runtime scanner or catalog behavior changes. What changed: - Infer reduced agent review coverage directly from non-ready agentReviewCoverage provider states. - Require medium/high AUTO_OK agent review artifacts to live under the active contract reviews root and be recorded in artifactLayout.artifactRefs. - Require ready agent review coverage to include a non-empty reviewer ID. - Add regression fixtures for the three review findings and refresh the Codex prompt eval baseline to 41 fixtures. Testing: - python3 -m py_compile evals/codex_prompt/checks_codex_prompt.py evals/codex_prompt/run_codex_prompt_eval.py evals/codex_prompt/compare_codex_prompt_eval.py - python3 evals/codex_prompt/run_codex_prompt_eval.py --json-output /tmp/codex-agent-review-fix.json - python3 evals/codex_prompt/compare_codex_prompt_eval.py --baseline evals/codex_prompt/baselines/current.json --candidate /tmp/codex-agent-review-fix.json - bash tests/test-codex-prompt-evals.sh - bash tests/test-codex-prompt-eval-compare.sh - bash tests/test-codex-plugin-metadata.sh - bash tests/test-policy-scanner-evals.sh - bash tests/test-policy-scanner-eval-compare.sh - bash tests/test-test-plan-check.sh - bash scripts/run-deterministic-tests.sh Risk: - narrow - only offline Codex prompt eval checks, fixtures, docs, and baseline are changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AUTO_OK.Issue
Related: #107
Test plan
python3 -m py_compile evals/codex_prompt/checks_codex_prompt.py evals/codex_prompt/run_codex_prompt_eval.py evals/codex_prompt/compare_codex_prompt_eval.pybash tests/test-codex-prompt-evals.shbash tests/test-codex-prompt-eval-compare.shbash tests/test-codex-plugin-metadata.shbash tests/test-policy-scanner-evals.shbash tests/test-policy-scanner-eval-compare.shbash tests/test-test-plan-check.shbash scripts/run-deterministic-tests.shScope
Notes
This PR adds prompt/eval guidance and offline invariant coverage. It does not yet implement automatic reviewer CLI orchestration inside Codex runtime.