Skip to content

Add hidden verifier patch support - #124

Merged
Jordak merged 4 commits into
mainfrom
codex/issue-91-hidden-verifier-patch
Jun 26, 2026
Merged

Add hidden verifier patch support#124
Jordak merged 4 commits into
mainfrom
codex/issue-91-hidden-verifier-patch

Conversation

@Jordak

@Jordak Jordak commented Jun 24, 2026

Copy link
Copy Markdown
Owner

Previously, task bundles could run public target graders after the agent phase, but they had no first-class way to apply verifier-only files after the model patch was captured. Behavioral graders either had to expose too much in public commands or rely on brittle deterministic source-shape checks.

This PR adds a strict optional hidden_verifier task field with one patch overlay and a non-empty command list. The runner now captures the model diff and edit metrics first, applies the hidden verifier overlay temporarily, runs hidden commands in the task environment, records those checks separately, and restores the workspace afterward. Hidden verifier failures affect the deterministic outcome, but hidden commands stay out of public commands_run, top-level checks, and top-level graders.

Reports and result JSON now distinguish public graders from hidden verifier checks. Reference verification runs hidden verifier checks too. Task cards show summary-only hidden verifier metadata, while prompts continue to omit hidden verifier patch paths and commands.

Readiness evidence: GitHub issue #91
Readiness verdict: Ready to Implement

Validation:

  • python3 -m unittest tests.test_tasks tests.test_agent_prompts tests.test_task_cards tests.test_task_execution tests.test_runner tests.test_reference tests.test_review_proposals
  • python3 -m unittest discover
  • python3 -m agentlab task validate --check-task-cards tasks
  • git diff --check

@Jordak
Jordak force-pushed the codex/issue-91-hidden-verifier-patch branch from 4e6a11e to 9b00a74 Compare June 24, 2026 22:46
@Jordak

Jordak commented Jun 26, 2026

Copy link
Copy Markdown
Owner Author

Agent Review

Scope: origin/main..34477ed, PR #124
Pass: full review-loop convergence (fresh + verification cycles, final fresh pass clean)
Review packet: chat packets from clean-context reviewer panels; final local report at /private/tmp/review-loop-pr124.html
Reviewers: 20 reviewer reports across correctness, tests-regressions, structural-depth, and issue-compliance lenses
Effort metadata: prescribed worker effort high via implement-github-issue / review-loop; effective reviewer model/effort not reported by harness

Issue families:

  1. [IF1] Severity: P2 - hidden verifier schema strictness
    Found by: P1-R1-F1, P1-R2-F2, P1-R3-F2, P1-R4-F1
    Decision: fixed
    Autopilot classification: auto-fix
    Smallest closing move: reject unknown hidden_verifier keys and blank command strings; add invalid-schema tests
    Fix commit: 2c0b45a

  2. [IF2] Severity: P1/P2 - workspace restore and symlink fidelity
    Found by: P1-R1-F2, P1-R3-F1
    Decision: fixed
    Autopilot classification: auto-fix
    Smallest closing move: preserve symlinks before directory handling; add directory-symlink restore regression
    Fix commit: 2c0b45a

  3. [IF3] Severity: P2 - task-card hidden verifier publication contract
    Found by: P1-R2-F1, P3-R1-F2, P3-R2-F2, P3-R3-F2, P3-R4-F2
    Decision: fixed
    Autopilot classification: auto-fix
    Smallest closing move: follow issue Add hidden verifier patch support for behavioral graders #91 exactly: task cards show patch path and command count only, not commands or patch contents
    Fix commits: 2c0b45a, 41a5bed

  4. [IF4] Severity: P1/P2 - hidden patch path symlink escape
    Found by: P3-R1-F1, P3-R2-F1, P3-R3-F1, P3-R4-F1
    Decision: fixed
    Autopilot classification: auto-fix
    Smallest closing move: require the resolved patch path to remain under the resolved task bundle directory; add symlink-escape load test
    Fix commit: 41a5bed

  5. [IF5] Severity: P3 - operability verifier-state propagation
    Found by: P3-R2-F3
    Decision: fixed
    Autopilot classification: auto-fix
    Smallest closing move: add hidden_verifier configured/check coverage facts and hidden-only operability test
    Fix commit: 41a5bed

  6. [IF6] Severity: P2 - reference hidden verifier persisted-artifact coverage
    Found by: P5-R2-F1
    Decision: fixed
    Autopilot classification: auto-fix
    Smallest closing move: extend reference hidden verifier test to write artifacts and assert hidden/public field separation in result JSON and report Markdown
    Fix commit: 34477ed

Declined issue families:

  • None.

Resolved ask-user decisions:

  • None.

Unresolved ask-user blockers:

  • None.

Validation notes:

  • python3 -m unittest tests.test_reference: 11 passed.
  • python3 -m unittest tests.test_tasks tests.test_task_cards tests.test_operability_evidence: 37 passed.
  • python3 -m unittest discover: 319 passed.
  • python3 -m agentlab task validate --check-task-cards tasks: 14 task files validated; task cards up to date.
  • git diff --check: passed.
  • GitHub Actions: branch hygiene, Python 3.9 validation, and Python 3.x validation succeeded.

Final readiness:

  • Final verification for IF6 reported no new findings.
  • Final eligible clean-context fresh pass with P7-R1 and P7-R2 reported no new findings.
  • PR Add hidden verifier patch support #124 is ready for human/integration-owner review from the review-loop perspective.

@Jordak
Jordak merged commit 570d95e into main Jun 26, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant