You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Previously, task bundles could run public target graders after the agent phase, but they had no first-class way to apply verifier-only files after the model patch was captured. Behavioral graders either had to expose too much in public commands or rely on brittle deterministic source-shape checks.
This PR adds a strict optional hidden_verifier task field with one patch overlay and a non-empty command list. The runner now captures the model diff and edit metrics first, applies the hidden verifier overlay temporarily, runs hidden commands in the task environment, records those checks separately, and restores the workspace afterward. Hidden verifier failures affect the deterministic outcome, but hidden commands stay out of public commands_run, top-level checks, and top-level graders.
Reports and result JSON now distinguish public graders from hidden verifier checks. Reference verification runs hidden verifier checks too. Task cards show summary-only hidden verifier metadata, while prompts continue to omit hidden verifier patch paths and commands.
Readiness evidence: GitHub issue #91
Readiness verdict: Ready to Implement
Scope: origin/main..34477ed, PR #124
Pass: full review-loop convergence (fresh + verification cycles, final fresh pass clean)
Review packet: chat packets from clean-context reviewer panels; final local report at /private/tmp/review-loop-pr124.html
Reviewers: 20 reviewer reports across correctness, tests-regressions, structural-depth, and issue-compliance lenses
Effort metadata: prescribed worker effort high via implement-github-issue / review-loop; effective reviewer model/effort not reported by harness
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, task bundles could run public target graders after the agent phase, but they had no first-class way to apply verifier-only files after the model patch was captured. Behavioral graders either had to expose too much in public commands or rely on brittle deterministic source-shape checks.
This PR adds a strict optional
hidden_verifiertask field with one patch overlay and a non-empty command list. The runner now captures the model diff and edit metrics first, applies the hidden verifier overlay temporarily, runs hidden commands in the task environment, records those checks separately, and restores the workspace afterward. Hidden verifier failures affect the deterministic outcome, but hidden commands stay out of publiccommands_run, top-levelchecks, and top-levelgraders.Reports and result JSON now distinguish public graders from hidden verifier checks. Reference verification runs hidden verifier checks too. Task cards show summary-only hidden verifier metadata, while prompts continue to omit hidden verifier patch paths and commands.
Readiness evidence: GitHub issue #91
Readiness verdict: Ready to Implement
Validation:
python3 -m unittest tests.test_tasks tests.test_agent_prompts tests.test_task_cards tests.test_task_execution tests.test_runner tests.test_reference tests.test_review_proposalspython3 -m unittest discoverpython3 -m agentlab task validate --check-task-cards tasksgit diff --check