Add hidden verifier patch support for behavioral graders

[Jordak]

This was generated by AI during triage.

Dependency

GitHub issue #90 is closed and GitHub issue #118 has landed, so hidden verifier support can build on base-only workspaces and the prompt-policy split where prompt-visible validation is explicit.

Summary

Add first-class support for hidden verifier patches so task authors can grade behavior with tests that are applied only after the agent finishes.

This should be implemented after #90, because base-only workspaces make the verifier flow easier to reason about: capture the model patch against a single synthetic base commit, apply the verifier patch temporarily, run behavioral checks, and keep grader files out of the model diff.

This should also be implemented after #118, because hidden verifier support should be layered on top of a contract where prompt-visible validation is explicit rather than an accidental property of target graders.

DeepSWE Inspiration

DeepSWE task bundles separate the agent instruction from hidden verifier assets. Its task format includes instruction.md, tests/test.sh, tests/test.patch, and a held-out solution/. The verifier checks observable behavior and the reference solution is not used at grading time.

Sources:

• DeepSWE official benchmark page: https://deepswe.datacurve.ai/
• DeepSWE GitHub repository and task format: https://github.com/datacurve-ai/deep-swe
• Harbor task format used by DeepSWE: https://www.harborframework.com/docs/tasks
• Pier runner used for DeepSWE: https://github.com/datacurve-ai/pier
• Example DeepSWE task directory shape: https://github.com/datacurve-ai/deep-swe/tree/main/tasks/happy-dom-abort-pending-body-reads

Current Local Context

Agent Eval Lab task bundles currently expose setup, baseline, target command lists, and explicit visible validation in task.yaml. Prompt-visible validation is now opt-in through visible_validation; target graders are not automatically printed in the agent prompt.

The existing runner captures final diffs and grader command results after the agent phase. Hidden verifier support should preserve that outcome-first architecture while making the verifier overlay temporary and clearly separated from model output.

Recent prompt-hint removal also exposed false negatives in string-shaped deterministic graders. Hidden verifier patches are the intended behavioral-grader path for cases where public source-token checks are too brittle. Follow-up issue #123 tracks migrating existing starter graders after this infrastructure exists.

Chosen Design

Task bundles may define one optional hidden verifier object:

hidden_verifier:
  patch: verifier.patch
  commands:
    - pytest tests/hidden_behavior.py

The object is strict for this implementation slice:

• hidden_verifier is an optional mapping.
• hidden_verifier.patch is required when hidden_verifier is present.
• hidden_verifier.commands is required and must be a non-empty string list.
• The patch path is relative to the task bundle, may not be absolute, may not escape the bundle, and should use a patch/diff-style extension.
• Static schema, path-safety, and missing-file problems fail task loading/validation.

Runtime semantics:

• The patch is a general overlay and may add or modify any repository file needed for verification.
• Hidden verifier details are never included in the agent prompt.
• After the agent finishes, the harness captures the model diff, changed files, and line metrics before applying the hidden verifier overlay.
• The harness applies the hidden verifier patch against the model-final workspace.
• Patch apply failure is represented as a failed hidden verifier check and fails the trial.
• If patch apply fails, hidden verifier commands do not run.
• Hidden verifier commands run in the same workspace cwd and task environment as normal graders.
• Hidden verifier checks are a distinct result category and affect overall deterministic success.
• Hidden verifier commands/checks stay out of top-level commands_run, checks, and graders; result.json gets a structured hidden_verifier field.
• report.md renders public graders separately from hidden verifier checks.
• Post-run artifacts may reveal hidden patch path, hidden commands, outputs, and pass/fail for maintainer debugging.
• diff.patch, files changed, and line-addition metrics describe only model changes, not temporary verifier overlay files.
• The run workspace is restored to model-only final state after hidden verifier grading. Restoration should be snapshot-style rather than relying only on git apply -R; restoration warnings should be reported without hiding the original hidden verifier result.
• Reference verification also applies/runs hidden verifier checks after capturing the reference diff.

Task-card rendering:

• Generated task cards may mention that a hidden verifier exists.
• Task cards should show patch path and command count only, not full hidden command strings or patch contents.
• Agent prompts remain clean.

Terminology:

• Keep the YAML field test: unchanged.
• Reports/cards may distinguish public graders from hidden verifier checks so “code-based graders” is no longer ambiguous.

Non-Goals

• Do not migrate existing starter tasks to hidden verifiers in this issue; that is tracked by Migrate brittle starter graders to hidden verifiers and semantic checks #123.
• Do not introduce an LLM grader in this issue.
• Do not implement a general AST/semantic structural-check framework in this issue.
• Do not solve ignored/generated artifact capture policy, such as whether __pycache__ should count against changed-file limits.
• Do not add multiple hidden verifier blocks or ordered verifier phases yet.

Acceptance Criteria

• Task loading supports the strict optional hidden_verifier schema described above.
• Task validation rejects malformed hidden verifier config and unsafe/missing patch paths.
• Hidden verifier details are not included in build_agent_prompt or manual-agent prompt text.
• The runner persists the model patch before applying the hidden verifier overlay.
• Hidden verifier patch apply and command checks affect the deterministic grader outcome.
• diff.patch, files changed, and line-addition metrics exclude verifier-only overlay files.
• Trial reports and result JSON make hidden verifier status clear without mixing hidden verifier commands into public command/check/grader fields.
• Run workspaces are restored to model-only state after hidden verification, with cleanup warnings surfaced when needed.
• Reference verification runs hidden verifier checks and reports them distinctly.
• Unit tests cover a synthetic task where a hidden patch adds tests after the agent phase.
• Unit tests prove that an agent modification to a visible/public test file cannot substitute for passing the hidden verifier.
• Unit tests cover hidden patch apply failure as a failed hidden verifier result.
• Task-card rendering is updated deliberately with summary-only hidden verifier metadata.

Validation Plan

• Add focused unit tests for task loading, prompt exclusion, task-card rendering, task execution, result JSON/report rendering, and reference verification.
• Run the focused unittest modules for touched surfaces.
• Run task bundle/card validation.
• Run git diff --check.

Deferred Follow-ups

• Migrate brittle starter graders to hidden verifiers and semantic checks #123 migrates brittle starter graders to hidden verifiers and semantic structural checks after this infrastructure lands.
• Artifact/change-capture policy for ignored generated files such as __pycache__ remains a separate decision.

Design readiness: ready to implement
Consensus provenance: grill-me design session in the Codex thread on 2026-06-24; the human user confirmed the design decisions one by one and then instructed “Make it so” to promote the agreed design into the durable issue before implementation.
Gate skipped: not applicable

[Jordak]

Recent prompt-hint removal and reruns of the starter-coding suite exposed three useful false-negative examples that seem directly relevant to this issue.

Current observed failures:

• click-should-strip-ansi-tests-001: the model wrote meaningful parametrized tests covering the requested ANSI/color/stream behavior, but the grader required exact source tokens such as color=True, color=False, sys.stdin, sys.stdout, and sys.stderr.
• remotion-audio-context-autoplay-muted-001: the model guarded the audio-context resume/wait path with an outer if (!muted), which appears behaviorally equivalent for muted playback, but the grader only accepted two specific textual shapes.
• click-help-option-refactor-001: the model shared help-option construction, but the grader scanned globally for a nested show_help callback string instead of checking whether the duplicate implementation was actually removed from the separate paths. This run also exposed a separate artifact-hygiene/change-capture question around ignored __pycache__ files.

These are good examples of why hidden verifier patch support matters. For behavioral requirements, deterministic graders should not need to duplicate the hidden verifier with brittle source-token assertions. Once hidden verifier patches are available, the intended shape should be:

• hidden verifier patches/tests check behavior that can be exercised;
• deterministic graders stay thinner for behavior, mostly running public commands, reporting hidden verifier outcomes, and enforcing scope/result hygiene;
• deterministic structural checks remain only where the task truly has a structural requirement;
• structural checks should prefer AST/semantic inspection over raw string matching when practical.

I opened follow-up issue #123 for migrating brittle starter graders after this infrastructure exists. My recommendation is to keep #91 focused on hidden verifier patch support and reporting mechanics, while #123 handles task-specific migration/simplification and any AST/semantic structural checker work.

[Jordak]

Completed by merged PR #124 on origin/main. Integration evidence: merge commit 570d95e is reachable from origin/main after fetch. Validation seen on PR #124: branch hygiene, Python 3.9, and Python 3.x GitHub checks passed; PR validation also recorded focused unittest modules, full unittest discovery, task bundle/card validation, and git diff --check. All acceptance criteria are checked in the issue body.