Split prompt-visible validation from target graders

[Jordak]

This was generated by AI during triage.

Summary

Separate agent-facing validation guidance from deterministic target graders. Today, task.test commands are both the commands run after the agent phase and the commands inserted into build_agent_prompt under "Validation commands that will be run after you finish." That makes test: unusable as a held-back behavioral grader and gives the harness-under-test exact knowledge of the checks that determine pass/fail.

This issue is a prerequisite for hidden verifier work in GitHub issue #91. Hidden verifier patches are still useful for held-out files/tests, but first the task contract should stop treating target graders as prompt text by default.

Current Behavior

• EvalTask.test is loaded from the test: field in task.yaml.
• agentlab.agents.prompts.build_agent_prompt includes every task.test command in the agent prompt.
• Codex and Claude adapters use that shared prompt builder.
• The manual adapter also prints task.test commands to the operator.
• The runner captures the model diff after the agent phase and then runs task.test as target checks.
• Task cards currently render setup, baseline, and target grader commands for maintainer review.

This visible-grader behavior can be useful for workflow-style tasks where the goal is to tell an agent the intended validation recipe. It is a poor default for eval tasks where target graders should measure behavior without becoming part of the prompt/spec.

Desired Behavior

• test: remains the deterministic target-grader command list run after the agent phase.
• test: commands are not injected into agent prompts by default.
• Add an explicit field for commands that task authors intentionally want to show to the agent, such as visible_validation: or prompt_validation:. The exact name can be chosen during implementation, but it should be clearly agent-facing.
• build_agent_prompt includes only the explicit visible-validation field, not task.test.
• The manual adapter follows the same visibility rule.
• Task-card rendering is deliberate: maintainer-facing cards can show both target graders and visible validation, but any agent-facing prompt remains clean.
• Existing task bundles are migrated or interpreted deliberately so their intended visible validation behavior is not silently changed.

Acceptance Criteria

• Task loading supports an optional explicit visible-validation command list separate from test:.
• build_agent_prompt no longer includes task.test commands.
• build_agent_prompt includes the new visible-validation commands when configured.
• The manual adapter no longer prints task.test commands as agent/operator guidance by default, and prints only explicit visible-validation commands if configured.
• Runner behavior for task.test remains unchanged: target checks still run after the agent phase and affect deterministic pass/fail.
• Reports/results still record target grader checks as grader evidence.
• Task cards are updated to distinguish visible validation from target graders in a maintainer-facing way.
• Existing committed task bundles are reviewed and migrated where the old prompt-visible test: behavior was intentional.
• Unit tests cover prompt exclusion for test: and prompt inclusion for the new explicit visible-validation field.

Non-goals

• Do not add hidden verifier patch application in this issue; that remains in GitHub issue Add hidden verifier patch support for behavioral graders #91.
• Do not remove visible validation as a capability; make it explicit instead of implicit.
• Do not change setup or baseline execution semantics except where names/rendering need to stay coherent.

Validation Plan

• Focused prompt-builder tests for hidden target graders and explicit visible validation.
• Task-loading tests for the new field.
• Manual-adapter or transcript tests for the visibility rule.
• Task-card rendering tests for the new maintainer-facing distinction.
• Existing runner/scoring tests proving test: checks still run and affect pass/fail.

Readiness

Design readiness: ready to implement
Consensus provenance: Current Codex conversation on 2026-06-22 with Jordan Harry; the user explicitly identified this as separate work from hidden verifier support and requested creating this GitHub issue and blocking #91 on it.
Gate skipped: not applicable

[Jordak]

Completed by merged PR #119 on origin/main. Integration evidence: merge commit fc25737 is reachable from origin/main after fetch. Validation seen on PR #119: branch hygiene, Python 3.9, and Python 3.x GitHub checks passed; worker also reported focused unit suites, full unittest discovery, task-card validation, git diff --check, and pre-commit task-card integrity. All acceptance criteria are checked in the issue body.