Migrate starter graders to hidden verifiers

[Jordak]

Summary

Issue: #123

This audits the full current tasks/starter/* suite and migrates every starter task to a hidden verifier-backed grader where the current target checks were behavioral or brittle. The public task cards now show only the hidden verifier patch path and command count, while the actual behavioral assertions live in per-task verifier.patch files applied after model diff capture.

Previous Behavior

Starter task target graders encoded most behavioral expectations directly in public test: commands. Some of those commands were brittle source/text checks; notably, click-should-strip-ansi-tests-001 could reject the curated reference solution because it used parametrized test data instead of literal strings such as color=True.

New Behavior

• All 12 current starter tasks now declare hidden_verifier.
• Behavioral target checks moved into hidden verifier scripts for all migrated tasks.
• Public target checks are now limited to intentional public surfaces: submitted-test execution for the Click test-writing task, git diff --check, and Vite's fixture staging before diff whitespace checking.
• click-should-strip-ansi-tests-001 now uses hidden mutation checks instead of source-token assertions, so equivalent submitted tests can pass if they catch broken production behavior.
• click-help-option-refactor-001 now includes a scoped AST-style hidden structure check for nested duplicate help callbacks plus hidden behavior checks.
• Reference reports/results were regenerated for the starter tasks whose full reference verification completed.

Starter Audit Outcome

Every current starter task was migrated rather than left unchanged:

• 2048-advanced-snake-params-001: hidden simulation behavior checks.
• click-default-map-nargs-001: hidden Click default-map behavior checks.
• click-help-option-refactor-001: hidden behavior checks plus semantic structure check.
• click-help-shadowed-option-001: hidden Click help-option behavior checks.
• click-should-strip-ansi-tests-001: public submitted pytest run plus hidden mutation tests.
• datawrapper-mcp-docker-requirements-001: hidden dependency/artifact consistency checks.
• httpx-verify-false-client-cert-001: hidden SSL/cert behavior checks.
• prettier-duplicate-dangling-comments-001: hidden formatter behavior checks; public diff whitespace check retained.
• react-tabs-selected-focus-overlay-001: hidden stylesheet behavior checks; public diff whitespace check retained.
• remotion-audio-context-autoplay-muted-001: hidden focused source-behavior/scope checks; public diff whitespace check retained to avoid expensive full Remotion test runs.
• todomvc-toggle-all-checkbox-001: hidden DOM/view behavior checks.
• vite-deno-workspace-root-001: hidden workspace-root behavior/spec checks; public fixture staging and diff whitespace checks retained.

Validation

• python3 -m unittest tests.test_tasks tests.test_task_cards tests.test_reference tests.test_runner passed: 50 tests.
• python3 -m agentlab task validate --check-task-cards tasks passed for 14 task files.
• git diff --check passed.
• python3 -m agentlab task verify-reference tasks/starter/click-should-strip-ansi-tests-001 --workspace-root /private/tmp/ael-ref-click-should passed.
• python3 -m agentlab task verify-reference tasks/starter --workspace-root /private/tmp/ael-ref-starter-123 was interrupted after 9 starter tasks had passed; the interruption happened while preparing the large Remotion workspace.
• python3 -m agentlab task verify-reference tasks/starter/todomvc-toggle-all-checkbox-001 tasks/starter/vite-deno-workspace-root-001 --workspace-root /private/tmp/ael-ref-starter-123-rest passed.
• Remotion full verify-reference was attempted twice and interrupted during synthetic-base materialization for the large repository; the hidden verifier was validated separately in /private/tmp/ael-remotion-mini-123 by applying the reference patch plus verifier patch to the two checked files and running python3 .agentlab_hidden/check_behavior.py and git diff --check, both passed.
• python3 -m agentlab task smoke-test --task tasks/starter/click-should-strip-ansi-tests-001 --agent codex --codex-reasoning-effort low --codex-timeout-seconds 900 --runs-dir /private/tmp/ael-smoke-123 passed one trial / one job. Report: /private/tmp/ael-smoke-123/20260626-103409-click-should-strip-ansi-tests-001-codex-d4757b53/report.md.

Readiness evidence: #123
Readiness verdict: Ready to Implement

[Jordak]

Agent Review

Scope: PR #125, origin/main..6377b58
Pass: cycle 1 fresh, cycle 2 verification, cycle 3 focused verification
Review packet: summarized in this comment; local report generated by orchestrator
Reviewers: P1-R1/P2-R1/P3-R1 correctness, P1-R2/P2-R2 tests-regressions, P1-R3/P2-R3 issue-compliance
Effort metadata: implement workflow prescribed high; review-pass prescribed high by review-loop convention; effective reviewer model/effort not reported

Issue families:

1. [IF1] Severity: P2 - Remotion verifier did not tie resume rejection resolution to the shared waiter
   Found by: P1-R1-F1, P2-R1-F1
   Evidence: tasks/starter/remotion-audio-context-autoplay-muted-001/verifier.patch
   Issue family: hidden verifier could falsely pass a solution where resumePromise.catch(...resolve...) existed outside the isResuming.current waiter, leaving getIsResumingAudioContext() able to hang.
   Related occurrences or sweep: Remotion hidden verifier reread; no sibling task with this invariant.
   Why it matters: this was the key behavioral invariant behind the Remotion task.
   Decision: fixed
   Autopilot classification: auto-fix
   Autopilot rationale: evidenced, in scope, local checker tightening only.
   Complexity posture: neutral
   Smallest closing move or lazy-human decision: tightened the hidden verifier to inspect the isResuming.current = new Promise...finally body.
   Suggested fix or fix commit: d066a9e, then final tightening in 6377b58

2. [IF2] Severity: P2 - Remotion reference artifacts were stale after migration
   Found by: P1-R1-F2, P1-R2-F1, P1-R3-F1
   Evidence: tasks/starter/remotion-audio-context-autoplay-muted-001/reference-report.md, reference-result.json
   Issue family: migrated task artifacts must reflect hidden verifier outcomes rather than old public grader shape.
   Related occurrences or sweep: all tasks/starter/*/reference-result.json and reference-report.md checked for hidden verifier coverage.
   Why it matters: stale generated artifacts contradicted the task YAML and weakened review/landing evidence.
   Decision: fixed
   Autopilot classification: auto-fix
   Autopilot rationale: evidenced artifact drift, local regeneration/substitute evidence only.
   Complexity posture: neutral
   Smallest closing move or lazy-human decision: regenerated current Remotion artifacts with explicit focused-substitute validation note.
   Suggested fix or fix commit: d066a9e, refreshed again in 6377b58

3. [IF3] Severity: P2 - Vite reference report hid the first hidden verifier command behind clipped output
   Found by: P1-R2-F2
   Evidence: tasks/starter/vite-deno-workspace-root-001/reference-report.md, verifier.patch
   Issue family: hidden verifier report output should remain readable and not start mid-command.
   Related occurrences or sweep: sibling report scan found only Vite with clipped command output.
   Why it matters: human-facing reference evidence looked corrupted despite passing checks.
   Decision: fixed
   Autopilot classification: auto-fix
   Autopilot rationale: localized report/readability issue with a simple verifier-script split.
   Complexity posture: neutral
   Smallest closing move or lazy-human decision: split giant inline Node snippets into short hidden .cjs scripts.
   Suggested fix or fix commit: d066a9e

Declined issue families:

• None.

Resolved ask-user decisions:

• None.

Unresolved ask-user blockers:

• None.

Validation notes:

• python3 -m unittest tests.test_tasks tests.test_task_cards tests.test_reference tests.test_runner passed, 50 tests.
• python3 -m agentlab task validate --check-task-cards tasks passed.
• git diff --check passed.
• python3 -m agentlab task verify-reference tasks/starter/vite-deno-workspace-root-001 --workspace-root /private/tmp/ael-ref-vite-123-fix4 passed.
• python3 -m agentlab task smoke-test --task tasks/starter/click-should-strip-ansi-tests-001 --agent codex --codex-reasoning-effort low --codex-timeout-seconds 900 --runs-dir /private/tmp/ael-smoke-123 passed one trial / one job.
• Remotion full verify-reference was attempted twice and interrupted during large-repo synthetic-base materialization; focused substitute validation applied the same reference/verifier patches to the two checked files and passed python3 .agentlab_hidden/check_behavior.py plus git diff --check. The committed Remotion reference artifact records this caveat.

Final review-loop status: clean after verification. No ready marker or merge action was performed by this workflow.