Spike hidden-validator process-validity fixture

[Jordak]

Parent

Related to #60, #77, and #114.

Current behavior / design

As of this issue being written, Eval Lab has deterministic graders, scope-oracle metadata, human review labels, and an open process-judge proposal in #77. Existing process-judge fixture work covers context-selection evidence in #114, but there is no narrow task fixture specifically designed around hidden or withheld process-validity checks where final visible tests are not enough.

Why change

Recent eval work emphasizes guarded environments and process validity: avoiding leakage, preserving protocol validity, recovering after targeted failures, stopping verification at a justified point, and not passing for the wrong operational reason. This is a useful next calibration target for Eval Lab because coding agents can pass final tests while still using invalid shortcuts or unreviewable process paths.

Sources:

• GRACE-DS
• RigorBench
• PORTICO: lingering authority
• Existing process-judge parent: Prototype automated process judge for trial artifacts #77
• Existing context-selection fixture issue: Add context-selection evidence fixtures for the process judge #114

What to build

Spike one tiny fixture or task design where success requires both an ordinary final outcome and process-validity evidence. The fixture should be small enough to run locally and should make invalid shortcuts observable without requiring a broad new benchmark.

Candidate fixture properties:

• visible tests can pass while a hidden/process-validity condition fails;
• forbidden or withheld fixture access is detectable;
• command/reproduction log is part of the evidence;
• authority or access can expire after a subgoal, and stale reuse is a graded failure;
• final handoff must report the validation path honestly.

Acceptance criteria

• Define one small candidate fixture with visible checks and hidden/process-validity checks.
• State which process-validity conditions belong in deterministic graders versus Prototype automated process judge for trial artifacts #77-style process-judge findings.
• Include expected evidence pointers such as command index, changed path, transcript snippet, withheld fixture access, or final-message excerpt.
• Include at least one case where final visible tests pass but the process-validity check should fail.
• Keep the fixture scoped to coding-agent trial validity; do not expand into a broad data-science or workflow benchmark.
• Decide whether implementation should wait for Prototype automated process judge for trial artifacts #77's evidence-packet shape.

Blocked by

#77 evidence-packet shape decision for process-judge integration. The fixture design can start immediately.