Parent
Related to #77.
Current behavior / design
As of this issue being written, #77 proposes a constrained process_judge.json artifact and fixtures for clean success, skipped verification, false completion claim, out-of-scope change, and unknown/insufficient evidence. It does not yet define fixtures that specifically test repository-exploration quality, context selection, or whether the agent's reads and searches were proportional to the task.
Why change
Recent agent-eval work separates final patch success from the path an agent took through the repository. For Eval Lab, this gives #77 a concrete calibration target: not a vibes-based process score, but evidence-backed checks for whether the agent found relevant context, avoided wasteful or irrelevant context, and used validation feedback honestly.
Sources:
What to build
Add a tiny fixture set for the process judge that focuses on context-selection evidence. The fixtures should exercise ranked/selected relevant files or line spans, irrelevant-but-tempting context, missing decisive context, validation feedback that should change the trajectory, and cases where the artifact evidence is insufficient.
Acceptance criteria
Blocked by
#77 evidence-packet shape decision
Parent
Related to #77.
Current behavior / design
As of this issue being written, #77 proposes a constrained
process_judge.jsonartifact and fixtures for clean success, skipped verification, false completion claim, out-of-scope change, and unknown/insufficient evidence. It does not yet define fixtures that specifically test repository-exploration quality, context selection, or whether the agent's reads and searches were proportional to the task.Why change
Recent agent-eval work separates final patch success from the path an agent took through the repository. For Eval Lab, this gives #77 a concrete calibration target: not a vibes-based process score, but evidence-backed checks for whether the agent found relevant context, avoided wasteful or irrelevant context, and used validation feedback honestly.
Sources:
What to build
Add a tiny fixture set for the process judge that focuses on context-selection evidence. The fixtures should exercise ranked/selected relevant files or line spans, irrelevant-but-tempting context, missing decisive context, validation feedback that should change the trajectory, and cases where the artifact evidence is insufficient.
Acceptance criteria
process_judge.json, such as command index, file path, line span, transcript snippet id, or grader/check output.present,absent, orunknownrather than a single quality score.Blocked by
#77 evidence-packet shape decision