Skip to content

Add context-selection evidence fixtures for the process judge #114

Description

@Jordak

Parent

Related to #77.

Current behavior / design

As of this issue being written, #77 proposes a constrained process_judge.json artifact and fixtures for clean success, skipped verification, false completion claim, out-of-scope change, and unknown/insufficient evidence. It does not yet define fixtures that specifically test repository-exploration quality, context selection, or whether the agent's reads and searches were proportional to the task.

Why change

Recent agent-eval work separates final patch success from the path an agent took through the repository. For Eval Lab, this gives #77 a concrete calibration target: not a vibes-based process score, but evidence-backed checks for whether the agent found relevant context, avoided wasteful or irrelevant context, and used validation feedback honestly.

Sources:

What to build

Add a tiny fixture set for the process judge that focuses on context-selection evidence. The fixtures should exercise ranked/selected relevant files or line spans, irrelevant-but-tempting context, missing decisive context, validation feedback that should change the trajectory, and cases where the artifact evidence is insufficient.

Acceptance criteria

  • Fixtures include at least one clean context-selection success and one missed-decisive-context case.
  • Fixtures include at least one case where the agent over-explores irrelevant context despite passing or nearly passing the final grader.
  • Each fixture has evidence pointers suitable for process_judge.json, such as command index, file path, line span, transcript snippet id, or grader/check output.
  • The expected judge output uses present, absent, or unknown rather than a single quality score.
  • The fixture vocabulary stays process-focused and does not replace deterministic graders.

Blocked by

#77 evidence-packet shape decision

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestneeds-triageMaintainer needs to evaluate this issue

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions