Parent
Related to #60 and #111.
Current behavior / design
As of this issue being written, Eval Lab records task metadata, run-surface metadata, scope-oracle metadata, transcripts/traces, result artifacts, and human review. It does not treat repository guidance files or prompt guidance as an explicit eval variable.
Why change
Recent repository-guidance research suggests that context files can materially change agent behavior, sometimes improving localization or workflow and sometimes adding bloat, cost, or misleading constraints. Eval Lab will miss an important source of trial variance if it cannot tell whether an agent used no guidance, stale guidance, normal maintained guidance, or refined guidance.
Sources:
What to build
Design a small post-starter task or fixture set that compares guidance conditions against the same pinned task. Candidate conditions: no project guidance, stale or misleading guidance, normal maintained guidance, and refined/minimal guidance. Preserve instruction source/provenance in the trial evidence and score file localization, validation-command selection, scope preservation, and final grader outcome.
Acceptance criteria
Blocked by
None - can start immediately
Parent
Related to #60 and #111.
Current behavior / design
As of this issue being written, Eval Lab records task metadata, run-surface metadata, scope-oracle metadata, transcripts/traces, result artifacts, and human review. It does not treat repository guidance files or prompt guidance as an explicit eval variable.
Why change
Recent repository-guidance research suggests that context files can materially change agent behavior, sometimes improving localization or workflow and sometimes adding bloat, cost, or misleading constraints. Eval Lab will miss an important source of trial variance if it cannot tell whether an agent used no guidance, stale guidance, normal maintained guidance, or refined guidance.
Sources:
What to build
Design a small post-starter task or fixture set that compares guidance conditions against the same pinned task. Candidate conditions: no project guidance, stale or misleading guidance, normal maintained guidance, and refined/minimal guidance. Preserve instruction source/provenance in the trial evidence and score file localization, validation-command selection, scope preservation, and final grader outcome.
Acceptance criteria
Blocked by
None - can start immediately