Skip to content

Add guidance-ablation and instruction-provenance task slice #112

Description

@Jordak

Parent

Related to #60 and #111.

Current behavior / design

As of this issue being written, Eval Lab records task metadata, run-surface metadata, scope-oracle metadata, transcripts/traces, result artifacts, and human review. It does not treat repository guidance files or prompt guidance as an explicit eval variable.

Why change

Recent repository-guidance research suggests that context files can materially change agent behavior, sometimes improving localization or workflow and sometimes adding bloat, cost, or misleading constraints. Eval Lab will miss an important source of trial variance if it cannot tell whether an agent used no guidance, stale guidance, normal maintained guidance, or refined guidance.

Sources:

What to build

Design a small post-starter task or fixture set that compares guidance conditions against the same pinned task. Candidate conditions: no project guidance, stale or misleading guidance, normal maintained guidance, and refined/minimal guidance. Preserve instruction source/provenance in the trial evidence and score file localization, validation-command selection, scope preservation, and final grader outcome.

Acceptance criteria

  • The design names the guidance conditions and how each is supplied to the agent harness.
  • The trial evidence records instruction/guidance source, version/provenance, and whether the guidance was intended to be authoritative.
  • The task or fixture can compare at least two guidance conditions without changing the underlying task target.
  • The rubric includes localization quality and validation-command selection in addition to final pass/fail.
  • The issue explicitly states whether this belongs under Draft PRD for harder post-starter evaluation tasks #60 harder-task design, Define task-prep and process-readiness rubric #111 process-readiness rubric, or a standalone task candidate.

Blocked by

None - can start immediately

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestneeds-triageMaintainer needs to evaluate this issue

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions