Spike a tiny reproduction-workflow task fixture

[Jordak]

Parent

Related to #109 and #110.

Current behavior / design

As of this issue being written, Eval Lab's near-term task roadmap covers eval-authoring (#109), complete-cycle/chained maintenance (#110), and harder post-starter code tasks (#60). It does not yet include a tiny workflow-reproduction fixture where the agent must run an existing computational workflow, compare the expected artifact, and report whether reproduction succeeded without over-claiming.

Why change

SocSci-Repro-Bench shows that frontier coding agents can be evaluated on reproduction workflows, but a broad social-science benchmark is too far from Eval Lab's current coding-agent scope. A tiny local fixture would test a useful adjacent skill: executing provided code/data, distinguishing reproducible from non-reproducible outputs, and resisting confirmatory prompt framing.

Source:

• AI Coding Agents Can Reproduce Social Science Findings
• Existing task-family parents: Spike eval-authoring task from Kaggle benchmark workflow #109 and Design complete-cycle and chained-maintenance tasks #110

What to build

Spike one small, code-local reproduction-workflow fixture. The fixture should be closer to a software artifact workflow than a broad scientific benchmark: provide a pinned repo or mini-project with code, data or fixture inputs, expected output evidence, and a deterministic or semi-deterministic reproduction check.

Candidate shape:

• one task where reproduction should succeed;
• one optional paired task where reproduction should fail because a required artifact is missing or inconsistent;
• final answer rubric that rewards accurate limitation reporting over confirmatory over-claiming.

Acceptance criteria

• Identify one tiny candidate workflow that can run locally without external accounts or private data.
• Define the expected output artifact and how Eval Lab checks it.
• Include a clear success/failure reporting rubric for the agent's final answer.
• Decide whether the fixture belongs under eval-authoring (Spike eval-authoring task from Kaggle benchmark workflow #109), complete-cycle workflow tasks (Design complete-cycle and chained-maintenance tasks #110), or should be parked.
• Explicitly reject expanding into a broad non-code social-science suite unless this tiny fixture proves useful.

Blocked by

None - can start immediately