Design complete-cycle and chained-maintenance tasks

[Jordak]

This was generated by AI during triage.

Context

The 2026-06-08 Radar review marked complete-cycle and chained-maintenance tasks as an "Investigate" item. Current Eval Lab starter tasks are valuable bounded maintenance tasks, but newer benchmarks are testing whether agents can preserve quality across environment setup, implementation, verification, and subsequent changes.

Sources

• SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle - evaluates environment reconstruction, code implementation, verification test generation, and an end-to-end full-cycle task.
• SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades - evaluates chained release-level upgrades where each transition builds on the agent's prior codebase and exposes regression risk across continuous maintenance.

Goal

Investigate a small post-starter Eval Lab task family for complete-cycle and chained-maintenance behavior.

Candidate Shapes

1. Complete-cycle task:

   • Start from a bare or lightly prepared pinned repository.
   • Agent must reconstruct the needed environment, implement the fix, and add or adjust verification tests.
   • Grade final behavior, setup reasonableness, and verification quality.

2. Chained-maintenance task:

   • Run two or three sequential requirement changes against the same evolving workspace.
   • Grade each round plus final regression count.
   • Preserve prior behavior as an explicit success criterion.

Eval Lab Implication

This would move beyond one-shot patch correctness toward reliability under realistic maintenance pressure: setup judgment, cross-phase dependencies, test generation, and regression preservation.

Suggested First Slice

Do not build the suite yet. First shortlist one tiny pinned real-repo issue that can be represented as either:

• one complete-cycle task with environment reconstruction and test-generation expectations, or
• a two-round chained task with a deterministic regression oracle.

Acceptance Criteria For The Investigation

• Identify one candidate repository/commit/issue for each shape, or explain why only one shape is feasible now.
• Define the minimum extra metadata needed beyond current task bundles.
• Define how pass/fail, regression count, and human review labels would be reported without mixing these tasks into the existing starter-suite metrics.
• Recommend whether to create a PRD under Draft PRD for harder post-starter evaluation tasks #60 or a standalone task-candidate issue.