Define task-prep and process-readiness rubric

[Jordak]

This was generated by AI during triage.

Context

The 2026-06-08 Radar review marked agentic task-prep practice as an "Investigate" item. The practical signal is simple but important: agent performance depends heavily on issue quality, feedback loops, single-task focus, and human-in-the-loop observation. Eval Lab should treat that setup work as measurable context instead of invisible background.

Sources

• AI Hero: Running Your AFK Agent - emphasizes HITL observation, clear issue descriptions, task-selection priority, feedback loops, single-task focus, TDD, and reviewing where agents struggle.
• mattpocock/skills - frames small, adaptable, composable skills as a way to reduce common coding-agent failure modes; highlights alignment/grilling, shared language/domain docs, and feedback loops.
• How Coding Agents Fail Their Users - source-backed reminder that failures include intent/rule/scope misunderstandings and inaccurate progress reporting, not just broken final code.

Goal

Define a small task-prep/process-readiness rubric that Eval Lab can use when curating issues, prompts, and future process-judge fixtures.

Candidate Rubric Dimensions

• Task has one clear intended outcome.
• Task boundaries are explicit enough for deterministic or human review.
• Acceptance criteria separate target behavior from implementation hints.
• Expected feedback loops are named: tests, type checks, linters, manual review, visual review, or reference verification.
• Domain vocabulary is available where jargon matters.
• Prompt ambiguity is intentional and bounded, not accidental.
• Human handoff expectations are clear: what changed, what was checked, and what remains uncertain.

Eval Lab Implication

This rubric could support:

• task-candidate triage;
• future issue-to-PRD/eval-authoring tasks;
• calibration fixtures for Prototype automated process judge for trial artifacts #77's process judge;
• clearer separation between agent failure and poor task setup.

Suggested First Slice

Write the rubric as a short design note or issue comment before adding code. Map it to the existing GitHub issue workflow and task-bundle concepts rather than creating a separate local backlog.

Acceptance Criteria For The Investigation

• Draft the rubric with 5-8 dimensions.
• Map each dimension to either task metadata, issue text, task-card content, human review, or process-judge evidence.
• Identify which dimensions can be checked deterministically and which require curator judgment.
• Decide whether this becomes documentation, a task-curation checklist, or a future report field.