Spike eval-authoring task from Kaggle benchmark workflow

[Jordak]

This was generated by AI during triage.

Context

The 2026-06-08 Radar review marked eval-authoring tasks as an "Investigate" item. The signal is that benchmark/task creation is moving into local coding-agent workflows, which gives Coding Agent Eval Lab a natural meta-eval: can an agent author a good eval task, not just solve one?

Sources

• Google: Kaggle is making AI benchmark creation effortless - local benchmark creation with coding agents, including writing, pushing, running, and downloading tasks from local environments.
• Kaggle/kaggle-skills - includes a write-kaggle-benchmarks skill and treats skills as folders of instructions/scripts/resources for specialized agent tasks.

Goal

Investigate a future Eval Lab task family where an agent must author a valid task bundle and grader from a task-design brief.

Candidate Shape

Give the agent a small source issue or bug description and ask it to produce:

• task.yaml;
• generated task-card.md;
• deterministic target checks;
• a reference artifact or reference patch plan;
• at least one hidden or non-obvious edge case;
• a short explanation of why the grader is not tautological.

Eval Lab Implication

This would measure evaluator quality: task clarity, grader validity, edge-case coverage, reference-verification readiness, and resistance to benchmark theater.

Suggested First Slice

Design one offline fixture that does not require uploading anything to Kaggle. Use Eval Lab's own task-bundle integrity and reference-verification concepts as the target artifact contract.

Acceptance Criteria For The Investigation

• Identify one concrete starter-sized source issue suitable for eval-authoring.
• Draft the expected output contract for the authored task bundle.
• Define a grading rubric for:
  • task-bundle validity;
  • non-tautological grader design;
  • hidden edge-case coverage;
  • reference-artifact feasibility;
  • readable task-card output.
• Decide whether this should become a real task family, a curator tool, or a parked idea.