Add SampleSource for dynamic sample generation within a task

[ransomr]

This PR contains:

• New features
• Changes to dev-tools e.g. CI config / github tooling
• Docs
• Bug fixes
• Code refactor

What is the current behavior? (You can also link to an open issue here)

Fixes meridianlabs-ai#36

A task's dataset is static: you pass a Dataset (or list of samples) and the task runs exactly those. There is no way to generate a task's samples dynamically based on results (for RL or adaptive evals) — the TaskSource pattern exists only at the eval level, where it produces whole tasks.

What is the new behavior?

SampleSource mirrors the TaskSource pattern at the sample level: pass one as the dataset argument to Task to generate samples from code while the task runs, all within one task / eval log.

class SampleSource:
    def initial_samples(self) -> list[Sample]: ...            # sync seed (may be empty)
    async def next_samples(self) -> list[Sample] | None: ...  # blocking pull; None ends the task
    async def sample_complete(self, sample: EvalSample) -> list[Sample] | None: ...  # observe + add

• Contract mirrors TaskSource: a synchronous seed, per-result follow-ups returned from sample_complete (the simplest sources need no next_samples()), and a blocking next_samples() for the explicit-pull case. SampleSource.from_samples(...) builds a source from a seed + callbacks without subclassing.
• Live injection: task_run gains a dispatcher (mirroring run_multiple's feed loop) — added samples start immediately on free capacity (bounded by max_samples); next_samples() is only awaited when fully idle, so there are no lost wakeups. Plain tasks keep the unchanged tg_collect path.
• enqueue_sample() imperative primitive (mirrors enqueue_task): add samples to the running task from any code (solver / scorer / tool), ContextVar-scoped per task. Raises a clear error outside a SampleSource-driven task.
• Ids / epochs / totals: injected samples get auto-ids continuing the seed's numbering (duplicate ids are a hard error), run for the task's configured epochs, and grow the planned totals (display denominator, fractional fail_on_error threshold, control-channel counters via a new record_samples_added).
• --limit / --sample-id: --limit caps the total samples (seed + added; additions past the cap are ignored with a warning) and --sample-id filters added samples the same way it filters the seed.
• Wake (re-armable wake signal) moved from _eval/run.py to _util/_async.py for reuse by the new dispatcher.

Docs: docs/sample-source.qmd (+ nav/reference entries). Design notes: design/sample-source.md. Tests: tests/test_sample_source.py (19 tests, including a liveness test that discriminates live injection from batch-at-a-time).

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

No. Task(dataset=...) accepts a new type; existing datasets behave identically (plain tasks take the unchanged fixed-set path). Task instances gain a sample_source attribute (None unless a source was passed).

Other information:

Verification: pytest tests/test_sample_source.py tests/test_task_source.py tests/test_enqueue_task.py tests/test_eval.py tests/_control tests/display tests/_eval tests/dataset tests/test_fail_on_error.py tests/test_retry_on_error.py tests/test_task_with.py tests/test_sample_id.py tests/test_eval_no_samples.py tests/test_disk_sample_store.py tests/test_score_on_error.py all pass; ruff check and mypy clean on changed files.

Known limits (documented in design/sample-source.md): task-level retries reuse completed injected samples only where the source regenerates the same ids (the same determinism contract as TaskSource + eval_set); early-stopping managers see only the seed at start_task.

🤖 Generated with Claude Code

[dragonstyle]

Is passing an initial samples required to get a sandbox started (or would it start downstream if I omitted this and provided next samples)? Any other configuration type things that would only be configured if initial_samples has content?

[ransomr]

Good question — as of b3d5e94, no: the seed isn't required for sandboxes, and there is no other configuration that only happens when initial_samples() has content.

Some background: per-sample sandbox creation (sample_init / compose up) always happens at sample run time, wherever the sample came from. What was keyed off the seed was the run-level startup pass (SandboxManager), which discovered sandbox configs only from the seed dataset — so a config first seen in an added sample (or the task-level config when the seed was empty) missed task_init: no up-front image build/pull (each sample's own compose up would pull on demand, racing concurrent samples), no fail-fast validation, and no registered task_cleanup at end of run.

b3d5e94 routes added samples through the same incremental sandbox startup pass used for injected tasks: the dynamic dispatcher awaits task_init for not-yet-started configs before spawning each added batch (already-started configs are a set-membership no-op, so the common same-config-as-seed case costs nothing). A source driven entirely by next_samples() now behaves identically to a seeded one.

Everything else was already handled equivalently downstream for added samples: id assignment/uniqueness, total-sample accounting (fail_on_error denominator, progress), --limit / --sample-id, and the max_sandboxes limiter (re-registered idempotently per sample). Updated the docstring here and added a Sandboxes section to docs/sample-source.qmd to document this.

[ransomr]

Follow-up: a review pass over the whole PR refined the "everything else is handled equivalently" part of the answer above — three interactions turned out not to be equivalent for dynamic tasks, and df9c1f5 changes their behavior:

• early_stopping is now rejected (PrerequisiteError, pre-run). Managers register a fixed sample set at start_task — samples the source adds were never registered — and samples a manager halts complete without notifying the source, which would hang a source blocked awaiting completions.
• Range --limit (start,end) is now rejected for SampleSource tasks. It selects seed samples by position, but added samples have no position (the previous behavior was incoherent: --limit 10,20 on a 3-sample seed ran zero seed samples yet admitted 10 additions). Plain numeric --limit still caps the total as documented.
• Fractional fail_on_error defers to the end-of-run check. The planned total grows while the task runs, so the mid-run check measured early errors against a transiently small denominator (a 1-sample seed erroring first tripped 1 >= 0.5*1 before the source produced anything). Absolute-count and any-error thresholds still abort mid-run.

Also in df9c1f5, related fixes surfaced by the same review: task retries now re-fire sample_complete for reused samples so completion-driven sources reconstruct their runs (previously their follow-up samples were silently absent from retry logs); the control channel no longer reports an idle dynamic eval (e.g. blocked in next_samples() with an empty seed) as finished, so ctl task cancel works during that window; and added samples' memory is bounded (slots released after their last epoch). Docs (sample-source.qmd) and the design doc cover the new constraints and the retry contract.