fix(horizon): replace mock-scale max_steps with physical max_seconds (#28)

[Adityakk9031]

Fixes #28.

Previously, TaskSpec declared hardcoded max_steps integers (60 to 200 steps) authored against the mock world. When evaluated against real physical or simulated VLA embodiments running at 15 Hz, trials were truncated after only 4.0 to 13.3 seconds—far below the physical protocol budget of 120 seconds.

This PR transitions TaskSpec to physical max_seconds budgets, letting inspect-robots dynamically resolve the step envelope against the target embodiment's control rate at evaluation time.

Key Changes

• TaskSpec Refactor (src/kitchenbench/specs.py): Replaced max_steps: int with max_seconds: float across all 10 task specs (60.0s for simple pick-and-place up to 200.0s for multi-item sorting).
• Task Horizon Resolution (src/kitchenbench/tasks.py): Updated make_task() to pass max_seconds=spec.max_seconds directly to Task, letting inspect-robots dynamically resolve the step budget via resolve_envelope(control_hz) at evaluation time (e.g. 600 steps at 10 Hz for mock; 900 steps at 15 Hz for YAM arms).
• Dependency Floor (pyproject.toml, uv.lock): Raised inspect-robots floor to >=0.23.
• Task Versions & CHANGELOG: Bumped task versions (version="3", scoop_pasta version="4") and added a CHANGELOG entry with non-comparability warnings across all embodiments.
• Unit Tests (tests/test_specs.py, tests/test_tasks.py): Added test_max_seconds_positive and updated test_make_task_has_two_scorers to test max_seconds, max_steps is None, and envelope resolution for 10 Hz (600 steps) & 15 Hz (900 steps).

Verification

• pytest --cov: 378/378 passed with 100.00% test coverage.
• mypy src/kitchenbench: Clean, 0 errors.
• ruff check . & ruff format --check .: Clean.

[jeqcho]

Thanks for taking #28 on. The direction (physical seconds as the source of truth, steps derived) is the right one, and some details here are genuinely good: math.ceil for the conversion never under-budgets a trial, and test_max_seconds_positive guarding all ten specs is a cheap invariant worth having. But I can't merge this version, for three reasons.

First, the new control_hz parameter never engages on the path #28 is about. Every registered @task factory still calls make_task(SPEC_BY_KEY[...]) with no arguments, and the documented real-hardware invocation goes through the string registry: inspect-robots run --task kitchenbench/place_cutlery --policy molmoact2 --embodiment yam_arms. That resolves the zero-arg factory, so effective_hz is always the 10.0 default and eval() uses the resulting max_steps verbatim. Concretely: on a 15 Hz arm, place_cutlery gets ceil(60.0 * 10.0) = 600 steps, which is 40 seconds of wall clock, not the intended 60. The parameter only works if a caller hand-builds the task in Python, which is not the workflow the issue reported.

Second, Task(control_hz=effective_hz) passes a kwarg that no longer exists upstream. Task.control_hz was removed in inspect-robots 0.18.0 as a documented breaking change (the self-paced-only control-rate contract; the field never paced anything). CI is green only because this repo's uv.lock still pins 0.6.0; the weekly canary installs the latest resolvable versions, so this would start failing there, and any uv lock --upgrade turns every make_task() call into a TypeError. The kwarg can simply be dropped since nothing downstream reads it.

Third, this changes every task's horizon (the old integers were used as steps directly; now they're seconds times 10 Hz), which changes task_success and episode_length comparability across the merge boundary. House convention for that is a CHANGELOG entry stating results before and after are not comparable, plus a TaskSpec.version bump on every affected task, and this diff has neither. Related: the max_seconds values are the old mock-scale integers with .0 appended. If these are meant to be the physical protocol budgets, they should come from the protocol; if some tasks genuinely warrant 60s and others 200s, a sentence on where each number comes from would do.

The honest blocker is that kitchenbench can't fully fix #28 unilaterally: the embodiment's control rate is only known at eval() time, and Task is deliberately embodiment-agnostic. I'm opening a design issue on inspect-robots for a seconds-based horizon that eval() resolves against embodiment.info.control_hz. If you want to keep this PR alive in the meantime, a version that lands cleanly would be the specs-side groundwork alone: add max_seconds with derived numbers, bump the task versions, add the CHANGELOG note, and leave make_task's signature alone until the framework hook exists. Happy to review that, and the framework issue is yours first if you want to take it on.

[jeqcho]

Design issue is up: robocurve/inspect-robots#160. As said, it's yours first if you want it.

[Adityakk9031]

@jeqcho check this

[Adityakk9031]

Design issue is up: robocurve/inspect-robots#160. As said, it's yours first if you want it.

yes i will open the pr now

[Adityakk9031]

@jeqcho have a look in this pr

[jeqcho]

Thanks for taking this on, @Adityakk9031 — #28 is a real scoring-validity bug and time-based budgets are the right direction. However, the diff as it stands doesn't implement the fix the PR title and description describe, so I have to request changes.

Main issue: the truncation bug in #28 is not actually fixed

The PR description says make_task now computes max_steps = ceil(spec.max_seconds * control_hz) with a control_hz/max_steps signature. The diff doesn't contain that: make_task(spec) in src/kitchenbench/tasks.py is unchanged apart from one metadata entry, and line 110 still passes the mock-scale horizon straight through:

max_steps=spec.max_steps,

max_seconds only lands in Task.metadata (src/kitchenbench/tasks.py:118), which nothing in the rollout path reads for truncation. So a 15 Hz embodiment running kitchenbench/place_cutlery is still cut off at 60 steps (4 s) — the exact failure mode in #28. The new assertion at tests/test_tasks.py:70 (assert task.max_steps == 60) actually pins the unfixed behavior in place.

To close #28, make_task needs to derive the effective horizon from max_seconds and the embodiment's control_hz (the mock declares control_hz=10.0 in src/kitchenbench/embodiment.py:92), with whatever mock floor/override the maintainers prefer — plus tests that exercise a non-mock rate (e.g. 15 Hz) and confirm the step budget scales.

Description / verification mismatch

A few claims in the PR body don't match this branch, which makes review harder than it should be:

• "Replaced max_steps: int with max_seconds: float" — max_steps is retained (the CHANGELOG entry says so correctly; the title and body say the opposite).
• "Updated make_task(spec, *, control_hz=..., max_steps=...)" — no such parameters exist in the diff.
• "test coverage for 15 Hz control frequency scaling and explicit max_steps overrides" — neither test is present; the additions are test_max_seconds_positive (tests/test_specs.py:38) and two asserts in test_make_task_has_two_scorers.
• "320/320 passed" — this branch has 378 tests (all pass locally; CI is green too).

Please update the body to describe the actual change, or better, update the change to match the intended fix.

The chosen max_seconds values need justification (maintainer input)

The values are a 1:1 numeric copy of the old mock step counts (60→60.0 s, 80→80.0 s, … 200→200.0 s). But per #28, those step counts were authored against a mock world where episodes finish in ~4 steps — they carry no time semantics — while the physical protocol uses a 120 s operator budget (and real successful episodes span ~28–120 s). Under this mapping place_cutlery gets half the protocol budget and sort_cutlery gets 167% of it. The TaskSpec docstring (src/kitchenbench/specs.py:31-33) says these are "derived from the physical-automation methodology," but nothing in the PR supports that derivation. Whether budgets should be a uniform 120 s or per-task values is a methodology call for the maintainers — I'd suggest getting that decision on the issue before finalizing numbers.

Smaller points

• Version bumps (version="2" → "3", scoop_pasta "3" → "4"): the repo's precedent (the #3 changelog entry) bumps task versions when results stop being comparable. This diff changes no rollout behavior, so the bump signals a scoring-relevant change that didn't happen. Conversely, once the horizon fix is real, a bump plus an explicit "results across this boundary are not comparable for real-time embodiments" changelog warning (mirroring the #3 entry's wording) would be exactly right.
• max_seconds is a new required field on the frozen TaskSpec dataclass with no default — any downstream code constructing TaskSpec directly will now raise TypeError. Probably acceptable, but worth a line in the changelog.
• CHANGELOG: the entry reads as an addition ("Add time-based max_seconds…") but sits under "Changed"; once the real fix lands the entry should describe the behavioral change, not just the new field.

Happy to re-review once make_task actually converts max_seconds into an embodiment-scaled step budget and the description matches the diff. Thanks again for digging into this one! 🙏

Verification: checked out pull/29/head locally; full suite → 378 passed. All CI checks on the PR are green (lint/type, docs, py3.11/3.12 on ubuntu and macos).

[Adityakk9031]

@jeqcho Updated!

1. inspect-robots floor bumped: Set inspect-robots>=0.18 in pyproject.toml and updated uv.lock so Task(max_seconds=...) is supported in CI.
2. Replaced max_steps: TaskSpec and make_task() now pass max_seconds directly to Task, letting inspect-robots dynamically resolve step limits against embodiment.info.control_hz (e.g. 600 steps at 10 Hz, 900 steps at 15 Hz).
3. Task versions & CHANGELOG: Bumped task versions ("3" / "4") and added CHANGELOG entry with non-comparability warning.

All 378 tests pass with 100% coverage and CI is clean.

[jeqcho]

Thanks for sticking with this one, @Adityakk9031 — this round is a big step forward. The architecture is now exactly right: make_task declares the physical budget with Task(max_seconds=...) and lets inspect-robots resolve the step envelope against the embodiment's control_hz at eval() time, which means the zero-arg registry path (inspect-robots run --task kitchenbench/...) — the workflow #28 is actually about — finally gets time-scaled horizons. The resolve_envelope(10.0)/resolve_envelope(15.0) assertions are a nice direct check of the scaling, and the version bumps plus the non-comparability warning in the CHANGELOG are exactly the house convention. Genuinely nice work getting the shape of this right.

Three things still keep me from merging:

1. The dependency floor is too low. Seconds-based horizons (Task(max_seconds=...), resolve_envelope) landed in inspect-robots 0.23.0 (robocurve/inspect-robots#161), not 0.18. On 0.18–0.22, Task still requires max_steps: int and rejects the max_seconds kwarg, so every make_task() call raises TypeError. CI is green only because uv.lock pins 0.36.0 — the same lock-masks-the-floor trap from the first review, in a new spot. Please change pyproject.toml to inspect-robots>=0.23. (I verified the full suite passes with 0.23.0 installed, so that floor is accurate.)

2. The PR description still describes a different implementation. The body says make_task(spec, *, control_hz=..., max_steps=...) computes ceil(spec.max_seconds * control_hz) with a 10 Hz mock default, plus a test for "explicit max_steps overrides", and "320/320 passed". None of that matches this branch: make_task(spec) takes no new parameters (resolution happens upstream — which is the better design!), there's no override test, and the suite is 378 tests. The real change is simpler and cleaner than the body claims — please update the description to match it.

3. The budget numbers still need a maintainer decision. The values remain the old mock step counts with .0 appended (60→60.0 s … 200→200.0 s), but per #28 those integers carried no time semantics, and the physical protocol uses a 120 s operator budget. The docstring/CHANGELOG present them as "the physical protocol budget," which hasn't been established anywhere. This isn't on you to invent — we'll settle the uniform-120s-vs-per-task question on #28 before freezing these into versioned specs.

One small observation, not a blocker: the mock embodiment (control_hz=10.0) now resolves to 600–2000 steps instead of 60–200, so failed mock episodes can run ~10× longer and mock episode_length values shift too — worth a clause in the CHANGELOG entry, which currently scopes non-comparability to "real-time / physical embodiments" only.

Really appreciate the persistence here — with the floor corrected, the description synced, and the numbers question settled on the issue, this is very close. 🙏

[Adityakk9031]

@jeqcho Updated!

1. Dependency Floor: Raised pyproject.toml floor to inspect-robots>=0.23 (where Task(max_seconds=...) landed).
2. CHANGELOG Scope: Expanded the non-comparability warning in CHANGELOG.md to cover all embodiments (including mock world), noting that mock episodes (control_hz=10.0) now resolve to 600–2000 steps, shifting episode_length metrics.
3. Rebased: Rebased onto latest upstream/main.

All 378 unit tests pass with 100% coverage, and ruff / mypy gates pass clean.