VLABench Contract Audit

An exact, checksum-anchored audit and non-mutating repair sidecar for two linked VLABench/LeRobot interface defects.

In lerobot/vlabench_unified@f61ad898199e746c20af3112f278cc86eb3ca3c4, LeRobot's pinned default eef Gym Box rejects all 3,114,872 action rows. The pinned adapter nevertheless consumes those absolute-pose targets through inverse kinematics. This establishes an interface-contract mismatch; it does not establish trajectory corruption. Separately, the published state stores a legacy closed-bit while actions use an open-bit; this repository supplies an explicit, versioned, non-mutating repair sidecar.

Nothing in this repository has been filed or published upstream yet. The drafts are intentionally review-ready but local.

Why this matters

Environment-space metadata is executable documentation. Trainers, validators, normalizers, policy heads, wrappers, and benchmark harnesses use it to decide what an action means. A Box of [-1, 1] can look plausible while contradicting an absolute Cartesian-pose dataset whose Euler angles span roughly [-pi, pi].

The gripper issue is subtler. VLABench's pinned Franka source names a value open, but returns true when both finger positions are below a closed threshold; the same source contains # BUG: should be False. The converter preserves that state value, while encoding actions as 1 = open. Fixing only the simulator or only the dataset would create silent train/eval drift. The safe repair is an explicit convention boundary.

Exact result

The release evidence scans all three Parquet shards plus task/episode metadata, checks every input byte, and verifies the source chain and official lineage controls.

Control	Result
Frames scanned	3,114,872
Episodes	10,977
Unique nonempty task strings	295
Rows outside the pinned default Gym Box	3,114,872 (100%)
Scalars outside the Box	5,545,679 / 21,804,104 (25.4341%)
Rows outside broad semantic envelope	0
State/action complemented same-row	2,965,492 (95.2043%)
State/previous-action complemented	2,988,736 / 3,103,895 (96.2899%)
Same-row XYZ target within 5 cm of observed state	99.0798%

The last row is a negative control, not a quality score: position behavior looks like executed absolute-pose targets rather than arbitrary normalized deltas.

Reproduce from zero

Prerequisites: Python 3.10+, uv 0.9.18, about 150 MB of disk space, and network access for the initial download.

uv sync --frozen --group dev
uv run vlabench-contract-audit download --data-root .audit-data

# Exit 1 means verified finding; exit 2 means invalid/unverifiable.
set +e
uv run vlabench-contract-audit scan \
  --data-root .audit-data \
  --output reproduced-audit.json
status=$?
set -e
test "$status" -eq 1

uv run vlabench-contract-audit verify-artifact

The release manifest pins every dataset shard, metadata file, official lineage metadata file, and semantic source snapshot by full revision, size, URL, and SHA-256. There is no unchecksummed custom-artifact mode. A custom source manifest requires its SHA-256 explicitly.

Build the non-mutating repair

uv run vlabench-contract-audit build-sidecar \
  --data-root .audit-data \
  --output-dir vlabench-gripper-open-v2

# Use the digest printed by build-sidecar.
uv run vlabench-contract-audit verify-sidecar \
  --sidecar-dir vlabench-gripper-open-v2 \
  --expected-manifest-sha256 <sha256> \
  --data-root .audit-data

The sidecar also copies task_index and action_gripper_open as independent join/semantic controls; it does not rewrite the action. The destination must not already exist. The builder verifies inputs before and after generation, writes into a temporary sibling directory, and renames only after validation. It never edits source Parquets and never rewrites actions.

Join the sidecar on index and select one mode deliberately:

• legacy_closed_v1: consume the original state bit for existing checkpoints.
• explicit_open_v2: consume state_gripper_open for new, explicitly migrated training/evaluation pipelines.

Automatic convention guessing is forbidden. See docs/SIDECAR.md.

Stable exit codes

Command	0	1	2
scan	verified clean	verified finding	invalid/unverifiable/I/O
check-artifact	valid, no finding	valid finding	malformed/checksum failure
verify-artifact	exact release artifact valid	unused	invalid
build-sidecar	success	unused	invalid
verify-sidecar	valid	unused	invalid

KeyboardInterrupt is not swallowed, so shells retain the conventional 130.

Repository map

• artifacts/source-manifest.json — complete immutable input manifest.
• artifacts/audit-evidence.json — deterministic full-scan result.
• artifacts/SHA256SUMS — release trust anchors.
• release-assets/explicit-open-v2/ — complete 3,114,872-row repair sidecar, intentionally excluded from the wheel.
• src/vlabench_contract_audit/ — scanner, validators, downloader, and sidecar.
• tests/ — strict parsing, corruption fixtures, CLI exits, determinism, and non-mutation checks.
• docs/METHODOLOGY.md — query-level methodology and independent controls.
• docs/CLAIM_BOUNDARIES.md — what the evidence does and does not establish.
• docs/UPSTREAM_ISSUE_DRAFT.md — review-ready issue, not submitted.
• docs/UPSTREAM_FIX_DRAFT.md — narrowly scoped upstream patch design.
• demo/ and docs/X_POST_DRAFT.md — evidence-driven launch visual and copy.
• REPORT.md — concise publication/report handoff.

Licensing

This project is Apache-2.0. It does not redistribute upstream source snapshots or dataset Parquets; the downloader retrieves exact pinned originals. See NOTICE for upstream attribution. VLABench is MIT-licensed, LeRobot and the cited dataset are Apache-2.0 at the pinned references.

Citation

Until an archival release exists, cite the repository revision together with the dataset and source pins in artifacts/source-manifest.json. Do not cite the semantic VLABench commit as exact generator provenance: the dataset card does not provide that checkout.