feat(examples): add Daft-native LeRobot and DROID reading scripts - #13
Open
everettVT wants to merge 17 commits into
Open
feat(examples): add Daft-native LeRobot and DROID reading scripts#13everettVT wants to merge 17 commits into
everettVT wants to merge 17 commits into
Conversation
everettVT
force-pushed
the
everettVT/add-dataset-scripts
branch
from
July 2, 2026 23:13
7d543b0 to
ed544d0
Compare
everettVT
force-pushed
the
everettVT/add-dataset-scripts
branch
from
July 3, 2026 23:27
ed544d0 to
8b335ed
Compare
everettVT
marked this pull request as ready for review
July 3, 2026 23:27
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
daft-physical-ai/examples/egodex_handtracking_lite/README.md
Lines 19 to 20 in 8b335ed
Update demo run paths after moving assets
After this commit moves the generated demo files under examples/egodex_handtracking_lite/, the documented commands still point to the removed examples/demo.py and examples/demo.ipynb paths. Users following this README from the repo root will fail before running the demo; update these commands to the new subdirectory path or tell users to cd into it first.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
everettVT
changed the title
everettVT
force-pushed
the
everettVT/add-dataset-scripts
branch
from
July 3, 2026 23:48
db3f5ab to
293b973
Compare
Policy evaluation is the workflow; failure detection is its payoff. The failure_modes/ module becomes daft_physical_ai/evals/ - the analysis half of the eval loop over the canonical step-row schema: - evals/failures.py: detect_regrasp + the terminal-failure taxonomy (moved) - evals/compare.py: episode_outcomes / success_rates / failure_counts / compare_policies, always grouped by policy - episode_id names the eval spec (suite/task/init_state/seed), so shared specs pair across policies instead of chimera-ing into phantom episodes - evals/protocol.py: the canonical LIBERO protocol (50 trials/task, seed 7, per-suite step caps) as constants + validate_run() so "we reproduced the benchmark" is checkable against the parquet itself Rollout generation (sim, policy stacks, Modal GPU apps) stays in the VLA-JEPA harness; the schema is the contract. Promoting the LIBERO runner as a [libero] extra is roadmapped, gated on a lerobot release. examples/ becomes the numbered researcher journey (01 reading data ... 08 policy evals) with an index README and per-stage stubs; existing examples move into their stages, and the marimo site gains a /demos/policy-evals page (old route redirects). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…is examples Land the OpenVLA vs VLA-JEPA benchmark data from the VLA-JEPA harness as in-repo parquet (examples/08_policy_evals/data/: 200 episodes, 23,283 step rows, ~2 MB zstd - byte-faithful compaction of the harness's per-episode parts), so every policy-eval example runs offline on CPU after a clone. Three new runnable examples against the real data: - success_rates.py: openvla 84% vs vla_jepa 99%, with the per-task table - compare_policies.py: paired per-spec view - vla_jepa wins 15 layouts openvla loses (and loses none openvla wins); every openvla loss ran to the 250-step cap while vla_jepa finished the same layout in 78-141 steps - validate_protocol.py: protocol check off the parquet (exits non-zero on deviation; this dataset is the 10-trials/task variant of the canonical 50) Docs updated to the real numbers (08 README, examples index, root README snippet now runnable as-is, marimo-site policy-evals page); the hosted- rollouts roadmap item is done, with HF/Multibase mirroring as follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
classify_failure() ports the harness notebook's tuned heuristic (close cycles, held fraction, max lift; HOLD 4mm / AIR 2mm) and label_failures() wraps it over ATTEMPT_KEYS. On the committed LIBERO-Spatial rollouts the previously-unlabeled failures resolve to: openvla 15x re_grasp + 1x no_grasp, vla_jepa 1x re_grasp - 16 of 17 failures are slip-then-regrasp fumble loops averaging ~11 grasp attempts before the 250-step cap. Adds grasp_no_lift to TERMINAL_FAILURE_LABELS (values only; harness tuple sync noted). detect_regrasp stays the object-pose path; precedence is documented. examples/08_policy_evals/label_failures.py prints the taxonomy and writes labels only as an optional sidecar - the committed rollout parquet stays byte-faithful. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rness daft_physical_ai.ingest.Hdf5Ingestor normalizes LIBERO demonstration HDF5 (the one physical-AI format Daft has no native reader for) into canonical Episode/Step rows: integer demo sort, success from sparse rewards/dones, obs-alias fallbacks, xyzw->axis-angle, 8-dim proprio state, gripper qpos differencing. One delta vs the harness: for LIBERO suites task_name comes from the file stem minus `_demo` - verified equal to the task_name rollouts record, so demos and rollouts join on (suite, task_name); problem_name stays in metadata. h5py ships as the [hdf5] extra and in the dev group so CI exercises the synthetic-fixture tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tep rows examples/02_episode_data/normalize_libero_demos.py downloads the original robomimic-style HDF5 release (yifengzhu-hf/LIBERO-datasets, ~6.2 GB) and normalizes it through Hdf5Ingestor; the committed slice is all 500 demos (10 tasks x 50) as signals-only step rows - 62,250 rows, ~4.7 MB zstd, one parquet per task. task_name joins exactly against the rollout data. Ingest fix surfaced by the real files: LIBERO releases record problem_info.domain_name="robosuite" (the engine), so the suite is now resolved from the bddl path / filename and domain_name is only trusted when it names a known suite (latent in the harness adapter too - sync noted). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
noop_mask/trim_span (numpy) define a no-op as negligible commanded motion with an unchanged gripper command; motion_trim wraps them over ATTEMPT_KEYS and reports the kept window, prefix/suffix trim, and interior noop_fraction (what an RLDS-style no-noops filter would drop). Measured verdict on the committed LIBERO-Spatial originals: only ~0.2% of the 62,250 demo steps are strict no-ops (median motion norm 0.82) - this suite barely needed the famous cleaning. The stage-04 example frames the op as the audit you point at your own teleop data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… examples daft_physical_ai/curation.py holds the three ways a training regime consumes graded episodes, as lazy Daft combinators: sft_view (successes only, motion-trimmed, weighted - views not copies), preference_pairs (shared specs with opposite outcomes -> (chosen, rejected) manifest; 15 pairs on the real rollouts), and acquisition_map (failures ranked by concentration; task 5 re_grasp on init states 1,3,4,8 tops the real plan). Examples close the no-GPU loop: 06 writes sft_steps + manifest + preference pairs (gitignored, deterministic from committed data); 07 streams the curated view into torch - (64,7) actions / (64,8) states via to_torch_dataloader + one as_tensor stack (list columns arrive as equal-length lists, the documented collation quirk); 08 gains the acquisition-map table. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
examples/README.md index and stage READMEs reflect reality: 02/04/06/07 are runnable on committed data, 08 gains label_failures + acquisition_map, and the intro states what ships in-repo (500 demos + 200 rollouts). AGENTS.md architecture adds ingest/, operations/, and curation; the roadmap's next item is the gated fine-tune step. Root README and the marimo policy-evals page pick up the new commands and the curation bridge. Also fixes two checks masked by earlier silenced pre-commit runs: ruff nits in ingest/hdf5.py and mypy narrowing in the ingest tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…away Daft v0.7.17 releases daft.datasets.lerobot (#7090), the gate two roadmap items sat behind. The daft floor bumps to 0.7.17, every nightly-install instruction is deleted (README, AGENTS.md, marimo gallery docs, and the generated demo's Modal image template), and the two gated examples land: - 01_reading_data/lerobot_episode_index.py: episode/task/frame views of a LeRobot v3 dataset via read_episodes/read_tasks/read - filtered without decoding any video. - 02_episode_data/merge_lerobot_datasets.py: merge two recording sessions into one training table by re-indexing episode_index and the global frame index before concat (same tiny v3 dataset read twice as sessions A/B - no second public v3 dataset exists yet; the mechanics are identical). Both run first-try against pepijn223/egodex-test on the released reader. Full suite (85 tests), pre-commit, and the data-touching examples verified on 0.7.17. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nario queries daft_physical_ai/pose/ lands the clean patterns from the daft-examples EgoDex pipeline, keeping its seams: - state.py / skeleton.py: pure-NumPy per-frame geometry over the 48-D hand state (wrist + rot6d + fingertips per hand) and the 204-D 68-joint body skeleton - rot6d orthonormalization, palm normals via best-fit planes, finger flexion chains, arm extension, hand-local frames. - features.py: episode-level track assembly - run the geometry once over the whole (N, ...) episode and take plain forward differences for rates (no explode, no window functions). EpisodeFeatureComputer takes arrays (state-only or state+skeleton); the raw-HDF5 FrameBuilder stays in daft-examples, so the dataset seam is the arrays. - query.py: scenario predicates as (tracks, thresholds) -> mask callables (writing/hammer grips, grasping, lifting, reaching, in-hand, twisting, openness), percentile calibration (DataFrame or plain arrays), and segment stitching. Examples 03/04 run live on the public LeRobot v3 EgoDex sample via the released daft.datasets.lerobot reader: per-episode curl/pinch/palm tracks, and grasp/lift segments stitched into time ranges. 16 new tests cover the geometry with known-answer synthetic hands and skeletons. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…the prototype The site's design held; its content lagged the repo by the whole loop. Home now pitches the starter kit (quick-start commands with the real numbers as comments, the package inventory, Daft owning the rest) instead of describing the website architecture; Examples mirrors the 01-08 collection with measured results on every card (84% vs 99%, 16/17 re-grasp loops, 0.2% no-ops, (64,7)/(64,8) torch batches) and gains pose + curation cards; Demos maps its topics to the numbered stages. Also fixes three lint/type nits pre-commit caught in the fresh pose port. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
everettVT
changed the title
…ow expressions daft-examples main moved every temporal rate in-plan (lead(1) over per-episode windows, euclidean_distance speeds, rows_between smoothing); this ports that as pose/temporal.py, the distributed twin of the NumPy TemporalFeatureComputer: - add_temporal_features(df, episode_keys=..., order_by=...) works on any per-frame table (LeRobot frames, exploded trajectories, canonical step rows) and adds only the rates whose spatial inputs are present - state_frame_features: the per-frame @daft.func twin of compute_raw_features, so 48-D state tables get the spatial columns without leaving the plan - forearm_roll: the one custom UDF, fed frame pairs by lead(1) An equivalence test pins the window rates to the NumPy computer per episode (and proves no cross-episode bleed); the new stage-03 example runs the whole thing lazily on the public EgoDex LeRobot sample - reader -> geometry UDF -> window rates -> scenario predicates as column expressions -> one collect - and its counts cross-check the array-path example exactly. Exercises the last untouched relational surfaces (Window, lead, euclidean_distance) and keeps geometry semantics identical to upstream (state/skeleton cores confirmed drift-free, docstrings aside). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The decorator returns an Expression at plan-build time while the body is annotated with its element type - route the test call sites through a cast helper, use lit() for the threshold comparison, and apply the pinned ruff format (the local venv had drifted onto WIP lock pins; a stale .mypy_cache masked the errors locally). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Splitting this into a stack of small PRs per review feedback - phase 1 (reading-data examples + taxonomy) is up. This PR stays open as the integration preview / conflict-free reference; it'll close when the stack lands. |
everettVT
marked this pull request as draft
July 11, 2026 00:03
The previous 01-08 scaffold conflicted with main and mixed episodes, evals, pose, ingest, and a demo site into one review. This PR now adds three copyable dataset scripts on current main and nothing else. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
everettVT
changed the title
everettVT
marked this pull request as ready for review
August 31, 2026 21:52
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37944efa11
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+36
to
+39
| merged = frames_a.concat( | ||
| frames_b.with_column("episode_index", col("episode_index") + lit(episode_offset)).with_column( | ||
| "index", col("index") + lit(frame_offset) | ||
| ) |
There was a problem hiding this comment.
When independently recorded sessions assign the same local task_index to different task strings—commonly both start at zero—this concatenation leaves those indices unchanged while making the other identifiers global. The merged table then maps one task index to multiple tasks, so downstream grouping, training, or export through the LeRobot task table can silently conflate supervision; retaining the strings does not make the conflicting integer column safe. Build a unified task mapping and rewrite task_index, or drop that column.
Useful? React with 👍 / 👎.
Comment on lines
+37
to
+39
| frames_b.with_column("episode_index", col("episode_index") + lit(episode_offset)).with_column( | ||
| "index", col("index") + lit(frame_offset) | ||
| ) |
There was a problem hiding this comment.
For session B, lerobot.read() broadcasts episode metadata such as dataset_from_index and dataset_to_index, but this chain increments only the row-level index. Those rows consequently no longer satisfy the canonical relationship index == dataset_from_index + frame_index used by examples/trim/demo.py:41, so consumers relying on the ranges can discard or seek the wrong frames from session B. Offset both range columns by frame_offset, or remove and recompute them during the merge.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Daft already reads robot data. This PR adds three copyable scripts for the patterns that were missing:
examples/lerobot_episode_index.py— episode / task / frame views of a LeRobot v3 dataset, filtered without decoding videoexamples/merge_lerobot_datasets.py— re-indexepisode_indexand the global frameindexbefore concatenating two recording sessionsexamples/droid_episode_index.py— lazy DROID episode index (wrist_cam_videocolumns, plan only)No new package surface. The previous 01–08 scaffold (episodes, evals, pose, ingest, parquet data, marimo site) is out of this PR;
mainalready has hands / rewards / trim / EgoDex.Test plan
uv run python examples/lerobot_episode_index.py(publicpepijn223/egodex-test)uv run python examples/merge_lerobot_datasets.py(same dataset twice as sessions A/B; unique globalindex)uv run python examples/droid_episode_index.py(prints a lazy plan, does not download)examples/README.mdlists the scripts next to the existing CLI demos