Replies: 1 comment 4 replies
|
Thank you @Sagar-024. You did the thing I asked for and then some: you took a negative result seriously instead of quietly switching to synthetic data and reporting only the timings. Yes to the LeRobot provider. Go for it. One steer on shape: build it as a general tool, not an hflow subsystem. LeRobot to canonical MCAP is useful to people who will never run this project, and we do not want to constrain it to our system or take on a dataset-format matrix inside core. Concretely, that means depending on the format rather than on our internals: A few things that follow from that:
Design questions that need a decision before you go far: how you map LeRobot's per-frame Parquet rows onto MCAP channels and timestamps, and what you do when the MP4 timestamps and the Parquet timestamps disagree. Post your intended mapping here before you write much of it and we will settle it in this thread, cheaply. Yes to the stress scripts as an examples entry, too. "The next stress test is one command" is worth real time, and your 200-episode numbers become reproducible instead of anecdotal. Send it separately from the converter so neither waits on the other. On the rest of the report: Awkward 1 is a real bug and I filed it as #86. I reproduced it. One correction worth making, because it changes the fix: the failure is not entirely silent. The Awkward 2, no local batch path: agreed, and it is the more interesting half. That you had to write a script to run 200 episodes locally is a real gap, and it is also why awkward 1 stayed invisible. Your stress-scripts contribution partly answers it; whether Awkward 3, WSL2 friction: partly fixed already. #75 added a platform section to On the 9 s per episode: that matches where I would expect the cost to sit, and One process note: if something you post here goes quiet, ping it. |
|
Thanks Kingston, and no apology needed, the reply was worth the wait. Got it on shape. I'll build it standalone: a plain callable plus a CLI under Before writing much of it I'll post the Parquet to MCAP mapping here so we can Stress scripts will come as a separate PR so neither blocks the other. And yes, Mapping post coming in a day or two. |
|
LeRobot to MCAP Mapping Plan (pusht, single camera) Target: Convert LeRobot pusht (Parquet + MP4, v3.0) to canonical MCAP per docs/FORMAT.md.
observation.image (MP4) becomes /observation.image with foxglove.CompressedVideo schema, protobuf encoding, cameras group. observation.state (Parquet) becomes /observation.state with custom float32[2] schema, CDR encoding, state group. action (Parquet) becomes /action with custom float32[2] schema, CDR encoding, state group. timestamp column is not a separate channel. It becomes log_time for all channels. observation.state and action are fixed_size_list[2] per the actual parquet schema. LeRobot doesn't define ROS message types, so I'll register custom schemas matching those feature definitions.
Both sources are frame by frame aligned at 10 Hz and start at 0.0 per episode. Parquet timestamp is float32 seconds from episode start. Sample values: 0.0, 0.10000000149, 0.20000000298, 0.30000001192, 0.40000000596. The trailing noise is float32 rounding. The underlying value is exact at 10 Hz. MP4 PTS is 1/10240 sec per tick (10240 = 1024 x 10 fps), pts_time in seconds matches the parquet column. Decision: Use parquet timestamp as canonical log_time_ns. Convert with log_time_ns = int(timestamp * 1e9). Lossless at 10 Hz since 0.1s = 100,000,000 ns is exactly representable.
Threshold: |parquet_timestamp - mp4_pts_time| > 50 ms for any frame means fail loud with ValueError naming the frame index and both timestamps. Rationale: at 10 Hz, frame interval = 100 ms. 50 ms = half frame. Anything larger is desync (dropped frames, variable rate, clock skew) and the converter shouldn't paper over it silently. I'll slice the per episode MP4 using ffmpeg -ss/-to on the canonical timestamps, guided by the videos/observation.image/from_timestamp and to_timestamp columns in the episodes parquet (both confirmed present as double type).
episode/v1: provenance/v1: pusht has no attachments. If a later dataset (e.g. aloha) has calibration or URDF, copy them through.
# examples/lerobot/prepare.py
def lerobot_to_mcap(
dataset_repo: str = "lerobot/pusht",
revision: str = "main",
output_dir: Path,
episode_index: int | None = None,
camera_key: str = "observation.image",
) -> list[Path]:
# 1. hf_hub_download parquet + MP4 for requested episodes
# 2. Load episodes parquet for per episode video timestamp ranges
# 3. Per episode:
# a. Slice MP4 via ffmpeg -ss/-to
# b. Filter data parquet by episode_index
# c. Write MCAP with CanonicalMcapWriter:
# - register_schema (foxglove.CompressedVideo + custom float32[2])
# - register_channel (group="cameras" / "state")
# - write_message: video (H.264 Annex B + AUD + SPS/PPS on keyframes)
# - write_message: state/action (CDR encoded float32[2])
# - add_metadata episode/v1 + provenance/v1
# - source_revision stamped into episode/v1
# 4. Return list of output MCAP pathsCLI: uv run python examples/lerobot/prepare.py \
--repo lerobot/pusht \
--revision main \
--output-dir ./data/lerobot_pusht \
--camera-key observation.image \
--episode-index 0Open questions for @kstonekuan
Acceptable, or is there a preferred preset for training friendly GOP structure?
Ready to implement once the design is settled. |
|
Good mapping. Channel split, parquet-as-canonical, and the 50 ms fail-loud rule are all right, and half a frame interval is the correct way to pick that threshold. Four answers, two corrections. 1. Schema: custom, not a JointState shape. Don't fake a ROS type LeRobot doesn't have. 2. Encoder: your flag set is missing the two that matter.
3. Multi-camera deferred: agreed. 4. Timestamps: the conversion isn't lossless. log_time_ns = round(index * 1_000_000_000 / fps)Exact, monotonic, and regular, so Correction 1: drop Correction 2: your transcode is load-bearing, which is why point 2 is not a style note. A channel already carrying Pipeline shape is right. Go build it. |
|
Got all six. Building now with:
Stress PR is already up (#92). Converter PR coming when pusht episode 0 |
Uh oh!
There was an error while loading. Please reload this page.
Context
In the PR #41 review, @kstonekuan asked me to run hflow against a real corpus like Egocentric-10K and report what breaks, what is slow, and what is awkward. This is that report. I want to be upfront about the path I took, because the most important finding happened before any timing was measured.
Machine: Windows 11 laptop, 16 GB RAM, no GPU, everything run inside WSL2 Ubuntu.
What breaks: the real corpora cannot get in
I started with the dataset named in the review, builddotai/Egocentric-10K, and with LeRobot as the most common public robot corpus.
hflow only reads canonical MCAP, and src/hflow/providers.py defines a single provider, NativeVideoProvider, which is for VLM servers such as vLLM. There is no ingestion provider for WebDataset or for Parquet plus MP4. So neither corpus can enter the pipeline at all. A team holding either dataset has to write an external converter before hflow sees a single episode.
This is the lead finding. The answer to "what breaks" on real data is the door. Everything downstream of ingestion worked once data was in MCAP.
What is slow: 200 episode timing run
Since real data could not enter, I generated a synthetic corpus to measure the pipeline itself: 200 canonical episodes, 175 MB, durations 2 to 10 s, 1 to 3 cameras at 10/15/30 Hz, 20% with black segments, 20% with joint jumps. Ingestion ran sequentially, one episode at a time.
Total about 34 minutes.
The ingestion bottleneck is camera_frame_stats, the ffmpeg decode plus signalstats pass per camera. At 9 s per episode, a 10K hour corpus is out of reach on a laptop and would need much more parallel decode on a server. DuckDB curation is fast and is not the bottleneck.
What is awkward
action_rate records nothing when registered the obvious way. The check requires the topics kwarg, so app.check()(action_rate) never produces a measurement. Ingestion still finishes and reports success, and I confirmed the ingest log contains zero mentions of action_rate. No error, no warning. The column is simply absent from the catalog. The fix is to wrap the check in a lambda that supplies topics. Nothing in the check registry or the docs flags this. I only noticed because my z-score query came back empty.
There is no local batch ingest path. hflow ingest only works through the Airflow runtime. For a local stress run I had to write a small script calling app.process() per file, which is not obvious from hflow --help. A local mode or an examples script would save the next person an hour.
WSL2 setup friction for anyone on Windows: apt asks for an interactive sudo password, pip3 install uv fails with PEP 668 so you need the official installer, and uv sync falls back to full file copies when the repo lives on /mnt/c because it crosses a filesystem boundary. Cloning inside the WSL home directory fixes the last one.
What I would do next
The highest leverage fix is a LeRobot ingestion provider, Parquet plus MP4 to canonical MCAP. It unlocks hundreds of public datasets and makes the real data stress test possible end to end. I would be glad to take that on.
If useful, I can also contribute my two stress scripts as an examples entry so the next stress test is one command.
All reactions