Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,3 +75,13 @@ conversion.

- Titles: Conventional Commits format; enforced by `.github/workflows/pr-labeller.yml`.
- Descriptions: follow `.github/pull_request_template.md`.

# Versioning & Publishing

Versions are derived from git tags via `hatch-vcs`. Tag releases as `v0.1.0`,
`v0.2.0`, etc.

Publishing a GitHub release triggers `.github/workflows/publish-package.yml`,
which builds a wheel and sdist with `uv build` and uploads both to PyPI via
[trusted publishing](https://docs.pypi.org/trusted-publishers/). Configure the
trusted publisher on PyPI for this repository before the first release.
142 changes: 81 additions & 61 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
# daft-physical-ai

Physical-AI data processing on [Daft](https://github.com/Eventual-Inc/Daft):
hand tracking and reward scoring. The methods run as Daft UDFs, so they slot
into any Daft pipeline and execute lazily, batched, and distributed.
hand tracking, reward scoring, and motion trimming. The methods run as Daft
UDFs and expressions, so they slot into any Daft pipeline and execute lazily,
batched, and distributed.

Available on [PyPI](https://pypi.org/project/daft-physical-ai/):

```bash
pip install "daft-physical-ai[mediapipe]"
pip install daft-physical-ai
```

## API
## Hand tracking

The package operates on a Daft image column and returns a hand-pose column. A
`track_hands` takes a Daft image column and returns a hand-pose column. A
LeRobot dataset is a natural source: Daft's native reader
`daft.datasets.lerobot` (added in [Daft #7090](https://github.com/Eventual-Inc/Daft/pull/7090))
decodes each camera into an image column with `load_video_frames`.
Expand All @@ -34,20 +35,6 @@ df = df.with_column("hands", track_hands(df["observation.image"], method="mediap
df.write_parquet("annotated/")
```

## Raw EgoDex releases

For Apple's original EgoDex HDF5+MP4 release, use the extension's lazy reader:

```python
from daft_physical_ai.datasets import egodex

episodes = egodex.raw("/data/egodex", tasks="fold_towel").limit(2)
poses = egodex.trajectory(episodes, fields=["transforms/leftHand", "transforms/rightHand"])
frames = egodex.camera_frames(poses, width=224, height=224, sample_interval_seconds=1.0)
```

EgoDex is CC-BY-NC-ND, so the package does not download, extract, or redistribute it. Download and extract the archives from the [official EgoDex repository](https://github.com/apple/ml-egodex), then point `raw()` at your copy. See [the runnable example](examples/egodex_raw_hdf5_video.py).

Install the method you need as an extra: `pip install "daft-physical-ai[mediapipe]"`
(CPU, 2D), `pip install "daft-physical-ai[wilor]"` (GPU, 3D), or
`pip install "daft-physical-ai[all]"` for both. WiLoR additionally needs a CUDA
Expand All @@ -56,7 +43,7 @@ Install the method you need as an extra: `pip install "daft-physical-ai[mediapip
extra because PyPI metadata can't carry direct references), plus a user-supplied
`MANO_RIGHT.pkl` ([research-gated](docs/mano.md)).

## Output schema
### Output schema

One unified output schema regardless of method: each frame yields a list of
0-2 detected hands. A single hand value (MediaPipe):
Expand All @@ -76,6 +63,46 @@ The Daft type is `list[struct{ handedness: string, confidence: float32, kp2d:
list[list[float32]], kp3d: list[list[float32]] }]`, defined as `HANDS_DTYPE` in
`daft_physical_ai/hands/schema.py`.

### Raw EgoDex releases

For Apple's original EgoDex HDF5+MP4 release, use the extension's lazy reader:

```python
from daft_physical_ai.datasets import egodex

episodes = egodex.raw("/data/egodex", tasks="fold_towel").limit(2)
poses = egodex.trajectory(episodes, fields=["transforms/leftHand", "transforms/rightHand"])
frames = egodex.camera_frames(poses, width=224, height=224, sample_interval_seconds=1.0)
```

EgoDex is CC-BY-NC-ND, so the package does not download, extract, or redistribute it. Download and extract the archives from the [official EgoDex repository](https://github.com/apple/ml-egodex), then point `raw()` at your copy. See [the runnable example](examples/egodex_raw_hdf5_video.py).

### Example

A complete walkthrough - read a dataset, run `track_hands` (MediaPipe), draw the
keypoints, and score against EgoDex ground truth:

![track_hands keypoints](examples/hands/demo_keypoints.png)

Available in three equivalent forms:

- **[examples/hands/demo.md](examples/hands/demo.md)** - read it start to finish; code and outputs inline.
- **[examples/hands/demo.ipynb](examples/hands/demo.ipynb)** - runnable notebook (outputs included).
- **[examples/hands/demo.py](examples/hands/demo.py)** - plain script.

Generate your own (other methods, a Modal GPU runtime, with/without eval) with the
`daft-physical-ai hands` command - run it with no flags for an interactive
walkthrough, or pass flags:

```bash
# No flags - interactive walkthrough that asks a few questions
uvx daft-physical-ai hands

# --no-input skips all prompts; flags supply the answers, the rest use defaults
uvx daft-physical-ai hands --method mediapipe --output-dir my-demo --no-input
uvx daft-physical-ai hands --method wilor --runtime modal --mano-path ./MANO_RIGHT.pkl --no-input
```

## Reward scoring

Score episodes with a reward model
Expand Down Expand Up @@ -114,6 +141,20 @@ struct {
}
```

### Example

[examples/rewards/](examples/rewards/) is the executed walkthrough - read
LIBERO episode metadata, score each episode with `score_rewards`, plot the
progress curves, and filter low-progress episodes with a Daft query. The
Robometer server scripts it talks to are committed next to it.

Generate your own (different dataset, episode count, frame budget):

```bash
daft-physical-ai rewards # interactive
daft-physical-ai rewards --episodes 10 --max-frames 8 --no-input
```

## Motion trimming

Find the dead frames in an episode - the operator setting up before anything
Expand Down Expand Up @@ -161,45 +202,36 @@ returns a DataFrame: a window is an aggregation across an episode's rows, not a
value each row can carry. `motion_energy` and `is_active` are ordinary
expressions.

## Example

A complete walkthrough - read a dataset, run `track_hands` (MediaPipe), draw the
keypoints, and score against EgoDex ground truth:

![track_hands keypoints](examples/hands/demo_keypoints.png)

Available in three equivalent forms:
### Example

- **[examples/hands/demo.md](examples/hands/demo.md)** - read it start to finish; code and outputs inline.
- **[examples/hands/demo.ipynb](examples/hands/demo.ipynb)** - runnable notebook (outputs included).
- **[examples/hands/demo.py](examples/hands/demo.py)** - plain script.
[examples/trim/](examples/trim/) is the executed walkthrough - one DROID shard
streamed from Hugging Face, scored per frame, reduced to trim windows, and
plotted. No GPU, no server.

Generate your own (other methods, a Modal GPU runtime, with/without eval) with the
`daft-physical-ai hands` command - run it with no flags for an interactive
walkthrough, or pass flags:
Generate your own (different dataset, state column, shard count):

```bash
# No flags - interactive walkthrough that asks a few questions
uvx daft-physical-ai hands

# --no-input skips all prompts; flags supply the answers, the rest use defaults
uvx daft-physical-ai hands --method mediapipe --output-dir my-demo --no-input
uvx daft-physical-ai hands --method wilor --runtime modal --mano-path ./MANO_RIGHT.pkl --no-input
daft-physical-ai trim # interactive
daft-physical-ai trim --dataset my/dataset --dims 6 --no-input
```

`uvx` runs the CLI without installing anything (scaffolding needs no inference
deps). If the [PyPI package](https://pypi.org/project/daft-physical-ai/) is
already installed (`pip install daft-physical-ai`), plain `daft-physical-ai hands`
works too; from a clone of this repo, `uv sync` installs it (`uv run daft-physical-ai`).
## The CLI

Each capability is its own subcommand - `daft-physical-ai hands` and
`daft-physical-ai rewards` so far (`daft-physical-ai` with no arguments lists
what's available). The `rewards` scaffold also writes the Robometer server
Each capability is its own subcommand - `daft-physical-ai hands`,
`daft-physical-ai rewards`, and `daft-physical-ai trim` (`daft-physical-ai`
with no arguments lists what's available). Each scaffolds a personalized,
runnable demo; the `rewards` scaffold also writes the Robometer server
scripts next to the demo, so one directory holds everything: score the
episodes, and serve the model locally or on Modal.

To *run* a generated demo you also need its inference stack. `uvx` covers that
too - one line, nothing installed:
`uvx daft-physical-ai <subcommand>` runs the CLI without installing anything
(scaffolding needs no inference deps). If the
[PyPI package](https://pypi.org/project/daft-physical-ai/) is already installed
(`pip install daft-physical-ai`), the plain command works too; from a clone of
this repo, `uv sync` installs it (`uv run daft-physical-ai`).

To *run* a generated demo you also need its runtime deps (inference libraries,
plotting). `uvx` covers that too - one line, nothing installed:

```bash
uvx --from jupyterlab --with "daft-physical-ai[mediapipe]" --with matplotlib --with scipy \
Expand All @@ -225,15 +257,3 @@ uv sync # set up env + install deps
uv run pre-commit install # install lint/format hooks
uv run pytest tests/ -v # run the test suite
```

## Versioning

Versions are derived from git tags via `hatch-vcs`. Tag releases as `v0.1.0`,
`v0.2.0`, etc.

## Publishing

Publishing a GitHub release triggers `.github/workflows/publish-package.yml`,
which builds a wheel and sdist with `uv build` and uploads both to PyPI via
[trusted publishing](https://docs.pypi.org/trusted-publishers/). Configure the
trusted publisher on PyPI for this repository before the first release.
7 changes: 6 additions & 1 deletion daft_physical_ai/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Physical-AI dataset access, hand tracking, and reward scoring for Daft DataFrames.
"""Physical-AI dataset access, hand tracking, reward scoring, and motion trimming for Daft DataFrames.

`track_hands(images, method=...)` takes an image column (a Daft expression) and
returns a hand-pose column, so it composes with any Daft pipeline. Every method
Expand All @@ -7,6 +7,11 @@
`score_rewards(...)` takes episode-metadata columns and returns a reward column
(per-frame task progress + success probability) scored against a Robometer eval
server you run - see `REWARD_DTYPE`.

`motion_energy(...)` / `is_active(...)` (in `daft_physical_ai.proprio`) score
per-frame motion from the robot's own state columns, and `trim_windows(...)`
(in `daft_physical_ai.trim`) reduces the flags to one trim window per episode -
see `TRIM_FIELDS`. Nothing decodes video.
"""

from __future__ import annotations
Expand Down
Loading