Skip to content

Inspect Robots scenes: references list with labels in data #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aris-zhu merged 1 commit into from
Aug 5, 2026
Merged

Inspect Robots scenes: references list with labels in data #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions scenes/inspect-robots/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,13 @@ Robots phone apps (iOS/Android): the app overlays a grid on the physical
bench, the operator taps the two arm-base positions, and the scene's objects
appear as labeled markers at their sampled poses.

The app's format stores anchors as an ordered `references` list of labeled
points; for these scenes the two anchors are the rig's arm bases, so they
carry the labels "LEFT arm" and "RIGHT arm" in the JSON itself. The apps
hardcode no arm wording; the placement prompts ("Tap the LEFT arm base")
come from these labels. Scenes in the older single `reference` object shape
still load, prompting by numbered position instead.

Each file is the task's **first instance** projected through the
`yam-bimanual` rig: epoch 0 is the default realization and epochs 1–4 are the
`variants` the app's **Resample** button draws from — so all five realizations
Expand All @@ -31,8 +38,8 @@ for id in place_cutlery/spoon-on-plate stack/cups place_in_rack/plate \
+ " · KitchenBench";
{format:"inspect-robots-setup", version:1, name:(.[0].instance_id|friendly),
instruction:.[0].instruction,
reference:{left_xy_cm:$rig[0].arms.left.base_xy_cm,
right_xy_cm:$rig[0].arms.right.base_xy_cm},
references:[{label:"LEFT arm", xy_cm:$rig[0].arms.left.base_xy_cm},
{label:"RIGHT arm", xy_cm:$rig[0].arms.right.base_xy_cm}],
objects:(.[0]|objs),
variants:[.[1:][] | {instruction:.instruction, objects:objs}]}' \
> "scenes/inspect-robots/${id%%/*}.json"
Expand Down
28 changes: 17 additions & 11 deletions scenes/inspect-robots/fold_cloth.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
{
"format": "inspect-robots-setup",
"version": 1,
"name": "Fold cloth: dish towel · KitchenBench",
"name": "Fold cloth: dish towel \u00b7 KitchenBench",
"instruction": "fold the cloth",
"reference": {
"left_xy_cm": [
-30.0,
-20.0
],
"right_xy_cm": [
30.0,
-20.0
]
},
"references": [
{
"label": "LEFT arm",
"xy_cm": [
-30.0,
-20.0
]
},
{
"label": "RIGHT arm",
"xy_cm": [
30.0,
-20.0
]
}
],
"objects": [
{
"label": "cloth",
Expand Down
28 changes: 17 additions & 11 deletions scenes/inspect-robots/handoff.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
{
"format": "inspect-robots-setup",
"version": 1,
"name": "Handoff: utensil · KitchenBench",
"name": "Handoff: utensil \u00b7 KitchenBench",
"instruction": "hand off the produce item from one arm to the other",
"reference": {
"left_xy_cm": [
-30.0,
-20.0
],
"right_xy_cm": [
30.0,
-20.0
]
},
"references": [
{
"label": "LEFT arm",
"xy_cm": [
-30.0,
-20.0
]
},
{
"label": "RIGHT arm",
"xy_cm": [
30.0,
-20.0
]
}
],
"objects": [
{
"label": "produce_item",
Expand Down
28 changes: 17 additions & 11 deletions scenes/inspect-robots/open_container.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
{
"format": "inspect-robots-setup",
"version": 1,
"name": "Open container: jar · KitchenBench",
"name": "Open container: jar \u00b7 KitchenBench",
"instruction": "open the food container",
"reference": {
"left_xy_cm": [
-30.0,
-20.0
],
"right_xy_cm": [
30.0,
-20.0
]
},
"references": [
{
"label": "LEFT arm",
"xy_cm": [
-30.0,
-20.0
]
},
{
"label": "RIGHT arm",
"xy_cm": [
30.0,
-20.0
]
}
],
"objects": [
{
"label": "food_container",
Expand Down
28 changes: 17 additions & 11 deletions scenes/inspect-robots/place_cutlery.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
{
"format": "inspect-robots-setup",
"version": 1,
"name": "Place cutlery: spoon on plate · KitchenBench",
"name": "Place cutlery: spoon on plate \u00b7 KitchenBench",
"instruction": "place the fork on the napkin",
"reference": {
"left_xy_cm": [
-30.0,
-20.0
],
"right_xy_cm": [
30.0,
-20.0
]
},
"references": [
{
"label": "LEFT arm",
"xy_cm": [
-30.0,
-20.0
]
},
{
"label": "RIGHT arm",
"xy_cm": [
30.0,
-20.0
]
}
],
"objects": [
{
"label": "fork",
Expand Down
28 changes: 17 additions & 11 deletions scenes/inspect-robots/place_in_rack.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
{
"format": "inspect-robots-setup",
"version": 1,
"name": "Place in rack: plate · KitchenBench",
"name": "Place in rack: plate \u00b7 KitchenBench",
"instruction": "place the plate into the dish rack",
"reference": {
"left_xy_cm": [
-30.0,
-20.0
],
"right_xy_cm": [
30.0,
-20.0
]
},
"references": [
{
"label": "LEFT arm",
"xy_cm": [
-30.0,
-20.0
]
},
{
"label": "RIGHT arm",
"xy_cm": [
30.0,
-20.0
]
}
],
"objects": [
{
"label": "plate",
Expand Down
28 changes: 17 additions & 11 deletions scenes/inspect-robots/pour_pasta.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
{
"format": "inspect-robots-setup",
"version": 1,
"name": "Pour pasta: measuring cup to bowl · KitchenBench",
"name": "Pour pasta: measuring cup to bowl \u00b7 KitchenBench",
"instruction": "pour the dry pasta into the bowl",
"reference": {
"left_xy_cm": [
-30.0,
-20.0
],
"right_xy_cm": [
30.0,
-20.0
]
},
"references": [
{
"label": "LEFT arm",
"xy_cm": [
-30.0,
-20.0
]
},
{
"label": "RIGHT arm",
"xy_cm": [
30.0,
-20.0
]
}
],
"objects": [
{
"label": "bowl",
Expand Down
28 changes: 17 additions & 11 deletions scenes/inspect-robots/scoop_pasta.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
{
"format": "inspect-robots-setup",
"version": 1,
"name": "Scoop pasta: spoon penne · KitchenBench",
"name": "Scoop pasta: spoon penne \u00b7 KitchenBench",
"instruction": "scoop about 118 g of the rigatoni with the spoon and transfer it to the container",
"reference": {
"left_xy_cm": [
-30.0,
-20.0
],
"right_xy_cm": [
30.0,
-20.0
]
},
"references": [
{
"label": "LEFT arm",
"xy_cm": [
-30.0,
-20.0
]
},
{
"label": "RIGHT arm",
"xy_cm": [
30.0,
-20.0
]
}
],
"objects": [
{
"label": "container",
Expand Down
28 changes: 17 additions & 11 deletions scenes/inspect-robots/seal_container.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
{
"format": "inspect-robots-setup",
"version": 1,
"name": "Seal container: food container · KitchenBench",
"name": "Seal container: food container \u00b7 KitchenBench",
"instruction": "seal the pot with its lid",
"reference": {
"left_xy_cm": [
-30.0,
-20.0
],
"right_xy_cm": [
30.0,
-20.0
]
},
"references": [
{
"label": "LEFT arm",
"xy_cm": [
-30.0,
-20.0
]
},
{
"label": "RIGHT arm",
"xy_cm": [
30.0,
-20.0
]
}
],
"objects": [
{
"label": "pot",
Expand Down
28 changes: 17 additions & 11 deletions scenes/inspect-robots/sort_cutlery.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
{
"format": "inspect-robots-setup",
"version": 1,
"name": "Sort cutlery: balanced pile · KitchenBench",
"name": "Sort cutlery: balanced pile \u00b7 KitchenBench",
"instruction": "sort the cutlery into the correct tray compartments",
"reference": {
"left_xy_cm": [
-30.0,
-20.0
],
"right_xy_cm": [
30.0,
-20.0
]
},
"references": [
{
"label": "LEFT arm",
"xy_cm": [
-30.0,
-20.0
]
},
{
"label": "RIGHT arm",
"xy_cm": [
30.0,
-20.0
]
}
],
"objects": [
{
"label": "tray",
Expand Down
28 changes: 17 additions & 11 deletions scenes/inspect-robots/stack.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
{
"format": "inspect-robots-setup",
"version": 1,
"name": "Stack: cups · KitchenBench",
"name": "Stack: cups \u00b7 KitchenBench",
"instruction": "stack the cups",
"reference": {
"left_xy_cm": [
-30.0,
-20.0
],
"right_xy_cm": [
30.0,
-20.0
]
},
"references": [
{
"label": "LEFT arm",
"xy_cm": [
-30.0,
-20.0
]
},
{
"label": "RIGHT arm",
"xy_cm": [
30.0,
-20.0
]
}
],
"objects": [
{
"label": "cup",
Expand Down
Loading