Skip to content

README.md

Latest commit

 

History

History
176 lines (139 loc) · 6.49 KB

README.md

File metadata and controls

176 lines (139 loc) · 6.49 KB
Raw

SimplerEnv

Framework for evaluating real-world robot manipulation policies (RT-1, RT-1-X, Octo) in simulation. Replicates common setups like Google Robot and WidowX+Bridge, with GPU-accelerated simulations (10-15x speedup). Offers visual matching and variant aggregation evaluation methods for robust policy assessment.

For more information, see the official repository.

Benchmark results

These values come from the default SimplerEnv evaluation runs (the per-task success rates are listed below).

Bridge (WidowX robot)

Provided checkpoints:

Task N1.6 success rate N1.7 success rate
widowx_spoon_on_towel 56/101 (55.4%) 78/100 (78.0%)
widowx_carrot_on_plate 46/100 (46.0%) 58/100 (58.0%)
widowx_put_eggplant_in_basket 89/100 (89.0%) 53/100 (53.0%)
widowx_stack_cube 5/100 (5.0%) 48/100 (48.0%)
widowx_put_eggplant_in_sink 33/100 (33.0%) 2/100 (2.0%)
widowx_close_drawer 73/100 (73.0%) 97/100 (97.0%)
widowx_open_drawer 95/100 (95.0%) 100/100 (100.0%)
Average 56.6% 62.3%

Fractal (Google Robot)

Provided checkpoints:

Task N1.6 success rate N1.7 success rate
google_robot_pick_coke_can 95/100 (95.0%) 100/100 (100.0%)
google_robot_pick_object 87/100 (87.0%) 94/100 (94.0%)
google_robot_move_near 81/100 (81.0%) 100/100 (100.0%)
google_robot_open_drawer 0/100 (0.0%) 65/100 (65.0%)
google_robot_close_drawer 44/100 (44.0%) 69/100 (69.0%)
google_robot_place_in_closed_drawer 5/100 (5.0%) 7/100 (7.0%)
Average 52.0% 72.5%

Fine-tune Simpler Env bridge dataset (WidowX robot)

To reproduce our finetune results, use the following commands to setup dataset and launch finetune experiments. Please remember to set WANDB_API_KEY since W&B logging is on by default (USE_WANDB=1 in examples/finetune.sh). If you don't have a WANDB account, prepend USE_WANDB=0 to the launch command to disable it:

uv run hf download \
    --repo-type dataset IPEC-COMMUNITY/bridge_orig_lerobot \
    --local-dir examples/SimplerEnv/bridge_orig_lerobot/

# Copy the patches and run the finetune script
cp examples/SimplerEnv/bridge_modality.json examples/SimplerEnv/bridge_orig_lerobot/meta/modality.json
NUM_GPUS=8 MAX_STEPS=20000 GLOBAL_BATCH_SIZE=1024 SAVE_STEPS=1000 uv run bash examples/finetune.sh \
    --base-model-path nvidia/GR00T-N1.7-3B \
    --dataset-path examples/SimplerEnv/bridge_orig_lerobot/ \
    --embodiment-tag SIMPLER_ENV_WIDOWX \
    --output-dir /tmp/bridge_finetune \
    --state-dropout-prob 0.8

Fine-tune Simpler Env fractal dataset (Google robot)

uv run hf download \
    --repo-type dataset IPEC-COMMUNITY/fractal20220817_data_lerobot \
    --local-dir examples/SimplerEnv/fractal20220817_data_lerobot/

# Copy the patches and run the finetune script
cp -r examples/SimplerEnv/fractal_modality.json examples/SimplerEnv/fractal20220817_data_lerobot/meta/modality.json
uv run python examples/SimplerEnv/convert_av1_to_h264.py --root examples/SimplerEnv/fractal20220817_data_lerobot --jobs 16
NUM_GPUS=8 MAX_STEPS=20000 GLOBAL_BATCH_SIZE=1024 SAVE_STEPS=1000 uv run bash examples/finetune.sh \
    --base-model-path nvidia/GR00T-N1.7-3B \
    --dataset-path examples/SimplerEnv/fractal20220817_data_lerobot/ \
    --embodiment-tag SIMPLER_ENV_GOOGLE \
    --output-dir /tmp/fractal_finetune \
    --state-dropout-prob 0.5

Evaluate checkpoint

First, complete the one-time simulation environment setup, then run this benchmark's setup script (only needed once per benchmark):

bash gr00t/eval/sim/SimplerEnv/setup_SimplerEnv.sh

Then, run client server evaluation under the project root directory in separate terminals:

Fractal (Google Robot) Evaluation

Terminal 1 - Server:

You can use either a local finetuned checkpoint path or the remote finetuned checkpoint (provided by us):

Option 1: Local finetuned checkpoint

uv run python gr00t/eval/run_gr00t_server.py \
    --model-path /tmp/fractal_finetune/checkpoint-30000 \
    --embodiment-tag SIMPLER_ENV_GOOGLE \
    --use-sim-policy-wrapper

Option 2: Remote finetuned checkpoint (directly runnable)

uv run python gr00t/eval/run_gr00t_server.py \
    --model-path nvidia/GR00T-N1.7-SimplerEnv-Fractal \
    --embodiment-tag SIMPLER_ENV_GOOGLE \
    --use-sim-policy-wrapper

Terminal 2 - Client:

gr00t/eval/sim/SimplerEnv/simpler_uv/.venv/bin/python gr00t/eval/rollout_policy.py \
    --n-episodes 10 \
    --policy-client-host 127.0.0.1 \
    --policy-client-port 5555 \
    --max-episode-steps 300 \
    --env-name simpler_env_google/google_robot_pick_coke_can \
    --n-action-steps 1 \
    --n-envs 5

Bridge (WidowX) Evaluation

Terminal 1 - Server:

Option 1: Local finetuned checkpoint

uv run python gr00t/eval/run_gr00t_server.py \
    --model-path /tmp/bridge_finetune/checkpoint-30000 \
    --embodiment-tag SIMPLER_ENV_WIDOWX \
    --use-sim-policy-wrapper

Option 2: Remote finetuned checkpoint (directly runnable)

uv run python gr00t/eval/run_gr00t_server.py \
    --model-path nvidia/GR00T-N1.7-SimplerEnv-Bridge \
    --embodiment-tag SIMPLER_ENV_WIDOWX \
    --use-sim-policy-wrapper

Terminal 2 - Client:

gr00t/eval/sim/SimplerEnv/simpler_uv/.venv/bin/python gr00t/eval/rollout_policy.py \
    --n-episodes 10 \
    --policy-client-host 127.0.0.1 \
    --policy-client-port 5555 \
    --max-episode-steps 300 \
    --env-name simpler_env_widowx/widowx_spoon_on_towel \
    --n-action-steps 4 \
    --n-envs 5

Other supported tasks are:

simpler_env_google/google_robot_pick_object
simpler_env_google/google_robot_move_near
simpler_env_google/google_robot_open_drawer
...
simpler_env_widowx/widowx_spoon_on_towel
simpler_env_widowx/widowx_carrot_on_plate
simpler_env_widowx/widowx_stack_cube

you can replace the env_name with the corresponding tasks listed in the SimplerEnv fork this repo pins at external_dependencies/SimplerEnv (see .gitmodules).