Gemini To SO101

ScreenRecording_03-16-2026.2.mp4

This example is a minimal VLM planning + SO101 inverse kinematics loop.

Files:

• run.py
• run_dry.sh
• run_execute.sh
• so101_new_calib.urdf

What The VLM Outputs

The VLM outputs only numeric robot actions:

{"plan":[{"dx":0.003,"dy":-0.002,"dz":0.000,"gripper_delta":0.0}]}

• dx
  • forward/backward end-effector delta in meters
• dy
  • left/right end-effector delta in meters
• dz
  • up/down end-effector delta in meters
• gripper_delta
  • gripper open/close delta

Architecture

Each cycle does this:

1. Read one camera frame.
2. Ask Gemini for a short numeric action plan.
3. Clamp the numeric output to local limits.
4. Convert the target end-effector pose to joint angles with IK.
5. Apply local safety checks before sending the command.
6. Send the joint command to SO101.

There is no memory, no experience analysis, no hypothesis generation, and no second model.

Safety

Safety is still local.

• action outputs are clipped
• end-effector jumps are bounded by EEBoundsAndSafety
• workspace bounds are clipped
• joint motion is bounded by max_relative_target
• if IK or safety fails, the script retries once with half-sized motion and then stops

Requirements

• iPhone with Iriun Webcam
• PC with Iriun Webcam Linux app
• SO101 follower arm configured and calibrated
• Gemini API key

Python packages:

pip install google-genai pillow

Environment variable:

export GEMINI_API_KEY=YOUR_API_KEY

Dry Run

bash examples/gemini_to_so101/run_dry.sh

Real Robot Execution

bash examples/gemini_to_so101/run_execute.sh

Important Flags

• --action-model
  • VLM used for numeric planning
• --plan-steps
  • number of action steps returned per VLM call
• --axis-step-limit-m
  • max absolute per-axis delta allowed from the VLM
• --gripper-step-limit
  • max absolute gripper change per step
• --max-ee-step-m
  • max allowed Cartesian jump per executed step
• --workspace-min
  • workspace lower xyz bound
• --workspace-max
  • workspace upper xyz bound
• --max-relative-target-deg
  • joint-space safety bound applied by the follower

Notes

This is intentionally simple. It does not do semantic success checking or long-horizon motion planning. It just repeats:

image -> VLM numeric plan -> IK -> safe execution