Say what to pick. Willy finds it, plans a collision-free 6-DoF grasp, refuses anything unsafe, executes it on a real or simulated arm, then verifies, recovers and logs.
See it work · Quick start · Your cell · Console · Architecture · Status · Reading
Workaholic-Willy turns a text prompt and a camera scene into a completed pick. It grounds the
prompt (GroundingDINO, SAM2, stereo or RGB-D depth), synthesizes and scores 6-DoF grasps, plans a
collision-free trajectory (cuRobo with exact-mesh collision), gates every motion through six
fail-closed guards, drives an arm and a gripper (UR, KUKA, Isaac Sim, Dummy, all behind one
Protocol), then verifies the grasp, recovers on failure, and logs each attempt as a frozen
GraspAttemptRecord.
It runs two end-effector modalities, a parallel jaw and suction, and is validated end to end in NVIDIA Isaac Sim on a UR5e cell.
flowchart LR
P["<b>prompt</b><br/><i>pick the red cube</i>"]:::io
C["<b>RGB-D or stereo</b>"]:::io
D["<b>GroundingDINO</b><br/>ground the words"]:::perc
S["<b>SAM2</b><br/>segment, then a point cloud"]:::perc
G["<b>Grasp calculator</b><br/>generate, score, rank"]:::grasp
F["<b>SafetyPreflight</b><br/>six fail-closed guards"]:::safe
M["<b>cuRobo</b><br/>collision-free plan"]:::safe
E["<b>Arm and gripper</b><br/>UR, KUKA, Isaac, Dummy"]:::exe
V["<b>verify, then recover</b>"]:::exe
L["<b>GraspAttemptRecord</b><br/>JSONL telemetry"]:::io
P --> D
C --> D --> S --> G --> F --> M --> E --> V --> L
V -.->|"retry, rescan, push"| G
classDef io fill:#1f2933,stroke:#63768d,color:#e4e7eb
classDef perc fill:#2b3a55,stroke:#5b8def,color:#e4e7eb
classDef grasp fill:#2d3b2f,stroke:#5fa463,color:#e4e7eb
classDef safe fill:#4a2f2f,stroke:#d16565,color:#e4e7eb
classDef exe fill:#3a3355,stroke:#8b7fd1,color:#e4e7eb
Note
A library and a CLI stack, with one optional service on top. The library under src/ imports
no web framework, and that is a contract rather than a habit: the operator console
(api/ and frontend/) is the single named exception, a
FastAPI application that depends on the library and that the library may never import back. There
is no ROS node in this repository.
Four scenarios, one stack: perceive, grasp, recover, refuse.
Four pillars, one loop each, and every clip below plays on its own:
| Vision-language perception | Autonomous decision |
|---|---|
 |
 |
| Prompt, then GroundingDINO, then SAM2, then a masked point cloud, then a 6-DoF grasp | The dense autonomous loop: perceive, refine, verify, recover. The AUTO decision gate is a separate opt-in; run_m1_pick --mode auto demonstrates it |
| Scene recovery | Fail-closed safety |
 |
 |
| A contact-redistribute push: the arm nudges a blocker aside near its centre of mass to free the target | Six ordered guards run before every motion (src/robot/safety/preflight.py). This clip's refusal is approach_path_blocked, from a seventh grasping-side check, and the pipeline fails closed either way |
Complementary end-effectors, jaw and suction on the same deep KLT bin: the jaw clears the shallow tray, suction reaches the floor. In the suction segment the object is carried kinematically on the wrist rather than held by a physics joint, because attaching one mid-play freezes Isaac's render. The jaw picks are real physics.
Full-length cuts, and the originals
The loops above are muted GIFs. The edited cuts and one original per capability are in
docs/assets/demo/, and GitHub plays them on click:
- The autonomy endgame, the whole run through the four pillars
- KLT, jaw and suction
docs/assets/demo/raw/: the wrist-camera pick, the seal-gated suction pick, exposing an occluded object, the planner working a bin, clearing a bin to empty, and a second gripper meeting a shape it cannot hold
Every one of them is reproducible on the workstation through the run_* recorders under
src/willy_sim/.
| Vision-language perception | A text prompt, then detection, then segmentation, then a masked point cloud, over stereo or RGB-D depth. A VLM route handles the attribute prompts a detector gets wrong. |
| 6-DoF grasp synthesis | Support-plane geometry, then antipodal, surface and dense contact sampling, then geometric, stability and reachability scoring, then a deterministic rank. Every formula: docs/grasping-math.md. |
| Fail-closed safety | Six ordered guards, workspace, joint limit, IK quality, self-collision, payload and motion continuity, run before every motion and outrank anything a model proposes. Every formula: docs/safety-math.md. |
| Collision-free motion | cuRobo for planning and Coal or fcl for vertex-exact mesh self-collision, with the engines installed once into a single local root. |
| Vendor-neutral drivers | UR over RTDE, KUKA over EthernetKRL, Isaac Sim and Dummy, all behind one RobotArm and Gripper Protocol, with vendor SDKs imported lazily inside the driver. |
| Gripper drivers | Robotiq over its URCap socket, OnRobot over Modbus, a jaw and a suction cup over digital I/O, the two simulated grippers, and Dummy and Null. Every wiring number is configuration, so bring-up is measuring rather than coding. |
| Multi-camera calibration and fusion | Per-camera eye-to-hand and eye-in-hand extrinsics through one central map, solved by AX=XB. Multi-camera geometry fusion runs inside the pick loop, and a worked two-camera cell ships as a configuration example. |
| Structured telemetry | Every attempt becomes a frozen GraspAttemptRecord, which feeds the KPI rollup, the soak gate, the failure taxonomy and offline reinforcement learning. |
| Offline reinforcement learning | Train, evaluate off-policy, promote through a gate. It runs shadow-only at run time and never overrides the safety mask, and check-dataset answers whether a log is trainable before the training run rather than after. |
| Operator console | FastAPI and React: preflight, cell, pick, history and configuration, over a typed HTTP surface with a live event stream. Driven end to end against real UR controller software. |
| Isaac Sim platform | A config-driven UR5e cell: known-pose, real-vision, eye-in-hand, multi-view and suction picks, hand-eye calibration, and the cinematic recorders that made the clips above. |
| Synthetic data engine | datagen/: path-traced scenes with a posed arm, analytic grasp labels and a physics reward. It is the independent reference the grasp calculator is measured against. |
Windows 10 or later, and Linux. The library and the test suite need no GPU, camera or robot. Only the Isaac Sim runners need the workstation.
# 1. install
python -m venv .venv
source .venv/bin/activate # Windows: .\.venv\Scripts\Activate.ps1
pip install -r requirements.txt # runtime, drivers, console and tooling in one file
# The cu128 wheels import without a GPU, so this is the default everywhere. If you would
# rather not pull them, requirements-cpu.txt installs the same set against CPU torch.
# 2. validate the whole configuration tree
python -m src.config --print
# 3. run the synthetic soak and KPI gate; exit 0 means every locked gate passed
python -m src.robot.grasping.replay --soak-report
# 4. rehearse the real-cell boot path end to end, on a dummy arm, commanding nothing
python -m src.robot.execution.real_cell --rehearse --runs 3Two rehearsals that need nothing but the clone, and print what they actually did:
python scripts/examples/cell/01_robot_setup.py # is this cell described coherently?
python scripts/examples/cell/03_first_pick.py # one grasp on a dummy arm, and which layers ranThe same pick from Python, which is what those examples call:
from src.config import load_config
from src.robot.execution.cell import Cell
from src.robot.execution.pick_run import PickRun, Recording
cell = Cell.rehearsal(load_config().robot) # or Cell.from_robot_config(...) for a real one
report = PickRun.from_cell(cell, runs=1, recording=Recording.off()).execute()
print(report.render()) # what happened, in words
report.exit_code # 0 passed, 1 refused, 2 picked and did not pass, 3 raisedNew here? Read the five-part guide, which goes from an empty directory to a
pick, or the shorter Quickstart. Getting the workstation ready for Isaac and
the motion engines: docs/isaac-ready.md.
Configuration is a Pydantic StrictModel tree with extra='forbid', so a typo fails loudly at load
rather than quietly at run time. The YAML lives in config/, and a profile is a set of
overlay files deep-merged onto it, selected by WILLY_PROFILE.
Four worked cells ship as profiles, and each one is commented key by key for the person standing at the bench:
| Profile | The cell |
|---|---|
sim |
The Isaac UR5e cell, the same tree the simulator boots from |
ur3e |
A UR3e, in simulation or on a bench. Every position is re-anchored, because a UR3e works a 500 mm sphere where a UR5e works 850 |
ur5e |
A real UR5e on a bench: the workspace box, the home pose, the bench as a collision fixture, and the planner's world |
ur5e,eth2 |
The same bench with two fixed RGB-D cameras, fused |
python -m src.config # validate the shipped tree
WILLY_PROFILE=ur5e,eth2 python -m src.config # validate a two-camera cell
python -m src.config explain robot.safety.self_collision.planner_margin_mmexplain reports a key's type, its default, which file and which layer set the winning value, the
whole override chain, and the comment written above that line.
Tip
The ur5e example draws a line worth copying. A value that fails closed if it is wrong ships
as a worked example with its assumption stated, because a workspace box that is too small refuses
a motion visibly. A value that fails open does not ship at all: a tool frame or a payload that
is merely plausible drives the arm into the bench and logs a success, so those keys are written out
as commented blocks with what to measure, and the shipped refusals stay armed.
Full reference: src/config/README.md. Every robot.grasping block, what it
does and which grasp mode it can fire in:
docs/grasping-config-reference.md.
A browser front end for the people who stand at the cell, not a demo shell. Five screens over a typed HTTP surface with a live event stream, served same-origin from one process.
python -m api --profile console_dummy # hardware-free; --profile ursim for a UR controller
# then open http://127.0.0.1:8000| Screen | Answers |
|---|---|
| Preflight | Is this cell runnable? Every check verbatim, with its fix instruction, nothing softened. |
| Cell | Build, preview the connect, connect, and watch live telemetry: TCP, joints, controller state, both stop flags. |
| Pick | Run an attempt against a live event stream. A refusal names the guard that refused, and why. |
| History | Every logged attempt, rolled up. |
| Config | The merged, validated tree, the same one the cell booted from. |
Tip
The console was driven end to end against URSim, which is real UR controller software running a robot that does not exist. That is what proved the motion-explanation path: a refused pick reported the guard, the pose and the reason rather than a generic failure.
Full surface, error envelope and event contract: api/README.md. UI internals:
frontend/README.md.
Twenty-nine executable examples, and each one covers a single decision you have to make: it shows the alternatives, runs them, and says what each costs. Where an answer fails, the example runs the failure, because a refusal is what you will actually meet. They call the library rather than shelling out, so every one is also a worked example of the API.
Nothing moves unless you pass
--live. Every example that can drive an arm defaults to a rehearsal: it loads the configuration, builds the real components, runs every check that needs no motion, and commands nothing.
The numbering runs across the folders, so reading in numeric order is a path from an unopened box to a trained model. The folders let you go straight to the question you have.
| Folder | The decisions in it |
|---|---|
cell/ |
which vendor, which gripper, one grasp end to end, and whether the arm plans or drives a straight line |
calibration/ |
a fixed camera, a wrist camera, and the two places a two-camera cell is wired wrong |
perception/ |
which detector, what to do when it is confidently wrong, and where depth comes from |
safety/ |
the six guards refusing on purpose, the exact meshes against the proxy, and declaring your bench |
grasping/ |
a jaw or a cup, the analytic ranker or the learned one, and the thirteen switches that ship off |
datagen/ |
your parts or public ones, what a scene decides, which engine, geometry against physics, and the corpus |
train/ |
the whole chain on your own parts, the public-corpus road, the two knobs, the report, and the limits |
sim/ |
the same pick where a wrong answer is free |
pipeline/ |
all of it, stopping at the first blocking stage |
Start with scripts/examples/cell/01_robot_setup.py. Full index:
scripts/examples/README.md.
A strict downward dependency stack, and nothing imports up.
flowchart TD
subgraph ENTRY["Entry points"]
direction LR
E1["<code>python -m src.config</code>"]
E2["<code>... grasping.replay</code><br/><code>... grasping.rl</code>"]
E3["<code>... execution.real_cell</code>"]
E4["<code>willy_sim.run_*</code><br/>Isaac validation and demos"]
E5["<code>python -m api</code><br/>operator console"]
end
subgraph EXEC["execution, the composition root"]
X["<b>Cell and AutonomousGraspService</b><br/>perceive, grasp, gate, move, verify, recover, log"]
end
subgraph MID["the three pillars"]
direction LR
GR["<b>grasping/</b><br/>generate, score, decide<br/>refine, verify, recover<br/>and rl, shadow-only"]
SA["<b>safety/</b><br/>SafetyPreflight<br/>six fail-closed guards<br/><i>outranks every model</i>"]
PE["<b>perception</b><br/>models, camera, calibration<br/>stereo and RGB-D depth"]
end
subgraph HW["robot, the vendor boundary"]
direction LR
CO["<b>core/</b><br/>RobotArm and Gripper Protocols"]
DR["<b>drivers/</b><br/>UR, KUKA, Isaac, Dummy"]
GP["<b>grippers/</b><br/>robotiq, onrobot, jaw_io, vacuum, sim, null"]
end
subgraph BASE["foundations"]
direction LR
GE["<b>geometry</b><br/>numpy SE(3): Frame, Pose, Transform"]
CF["<b>config</b><br/>Pydantic StrictModel and YAML overlays"]
end
OFF["<b>offline</b>: replay, KPI rollup, the soak gate, RL training<br/><i>never imported back</i>"]
ENTRY --> EXEC --> MID --> HW --> BASE
EXEC -. "JSONL telemetry" .-> OFF
classDef entry fill:#1f2933,stroke:#63768d,color:#e4e7eb
classDef exe fill:#3a3355,stroke:#8b7fd1,color:#e4e7eb
classDef mid fill:#2b3a55,stroke:#5b8def,color:#e4e7eb
classDef hw fill:#2d3b2f,stroke:#5fa463,color:#e4e7eb
classDef base fill:#4a3f2f,stroke:#d1a065,color:#e4e7eb
class E1,E2,E3,E4,E5,OFF entry
class X exe
class GR,SA,PE mid
class CO,DR,GP hw
class GE,CF base
Important
The default pick is open-loop: perceive, rank geometrically, gate, move, log. The AUTO decision gate, closed-loop refine and verify, recovery, multi-view fusion, the learned success model and reinforcement learning are all built, opt-in and default-off. A switch that would read as on while doing nothing is refused by the schema rather than silently accepted.
Repository layout
config/ the YAML tree every cell is described in, plus the shipped profiles
src/
contracts/ the calling convention: factories, one verb, frozen reports, UNSET
geometry/ numpy SE(3): Frame, Pose, Transform, quaternions
calibration/ extrinsics, eye-hand through AX=XB, stereo, versioned artifacts
camera/ capture rigs, frame providers, streaming over OpenCV or RealSense
models/ GroundingDINO, SAM2, RT-DETR, a VLM route, hand and speech wrappers
config/ the loader, the schema and the CLI that reads the tree above
robot/
core/ RobotArm and Gripper Protocols, typed motion objects, vendor enums
drivers/ UR over RTDE, KUKA over EKI, Isaac Sim, Dummy, and two empty slots
grippers/ robotiq, onrobot, jaw_io, vacuum, the two simulated ones, dummy, null
safety/ the ordered fail-closed preflight, and the cuRobo binding
perception/ real-camera RGB-D into a PerceptionFrame
grasping/ generate, score, decide, refine, verify, recover, log
execution/ the composition root, the real cell, the calibration routine
willy_sim/ the Isaac Sim runners, the full-motion validation platform
utility/ paths, device selection, atomic IO, logging, unit scaling
api/ the console backend: FastAPI, seven routers, an event hub
frontend/ the console UI: React and Vite, built into api/static/
datagen/ synthetic scenes, analytic grasp labels, a physics reward
scripts/ the examples, the URSim probes, the build and bake tools
ext_deps/ the single install root for Coal and cuRobo; the payload is ignored
tests/ the suite, torch-free, gated at 80 percent coverage
docs/ the guide, the runbooks, the math references, and the media
Continuous integration runs lint, types, tests, coverage and the soak gate on every change. Locally:
ruff check src api datagen tests scripts
mypy src api datagen scripts
pytest tests --cov=src --cov=api --cov=datagen --cov-fail-under=80
python -m src.robot.grasping.replay --soak-reportThe suite mocks every model constructor and every robot and camera connection, so it needs no GPU, no camera and no robot, and it is torch-free. The Isaac imports are lazy, so the mock-mode simulation tests stay green on a machine that has no Isaac at all.
Three levels of evidence run through this repository, and they mean exactly this:
| measured in simulation | Isaac Sim, on the box, with numbers |
| measured against real controller software | a real protocol or a real controller, no physical motion |
| never touched hardware | code complete, behaviour unproven |
| Area | State |
|---|---|
| Vision-language perception | measured in simulation: prompt, detect, segment, masked cloud |
| Generate, score, decide, execute, verify, recover, log | measured in simulation: the core pipeline |
| Fail-closed safety preflight | measured in simulation: runs before every motion, outranks every model |
| Motion: cuRobo with exact-mesh collision | measured in simulation: the standard path |
| Isaac Sim validation on a UR5e with a 2F-85 | measured in simulation: known-pose, real-vision, eye-in-hand and multi-view picks, plus calibration |
| Jaw and suction | measured in simulation: both |
| Driver: UR over RTDE | measured against real controller software: connect, power, motion, digital I/O, tool frame, payload, protective stop |
| Drivers: Dummy and Isaac Sim | measured in simulation |
| Driver: KUKA over EthernetKRL | never touched hardware: software-complete, unvalidated |
| Drivers: Franka and ROS2 | reserved empty vendor slots |
| Grippers: Robotiq, OnRobot, jaw and suction over digital I/O | never touched hardware: the drivers are complete and the digital-I/O pins were measured switching on a real controller, the grippers were not |
| Real-camera perception over RealSense | never touched hardware: complete and reachable from configuration, never fed a real frame |
| Operator console | measured against real controller software: five screens, driven end to end |
| Synthetic data engine | measured in simulation: proof run complete, verifier clean |
| Offline reinforcement learning | measured in simulation: the chain closes on measured physics outcomes, and the committed policies abstain rather than pretend |
| Physical arm motion | no line of this code has ever executed on a physical robot |
| A ROS node, or a voice API | does not exist, by design |
Important
Honesty first. The advanced grasping layers ship default-off and byte-identical, and the default pick is open-loop. The soak gate is a synthetic contract self-check: it proves telemetry and KPI consistency, not grasp quality, and says so in its own provenance block. Software collision avoidance is not certified functional safety; a real cell needs the vendor's safety-rated stop, not this guard. On the clips: the endgame recorder retries each segment and keeps the first take that lifts clear, so the reel shows representative successful takes rather than a per-attempt rate, and the on-frame banner never claims a recovery that did not happen.
New here? Start with the guide. Five sequential walkthroughs that go from an empty directory to a robot picking an object: configuration, models, calibration, robot and safety, the pick loop. The package READMEs below are the per-module reference you reach for afterwards.
The two math references, every formula the stack evaluates and how each one fails: grasping, from prompt to point cloud to grasp pose, and safety, workspace, forward kinematics, Jacobian, capsules and mesh distance.
| Foundations | Vision and calibration |
|---|---|
|
|
| The robot | Drivers and grippers |
|
|
| Grasping | Simulation, data and operations |
|
|
Workaholic-Willy · perceive · grasp · verify · recover
by Alet-Robotics · never clocks off · currently employed exclusively in simulation