MoveIt manipulation with a perception-fed planning scene, related 3D lifting of 2D vision output and improvements to motion detection.

[aleph-ra]

Adds classical, collision-aware manipulation to EmbodiedAgents: a MoveIt component that plans and executes through a running move_group, and the perception pipeline that lets it plan around what the robot's cameras actually see. Requires sugarcoat with the external-launch support (automatika-robotics/sugarcoat#61, merged).

The MoveIt component (MoveManipulator action server)

• Six goal modes: pose, joints, named (SRDF targets read from the live move_group), cartesian (fraction-checked), and the sequence modes pick and place. Pose goals carrying an explicit orientation plan on a dedicated oriented IK group with a snug tolerance (oriented_group_name, oriented_orientation_tolerance) — the default tolerance stays deliberately wide so unoriented goals remain solvable on underactuated arms, which would otherwise make an explicit attitude request vacuous.
• pick: resolve the target (scene id → detection label by best rank → nearest to a pose within target_match_radius) → approach → open → straight-line descent with contact permitted, holding the orientation the approach achieved → grasp → attach with SRDF-derived touch_links at the measured extents (planning padding comes off the thing the arm holds) → lift. approach_mode="side" serves grippers whose mouths take objects horizontally: pre-grasp behind the target at grasp height along the base→target bearing, the descent becomes the slide in, and the lift goes straight up, position-only. grasp_orientation gives unoriented picks the arm's known-good attitude, rotated to the target's bearing so one calibration serves every direction. place is the inverse; the released object stays in the scene where it was set down.
• The grasp contract, as MoveIt's own pipelines write it: a targeted allowed-collision entry between the gripper's touch links and the target only, granted for the grasp and retired when the object leaves the world — mirrored in place around detach→retreat. After an interrupted pick or place the allowance and a scene freeze persist until a motion succeeds, so the arm can always retreat from the object it is parked against (no more start-state-in-collision deadlocks).
• Underactuated-arm support throughout: cartesian_group_name pairs position-only IK for pose goals with orientation-tracking IK for the Cartesian validator's hardcoded per-step precision; an FK client holds the current end-effector orientation wherever none is requested.
• Gripper control via planning group or GripperCommand; all capabilities exposed as component actions, discoverable as tools by Cortex.

The planning scene

• Optional Detections3D input mirrored into move_group as named collision objects (det__<label>_<rank>), refreshed manually, on_goal, or continuously — always as a single diff, with TTL eviction bridging detection dropouts and a label allow-list (scene_detection_labels) keeping background furniture out. Scene failures degrade, never abort a motion.
• The scene freezes while an object is held and while a grasp-contact allowance is active: detection ids are score ranks, not identities, so a refresh could re-rank the id an allowance or attachment is keyed to onto a different object.
• Manual scene management, attach/detach, clear_octomap as component actions.

Perception: 2D → metric 3D

• New Bbox3D/Detections3D messages (map 1:1 onto vision_msgs/BoundingBox3D and MoveIt BOX primitives), with callback, type wrapper and frame ids on all vision messages.
• Vision lifts detections to 3D boxes via kompass-core (optional dependency, ≥ 0.8.4 — the pose is passed in the optical convention the new API defines, and older releases are refused since they would rotate every box silently): registered depth + CameraInfo aux inputs, robust depth statistics, per-box depth_validity, boxes published directly in the consumer's planning frame.
• Single-view surface-bias correction: a depth camera sees an object's front skin, so centers are pushed away from the camera along the view ray by half the box's smallest extent — exact for spheres, provably never overshooting a convex object's center.
• MLLM grounds on demand: depth is latched with the picture the VLM saw, lifted boxes (and now the 2D ones too) are labeled with the query that grounded them — "ground the red mug" → det__the red mug_0 → pick, target_object: "the red mug" with no glue. A run_task component action runs the configured task once on a planner's query and returns what was published, including located objects with metric centers; describe stays pinned to general VQA.

Memory and language context

• The Memory component takes a Detections3D layer and stores one observation per box at the object's own position instead of one text blob at the robot's — which also makes emem's entity merging by spatial radius behave as designed. Contract: publish the layer in the position topic's world frame (the lift's detections_frame does the transform).
• LLM and MLLM accept Detections3D as prompt context; its context string carries the metric positions ("In odom: orange at (2.00, 3.00, 0.05)"). The 2D detection strings were simplified to plain label joins, retiring the vendored pluralization module (−600 lines).
• RoboBrain2 defaults to the BAAI/RoboBrain2.0-3B checkpoint.

Motion detection

• Image motion estimation defaults to frame_difference, and an unset or unknown estimator is rejected at configuration — previously the default None silently declared motion on every frame.
• frame_difference is now the textbook version: blur, signed difference, a fixed changed-pixel threshold, and a gated global-shift removal so auto-exposure/white-balance steps and lights switching — which move the whole frame at once — do not register as motion. optical_flow thresholds on flow magnitude, so leftward and upward motion count and both estimators share the meaning of threshold.
• Single-channel streams (mono8, RealSense infrared) and 16-bit mono are processed (fixed rescale) instead of crashing in the color conversion.
• The ego-motion gate also pauses on turn rate (ego_turn_threshold, derived from odometry heading — a robot turning in place has no linear speed while the whole image sweeps), and resets its reference frame during the pause so stopping does not register as motion.
• Motion videos open with a pre-roll (video_preroll_frames, message references only, bounded) and close with the debounce tail, so a clip shows the scene before and after the event.
• image_scale downscales frames before estimation (default 0.5: optical flow ~5× cheaper at VGA, the ROI polygon scales along). With a position topic wired, cloud history is not accumulated before the first odometry reading, which used to produce a false motion burst at startup.