feat(temporal): collision-response workflow + MQTT bridge - #15
Merged
Conversation
Pulls the simulator out of the bundled "sim" container into its own
service. The robot's always-on ROS infrastructure (bridge_node +
sim_battery) moves into a sibling "robot" service. The OTA-swappable
robot software (sim/controllers/*) is unchanged — those images still
run as agent-managed containers on the lab network.
Three services, all on the same lab network and ROS DDS domain
(ROS_DOMAIN_ID=42, RMW=cyclonedds):
gazebo - ign gazebo + ros_gz_bridge + Xvfb/x11vnc/noVNC GUI
stack. Subscribes to /cmd_vel on ROS DDS and forwards
to the gz diff_drive plugin. Now also bridges
/perseverance/contacts so collision events are visible
ROS-side (groundwork for the Temporal collision-response
workflow).
robot - bridge_node (rclpy → gRPC) + sim_battery. The agent's
persistent peer.
agent - unchanged otherwise. BRIDGE_ADDR moves from a Unix
socket on a shared volume to TCP `robot:50051` over
the lab network. BROKER_URL stays on tcp://mqtt:1883
(lab broker, port 14883 host-side).
bridge_node/server.py: new --listen flag accepting either bare
host:port (TCP) or unix://path. --socket retained for back-compat.
moon.sdf: a 0.9 m boulder at x=8 directly in the rover's forward
path. Rover deck has a contact sensor (`bumper`) tied to the
gz-sim contact system; it emits Contacts messages on
/perseverance/contacts whenever the deck collides.
docker/sim/ removed; replaced by docker/gazebo/ + docker/robot/.
Worlds + models live under docker/gazebo/ now.
Verified locally on Apple Silicon + podman:
- gazebo, robot, agent all Up
- robot: "bridge listening on 0.0.0.0:50051"
- agent: "bridge connected addr=robot:50051"
- ros_gz_bridge: 6 bidirectional bridges including
/perseverance/contacts ↔ ignition.msgs.Contacts
- Lab MQTT, Postgres, Temporal, Registry all healthy
User asked for a Temporal job that listens for a robot collision and
backs the rover up, turns right 90 deg, drives forward again. Wires
the full chain.
Robot: ros-humble-ros-gz-interfaces + python3-paho-mqtt; new
collision_publisher (ROS /contacts -> MQTT events/{robot_id}/collision,
2s debounce) and twist_subscriber (MQTT cmd/{robot_id}/twist -> ROS
/cmd_vel). paho 1.x API for jammy.
Cloud cloud/internal/collision + cmd/collision-worker:
- CollisionResponse workflow: back -0.3 m/s 3s, settle, turn -0.5
rad/s 3.14s (~90 deg right), settle, forward 0.4 m/s 5s, stop.
- SendTwist activity republishes at 10Hz so DiffDrive's ~0.5s
command timeout does not stall the rover.
- mqttbridge subscribes to events/+/collision and starts a
workflow per inbound event.
Make: workers-up / workers-down / workers-status to run the host-side
ota-worker + collision-worker. 'make collide' triggers a fake event.
PID files in .run/.
Verified end-to-end: collision-{robot_id}-{ts} workflow visible at
http://localhost:14080; rover odometry advances through the phases.
Demo glue. Rounds out the host-side runner so the OTA story is one
command per scenario, mirroring 'make collide' for the collision
demo.
Targets:
controlplane-up / -down / -status — start/stop the controlplane HTTP
API binary against the lab Temporal + TSDB. PID file in .run/.
ota-circle — build sim/controllers/drive-circle, push to the
lab registry (localhost:14050), and POST a rollout to
/v1/ota/rollouts targeting sim-robot-01.
ota-figure-eight — same for drive-figure-eight.
ota-status — GET /v1/ota/rollouts (jq if present).
Override OTA_REGISTRY / OTA_ROBOT_ID / OTA_CP_HOST to retarget.
Verified: 'make controlplane-up' brings the API up; healthz 200;
ota-status returns null pre-rollout.
User reported the OTA rollout was stuck. Two fixes:
1. agent/internal/ota/docker.go — auto-detect docker vs podman.
AGENT_CONTAINER_BIN env override > docker on PATH > podman.
Pull(): if podman, append --tls-verify=false so localhost:14050
(HTTP-only lab registry) doesn't get rejected.
2. New native-host agent mode. Rootless podman bind-mounts the
daemon socket as a non-stat-able file (idmap remaps owner
UID); even userns_mode=keep-id + chmod 666 didn't get container
root through. Side-stepping by running the agent as a Go binary
on the host:
- docker-compose.sim.yml: robot publishes :50051 to host;
in-container agent profile-gated under in-container-agent.
- Makefile: agent-up / agent-down / agent-status targets.
Verified: make agent-up + make ota-circle ->
rollout completed; robot-app container Up running
localhost:14050/robot-app:circle-v1; rover circles in noVNC.
User reported rollout jobs not finishing. Root cause: the cloud
MQTT bridge routes acks like this:
PHASE_ROLLED_BACK -> {rollout_id}-robot-{robot_id}-rollback
(everything else) -> {rollout_id}-robot-{robot_id}
When a rollback command was issued for the very first OTA on a
robot (or any case where rollback's prev-digest is empty), the
agent's runRollback:
- tried Docker.Rollback(ctx, prev='') -> error
- published PHASE_FAILED with detail
- PHASE_FAILED was routed to OTASingleRobot (already exited),
NOT to the -rollback workflow that was waiting for a phase
signal
- The rollback workflow stayed blocked on its 5-minute timer
- The parent OTASingleRobot was waiting on the rollback child's
completion -> the parent OTARollout never RecordRolloutEnded
- The DB row showed 'pending' for ~5 minutes per failed rollout
Fix: agent's runRollback always emits PHASE_ROLLED_BACK, with the
failure mode captured in detail when rollback couldn't actually
restore. The rollback workflow always terminates within ms of the
agent processing the command; the parent unblocks; the rollout
status moves to its terminal value (canary_failed / aborted /
completed) immediately.
Verified: post-fix, ota-circle and ota-figure-eight runs land in
~1 second with status=completed. Stale 'pending' rows from before
this fix will resolve naturally as their 5-minute timers fire,
or via curl POST /v1/ota/rollouts/{id}/abort.
…t make targets
The make-target inventory has grown a lot since the last doc pass:
agent-up/-down/-status, workers-up/-down/-status, controlplane-up/-
down/-status, ota-circle, ota-figure-eight, ota-status, collide,
sim-drive-{fwd,back,left,right,stop}, sim-gui, sim-up-headless,
container-info, plus the host-vs-container split (the agent now
runs as a native macOS Go binary by default).
README.md
- New service-shape diagram showing gazebo + robot + agent + workers
+ controlplane + lab cluster.
- Lab quickstart is now four make targets in sequence (sim-up,
agent-up, workers-up, controlplane-up).
- Drive demo, OTA demo, Collision demo each get a one-command
recipe.
- Full make-target reference grouped by purpose.
- Default lab ports table includes the new robot:50051, controlplane
:8081, gazebo :14680/:14900.
ONBOARDING.md
- Section 7 (run end-to-end) replaced with the new make-target
flow + .run/ PID file convention.
- Section 9 covers OTA + Collision demos as one command each.
- Troubleshooting table refreshed with the issues actually hit
on this machine: docker-not-on-PATH, podman HTTPS pull rejection,
rollouts stuck pending, robot port not published.
CLAUDE.md
- Build/test/run section rewritten to enumerate the host-side
processes and demo helpers.
installer/README.md
- Stale 5xxx-port table replaced with the current 14xxx lab + 2xxxx
CI matrix, plus a sim-only ports section (14680, 14900, 50051)
and a host-side processes section (8081).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wires the full demo: contact sensor in Gazebo → MQTT collision event → Temporal workflow → MQTT twist commands → rover backs up + turns right 90° + drives forward.
Stacks on PR #14 (gazebo + robot split). Merge order: #14 then this.
What's in
Robot container (
docker/robot/Dockerfile)ros-humble-ros-gz-interfaces,ros-humble-geometry-msgs,python3-paho-mqtt.bridge_node/collision_publisher.py— subscribes to ROS/contacts(republished from gz/perseverance/contacts), publishes one MQTT event onevents/{robot_id}/collisionper real impact (2 s debounce).bridge_node/twist_subscriber.py— subscribes to MQTTcmd/{robot_id}/twist, republishes onto ROS/cmd_vel.CallbackAPIVersion).Cloud (
cloud/internal/collision/+cloud/cmd/collision-worker/)CollisionResponseworkflow — back -0.30 m/s 3 s → settle → turn -0.50 rad/s 3.14 s (~90° right) → settle → forward 0.40 m/s 5 s → stop.SendTwistactivity — republishes the twist at 10 Hz oncmd/{robot_id}/twistso gz DiffDrive's ~0.5 s command timeout doesn't stall the rover; final 0,0 frame at end.MQTTBridge— subscribes toevents/+/collision, starts one workflow per inbound event (workflow IDcollision-{robot_id}-{ts}).collision-workercmd hosts both the Temporal worker and the MQTT bridge.Make targets
workers-up/workers-down/workers-status— start, stop, inspect the host-side ota-worker + collision-worker. PID files in.run/.collide— publish a fake collision event via the robot container's paho client (no host MQTT client needed) so the workflow runs without a real impact.Verified locally
Test plan
make collide