Reading the Room demonstrates how a frontier multimodal policy (NVIDIA Cosmos) can be augmented with SpatioTemporal’s LSTM (Large SpatioTemporal Model) based human intent modeller to yield safer, more socially aware robot navigation in crowded indoor scenarios.
- Scenarios covered: straight hallway passing, hallway crossroads, a minimally-crowded room, and 6m × 6m “party” scenario packed with 20 humans.
- Cosmos pure / Cosmos + intent: the critial facet of the experiment is to control a robot's motion via the Cosmos Reason engine, using a step-by-step data frame as input, and compare that to the same setup, but adding in 'intent' signals for each human in the scenario.
- Seed library: 100 procedurally generated seeds per scenario, each with deterministic human motion (via LSTM motion completion or scripted behaviors) and precomputed intent metadata. Humans are 'on rails', with no further inputs. Collisions and intersections are possible - this is a naive setup: no 3D modelling of robot nor human volumes are being done - we're simply interested in the outcomes of the any quantity of runs (rollouts) times 100 random seeds times 2 setups (with and without intent labels) times 4 increasingly complex scenarios.
- Evaluation harness: an OpenAI-compatible vLLM deployment of Cosmos-Reason2-8B drives the robot in simulation. We run paired experiments (plain vs intent-aware prompt) for every seed, logging Cosmos’ chosen actions, resulting trajectories, and safety metrics.
- Visualization: a Three.js web viewer replays any seed with per-agent trails, live intent labels, motion tokens, and speed annotations.
- Metrics: per-run time-to-goal, timeout rate, minimum human-robot separation, counts of near-misses, and count of humans within 0.5 m for qualitative “comfort” analysis.
- Dynamic intent prompting – LSTM-backed intent posteriors are summarized into natural language and appended to Cosmos prompts, producing clear changes in behavior (e.g., stronger yielding when the human is “assertive”).
- Human motion via LSTM – For hallway/crossing seeds, motion tokens from the same LSTM model generate realistic microvariations in human speed and lateral drift.
- Crowd stress tests – Scenarios 3 and 4 force the robot to traverse all four quadrants of a dense room, interacting with an increasing number of humans, who exhibit mixed behaviors (some are static, some wander, some pace, some cross the room to achieve their own goal).
- Web replay + seed randomizer – Browser UI with 3D visualisation of the scenario, pause/play/speed controls, with motion trails for clarity. the Interface has per-agent labels for the humans - eg: (Assertive #2345 1.1 m/s) - where the syntax is ([highest intent classification] [current LSTM token] [current speed])
- Cosmos run matrix – Automated manifest builder and rollout script execute 800 run batches per prompt variant (4 scenarios × 2 conditions × 100 seeds × 1-n rollouts) and archives the outputs for analysis.
- Single robot + single human
- Long narrow corridor
- Used for baseline interaction behavior and easy visual sanity checks. With random starting points and end goals in each of the 100 seed files, sometimes a confrontation is engineered / other times, straight paths to each agent's goals ensure a simple pass-by
- Perpendicular corridors with a forced collision tendency near the center :)
- Corridor width is widened, as compared to hallway, to allow for more decision time - a few more frames where the robot can see the human. No need to brute-force a collision every time - we're hoping for those 'awkward situations' where sometimes the robot arrives before the human, and other times when the robot will miss the human, but needs to adjust subtlely. Humans don't change their path / they don't react to the robot - they're 'on rails' - prebaked movements, supplied by the LSTM.
- Single robot with multi-goal route across all quadrants of a 10m x 10m room
- Multiple humans, each with visible per-tick inferred intent
- The simulation is naive: humans regularly walk through each other / have no bounding boxes nor volumes to check for local collisions - we're simply looking at the bigger picture: closeness, yielding, and comfort levels.
- Room set to 6m × 6m with 20 humans. Party on, Wayne.
- Mix of standing, pacing, wandering agents to create density, and invite the need for negotiation - even though the humans are not true agents / do not react to the robot's movements.
- Robot still visits four quadrants but must weave through a tighter footprint, making near-miss/collision metrics more telling
Reading the Room equips a frozen frontier policy (Cosmos) with human intent awareness derived from a dedicated LSTM trained on spatiotemporal trajectories. The pipeline precomputes human motions/intents, then re-simulates each seed while Cosmos controls the robot. We compare plain vs intent-aware prompts across hallway, crossing, and crowd settings, tracking both productivity (time-to-goal) and social comfort (minimum distance, near-misses). Early results show intent-aware prompts lead to longer but dramatically safer trajectories, hinting that “reading the room” improves perceived comfort even if raw speed drops.
Offline intent notes: The viewer does not require live LSTM inference. Every seed JSON embeds per-human intent snapshots (produced during generation), so you can scrub through the episode and see intent changes across time without running the intent model. Cosmos rollouts reuse those baked labels for replay, while GPU experiments optionally recompute them for higher fidelity.
- Intent inference trades speed for foresight. In every scenario, enabling intent increases path length/time (≈25–52 %) because the robot deliberately slows to reason about humans.
- Hallway and Crossing benefit most in safety terms. Collisions fall from 56 %→25 % (Hallway) and 24 %→2 % (Crossing) while minimum pass distance shrinks sharply, showing the planner is willing to brush closer when it trusts intent signals.
- Dense rooms behave differently. In the 10m “Crowd” room and the 6m “Party” scene, collisions stay effectively 100% because the robot inevitably scrapes against humans; intent still reduces close calls with the humans but sacrifices comfort (close-but-safe frames rise 7–25 %).
- Comfort vs. throughput is the key. Analysis of the Party scenario clarifies that background bumps dominate the collision metric now that we have many actors; future releases should incorporate crowd-comfort heuristics rather than binary collision counts.
- Close-but-safe is a new measure of human 'confort'. Close-but-safe exposure is scenario-specific: intent slashes near missed in Hallway (−43 %) and Crossing (−72 %), but it actually increases them in the two crowd-heavy rooms (+7 % in 10m Crowd, +25 % in Party). That hints the planner is willingly riding the 0.45 m envelope to keep moving — which is good for throughput, but it calls for better comfort metrics in dense scenes.
- Key Takeaway. Intent inference is doing its job to avoid collisions early, and decrease near-misses, but the dense crowds still force body contact. This is the clearest proof yet that a planner which models human intent earns safer trajectories even if it has to spend a few more seconds per mission—intent awareness. Improving the motion-intent (LSTM - Large SpatioTemporal Model) loop and integrating this into future NVIDIA Cosmos/Alpamayo architectures is a vital next step.
| Scenario | Condition | Average Steps | Average Time (s) | Average Speed (m/s) | Collision <0.22 m | Close-but-safe Steps (<0.45 m) | Min Pass (m) |
|---|---|---|---|---|---|---|---|
| Hallway | Plain | 103.9 | 10.3 | 0.74 | 56% | 2.76 | 0.36 |
| +Intent | 158.0 | 15.7 | 0.48 | 25% | 1.57 | 0.51 | |
| Crossing | Plain | 99.6 | 9.86 | 0.77 | 24% | 2.13 | 1.03 |
| +Intent | 144.1 | 14.31 | 0.53 | 2% | 0.60 | 1.65 | |
| Crowd (Room) | Plain | 458.3 | 45.7 | 0.63 | 0% | 36.4 | 2.31 |
| +Intent | 502.7 | 50.2 | 0.60 | 0% | 39.1 | 2.26 | |
| Party | Plain | 268.0 | 26.7 | 1.01 | 100% | 135.5 | 0.047 |
| +Intent | 336.5 | 33.5 | 0.80 | 100% | 169.6 | 0.0397 |
This repo is the generation and analysis pipeline for four scenarios, with paired outputs:
plain: robot sees room geometry/the robot's own goal/human postions - onlywith_intent: same as above, plus LSTM-based human-intent estimates each frame, for each human.
The benchmark objective is causal and narrow: compare robot time-to-goal and failure behavior with and without intent sensing under identical physical conditions.
- Scenario 1: hallway passing (
hallway_passing) - Scenario 2: hallway crossing (
hallway_crossing) - Scenario 3: crowded room (
crowd_realworld) - Scenario 4: party / dense crowd (
crowd_party, 6m × 6m room, ~20 humans) - Seed set:
100seeds per scenario - Conditions:
2(plain,with_intent) - Deterministic mode:
--mode-policy seededfor stable hidden human mode per seed
| Path | Purpose |
|---|---|
generate_episodes.py |
Main episode generator (all scenarios, plain/intent, seeded runs) |
rtr/env |
Scenario dynamics (hallway.py, crossing.py, crowd.py) |
rtr/policies |
Robot/human baseline policies |
rtr/intent |
Stub + LSTM intent adapter + LSTM motion policy + token tools |
scripts/export_three_episode.py |
JSONL -> Three.js viewer payload export |
scenario-seeds |
Seed JSONLs (plain + with_intent) for hallway/crossing/crowd/party runs |
manifests |
JSON/CSV manifests, used to drive run_cosmos_rollouts.py |
web/three_viewer |
Browser replay UI |
python -m http.server 9000
Open http://localhost:9000 and use query params to jump straight into Cosmos rollouts:
Example (single run per seed with baked intents and scripted human motion):
PYTHONPATH=. python scripts/run_cosmos_rollouts.py \
--manifest manifests/a100_run_matrix_v1.jsonl \
--seeds-root scenario-seeds \
--base-url http://127.0.0.1:8000/v1 \
--model nvidia/Cosmos-Reason2-8B \
--intent-backend baked \
--human-motion-backend scripted \
--rollouts-per-seed 1 \
--skip-existingTo stress test with more samples per seed, raise --rollouts-per-seed. If you later add a live LSTM checkpoint, switch to --intent-backend lstm and optionally --human-motion-backend lstm to regenerate motion tokens.