Skip to content

feat(alpasim-integration): add alpasim driver plugin for closed-loop operation - #175

Open
riita10069 wants to merge 12 commits into
mainfrom
restore/pr-166-closed-loop-integration
Open

feat(alpasim-integration): add alpasim driver plugin for closed-loop operation#175
riita10069 wants to merge 12 commits into
mainfrom
restore/pr-166-closed-loop-integration

Conversation

@riita10069

@riita10069 riita10069 commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR implements the AlpaSim Driver Plugin (Model/plugins/alpasim_driver/) and the real-time Observation Stream Parser (Model/data_parsing/alpasim_stream/) to resolve #140.

It enables closed-loop evaluation and training rollouts of the AutoE2E policy within NVIDIA AlpaSim over the KitScenes 7-camera sensor topology at 10 Hz.

  • AlpaSim Driver Plugin: Implements AutoE2EDriver (BaseTrajectoryModel subclass) and AutoE2EAlpaSimConfig, registering autoe2e under Python entry point groups alpasim.models and alpasim.configs.
  • Stream Parser: AlpasimStreamParser preprocessor converting real-time PredictionInput payloads to normalized model observation tensors.
  • Offline Parity Guarantee: parity with offline Model/data_parsing/kit_scenes/dataset.py.
  • Standalone Examples & Tests: 50-step closed-loop simulation demonstration (examples/run_closed_loop.py), local smoke test (smoke_test_alpasim.py), and parity unit tests (test_alpasim_stream.py).

Architecture & Data Flow

AlpaSim Runtime
  ↓ PredictionInput (7 RGB camera frames, speed, acceleration, DriveCommand)
AutoE2EDriver (Model/plugins/alpasim_driver/plugin.py)
  ↓ AlpasimStreamParser (Model/data_parsing/alpasim_stream/parser.py)
AutoE2E Model Inference (normalized visual_tiles, egomotion_history)
  ↓ Waypoints [64, 2] + Headings [64]
ModelPrediction (trajectory_xy, headings in rig frame)
  ↓ AlpaSim Physics & Kinematic State Update

Registered Entry Points (pyproject.toml)

[project.entry-points."alpasim.models"]
autoe2e = "alpasim_autoe2e_plugin:AutoE2EDriver"

[project.entry-points."alpasim.configs"]
autoe2e = "alpasim_autoe2e_config:AutoE2EAlpaSimConfig"

Validation

  • pytest, ruff, and mypy clean
  • Coverage: 100% coverage on alpasim_stream/parser.py (49/49 statements), 84% total coverage across the plugin suite.
  • Closed-Loop Rollout Verification: Executed 50 steps at 10 Hz (5.0s simulated, 76.72m traveled, mean policy latency 10.60 ms/step) using AlpaSim physics feedback. I did not load actual checkpoint weights, so this was done for dummy inputs.

Acceptance Criteria

  • Model/data_parsing/alpasim_stream/parser.py is implemented.
  • Live observations parsed from stream are dimensionally and typographically identical to offline KitScenes dataset outputs.
  • Unit tests added to Model/tests/ verifying parser compliance against offline loader.
  • Local smoke test run successfully generating world model visualization (smoke_test_alpasim.py).
  • Standalone 50-step closed-loop simulation demonstration added (Model/plugins/alpasim_driver/examples/run_closed_loop.py).

Closes #140

@riita10069 @m-zain-khawaja

P.S. I also added Pillow to our requirements since it is used in the visualization

…n for issue #140

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
…ack in test_alpasim_stream.py

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
…int inference, and attach camera params in stream parser

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
…ing requirements, and add closed-loop smoke test

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
…tion example

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
…tion

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Closed-Loop Integration with AlpaSim and OmniDreams

2 participants