Skip to content

feat(alpasim-integration): add alpasim driver plugin for closed-loop operation - #166

Closed
Arseni10Lk wants to merge 2277 commits into
autowarefoundation:mainfrom
Arseni10Lk:closed-loop-integration
Closed

feat(alpasim-integration): add alpasim driver plugin for closed-loop operation#166
Arseni10Lk wants to merge 2277 commits into
autowarefoundation:mainfrom
Arseni10Lk:closed-loop-integration

Conversation

@Arseni10Lk

Copy link
Copy Markdown
Contributor

Summary

This PR implements the AlpaSim Driver Plugin (Model/plugins/alpasim_driver/) and the real-time Observation Stream Parser (Model/data_parsing/alpasim_stream/) to resolve #140.

It enables closed-loop evaluation and training rollouts of the AutoE2E policy within NVIDIA AlpaSim over the KitScenes 7-camera sensor topology at 10 Hz.

  • AlpaSim Driver Plugin: Implements AutoE2EDriver (BaseTrajectoryModel subclass) and AutoE2EAlpaSimConfig, registering autoe2e under Python entry point groups alpasim.models and alpasim.configs.
  • Stream Parser: AlpasimStreamParser preprocessor converting real-time PredictionInput payloads to normalized model observation tensors.
  • Offline Parity Guarantee: parity with offline Model/data_parsing/kit_scenes/dataset.py.
  • Standalone Examples & Tests: 50-step closed-loop simulation demonstration (examples/run_closed_loop.py), local smoke test (smoke_test_alpasim.py), and parity unit tests (test_alpasim_stream.py).

Architecture & Data Flow

AlpaSim Runtime
  ↓ PredictionInput (7 RGB camera frames, speed, acceleration, DriveCommand)
AutoE2EDriver (Model/plugins/alpasim_driver/plugin.py)
  ↓ AlpasimStreamParser (Model/data_parsing/alpasim_stream/parser.py)
AutoE2E Model Inference (normalized visual_tiles, egomotion_history)
  ↓ Waypoints [64, 2] + Headings [64]
ModelPrediction (trajectory_xy, headings in rig frame)
  ↓ AlpaSim Physics & Kinematic State Update

Registered Entry Points (pyproject.toml)

[project.entry-points."alpasim.models"]
autoe2e = "alpasim_autoe2e_plugin:AutoE2EDriver"

[project.entry-points."alpasim.configs"]
autoe2e = "alpasim_autoe2e_config:AutoE2EAlpaSimConfig"

Validation

  • pytest, ruff, and mypy clean
  • Coverage: 100% coverage on alpasim_stream/parser.py (49/49 statements), 84% total coverage across the plugin suite.
  • Closed-Loop Rollout Verification: Executed 50 steps at 10 Hz (5.0s simulated, 76.72m traveled, mean policy latency 10.60 ms/step) using AlpaSim physics feedback. I did not load actual checkpoint weights, so this was done for dummy inputs.

Acceptance Criteria

  • Model/data_parsing/alpasim_stream/parser.py is implemented.
  • Live observations parsed from stream are dimensionally and typographically identical to offline KitScenes dataset outputs.
  • Unit tests added to Model/tests/ verifying parser compliance against offline loader.
  • Local smoke test run successfully generating world model visualization (smoke_test_alpasim.py).
  • Standalone 50-step closed-loop simulation demonstration added (Model/plugins/alpasim_driver/examples/run_closed_loop.py).

Closes #140

@riita10069 @m-zain-khawaja

P.S. I also added Pillow to our requirements since it is used in the visualization

Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
riita10069 and others added 18 commits July 27, 2026 00:46
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
Signed-off-by: riita10069 <ryota10069.tech5.jizi@gmail.com>
…t/bev-diagnostics-dashboard

Add per-encoder BEV diagnostics to DataModelConsole
…n for issue autowarefoundation#140

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
…ack in test_alpasim_stream.py

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
…int inference, and attach camera params in stream parser

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
…ing requirements, and add closed-loop smoke test

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
…tion example

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
…tion

Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
@Arseni10Lk
Arseni10Lk force-pushed the closed-loop-integration branch from ee8b7c5 to 9af5d25 Compare July 29, 2026 14:53
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
Signed-off-by: Arseni10Lk <arseniy230606@gmail.com>
@riita10069

Copy link
Copy Markdown
Collaborator

Great...! I will review this PR.

@Arseni10Lk

Copy link
Copy Markdown
Contributor Author

@riita10069

I am just making more changes right now (in accordance with the yesterday's meeting), so please give me a second, I will tag you as soon as it is complete

@riita10069

Copy link
Copy Markdown
Collaborator

I’m sorry for the disruption. Environment-specific infrastructure identifiers, including instance IDs, were inadvertently committed to the repository history, so we had to rewrite the history and force-push the cleaned branches. This caused this pull request to close automatically.

I restored your commits in replacement PR #175 from the sanitized history, with the original commit authorship preserved:
#175

Before continuing work, please back up any uncommitted changes outside the repository, remove your existing local clone, and clone the repository again. Reusing or pushing from an old clone could reintroduce the removed history.

I apologize for the unexpected closure and the additional review noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Closed-Loop Integration with AlpaSim and OmniDreams

2 participants