feat(trajectory_planning): add BEV-only multi-sample trajectory scorer and shifted inference schedule - #170
Open
riita10069 wants to merge 4 commits into
Open
feat(trajectory_planning): add BEV-only multi-sample trajectory scorer and shifted inference schedule#170riita10069 wants to merge 4 commits into
riita10069 wants to merge 4 commits into
Conversation
…r and shifted inference schedule Signed-off-by: Flagbusted <justthefourofus@proton.me>
…lanner Returns [B, K, 128] via repeat_interleave + single forward() call. K samples per scene are independent — each row draws fresh x_0 ~ N(0, I_128) inside forward(). 12 unit tests added. When issue #17 (BEV seg aux loss) lands, drivable_area_compliance can upgrade from pixel-colour heuristic to label-based lookup with no scorer API change. Implements #75 Signed-off-by: FLagbusted <justthefourofus@proton.me>
…real contract sample_and_score unpacked self.planner(...) as (trajectory, ego_hidden) — BasePlanner.forward() has only ever returned a single trajectory tensor (see base.py's own docstring). Every test here passed anyway because the local FakePlanner test double returned a 2-tuple to match the WRONG contract, so this was never exercised against a real planner. Wiring TrajectoryComplianceScorer(real_flow_matching_planner, ...) and calling sample_and_score would have failed immediately with an unpack error. FutureState (the only place ego_hidden was ever consumed) isn't called from AutoE2E.forward() any more — WorldActionModel.predict_future superseded it — so there's nothing downstream expecting a second value. sample_and_score now returns (trajectory, scores). Also, per @riita10069's review on #76: - decode_trajectory_to_xy now wraps the canonical Model.evaluation.metrics.integrate_trajectory instead of duplicating the bicycle-model integration. - Both decode_trajectory_to_xy and kinematic_comfort_score now take a real per-row initial_speed (extract_initial_speed(), reading the speed channel out of egomotion_history) instead of a fixed ScorerConfig.initial_speed=5.0 placeholder applied to every sample regardless of how fast the ego actually was. Added a regression test (test_different_initial_speeds_change_selection) that would have caught this. - Flagged in the module docstring: do not calibrate pixels_per_meter/ego_row/ego_col/drivable_rgb against Model/data_parsing/kit_scenes/map.py as it stands today without checking #148/#149 first — both open, both assigned to riita10069, both change the geometry this scorer would be calibrated against. Moved tests/test_trajectory_scorer.py -> Model/tests/ (per review) and discovered why it mattered beyond tidiness: python -m pytest Model/tests -v ============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0 -- /usr/bin/python cachedir: .pytest_cache rootdir: /home/claude/auto_fsd2/Model configfile: pytest.ini plugins: anyio-4.14.2 collecting ... collected 596 items / 3 errors / 6 deselected / 6 skipped / 590 selected ==================================== ERRORS ==================================== ___________ ERROR collecting tests/test_dataset_publication_tasks.py ___________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_dataset_publication_tasks.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_dataset_publication_tasks.py:10: in <module> from botocore.exceptions import ClientError E ModuleNotFoundError: No module named 'botocore' _________________ ERROR collecting tests/test_overlay_tasks.py _________________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_overlay_tasks.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_overlay_tasks.py:9: in <module> from botocore.exceptions import ClientError E ModuleNotFoundError: No module named 'botocore' ______________ ERROR collecting tests/test_training_checkpoint.py ______________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_training_checkpoint.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_training_checkpoint.py:11: in <module> from botocore.exceptions import ClientError E ModuleNotFoundError: No module named 'botocore' =========================== short test summary info ============================ ERROR Model/tests/test_dataset_publication_tasks.py ERROR Model/tests/test_overlay_tasks.py ERROR Model/tests/test_training_checkpoint.py !!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!! ================== 6 skipped, 6 deselected, 3 errors in 3.18s ================== / CI only run ============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0 rootdir: /home/claude/auto_fsd2/Model configfile: pytest.ini plugins: anyio-4.14.2 collected 518 items / 12 errors / 6 deselected / 6 skipped / 512 selected ==================================== ERRORS ==================================== ______________ ERROR collecting tests/test_dataset_publication.py ______________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_dataset_publication.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_dataset_publication.py:10: in <module> from Platform.pipelines.dataset_publication import ( E ModuleNotFoundError: No module named 'Platform' ___________ ERROR collecting tests/test_dataset_publication_tasks.py ___________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_dataset_publication_tasks.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_dataset_publication_tasks.py:10: in <module> from botocore.exceptions import ClientError E ModuleNotFoundError: No module named 'botocore' _______________ ERROR collecting tests/test_grad_accumulation.py _______________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_grad_accumulation.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_grad_accumulation.py:23: in <module> from Platform.pipelines.training_checkpoint import ( E ModuleNotFoundError: No module named 'Platform' ______________ ERROR collecting tests/test_kitscenes_recovery.py _______________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_kitscenes_recovery.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_kitscenes_recovery.py:9: in <module> from Platform.pipelines.kitscenes_recovery import ( E ModuleNotFoundError: No module named 'Platform' _______________ ERROR collecting tests/test_overlay_artifact.py ________________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_overlay_artifact.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_overlay_artifact.py:10: in <module> from Platform.pipelines.overlay import ( E ModuleNotFoundError: No module named 'Platform' _______________ ERROR collecting tests/test_overlay_inference.py _______________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_overlay_inference.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_overlay_inference.py:8: in <module> from Platform.pipelines.inference import ( E ModuleNotFoundError: No module named 'Platform' ____________ ERROR collecting tests/test_overlay_reproducibility.py ____________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_overlay_reproducibility.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_overlay_reproducibility.py:7: in <module> from Platform.pipelines.reproducibility import ( E ModuleNotFoundError: No module named 'Platform' _________________ ERROR collecting tests/test_overlay_store.py _________________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_overlay_store.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_overlay_store.py:7: in <module> from Platform.pipelines.overlay_store import ( E ModuleNotFoundError: No module named 'Platform' _________________ ERROR collecting tests/test_overlay_tasks.py _________________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_overlay_tasks.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_overlay_tasks.py:9: in <module> from botocore.exceptions import ClientError E ModuleNotFoundError: No module named 'botocore' ______________ ERROR collecting tests/test_training_checkpoint.py ______________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_training_checkpoint.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_training_checkpoint.py:11: in <module> from botocore.exceptions import ClientError E ModuleNotFoundError: No module named 'botocore' _______________ ERROR collecting tests/test_trajectory_scorer.py _______________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_trajectory_scorer.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_trajectory_scorer.py:17: in <module> from Model.model_components.trajectory_planning.trajectory_scorer import ( E ModuleNotFoundError: No module named 'Model' ___________ ERROR collecting tests/test_trajectory_visualization.py ____________ ImportError while importing test module '/home/claude/auto_fsd2/Model/tests/test_trajectory_visualization.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib/python3.12/importlib/__init__.py:90: in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Model/tests/test_trajectory_visualization.py:17: in <module> from Platform.pipelines.overlay import write_overlay E ModuleNotFoundError: No module named 'Platform' =========================== short test summary info ============================ ERROR Model/tests/test_dataset_publication.py ERROR Model/tests/test_dataset_publication_tasks.py ERROR Model/tests/test_grad_accumulation.py ERROR Model/tests/test_kitscenes_recovery.py ERROR Model/tests/test_overlay_artifact.py ERROR Model/tests/test_overlay_inference.py ERROR Model/tests/test_overlay_reproducibility.py ERROR Model/tests/test_overlay_store.py ERROR Model/tests/test_overlay_tasks.py ERROR Model/tests/test_training_checkpoint.py ERROR Model/tests/test_trajectory_scorer.py ERROR Model/tests/test_trajectory_visualization.py !!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!! ================= 6 skipped, 6 deselected, 12 errors in 2.21s ================== (see Makefile), so this entire file — all 15 original tests — was never collected by CI. Neither the contract bug above nor a pre-existing off-by-one in the map fixture (the drivable rectangle stopped one row short of ego_row, so the 'origin should always score compliant' assumption was silently false) had ever actually run. Fixed the fixture and added 5 new tests (extract_initial_speed coverage, initial-speed-changes-decode, the contract regression test above); 20/20 pass now, none of them previously executed by CI. Removed PROPOSAL_diffusion_driving_policy.md from repo root per review — belongs in Discussion #75, not a tracked file in this PR. Full re-run: Model/tests/test_trajectory_scorer.py (20), Model/tests/test_trajectory_planning.py + test_reasoning_coupling.py (51 passed, 1 GPU-only skip). ruff clean. Signed-off-by: FLagbusted <justthefourofus@proton.me>
… not guessed calibration (#161) #149/#148 landed via #161 (route-conditioned navigation inputs) while this PR was waiting on calibration. That changes what needs fixing here from 'verify a guessed calibration' to 'the input format itself changed': map_input (a rendered RGB image) is now map_context, a 14-channel semantic raster (Model.navigation.geometry.MapChannel) -- there is no pixel colour to fuzzy-match against any more. Per riita10069's comment on #17 (BEV segmentation labels are being redirected to a separate PandaSet pretraining track, not landing on KITScenes route-training soon), this can't wait for learned segmentation labels either -- rebuilt directly on what #161 already provides: - project_xy_to_bev_pixel now reimplements NavigationRasterGeometry.ego_to_pixel's exact formula (same signs, same -0.5 pixel-center offset) in torch, instead of a hand-derived formula against guessed constants. Cross-checked against the real geometry object directly in tests, not just self-consistency. - ScorerConfig's calibration fields are now sourced directly from Model.navigation.geometry.DEFAULT_NAVIGATION_GEOMETRY (geometry_id=kitscenes-v3-bev-1m-v1) -- the actual values map_context is rasterized with, not a placeholder. - drivable_area_compliance reads map_context's DRIVABLE_AREA channel (a real binary 0/1 mask, one of BINARY_MAP_CHANNELS) directly, replacing the RGB-tolerance colour-distance heuristic entirely. - Renamed the scorer's own map_input param to map_context throughout, matching ReactiveE2E.forward()'s naming post-#161. Rebased onto main post-#161 -- clean, no conflicts (this branch's only overlap with #161's changes was via flow_matching_planner.py, and sample_k_trajectories doesn't touch anything #161 changed). Tests rewritten against the new 14-channel shape and real geometry: added a direct cross-check against DEFAULT_NAVIGATION_GEOMETRY.ego_to_pixel on both the origin and random points (not just internal self-consistency), and a test proving compliance now reads the binary channel rather than any color-adjacent heuristic. 22/22 passing, full test_trajectory_planning.py still 38 passed/1 skipped. ruff clean. Signed-off-by: FLagbusted <justthefourofus@proton.me>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1, training-free upgrade to the flow-matching driving policy: multi-sample
scoring against the BEV/map data we already have. Implements #75.
What's in here
BasePlanner, re-ranks by drivable-area compliance + kinematic comfort.
draws per scene in one forward call.
Round 1 — riita10069's review, contract bug found
BasePlanner.forward() only ever returns one tensor. Never caught because the
test's FakePlanner matched the same wrong contract, and the whole test file
lived at repo-root tests/ — never collected by CI at all (make test only
globs Model/tests/). Fixed both the contract and a second latent bug (map
fixture off-by-one) this uncovered; moved the file into Model/tests/.
using each sample's real starting speed from egomotion_history instead of a
flat 5 m/s guess for every scene.
Round 2 — #161 landed, calibration redesigned on real geometry
#161 (route-conditioned navigation inputs) resolved #148/#149, but changed
more than the calibration: map_input (rendered RGB) doesn't exist any more —
map_context is a 14-channel semantic raster now. Separately, riita10069's
comment on #17 confirmed BEV segmentation labels are headed to a separate
PandaSet track, not landing on KITScenes route-training soon — so this
couldn't wait on that either. Rebuilt directly on what #161 already ships:
(Model.navigation.geometry.MapChannel, a real binary mask) instead of an
RGB-tolerance colour heuristic.
exact formula for batched torch use — cross-checked against the real
geometry object in tests, not just internal consistency.
Model.navigation.geometry.DEFAULT_NAVIGATION_GEOMETRY
(geometry_id="kitscenes-v3-bev-1m-v1", 256×256 @ 1.0 m/px) — Correct me If I am Wrong.
Testing done
test_trajectory_scorer.py — 22 tests, including a direct cross-check against
DEFAULT_NAVIGATION_GEOMETRY.ego_to_pixel and a test proving compliance reads
the binary channel, not anything colour-adjacent. test_trajectory_planning.py
still 38 passed/1 GPU-only skip. Rebased cleanly onto main post-#161 (only
overlap was flow_matching_planner.py, non-conflicting). ruff clean.
Checklist