feat(training): let the camera BEV grid be set, so a consumer GPU can train - #188
feat(training): let the camera BEV grid be set, so a consumer GPU can train#188gcordova10 wants to merge 1 commit into
Conversation
… train The KITScenes navigation geometry pins the camera BEV grid at 256x256, and train_il exposes no way to change it. That allocates 65,536 BEV queries, which does not fit in 6 GB of VRAM: measured on an RTX 3060 with 6 GB, train_il runs out of memory at batch_size 1 before finishing an epoch, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -- which the traceback suggests and which has helped on a 12 GB card -- does not close the gap. The workaround available today is editing the code, and contributors have been doing exactly that. train_il and wf_train_il now take an optional camera_bev_size. Left unset the run is unchanged, so this is opt-in and no existing configuration moves. Set, it resizes the grid and nothing else: pc_range is deliberately untouched, so the BEV still covers the ground area the geometry defines and the map BEV stays aligned with it -- the cells are coarser, not fewer over a smaller patch. The resize is a small pure helper rather than inline logic so that property is directly testable. Measured on the same 6 GB card: at 64x64 an epoch completes and the checkpoint uploads. Whether the coarser grid costs accuracy was measured separately on a 40-scene bring-up subset, three seeds per configuration, changing only this parameter: 65,536 / 4,096 / 64 cells gave ADE ranges that all overlap, so no difference was measurable. That is a bring-up subset rather than the protocol split, so it is a direction and not a number -- but it means shrinking the grid to fit a card is not obviously trading accuracy for memory. Three tests, none of which pass by accident: resizing changes bev_h/bev_w and nothing else in the kwargs (verified by diffing the whole dict, and confirmed to fail if pc_range is moved), the caller's dict is not mutated, non-positive sizes raise, and the parameter stays opt-in. Signed-off-by: GABRIELA CORDOVA <100548769@alumnos.uc3m.es>
|
Setting the BEV features to 64 is far too small. At this point, it is not at a memory scale that would cause OOM on a single node. Model parameters should not be changed dynamically. |
|
Thanks for looking. One measurement in case it is useful, since the target here is consumer hardware rather than a node: On an RTX 3060 Laptop (6 GB), train_il at batch_size 1 with the default 256x256 grid goes out of memory before finishing an epoch, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True does not close the gap. At 64 an epoch over the frozen 404-partition split completes in 2 h 20 min, peaking at 5.7 GB of the 6.1 available — so it fits, but without much headroom. That said, I take the point that architecture should not be a CLI parameter, and I have no evidence that 64 is the right value rather than just a value that fits. Now that #184 has landed, is deformable fusion the intended answer for small cards? If so this is not needed and I will close it. |
Problem
The KITScenes navigation geometry pins the camera BEV grid at 256x256 and
train_ilexposes no way to change it. That allocates 65,536 BEV queries, which does not fit in
6 GB of VRAM: measured on an RTX 3060,
train_ilruns out of memory atbatch_size 1before finishing an epoch, andPYTORCH_CUDA_ALLOC_CONF=expandable_segments:True— which the traceback suggests, andwhich has helped on a 12 GB card — does not close the gap. The only workaround today
is editing the code, and contributors have been doing exactly that.
Fix
train_ilandwf_train_iltake an optionalcamera_bev_size. Left unset the run isunchanged, so this is opt-in and no existing configuration moves.
Set, it resizes the grid and nothing else.
pc_rangeis deliberately untouched, sothe BEV still covers the ground area the geometry defines and the map BEV stays
aligned with it — the cells are coarser, not fewer over a smaller patch. The resize is
a small pure helper rather than inline logic so that property is directly testable.
Verification
Run on the frozen protocol split, on the 6 GB card that cannot hold the default
grid: with
--camera_bev_size 64an epoch over all 404 validation-eligiblepartitions completed in 2 h 20 min and uploaded its checkpoint. The run logs the
override it applied, so the effect is checkable in the output rather than inferred:
It fits, but not with room to spare: peak 5.7 GB of the 6.1 available, with
recoverable
memory allocation failed with OOMwarnings from the caching allocatoralong the way. Worth stating, because "it fits" and "it fits comfortably" are
different claims and only the first is supported.
Whether the coarser grid costs accuracy was measured separately on a 40-scene
bring-up subset, three seeds per configuration, changing only this parameter:
65,536 / 4,096 / 64 cells gave ADE ranges that all overlap, so no difference was
measurable. That is a bring-up subset rather than the protocol split, so please read
it as a direction and not a number — but it means shrinking the grid to fit a card
is not obviously trading accuracy for memory.
Three tests, five cases with the parametrisation: resizing changes
bev_h/bev_wand nothing else in the kwargs (verified by diffing the whole dict), the caller's
dict is not mutated, non-positive sizes raise, and the parameter stays opt-in.
ruff checkandcd Model && mypy .clean;pytest Model/tests/test_workflow_training_lifecycle.py— 47 passed.This is orthogonal to the deformable map fusion that just landed. That makes the
map-to-BEV attention affordable at the current grid; this makes the grid itself
settable, for any fusion mode, when the 65,536 camera BEV queries are what does not
fit.
Companion PR: #187 adds the corpus tooling and a local-training guide. That guide
currently has to tell readers with a small card that their only option is editing the
code; this is what removes that. They are independent — each is green on its own and
neither depends on the other landing.