Skip to content

feat(training): let the camera BEV grid be set, so a consumer GPU can train - #188

Open
gcordova10 wants to merge 1 commit into
autowarefoundation:mainfrom
gcordova10:feat/configurable-camera-bev-grid
Open

feat(training): let the camera BEV grid be set, so a consumer GPU can train#188
gcordova10 wants to merge 1 commit into
autowarefoundation:mainfrom
gcordova10:feat/configurable-camera-bev-grid

Conversation

@gcordova10

Copy link
Copy Markdown
Contributor

Problem

The KITScenes navigation geometry pins the camera BEV grid at 256x256 and train_il
exposes no way to change it. That allocates 65,536 BEV queries, which does not fit in
6 GB of VRAM: measured on an RTX 3060, train_il runs out of memory at
batch_size 1 before finishing an epoch, and
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True — which the traceback suggests, and
which has helped on a 12 GB card — does not close the gap. The only workaround today
is editing the code, and contributors have been doing exactly that.

Fix

train_il and wf_train_il take an optional camera_bev_size. Left unset the run is
unchanged, so this is opt-in and no existing configuration moves.

Set, it resizes the grid and nothing else. pc_range is deliberately untouched, so
the BEV still covers the ground area the geometry defines and the map BEV stays
aligned with it — the cells are coarser, not fewer over a smaller patch. The resize is
a small pure helper rather than inline logic so that property is directly testable.

Verification

  • Run on the frozen protocol split, on the 6 GB card that cannot hold the default
    grid: with --camera_bev_size 64 an epoch over all 404 validation-eligible
    partitions completed in 2 h 20 min and uploaded its checkpoint. The run logs the
    override it applied, so the effect is checkable in the output rather than inferred:

    Camera BEV grid overridden: 64x64 (4096 cells; geometry default is 256x256)
    

    It fits, but not with room to spare: peak 5.7 GB of the 6.1 available, with
    recoverable memory allocation failed with OOM warnings from the caching allocator
    along the way. Worth stating, because "it fits" and "it fits comfortably" are
    different claims and only the first is supported.

  • Whether the coarser grid costs accuracy was measured separately on a 40-scene
    bring-up subset, three seeds per configuration, changing only this parameter:
    65,536 / 4,096 / 64 cells gave ADE ranges that all overlap, so no difference was
    measurable. That is a bring-up subset rather than the protocol split, so please read
    it as a direction and not a number — but it means shrinking the grid to fit a card
    is not obviously trading accuracy for memory.

  • Three tests, five cases with the parametrisation: resizing changes bev_h/bev_w
    and nothing else in the kwargs (verified by diffing the whole dict), the caller's
    dict is not mutated, non-positive sizes raise, and the parameter stays opt-in.
    ruff check and cd Model && mypy . clean; pytest Model/tests/test_workflow_training_lifecycle.py — 47 passed.

This is orthogonal to the deformable map fusion that just landed. That makes the
map-to-BEV attention affordable at the current grid; this makes the grid itself
settable, for any fusion mode, when the 65,536 camera BEV queries are what does not
fit.

Companion PR: #187 adds the corpus tooling and a local-training guide. That guide
currently has to tell readers with a small card that their only option is editing the
code; this is what removes that. They are independent — each is green on its own and
neither depends on the other landing.

… train

The KITScenes navigation geometry pins the camera BEV grid at 256x256, and
train_il exposes no way to change it. That allocates 65,536 BEV queries, which
does not fit in 6 GB of VRAM: measured on an RTX 3060 with 6 GB, train_il runs
out of memory at batch_size 1 before finishing an epoch, and
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -- which the traceback suggests
and which has helped on a 12 GB card -- does not close the gap. The workaround
available today is editing the code, and contributors have been doing exactly
that.

train_il and wf_train_il now take an optional camera_bev_size. Left unset the
run is unchanged, so this is opt-in and no existing configuration moves. Set,
it resizes the grid and nothing else: pc_range is deliberately untouched, so the
BEV still covers the ground area the geometry defines and the map BEV stays
aligned with it -- the cells are coarser, not fewer over a smaller patch. The
resize is a small pure helper rather than inline logic so that property is
directly testable.

Measured on the same 6 GB card: at 64x64 an epoch completes and the checkpoint
uploads. Whether the coarser grid costs accuracy was measured separately on a
40-scene bring-up subset, three seeds per configuration, changing only this
parameter: 65,536 / 4,096 / 64 cells gave ADE ranges that all overlap, so no
difference was measurable. That is a bring-up subset rather than the protocol
split, so it is a direction and not a number -- but it means shrinking the grid
to fit a card is not obviously trading accuracy for memory.

Three tests, none of which pass by accident: resizing changes bev_h/bev_w and
nothing else in the kwargs (verified by diffing the whole dict, and confirmed to
fail if pc_range is moved), the caller's dict is not mutated, non-positive sizes
raise, and the parameter stays opt-in.

Signed-off-by: GABRIELA CORDOVA <100548769@alumnos.uc3m.es>
@riita10069

Copy link
Copy Markdown
Collaborator

Setting the BEV features to 64 is far too small. At this point, it is not at a memory scale that would cause OOM on a single node. Model parameters should not be changed dynamically.

@gcordova10

Copy link
Copy Markdown
Contributor Author

Thanks for looking. One measurement in case it is useful, since the target here is consumer hardware rather than a node:

On an RTX 3060 Laptop (6 GB), train_il at batch_size 1 with the default 256x256 grid goes out of memory before finishing an epoch, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True does not close the gap. At 64 an epoch over the frozen 404-partition split completes in 2 h 20 min, peaking at 5.7 GB of the 6.1 available — so it fits, but without much headroom.

That said, I take the point that architecture should not be a CLI parameter, and I have no evidence that 64 is the right value rather than just a value that fits. Now that #184 has landed, is deformable fusion the intended answer for small cards? If so this is not needed and I will close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants