fix(predict): co-locate load-added modules onto the model's device - #26
Merged
Conversation
A model's custom Network.load() may append checkpoint-sized modules -- e.g. a head sized from the checkpoint's class count -- after the model was already placed on its device. Those default to CPU, and because Network.to only descends into still-unplaced (gpu=="cpu") parents, a module added inside an already-placed parent is never moved, so the forward raises "Input type (...cuda) and weight type (...CPU) should be the same". After each per-model load in ModelComposite._ensure_model_loaded, re-home any fully-CPU leaf onto the device the rest of the model lives on; modules already on a device (including model-parallel splits) are left untouched. Reproduces the TotalSegmentator ensemble crash (Head.Conv added on CPU in load()); adds a regression test.
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a GPU crash for ensemble models whose custom
Network.load()appends modules at load time(e.g. TotalSegmentator, whose
load()adds aHead.Convsized from the checkpoint's class count).The bug
Network.to(network.py) places modules by walking the tree and moving each one whose placement markeris still
gpu == "cpu"— but it only descends into a child when that child itself is unplaced. So amodule added inside an already-placed parent (
Head) after the initial placement is never reached andstays on CPU. Since
ModelComposite._ensure_model_loadedre-runs the customload()per model swap, thefreshly-added head is recreated on CPU every time, and the forward pass crashes.
The reported error depends on the path: under
autocast: true(fp16) torch reports the dtype mismatchfirst —
— and in fp32 the device mismatch —
Both are the same root cause (a CPU-resident module receiving a GPU tensor); there is no separate
autocast bug.
The fix
After each per-model
load()inModelComposite._ensure_model_loaded, co-locate any fully-CPU leafonto the device the rest of the model already lives on. Modules already on a device — including
model-parallel splits across several GPUs — are left untouched. The base
nn.Module.parameters/buffersare used to read each module's own tensors, since
ModuleArgsDictoverrides those without arecursekwarg.Test plan
tests/unit/test_model_load_device.py: a model whoseload()appends a head is placed on the GPU,then asserted to have the head co-located (GPU-gated) + a CPU no-op guard.
test_predictor_memory.pyandtest_perf_hot_paths.py(both exerciseModelComposite) still pass.TotalSegmentator-KonfAI:total-3mmnow runs to completion on GPU underautocast: trueand writes a valid whole-body segmentation (59 labels present, range 0–117). Before this fix it crashed
on the first forward.