perf: keep the LoGeR input batch on the host

[HuguesSib]

Closes #19. Thanks @Matt-Tav for the report and the diagnosis, which was correct.

Why

process_sequence preloaded the entire resized sequence onto the GPU:

batch_t = torch.from_numpy(batch).permute(0, 3, 1, 2).unsqueeze(0).to(self._device)

That was never needed. Pi3.forward slices the sequence into sliding windows and moves each one to the model's device itself:

# third_party/LoGeR/loger/models/pi3.py:671
imgs_w = imgs[:, start_idx:end_idx]
imgs_w = imgs_w.to(self.image_mean.device)

image_mean is a registered buffer, so it tracks model.to(device) and the window always lands on the right device. Only the window being decoded ever had to be resident. The rest of the sequence just held VRAM.

No submodule bump is required. The per-window .to() is already present at our pin (13d78a0). The upstream commit linked in the issue (7685b7a) touches only demo_viser.py, and applies the same one-line change to LoGeR's own demo script.

What

• Drop the .to(self._device) on the input batch.
• Remove the now-dead batch_t = batch_t.cpu() from the MPS fallback. Moving the model is what that path needs; the batch is already on the host.
• Add test_loger_hands_the_model_a_host_resident_batch, which records Tensor.to calls on the 5-D batch and asserts the model receives it on CPU. Verified to fail if the preload is restored.

Measurements

RTX 3090, 600 frames at the default 504x280, window_size=32, overlap_size=3.

	peak allocated	peak reserved	inference	output sha256
before	12561 MiB	13594 MiB	59.5 s	bbf026eb0989914c
after	11587 MiB	12630 MiB	58.8 s	bbf026eb0989914c

The 974 MiB saving is exactly the batch tensor (600 x 504 x 280 x 3 x 4 B = 969 MiB), so it scales linearly at 1.6 MiB/frame. A 1500-frame run saves about 2.4 GiB.

The checksum covers depth_maps, poses_w_c, local_points, world_points and confidence. It is unchanged, so this shifts no numeric output and the changelog entry sits under Fixed rather than Changed.

Inference time is unchanged, confirming the report's observation that the per-window host-to-device copy is fully hidden behind compute.

On .pin_memory()

The issue also suggested pinning the batch. I left it out. The transfer at pi3.py:671 is synchronous (no non_blocking=True), so pinning cannot overlap it with compute, and the timing above shows nothing to recover. Page-locking a multi-GB host tensor is also a poor trade on the low-RAM field machines this tool targets. Worth revisiting only alongside an async non_blocking=True copy.

What this does not fix

The input batch is only about 12% of the per-frame VRAM cost. Peak allocated decomposes as:

term	600-frame size	scaling
Pi3 weights	4786 MiB	fixed
input batch	969 MiB	1.6 MiB/frame (removed here)
window predictions held on GPU	6801 MiB	11.3 MiB/frame

The third term is the real cap on sequence length. _merge_windowed_predictions keeps every window's points, local_points and conf on the device until the merge at the end. Offloading each window's predictions to the host as it completes is the larger win, and is a separate change. Filing that as a follow-up issue.

🤖 Generated with Claude Code