…tch_model (#6) Resolves #2, closes #6. ## Why Upstream (OpenBMB#13 collaborator answer) confirms the demo needs ~28 GB of VRAM to run without runtime OOM, and that the demo does not natively integrate multi-GPU. On 2× RTX 3090 (24 GB each), single-GPU placement leaves ~290 MB free after model load, and Duplex prepare's `_reset_token2wav_for_new_turn` allocates 600 MB per fresh turn — the second turn races against an incomplete `finalize_unit` and the worker raises: RuntimeError: streaming_prefill called before finalize_unit()! Same symptom as upstream OPEN issues OpenBMB#28 (no-feedback turns), OpenBMB#30 (audio output stutter on Nvidia L20), OpenBMB#29 (concurrency). ## What `core/processors/unified.py:1376` — when `torch.cuda.device_count() > 1`, build a `device_map` via `accelerate.infer_auto_device_map` with asymmetric `max_memory={0: "8GiB", 1: "22GiB"}` and dispatch the model across both GPUs. Pin the omni-specific modules (`vpm`, `apm`, `tts`, `token2wav`, `resampler`, `audio_tokenizer`, `embed_tokens`, `lm_head`) to device 0 because: - `init_token2wav` calls `self.flow.cuda()` unconditionally — it must land on device 0 or we end up duplicating it - `DuplexCapability(device=...)` only accepts a single-device string - the small omni encoders are touched on every input/output forward — cross-device transfer would dominate latency The asymmetric cap (8 GB on device 0) is necessary: with a symmetric 22 GiB cap, accelerate decides everything fits on device 0 and emits `{'': 0}` (single-device map). 8 GB forces the LLM decoder layers to spill into device 1. ## Verified Single worker (PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, CUDA_VISIBLE_DEVICES=0,1): - device_map: 46 entries, 5 pinned-to-0 - GPU 0 ≈ 15.2 GB (omni modules + LLM layers 0–4 + lm_head) - GPU 1 ≈ 8.1 GB (LLM layers 5–end + norm) - Multi-turn duplex: 25 `> speak:` tokens across multiple turns, **0 `streaming_prefill before finalize_unit` errors** - Phase 1 acceptance: single-turn audio playback (Playwright) AND multi-turn live conversation both pass ## Trade-off Cross-device tensor transfer adds ~1-2 ms per forward token. Late turns in a long conversation feel slightly slower than on a single >28 GB GPU. This is the cost of running the demo on under-spec hardware; latency remains acceptable for reference-quality use. Production use should prefer a single 28 GB+ GPU or vllm-omni (paged attention). ## Other files in this commit - `ROADMAP.md` — phase plan + issue links (#1 epic, #2-#6 children) - `pytest.ini` — `phase1` / `phase2` / `phase3` markers + asyncio_mode - `tests/helpers/__init__.py`, `tests/helpers/phase_adapters.py` — cross-phase test infrastructure (`PhaseAdapter`, `Phase1HFAdapter`, `Phase2VllmOmniAdapter`, `Phase3NaiaTsClientAdapter`) - `tests/test_cross_phase_consistency.py` — pytest entry, reachability skips per phase - `tests/cases/cross_phase/multi_turn_image_audio_en.json` — scenario - `tests/cases/cross_phase/fixtures/turn1_image.png`, `turn1_user_audio.wav`, `turn2_user_audio.wav` — fixtures

Uh oh!

全双工语音输出有卡顿 #30

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions