[None][fix] Log and validate transceiver runtime selection for hybrid managers#4
Open
lingjiew wants to merge 1 commit into
Conversation
… managers Two silent-selection gaps in create_kv_cache_transceiver: 1. When the KV manager is V2MambaHybridCacheManager, the factory overrides an explicit transceiver_runtime='CPP' and instantiates KvCacheTransceiverV2 with no message — the LlmArgs dump still shows runtime='CPP', which misleads debugging. Emit a warning when the explicit CPP request is overridden. 2. transceiver_runtime='PYTHON' combined with CppMambaHybridCacheManager (e.g. via TLLM_MAMBA_MANAGER_PREFERENCE=CPP) fails deep inside the python transceiver on an unsupported manager type. Fail fast with an actionable ValueError instead. Signed-off-by: Lingjie Wu <lingjiew@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
Two silent-selection gaps in
create_kv_cache_transceiver:Silent CPP override. With
V2MambaHybridCacheManager, the isinstance branch overrides an explicittransceiver_runtime='CPP'and instantiatesKvCacheTransceiverV2with no message. The LlmArgs dump in the worker log still printstransceiver_runtime='CPP', so the log actively misleads anyone debugging which transceiver ran (we lost time to exactly this on the Qwen3.5 GB300 bring-up — the config echo said CPP while the runtime was the python transceiver).Late failure on an unsupported pairing.
transceiver_runtime='PYTHON'combined withCppMambaHybridCacheManager(reachable viaTLLM_MAMBA_MANAGER_PREFERENCE=CPP) fails deep inside the python transceiver on an unsupported manager type instead of at selection time.Fix
logger.warningwhen an explicit'CPP'request is overridden by the V2-hybrid isinstance route.ValueErrorforPYTHON+CppMambaHybridCacheManagerat the factory, naming the two valid alternatives.No behavior change for any currently-working configuration — additions are a warning and an early error on a pairing that could not work anyway.
Related: #2, and a sibling PR removing the stale PYTHON->Mixed manager rule in
_util.py.🤖 Generated with Claude Code