perf: default Xavier PD to GPU handoff with tiered KV history

.github/workflows/pd-gpu.yaml

@@ -41,11 +41,11 @@ jobs:
         run: |
           python -m venv .venv-pd
           .venv-pd/bin/python -m pip install -U pip 'setuptools<82'
-          .venv-pd/bin/python -m pip install -e '.[dev]' "vllm==$VLLM_VERSION" "nixl==$NIXL_VERSION" openai
+          .venv-pd/bin/python -m pip install -e '.[dev]' "vllm==$VLLM_VERSION" "nixl==$NIXL_VERSION" "xoscar[nixl]>=0.11.1" openai
       - name: Verify PD with actual KV transfer
         run: |
           source .venv-pd/bin/activate
-          python -m pytest --timeout=1800 -vv xinference/model/llm/vllm/xavier/test/test_pd_gpu.py xinference/model/llm/vllm/xavier/test/test_v1_cache_layout.py
+          python -m pytest --timeout=1800 -vv xinference/model/llm/vllm/xavier/test/test_pd_gpu.py xinference/model/llm/vllm/xavier/test/test_v1_cache_layout.py xinference/model/llm/vllm/xavier/test/test_direct_history.py
       - name: Upload runtime logs
         if: always()
         uses: actions/upload-artifact@v4

.github/workflows/python.yaml

@@ -380,6 +380,8 @@ jobs:
               xinference/model/llm/vllm/xavier/test/test_v1_engine.py \
               xinference/model/llm/vllm/xavier/test/test_gpu_transfer.py \
               xinference/model/llm/vllm/xavier/test/test_kv_schema.py \
+              xinference/model/llm/vllm/xavier/test/test_direct_handoff.py \
+              xinference/model/llm/vllm/xavier/test/test_direct_history.py \
               xinference/model/llm/vllm/xavier/test/test_tiered_snapshot.py \
               xinference/model/llm/vllm/xavier/test/test_request_transfer.py \
               xinference/model/llm/vllm/xavier/test/test_transfer_blocks.py \

doc/source/locale/de/LC_MESSAGES/user_guide/pd_separation.po

@@ -299,3 +299,84 @@ msgstr ""
 #: ../../source/user_guide/pd_separation.rst:145
 msgid "Transfer failures and incompatible KV cache layouts raise errors instead of silently recomputing the request."
 msgstr "Übertragungsfehler und inkompatible KV-Cache-Layouts lösen Fehler aus, statt die Anfrage stillschweigend neu zu berechnen."
+
+#: user_guide/pd_separation.rst:1
+msgid ""
+"Xavier P/D uses GPU-to-GPU handoff by default. It requires Linux, NVIDIA "
+"GPUs, vLLM >= 0.21.0 and ``xoscar[nixl]>=0.11.1`` in both worker and model "
+"environments. Use a reachable host address (not ``0.0.0.0``). Missing NIXL "
+"fails the launch; there is no CPU fallback."
+msgstr ""
+"Xavier P/D verwendet standardmäßig direkte GPU-zu-GPU-Übertragung. "
+"Erforderlich sind Linux, NVIDIA-GPUs, vLLM >= 0.21.0 und "
+"``xoscar[nixl]>=0.11.1`` in der Worker- und Modellumgebung. Verwenden Sie "
+"eine erreichbare Hostadresse (nicht ``0.0.0.0``). Ohne NIXL schlägt der "
+"Start fehl; es gibt keinen CPU-Fallback."
+
+
+#: user_guide/pd_separation.rst:1
+msgid ""
+"``xavier_gpu_cache_bytes`` defaults to ``268435456`` (256 MiB per replica) "
+"for P/D. History uses GPU first and spills to CPU when GPU capacity is "
+"exhausted. CPU history is bounded by the engine KV block capacity and can "
+"approach that cache size in host memory. CPU hits restore KV to GPU; misses "
+"compute locally. ``0`` disables history, retaining direct transfer. Transfer"
+" failures raise errors."
+msgstr ""
+"``xavier_gpu_cache_bytes`` beträgt für P/D standardmäßig ``268435456`` (256 "
+"MiB pro Replikat). Der Verlauf nutzt zuerst die GPU und wird bei erschöpfter"
+" GPU-Kapazität auf die CPU ausgelagert. Der CPU-Verlauf ist durch die KV-"
+"Blockkapazität der Engine begrenzt und kann im Hauptspeicher deren Cache-"
+"Größe erreichen. CPU-Treffer stellen KV auf der GPU wieder her; Misses "
+"werden lokal berechnet. ``0`` deaktiviert den Verlauf, behält aber die "
+"direkte Übertragung bei. Übertragungsfehler lösen Fehler aus."
+
+
+#: user_guide/pd_separation.rst:1
+msgid "Direct handoff and tiered history"
+msgstr "Direkte Übertragung und gestufter Verlauf"
+
+
+#: user_guide/pd_separation.rst:194
+msgid "Direct handoff currently requires ``n=1``."
+msgstr "Die direkte Übertragung erfordert derzeit ``n=1``."
+
+
+#: user_guide/pd_separation.rst
+msgid ""
+"History is restored on P before the handoff to D. Long prompts may retain "
+"only a prefix; cache retention is best effort."
+msgstr ""
+"Der Verlauf wird vor der Übergabe an D auf P wiederhergestellt. Bei langen "
+"Prompts wird möglicherweise nur ein Präfix behalten; die Aufbewahrung aller "
+"Cache-Daten ist nicht garantiert."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Unclaimed handoffs expire after 120 seconds. If a handoff expires before D "
+"accepts it, D recomputes locally. Once D accepts it, the source cache stays "
+"reserved until the load finishes or the request is cancelled before loading."
+msgstr ""
+"Nicht angenommene Übergaben verfallen nach 120 Sekunden. Verfällt eine "
+"Übergabe, bevor D sie annimmt, berechnet D lokal neu. Nach der Annahme "
+"bleibt der Quellcache reserviert, bis das Laden abgeschlossen ist oder die "
+"Anfrage vor dem Laden abgebrochen wird."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Direct transfers and history loads can overlap with response generation for "
+"other requests."
+msgstr ""
+"Direkte Übertragungen und das Laden des Verlaufs können parallel zur "
+"Antwortgenerierung anderer Anfragen erfolgen."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Unclaimed handoffs expire after 120 seconds. Claimed handoffs expire after "
+"10 minutes without a scheduling retry or transfer progress. Active copies "
+"finish before release. Expired handoffs are recomputed locally on D."
+msgstr ""
+"Nicht angenommene Übergaben verfallen nach 120 Sekunden. Angenommene "
+"Übergaben verfallen nach 10 Minuten ohne erneuten Planungsversuch oder "
+"Übertragungsfortschritt. Laufende Kopien werden vor der Freigabe "
+"abgeschlossen. Verfallene Übergaben werden lokal auf D neu berechnet."

doc/source/locale/es/LC_MESSAGES/user_guide/pd_separation.po

@@ -297,3 +297,83 @@ msgstr ""
 #: ../../source/user_guide/pd_separation.rst:145
 msgid "Transfer failures and incompatible KV cache layouts raise errors instead of silently recomputing the request."
 msgstr "Los fallos de transferencia y los formatos de caché KV incompatibles generan errores en lugar de volver a calcular la solicitud silenciosamente."
+
+#: user_guide/pd_separation.rst:1
+msgid ""
+"Xavier P/D uses GPU-to-GPU handoff by default. It requires Linux, NVIDIA "
+"GPUs, vLLM >= 0.21.0 and ``xoscar[nixl]>=0.11.1`` in both worker and model "
+"environments. Use a reachable host address (not ``0.0.0.0``). Missing NIXL "
+"fails the launch; there is no CPU fallback."
+msgstr ""
+"Xavier P/D utiliza por defecto transferencia directa entre GPU. Requiere "
+"Linux, GPU NVIDIA, vLLM >= 0.21.0 y ``xoscar[nixl]>=0.11.1`` en los entornos"
+" del worker y del modelo. Use una dirección de host accesible (no "
+"``0.0.0.0``). Sin NIXL, el inicio falla; no hay alternativa por CPU."
+
+
+#: user_guide/pd_separation.rst:1
+msgid ""
+"``xavier_gpu_cache_bytes`` defaults to ``268435456`` (256 MiB per replica) "
+"for P/D. History uses GPU first and spills to CPU when GPU capacity is "
+"exhausted. CPU history is bounded by the engine KV block capacity and can "
+"approach that cache size in host memory. CPU hits restore KV to GPU; misses "
+"compute locally. ``0`` disables history, retaining direct transfer. Transfer"
+" failures raise errors."
+msgstr ""
+"Para P/D, ``xavier_gpu_cache_bytes`` vale por defecto ``268435456`` (256 MiB"
+" por réplica). El historial usa primero GPU y pasa a CPU cuando se agota su "
+"capacidad. El historial CPU está limitado por la capacidad de bloques KV del"
+" motor y puede ocupar un tamaño similar en memoria del host. Los aciertos "
+"CPU restauran KV en GPU; los fallos se calculan localmente. ``0`` desactiva "
+"el historial y mantiene la transferencia directa. Los fallos de "
+"transferencia generan errores."
+
+
+#: user_guide/pd_separation.rst:1
+msgid "Direct handoff and tiered history"
+msgstr "Transferencia directa e historial por niveles"
+
+
+#: user_guide/pd_separation.rst:194
+msgid "Direct handoff currently requires ``n=1``."
+msgstr "La transferencia directa requiere actualmente ``n=1``."
+
+
+#: user_guide/pd_separation.rst
+msgid ""
+"History is restored on P before the handoff to D. Long prompts may retain "
+"only a prefix; cache retention is best effort."
+msgstr ""
+"El historial se restaura en P antes de transferirlo a D. Los prompts largos "
+"pueden conservar solo un prefijo; no se garantiza la retención de toda la "
+"caché."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Unclaimed handoffs expire after 120 seconds. If a handoff expires before D "
+"accepts it, D recomputes locally. Once D accepts it, the source cache stays "
+"reserved until the load finishes or the request is cancelled before loading."
+msgstr ""
+"Las transferencias no aceptadas caducan tras 120 segundos. Si caducan antes "
+"de que D las acepte, D vuelve a calcular localmente. Tras la aceptación, la "
+"caché de origen permanece reservada hasta que termine la carga o se cancele "
+"la solicitud antes de cargar."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Direct transfers and history loads can overlap with response generation for "
+"other requests."
+msgstr ""
+"Las transferencias directas y las cargas del historial pueden solaparse con "
+"la generación de respuestas de otras solicitudes."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Unclaimed handoffs expire after 120 seconds. Claimed handoffs expire after "
+"10 minutes without a scheduling retry or transfer progress. Active copies "
+"finish before release. Expired handoffs are recomputed locally on D."
+msgstr ""
+"Las transferencias no aceptadas caducan tras 120 segundos. Las aceptadas "
+"caducan tras 10 minutos sin reintentos de planificación ni progreso de "
+"transferencia. Las copias en curso terminan antes de liberar los recursos. D"
+" vuelve a calcular localmente las transferencias caducadas."

doc/source/locale/fr/LC_MESSAGES/user_guide/pd_separation.po

@@ -300,3 +300,84 @@ msgstr ""
 #: ../../source/user_guide/pd_separation.rst:145
 msgid "Transfer failures and incompatible KV cache layouts raise errors instead of silently recomputing the request."
 msgstr "Les échecs de transfert et les dispositions incompatibles du cache KV provoquent des erreurs au lieu de recalculer silencieusement la requête."
+
+#: user_guide/pd_separation.rst:1
+msgid ""
+"Xavier P/D uses GPU-to-GPU handoff by default. It requires Linux, NVIDIA "
+"GPUs, vLLM >= 0.21.0 and ``xoscar[nixl]>=0.11.1`` in both worker and model "
+"environments. Use a reachable host address (not ``0.0.0.0``). Missing NIXL "
+"fails the launch; there is no CPU fallback."
+msgstr ""
+"Xavier P/D utilise par défaut le transfert direct entre GPU. Il nécessite "
+"Linux, des GPU NVIDIA, vLLM >= 0.21.0 et ``xoscar[nixl]>=0.11.1`` dans les "
+"environnements du worker et du modèle. Utilisez une adresse hôte accessible "
+"(pas ``0.0.0.0``). Sans NIXL, le démarrage échoue ; aucun repli CPU n’est "
+"prévu."
+
+
+#: user_guide/pd_separation.rst:1
+msgid ""
+"``xavier_gpu_cache_bytes`` defaults to ``268435456`` (256 MiB per replica) "
+"for P/D. History uses GPU first and spills to CPU when GPU capacity is "
+"exhausted. CPU history is bounded by the engine KV block capacity and can "
+"approach that cache size in host memory. CPU hits restore KV to GPU; misses "
+"compute locally. ``0`` disables history, retaining direct transfer. Transfer"
+" failures raise errors."
+msgstr ""
+"Pour P/D, ``xavier_gpu_cache_bytes`` vaut par défaut ``268435456`` (256 MiB "
+"par réplique). L’historique utilise d’abord le GPU puis déborde vers le CPU "
+"lorsque sa capacité est épuisée. L’historique CPU est limité par la capacité"
+" en blocs KV du moteur et peut approcher cette taille de cache en mémoire "
+"hôte. Les accès CPU restaurent les KV sur le GPU ; les défauts sont calculés"
+" localement. ``0`` désactive l’historique tout en conservant le transfert "
+"direct. Les échecs de transfert produisent des erreurs."
+
+
+#: user_guide/pd_separation.rst:1
+msgid "Direct handoff and tiered history"
+msgstr "Transfert direct et historique hiérarchisé"
+
+
+#: user_guide/pd_separation.rst:194
+msgid "Direct handoff currently requires ``n=1``."
+msgstr "Le transfert direct nécessite actuellement ``n=1``."
+
+
+#: user_guide/pd_separation.rst
+msgid ""
+"History is restored on P before the handoff to D. Long prompts may retain "
+"only a prefix; cache retention is best effort."
+msgstr ""
+"L’historique est restauré sur P avant le transfert vers D. Pour les longs "
+"prompts, seul un préfixe peut être conservé ; la conservation de tout le "
+"cache n’est pas garantie."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Unclaimed handoffs expire after 120 seconds. If a handoff expires before D "
+"accepts it, D recomputes locally. Once D accepts it, the source cache stays "
+"reserved until the load finishes or the request is cancelled before loading."
+msgstr ""
+"Les transferts non acceptés expirent après 120 secondes. Si un transfert "
+"expire avant que D ne l’accepte, D recalcule localement. Après acceptation, "
+"le cache source reste réservé jusqu’à la fin du chargement ou à l’annulation"
+" de la requête avant le chargement."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Direct transfers and history loads can overlap with response generation for "
+"other requests."
+msgstr ""
+"Les transferts directs et le chargement de l’historique peuvent se dérouler "
+"en parallèle de la génération des réponses à d’autres requêtes."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Unclaimed handoffs expire after 120 seconds. Claimed handoffs expire after "
+"10 minutes without a scheduling retry or transfer progress. Active copies "
+"finish before release. Expired handoffs are recomputed locally on D."
+msgstr ""
+"Les transferts non acceptés expirent après 120 secondes. Les transferts "
+"acceptés expirent après 10 minutes sans nouvelle tentative de planification "
+"ni progression du transfert. Les copies en cours se terminent avant la "
+"libération. Les transferts expirés sont recalculés localement sur D."

doc/source/locale/it/LC_MESSAGES/user_guide/pd_separation.po

@@ -297,3 +297,84 @@ msgstr ""
 #: ../../source/user_guide/pd_separation.rst:145
 msgid "Transfer failures and incompatible KV cache layouts raise errors instead of silently recomputing the request."
 msgstr "Gli errori di trasferimento e i layout incompatibili della cache KV generano errori anziché ricalcolare silenziosamente la richiesta."
+
+#: user_guide/pd_separation.rst:1
+msgid ""
+"Xavier P/D uses GPU-to-GPU handoff by default. It requires Linux, NVIDIA "
+"GPUs, vLLM >= 0.21.0 and ``xoscar[nixl]>=0.11.1`` in both worker and model "
+"environments. Use a reachable host address (not ``0.0.0.0``). Missing NIXL "
+"fails the launch; there is no CPU fallback."
+msgstr ""
+"Xavier P/D usa per impostazione predefinita il trasferimento diretto tra "
+"GPU. Richiede Linux, GPU NVIDIA, vLLM >= 0.21.0 e ``xoscar[nixl]>=0.11.1`` "
+"negli ambienti del worker e del modello. Usare un indirizzo host "
+"raggiungibile (non ``0.0.0.0``). Senza NIXL, l’avvio fallisce; non è "
+"previsto il ripiego sulla CPU."
+
+
+#: user_guide/pd_separation.rst:1
+msgid ""
+"``xavier_gpu_cache_bytes`` defaults to ``268435456`` (256 MiB per replica) "
+"for P/D. History uses GPU first and spills to CPU when GPU capacity is "
+"exhausted. CPU history is bounded by the engine KV block capacity and can "
+"approach that cache size in host memory. CPU hits restore KV to GPU; misses "
+"compute locally. ``0`` disables history, retaining direct transfer. Transfer"
+" failures raise errors."
+msgstr ""
+"Per P/D, ``xavier_gpu_cache_bytes`` vale per impostazione predefinita "
+"``268435456`` (256 MiB per replica). La cronologia usa prima la GPU e passa "
+"alla CPU quando la capacità GPU è esaurita. La cronologia CPU è limitata "
+"dalla capacità dei blocchi KV del motore e può avvicinarsi a tale dimensione"
+" della cache nella memoria host. Gli hit CPU ripristinano KV sulla GPU; i "
+"miss vengono calcolati localmente. ``0`` disabilita la cronologia mantenendo"
+" il trasferimento diretto. I guasti di trasferimento generano errori."
+
+
+#: user_guide/pd_separation.rst:1
+msgid "Direct handoff and tiered history"
+msgstr "Trasferimento diretto e cronologia a livelli"
+
+
+#: user_guide/pd_separation.rst:194
+msgid "Direct handoff currently requires ``n=1``."
+msgstr "Il trasferimento diretto richiede attualmente ``n=1``."
+
+
+#: user_guide/pd_separation.rst
+msgid ""
+"History is restored on P before the handoff to D. Long prompts may retain "
+"only a prefix; cache retention is best effort."
+msgstr ""
+"La cronologia viene ripristinata su P prima del trasferimento a D. Per i "
+"prompt lunghi può essere conservato solo un prefisso; la conservazione di "
+"tutta la cache non è garantita."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Unclaimed handoffs expire after 120 seconds. If a handoff expires before D "
+"accepts it, D recomputes locally. Once D accepts it, the source cache stays "
+"reserved until the load finishes or the request is cancelled before loading."
+msgstr ""
+"I trasferimenti non accettati scadono dopo 120 secondi. Se un trasferimento "
+"scade prima che D lo accetti, D ricalcola localmente. Dopo l’accettazione, "
+"la cache di origine rimane riservata fino al completamento del caricamento o"
+" all’annullamento della richiesta prima del caricamento."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Direct transfers and history loads can overlap with response generation for "
+"other requests."
+msgstr ""
+"I trasferimenti diretti e il caricamento della cronologia possono avvenire "
+"in parallelo alla generazione delle risposte ad altre richieste."
+
+#: user_guide/pd_separation.rst
+msgid ""
+"Unclaimed handoffs expire after 120 seconds. Claimed handoffs expire after "
+"10 minutes without a scheduling retry or transfer progress. Active copies "
+"finish before release. Expired handoffs are recomputed locally on D."
+msgstr ""
+"I trasferimenti non accettati scadono dopo 120 secondi. Quelli accettati "
+"scadono dopo 10 minuti senza nuovi tentativi di pianificazione o avanzamento"
+" del trasferimento. Le copie in corso terminano prima del rilascio. I "
+"trasferimenti scaduti vengono ricalcolati localmente su D."