From b2c1cb29a13a825f9df67ed44d35f93f3161a023 Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Wed, 22 Jul 2026 10:52:09 -0700 Subject: [PATCH 1/7] docs: Update LLM getting-started guide to PyTorch backend (DeepSeek-V4-Flash) Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- docs/getting_started/llm.md | 1802 +++-------------------------------- 1 file changed, 112 insertions(+), 1690 deletions(-) diff --git a/docs/getting_started/llm.md b/docs/getting_started/llm.md index 2ae5f82290..ccf586035a 100644 --- a/docs/getting_started/llm.md +++ b/docs/getting_started/llm.md @@ -26,1716 +26,138 @@ # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. --> -# Deploying Phi-3 Model with Triton and TRT-LLM - -This guide captures the steps to build Phi-3 with TRT-LLM and deploy with Triton Inference Server. It also shows a shows how to use GenAI-Perf to run benchmarks to measure model performance in terms of throughput and latency. - -This guide is tested on A100 80GB SXM4 and H100 80GB PCIe. It is confirmed to work with Phi-3-mini-128k-instruct and Phi-3-mini-4k-instruct (see [Support Matrix](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/models/core/phi) for full list) using TRT-LLM v0.11 and Triton Inference Server 24.07. - -- [Build and test TRT-LLM engine](#build-and-test-trt-llm-engine) -- [Deploy with Triton Inference Server](#deploy-with-triton-inference-server) +# Deploying an LLM with Triton and TRT-LLM + +This guide walks through serving a Hugging Face LLM with Triton Inference Server +using the TensorRT-LLM PyTorch backend (LLM API), and shows how to use GenAI-Perf +to benchmark throughput and latency. The PyTorch backend serves any Hugging Face +model directly — no TensorRT engine compilation required. + +> [!NOTE] +> The legacy TensorRT engine-build workflow (`convert_checkpoint.py` + +> `trtllm-build` and the `inflight_batcher_llm` ensemble model layout) is +> deprecated and is being removed from TensorRT-LLM. This guide uses the modern +> LLM API / PyTorch backend instead. See the +> [TensorRT-LLM Backend README](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/README.md) +> for the full set of configuration and deployment options. + +This guide uses [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) +as the example model, but you can serve any Hugging Face model by changing a +single line in `model.yaml`. + +- [Serve the model with Triton](#serve-the-model-with-triton) +- [Send an inference request](#send-an-inference-request) - [Benchmark with GenAI-Perf](#benchmark-with-genai-perf) -- [Reference Configurations](#reference-configurations) - - -## Build and test TRT-LLM engine - -Reference: - -1. ## Retrieve and launch the Docker container (optional) - - - - # Pre-install the environment using the NVIDIA Container Toolkit to avoid manual environment configuration - docker run --rm --ipc=host --runtime=nvidia --gpus '"device=0"' --entrypoint /bin/bash -it nvidia/cuda:12.4.1-devel-ubuntu22.04 - -2. ## Install TensorRT-LLM - - - - # Install dependencies, TensorRT-LLM requires Python 3.10 - apt-get update && apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev git git-lfs - - # Install TensorRT-LLM (v0.11.0) - pip3 install tensorrt_llm==0.11.0 --extra-index-url https://pypi.nvidia.com - - # Check installation - python3 -c "import tensorrt_llm" - -3. ## Clone the TRT-LLM repo with the Phi-3 conversion script - - - - git clone -b v0.11.0 https://github.com/NVIDIA/TensorRT-LLM.git - cd TensorRT-LLM/examples/phi/ - - # only need to install requirements.txt if you want to test the summarize.py example - # if so, modify requirements.txt such that tensorrt_llm==0.11.0 - # pip install -r requirements.txt - - -## Build the TRT-LLM Engine - -Reference: - -4. ## Download Phi-3-mini-4k-instruct - - - - git lfs install - git clone https://huggingface.co/microsoft/Phi-3-mini-4k-instruct - -5. ## Convert weights from HF Transformers to TensorRT-LLM format - - - - python3 ./convert_checkpoint.py \ - --model_dir ./Phi-3-mini-4k-instruct \ - --output_dir ./phi-checkpoint \ - --dtype float16 - -6. ## Build TensorRT engine(s) - - - - # Build a float16 engine using a single GPU and HF weights. - # Enable several TensorRT-LLM plugins to increase runtime performance. It also helps with build time. - # --tp_size and --pp_size are the model shard size - trtllm-build \ - --checkpoint_dir ./phi-checkpoint \ - --output_dir ./phi-engine \ - --gemm_plugin float16 \ - --max_batch_size 8 \ - --max_input_len 1024 \ - --max_seq_len 2048 \ - --tp_size 1 \ - --pp_size 1 - -7. ## Run the model - - - - python3 ../run.py --engine_dir ./phi-engine \ - --max_output_len 500 \ - --tokenizer_dir ./Phi-3-mini-4k-instruct \ - --input_text "How do I count to nine in French?" - -8. ## Summarization test using the Phi model - -The TensorRT-LLM Phi model can be tested to summarize the articles from the [cnn\_dailymail](https://huggingface.co/datasets/cnn_dailymail) dataset. For each summary, the script can compute the [ROUGE](https://en.wikipedia.org/wiki/ROUGE_\(metric\)) scores and use the ROUGE-1 score to validate the implementation. The script can also perform the same summarization using the HF Phi model. - - # Run the summarization task using a TensorRT-LLM model and a single GPU. - python3 ../summarize.py --engine_dir ./phi-engine \ - --hf_model_dir ./Phi-3-mini-4k-instruct \ - --batch_size 1 \ - --test_trt_llm \ - --test_hf \ - --data_type fp16 \ - --check_accuracy \ - --tensorrt_llm_rouge1_threshold=20 - - -## Deploy with Triton Inference Server - -9. ## Copy engine files from the Docker container to the host - - - - # In another terminal instance, before exiting the current container - docker cp : - - # For example - docker cp 452ee1c1d8a1:/TensorRT-LLM/examples/phi/phi-engine /home/user/phi-engine - -10. ## Copy the compiled model to the skeleton repository with TRT-LLM backend - - - - # After exiting the TensorRT-LLM Docker container - git clone https://github.com/triton-inference-server/tensorrtllm_backend.git - cd tensorrtllm_backend - cp ../phi-engine/* all_models/inflight_batcher_llm/tensorrt_llm/1/ - -11. ## Modify the configuration files from the model repository - -The following configuration files need to be updated: - -- ensemble/config.pbtxt - -- postprocessing/config.pbtxt - -- preprocessing/config.pbtxt - -- tensorrt\_llm/config.pbxt - -- tensorrt\_llm/1/config.json - - -### Update ensemble/config.pbtxt - - python3 tools/fill_template.py --in_place \ - all_models/inflight_batcher_llm/ensemble/config.pbtxt \ - triton_max_batch_size:128 - +- [References](#references) -### Update preprocessing/config.pbtxt +## Serve the model with Triton - python3 tools/fill_template.py --in_place \ - all_models/inflight_batcher_llm/postprocessing/config.pbtxt \ - tokenizer_type:auto,\ - tokenizer_dir:../Phi-3-mini-4k-instruct,\ - triton_max_batch_size:128,\ - postprocessing_instance_count:2 +### 1. Launch the container +```bash +docker run --rm -it --net host --shm-size=2g --ulimit memlock=-1 --gpus all \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + nvcr.io/nvidia/tritonserver:26.03-trtllm-python-py3 bash +``` -### Update postprocessing/config.pbtxt +Replace `26.03` with the latest tag from +[NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/tags). +For gated models, set your token first: `export HF_TOKEN=hf_...` - python3 tools/fill_template.py --in_place \ - all_models/inflight_batcher_llm/preprocessing/config.pbtxt \ - tokenizer_type:auto,\ - tokenizer_dir:../Phi-3-mini-4k-instruct,\ - triton_max_batch_size:128,\ - preprocessing_instance_count:2 +### 2. Configure your model +```bash +git clone https://github.com/NVIDIA/TensorRT-LLM.git +``` -### Update tensorrt\_llm/config.pbxt +Edit `TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml` +and set `model:` to any Hugging Face model ID or local path: - python3 tools/fill_template.py --in_place \ - all_models/inflight_batcher_llm/tensorrt_llm/config.pbtxt \ - decoupled_mode:true,\ - engine_dir:/all_models/inflight_batcher_llm/tensorrt_llm/1,\ - max_tokens_in_paged_kv_cache:,\ - batch_scheduler_policy:guaranteed_completion,\ - kv_cache_free_gpu_mem_fraction:0.2,\ - max_num_sequences:4,\ - triton_backend:tensorrtllm,\ - triton_max_batch_size:128,\ - max_queue_delay_microseconds:10,\ - max_beam_width:1,\ - batching_strategy:inflight_fused_batching,\ - engine_dir:/opt/all_models/inflight_batcher_llm/tensorrt_llm/1,\ - max_tokens_in_paged_kv_cache:1,\ - batch_scheduler_policy:guaranteed_completion,\ - kv_cache_free_gpu_mem_fraction:0.2 +```yaml +model: deepseek-ai/DeepSeek-V4-Flash +``` +All keys in `model.yaml` map directly to the +[`LLM()` constructor arguments](https://nvidia.github.io/TensorRT-LLM/llm-api/). +This is where you configure KV cache, quantization, and parallelism. +DeepSeek-V4-Flash is a Mixture-of-Experts model that runs on a single multi-GPU +node (for example 8x B200) — set the parallelism to match your hardware: - # manually access tensort_llm/config.pbtxt and change the CPU instances to > 1 - # unfortunately this was hard-coded and cannot be update with the above script +```yaml +model: deepseek-ai/DeepSeek-V4-Flash +tensor_parallel_size: 8 +``` - # instance_group [ - # { - # count: 2 - # kind : KIND_CPU - # } - # ] +For a quick single-GPU trial, swap in a smaller model such as +[`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B). +### 3. Launch the server -#### Max Tokens in Paged KV Cache +Run the launch script from the parent of `TensorRT-LLM/` (running it from inside +the cloned folder causes `ModuleNotFoundError: No module named +'tensorrt_llm.bindings'`): -This is only required for Phi-3-mini-128k-instruct, and it is not necessary to modify this parameter for Phi-3-mini-4k-instruct. +```bash +python3 TensorRT-LLM/triton_backend/scripts/launch_triton_server.py \ + --model_repo=TensorRT-LLM/triton_backend/all_models/llmapi/ +``` -To accommodate for the 128k context, remove the following from tensorrt\_llm/config.pbxt - which will allow the max tokens to be determined by the KV cache manager. If you don’t want to remove it, you can also set maxTokensInPagedKvCache such that it is large enough (e.g. 4096) to process at least 1 sequence to completion (i.e. must be larger than beam\_width \* tokensPerBlock \* maxBlocksPerSeq) +You should see the following logs once the server is ready: - parameters: { - key: "max_tokens_in_paged_kv_cache" - value: { - string_value: "4096" - } - } +``` +I0503 22:01:25.210518 1175 grpc_server.cc:2463] Started GRPCInferenceService at 0.0.0.0:8001 +I0503 22:01:25.211612 1175 http_server.cc:4692] Started HTTPService at 0.0.0.0:8000 +I0503 22:01:25.254914 1175 http_server.cc:362] Started Metrics Service at 0.0.0.0:8002 +``` +To stop Triton Server inside the container, run `pkill tritonserver`. -### Update tensorrt\_llm/1/config.json - -In the engine config (tensorrtllm\_backend/all\_models/inflight\_batcher\_llm/tensorrt\_llm/1/config.json), add the following under plugin\_config - - "Use_context_fmha_for_generation": false - - # for example: - "plugin_config": { - "dtype": "float16", - "bert_attention_plugin": "auto", - "streamingllm": false, - "Use_context_fmha_for_generation": false - -The above needs to be done manually with your favorite editor. Once finished, please be sure your working directory is \~/tensorrtllm\_backend - -12. ## Delete tensorrt\_llm\_bls - - - - # Recommended to remove the BLS directory if not needed - rm -rf all_models/inflight_batcher_llm/tensorrt_llm_bls/ - -13. ## Download model repository - - - - # for tokenizer - git lfs install - git clone https://huggingface.co/microsoft/Phi-3-mini-4k-instruct - -14. ## Launch Triton Inference Server (trtllm-python3-py3) - - - - docker run -it --rm --gpus all --network host --shm-size=1g \ - -v $(pwd)/all_models:/opt/all_models \ - -v $(pwd)/scripts:/opt/scripts \ - -v $(pwd)/Phi-3-mini-4k-instruct:/opt/Phi-3-mini-4k-instruct \ - nvcr.io/nvidia/tritonserver:26.06-trtllm-python-py3 - - # Launch Server - python3 ../scripts/launch_triton_server.py --model_repo ../all_models/inflight_batcher_llm --world_size 1 - -15. ## Send Requests - - - - curl -X POST localhost:8000/v2/models/ensemble/generate -d \ - '{ - "text_input": "A farmer with a wolf, a goat, and a cabbage must cross a river by boat. The boat can carry only the farmer and a single item. If left unattended together, the wolf would eat the goat, or the goat would eat the cabbage. How can they cross the river without anything being eaten?", - "parameters": { - "max_tokens": 256, - "bad_words":[""], - "stop_words":[""] - } - }' | jq +## Send an inference request +```bash +curl -X POST localhost:8000/v2/models/tensorrt_llm/generate \ + -d '{"text_input": "How do I count to nine in French?", "sampling_param_max_tokens": 256}' | jq +``` ## Benchmark with GenAI-Perf -16. ## Launch Triton Inference Server (py3-sdk) - - - - export RELEASE="26.06" - docker run -it --net=host --gpus '"device=0"' nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk - -17. ## Download the Phi-3 tokenizer - -Login to Hugging Face (with User Access Tokens) to get the Phi-3 tokenizer. This step is not necessary but helps with interpreting token metrics from prompts and responses. If you skip this step, be sure to remove the --tokenizer flag from the GenAI-Perf script in Step 18. - - git lfs install - git clone https://huggingface.co/microsoft/Phi-3-mini-4k-instruct - - pip install huggingface_hub - huggingface-cli login --token hf_*** - -18. ## Run GenAI-Perf - - - - export INPUT_SEQUENCE_LENGTH=128 - export OUTPUT_SEQUENCE_LENGTH=128 - export CONCURRENCY=25 - - genai-perf \ - -m ensemble \ - --service-kind triton \ - --backend tensorrtllm \ - --random-seed 123 \ - --synthetic-input-tokens-mean $INPUT_SEQUENCE_LENGTH \ - --synthetic-input-tokens-stddev 0 \ - --streaming \ - --output-tokens-mean $OUTPUT_SEQUENCE_LENGTH \ - --output-tokens-stddev 0 \ - --output-tokens-mean-deterministic \ - --concurrency $CONCURRENCY \ - --tokenizer microsoft/Phi-3-mini-4k-instruct \ - --measurement-interval 4000 \ - --url localhost:8001 - -More details on performance benchmarking with GenAI-Perf can be found [here](https://github.com/triton-inference-server/perf_analyzer/blob/main/genai-perf/README.md). - -## Reference Configurations - -All config files inside /tensorrtllm\_backend/all\_models/inflight\_batcher\_llm are shown below. - -
- ensemble/config.pbtxt - - # Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # - # Redistribution and use in source and binary forms, with or without - # modification, are permitted provided that the following conditions - # are met: - # * Redistributions of source code must retain the above copyright - # notice, this list of conditions and the following disclaimer. - # * Redistributions in binary form must reproduce the above copyright - # notice, this list of conditions and the following disclaimer in the - # documentation and/or other materials provided with the distribution. - # * Neither the name of NVIDIA CORPORATION nor the names of its - # contributors may be used to endorse or promote products derived - # from this software without specific prior written permission. - # - # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY - # EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR - # PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR - # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, - # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, - # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR - # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY - # OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - name: "ensemble" - platform: "ensemble" - max_batch_size: 128 - input [ - { - name: "text_input" - data_type: TYPE_STRING - dims: [ 1 ] - }, - { - name: "decoder_text_input" - data_type: TYPE_STRING - dims: [ 1 ] - optional: true - }, - { - name: "image_input" - data_type: TYPE_FP16 - dims: [ 3, 224, 224 ] - optional: true - }, - { - name: "max_tokens" - data_type: TYPE_INT32 - dims: [ 1 ] - }, - { - name: "bad_words" - data_type: TYPE_STRING - dims: [ -1 ] - optional: true - }, - { - name: "stop_words" - data_type: TYPE_STRING - dims: [ -1 ] - optional: true - }, - { - name: "end_id" - data_type: TYPE_INT32 - dims: [ 1 ] - optional: true - }, - { - name: "pad_id" - data_type: TYPE_INT32 - dims: [ 1 ] - optional: true - }, - { - name: "top_k" - data_type: TYPE_INT32 - dims: [ 1 ] - optional: true - }, - { - name: "top_p" - data_type: TYPE_FP32 - dims: [ 1 ] - optional: true - }, - { - name: "temperature" - data_type: TYPE_FP32 - dims: [ 1 ] - optional: true - }, - { - name: "length_penalty" - data_type: TYPE_FP32 - dims: [ 1 ] - optional: true - }, - { - name: "repetition_penalty" - data_type: TYPE_FP32 - dims: [ 1 ] - optional: true - }, - { - name: "min_length" - data_type: TYPE_INT32 - dims: [ 1 ] - optional: true - }, - { - name: "presence_penalty" - data_type: TYPE_FP32 - dims: [ 1 ] - optional: true - }, - { - name: "frequency_penalty" - data_type: TYPE_FP32 - dims: [ 1 ] - optional: true - }, - { - name: "random_seed" - data_type: TYPE_UINT64 - dims: [ 1 ] - optional: true - }, - { - name: "return_log_probs" - data_type: TYPE_BOOL - dims: [ 1 ] - optional: true - }, - { - name: "return_context_logits" - data_type: TYPE_BOOL - dims: [ 1 ] - optional: true - }, - { - name: "return_generation_logits" - data_type: TYPE_BOOL - dims: [ 1 ] - optional: true - }, - { - name: "beam_width" - data_type: TYPE_INT32 - dims: [ 1 ] - optional: true - }, - { - name: "stream" - data_type: TYPE_BOOL - dims: [ 1 ] - optional: true - }, - { - name: "prompt_embedding_table" - data_type: TYPE_FP16 - dims: [ -1, -1 ] - optional: true - }, - { - name: "prompt_vocab_size" - data_type: TYPE_INT32 - dims: [ 1 ] - optional: true - }, - { - name: "embedding_bias_words" - data_type: TYPE_STRING - dims: [ -1 ] - optional: true - }, - { - name: "embedding_bias_weights" - data_type: TYPE_FP32 - dims: [ -1 ] - optional: true - } - ] - output [ - { - name: "text_output" - data_type: TYPE_STRING - dims: [ -1 ] - }, - { - name: "cum_log_probs" - data_type: TYPE_FP32 - dims: [ -1 ] - }, - { - name: "output_log_probs" - data_type: TYPE_FP32 - dims: [ -1, -1 ] - }, - { - name: "context_logits" - data_type: TYPE_FP32 - dims: [ -1, -1 ] - }, - { - name: "generation_logits" - data_type: TYPE_FP32 - dims: [ -1, -1, -1 ] - }, - { - name: "batch_index" - data_type: TYPE_INT32 - dims: [ 1 ] - } - ] - ensemble_scheduling { - step [ - { - model_name: "preprocessing" - model_version: -1 - input_map { - key: "QUERY" - value: "text_input" - } - input_map { - key: "DECODER_QUERY" - value: "decoder_text_input" - } - input_map { - key: "IMAGE" - value: "image_input" - } - input_map { - key: "REQUEST_OUTPUT_LEN" - value: "max_tokens" - } - input_map { - key: "BAD_WORDS_DICT" - value: "bad_words" - } - input_map { - key: "STOP_WORDS_DICT" - value: "stop_words" - } - input_map { - key: "EMBEDDING_BIAS_WORDS" - value: "embedding_bias_words" - } - input_map { - key: "EMBEDDING_BIAS_WEIGHTS" - value: "embedding_bias_weights" - } - input_map { - key: "END_ID" - value: "end_id" - } - input_map { - key: "PAD_ID" - value: "pad_id" - } - input_map { - key: "PROMPT_EMBEDDING_TABLE" - value: "prompt_embedding_table" - } - output_map { - key: "REQUEST_INPUT_LEN" - value: "_REQUEST_INPUT_LEN" - } - output_map { - key: "INPUT_ID" - value: "_INPUT_ID" - } - output_map { - key: "REQUEST_DECODER_INPUT_LEN" - value: "_REQUEST_DECODER_INPUT_LEN" - } - output_map { - key: "DECODER_INPUT_ID" - value: "_DECODER_INPUT_ID" - } - output_map { - key: "REQUEST_OUTPUT_LEN" - value: "_REQUEST_OUTPUT_LEN" - } - output_map { - key: "STOP_WORDS_IDS" - value: "_STOP_WORDS_IDS" - } - output_map { - key: "BAD_WORDS_IDS" - value: "_BAD_WORDS_IDS" - } - output_map { - key: "EMBEDDING_BIAS" - value: "_EMBEDDING_BIAS" - } - output_map { - key: "OUT_END_ID" - value: "_PREPROCESSOR_END_ID" - } - output_map { - key: "OUT_PAD_ID" - value: "_PREPROCESSOR_PAD_ID" - } - output_map { - key: "OUT_PROMPT_EMBEDDING_TABLE" - value: "out_prompt_embedding_table" - } - }, - { - model_name: "tensorrt_llm" - model_version: -1 - input_map { - key: "input_ids" - value: "_INPUT_ID" - } - input_map { - key: "decoder_input_ids" - value: "_DECODER_INPUT_ID" - } - input_map { - key: "input_lengths" - value: "_REQUEST_INPUT_LEN" - } - input_map { - key: "decoder_input_lengths" - value: "_REQUEST_DECODER_INPUT_LEN" - } - input_map { - key: "request_output_len" - value: "_REQUEST_OUTPUT_LEN" - } - input_map { - key: "end_id" - value: "_PREPROCESSOR_END_ID" - } - input_map { - key: "pad_id" - value: "_PREPROCESSOR_PAD_ID" - } - input_map { - key: "embedding_bias" - value: "_EMBEDDING_BIAS" - } - input_map { - key: "runtime_top_k" - value: "top_k" - } - input_map { - key: "runtime_top_p" - value: "top_p" - } - input_map { - key: "temperature" - value: "temperature" - } - input_map { - key: "len_penalty" - value: "length_penalty" - } - input_map { - key: "repetition_penalty" - value: "repetition_penalty" - } - input_map { - key: "min_length" - value: "min_length" - } - input_map { - key: "presence_penalty" - value: "presence_penalty" - } - input_map { - key: "frequency_penalty" - value: "frequency_penalty" - } - input_map { - key: "random_seed" - value: "random_seed" - } - input_map { - key: "return_log_probs" - value: "return_log_probs" - } - input_map { - key: "return_context_logits" - value: "return_context_logits" - } - input_map { - key: "return_generation_logits" - value: "return_generation_logits" - } - input_map { - key: "beam_width" - value: "beam_width" - } - input_map { - key: "streaming" - value: "stream" - } - input_map { - key: "prompt_embedding_table" - value: "out_prompt_embedding_table" - } - input_map { - key: "prompt_vocab_size" - value: "prompt_vocab_size" - } - input_map { - key: "stop_words_list" - value: "_STOP_WORDS_IDS" - } - input_map { - key: "bad_words_list" - value: "_BAD_WORDS_IDS" - } - output_map { - key: "output_ids" - value: "_TOKENS_BATCH" - } - output_map { - key: "sequence_length" - value: "_SEQUENCE_LENGTH" - }, - output_map { - key: "cum_log_probs" - value: "_CUM_LOG_PROBS" - } - output_map { - key: "output_log_probs" - value: "_OUTPUT_LOG_PROBS" - }, - output_map { - key: "context_logits" - value: "_CONTEXT_LOGITS" - }, - output_map { - key: "generation_logits" - value: "_GENERATION_LOGITS" - }, - output_map { - key: "batch_index" - value: "_BATCH_INDEX" - } - }, - { - model_name: "postprocessing" - model_version: -1 - input_map { - key: "TOKENS_BATCH" - value: "_TOKENS_BATCH" - } - input_map { - key: "CUM_LOG_PROBS" - value: "_CUM_LOG_PROBS" - } - input_map { - key: "OUTPUT_LOG_PROBS" - value: "_OUTPUT_LOG_PROBS" - } - input_map { - key: "CONTEXT_LOGITS" - value: "_CONTEXT_LOGITS" - } - input_map { - key: "GENERATION_LOGITS" - value: "_GENERATION_LOGITS" - } - input_map { - key: "SEQUENCE_LENGTH" - value: "_SEQUENCE_LENGTH" - } - input_map { - key: "BATCH_INDEX" - value: "_BATCH_INDEX" - } - output_map { - key: "OUTPUT" - value: "text_output" - } - output_map { - key: "OUT_OUTPUT_LOG_PROBS" - value: "output_log_probs" - } - output_map { - key: "OUT_CUM_LOG_PROBS" - value: "cum_log_probs" - } - output_map { - key: "OUT_CONTEXT_LOGITS" - value: "context_logits" - } - output_map { - key: "OUT_GENERATION_LOGITS" - value: "generation_logits" - } - output_map { - key: "OUT_BATCH_INDEX" - value: "batch_index" - } - } - ] - } -
- -
-postprocessing/config.pbtxt - - # Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # - # Redistribution and use in source and binary forms, with or without - # modification, are permitted provided that the following conditions - # are met: - # * Redistributions of source code must retain the above copyright - # notice, this list of conditions and the following disclaimer. - # * Redistributions in binary form must reproduce the above copyright - # notice, this list of conditions and the following disclaimer in the - # documentation and/or other materials provided with the distribution. - # * Neither the name of NVIDIA CORPORATION nor the names of its - # contributors may be used to endorse or promote products derived - # from this software without specific prior written permission. - # - # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY - # EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR - # PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR - # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, - # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, - # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR - # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY - # OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - name: "postprocessing" - backend: "python" - max_batch_size: 128 - input [ - { - name: "TOKENS_BATCH" - data_type: TYPE_INT32 - dims: [ -1, -1 ] - }, - { - name: "SEQUENCE_LENGTH" - data_type: TYPE_INT32 - dims: [ -1 ] - }, - { - name: "CUM_LOG_PROBS" - data_type: TYPE_FP32 - dims: [ -1 ] - optional: true - }, - { - name: "OUTPUT_LOG_PROBS" - data_type: TYPE_FP32 - dims: [ -1, -1 ] - optional: true - }, - { - name: "CONTEXT_LOGITS" - data_type: TYPE_FP32 - dims: [ -1, -1 ] - optional: true - }, - { - name: "GENERATION_LOGITS" - data_type: TYPE_FP32 - dims: [ -1, -1, -1 ] - optional: true - }, - { - name: "BATCH_INDEX" - data_type: TYPE_INT32 - dims: [ 1 ] - optional: true - } - ] - output [ - { - name: "OUTPUT" - data_type: TYPE_STRING - dims: [ -1 ] - }, - { - name: "OUT_CUM_LOG_PROBS" - data_type: TYPE_FP32 - dims: [ -1 ] - }, - { - name: "OUT_OUTPUT_LOG_PROBS" - data_type: TYPE_FP32 - dims: [ -1, -1 ] - }, - { - name: "OUT_CONTEXT_LOGITS" - data_type: TYPE_FP32 - dims: [ -1, -1 ] - }, - { - name: "OUT_GENERATION_LOGITS" - data_type: TYPE_FP32 - dims: [ -1, -1, -1 ] - }, - { - name: "OUT_BATCH_INDEX" - data_type: TYPE_INT32 - dims: [ 1 ] - } - ] - - parameters { - key: "tokenizer_dir" - value: { - string_value: "../Phi-3-mini-4k-instruct" - } - } - - parameters { - key: "skip_special_tokens" - value: { - string_value: "${skip_special_tokens}" - } - } - - instance_group [ - { - count: 4 - kind: KIND_CPU - } - ] -
- -
- preprocessing/config.pbtxt - - # Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # - # Redistribution and use in source and binary forms, with or without - # modification, are permitted provided that the following conditions - # are met: - # * Redistributions of source code must retain the above copyright - # notice, this list of conditions and the following disclaimer. - # * Redistributions in binary form must reproduce the above copyright - # notice, this list of conditions and the following disclaimer in the - # documentation and/or other materials provided with the distribution. - # * Neither the name of NVIDIA CORPORATION nor the names of its - # contributors may be used to endorse or promote products derived - # from this software without specific prior written permission. - # - # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY - # EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR - # PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR - # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, - # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, - # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR - # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY - # OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - name: "preprocessing" - backend: "python" - max_batch_size: 128 - input [ - { - name: "QUERY" - data_type: TYPE_STRING - dims: [ 1 ] - }, - { - name: "DECODER_QUERY" - data_type: TYPE_STRING - dims: [ 1 ] - optional: true - }, - { - name: "IMAGE" - data_type: TYPE_FP16 - dims: [ 3, 224, 224 ] - optional: true - }, - { - name: "REQUEST_OUTPUT_LEN" - data_type: TYPE_INT32 - dims: [ 1 ] - }, - { - name: "BAD_WORDS_DICT" - data_type: TYPE_STRING - dims: [ -1 ] - optional: true - }, - { - name: "STOP_WORDS_DICT" - data_type: TYPE_STRING - dims: [ -1 ] - optional: true - }, - { - name: "EMBEDDING_BIAS_WORDS" - data_type: TYPE_STRING - dims: [ -1 ] - optional: true - }, - { - name: "EMBEDDING_BIAS_WEIGHTS" - data_type: TYPE_FP32 - dims: [ -1 ] - optional: true - }, - { - name: "END_ID" - data_type: TYPE_INT32 - dims: [ 1 ] - optional: true - }, - { - name: "PAD_ID" - data_type: TYPE_INT32 - dims: [ 1 ] - optional: true - }, - { - name: "PROMPT_EMBEDDING_TABLE" - data_type: TYPE_FP16 - dims: [ -1, -1 ] - optional: true - allow_ragged_batch: true - } - ] - output [ - { - name: "INPUT_ID" - data_type: TYPE_INT32 - dims: [ -1 ] - }, - { - name: "REQUEST_INPUT_LEN" - data_type: TYPE_INT32 - dims: [ 1 ] - }, - { - name: "DECODER_INPUT_ID" - data_type: TYPE_INT32 - dims: [ -1 ] - }, - { - name: "REQUEST_DECODER_INPUT_LEN" - data_type: TYPE_INT32 - dims: [ 1 ] - }, - { - name: "BAD_WORDS_IDS" - data_type: TYPE_INT32 - dims: [ 2, -1 ] - }, - { - name: "STOP_WORDS_IDS" - data_type: TYPE_INT32 - dims: [ 2, -1 ] - }, - { - name: "EMBEDDING_BIAS" - data_type: TYPE_FP32 - dims: [ -1 ] - }, - { - name: "REQUEST_OUTPUT_LEN" - data_type: TYPE_INT32 - dims: [ -1 ] - }, - { - name: "OUT_END_ID" - data_type: TYPE_INT32 - dims: [ 1 ] - }, - { - name: "OUT_PAD_ID" - data_type: TYPE_INT32 - dims: [ 1 ] - }, - { - name: "OUT_PROMPT_EMBEDDING_TABLE" - data_type: TYPE_FP16 - dims: [ -1, -1 ] - } - ] - - parameters { - key: "tokenizer_dir" - value: { - string_value: "../Phi-3-mini-4k-instruct" - } - } - - parameters { - key: "add_special_tokens" - value: { - string_value: "${add_special_tokens}" - } - } - - parameters { - key: "visual_model_path" - value: { - string_value: "${visual_model_path}" - } - } - - parameters: { - key: "gpt_model_path" - value: { - string_value: "${engine_dir}" - } - } - - instance_group [ - { - count: 4 - kind: KIND_CPU - } - ] - -
- -
- tensorrt_llm/config.pbtxt - - - # Copyright (c) 2024-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # - # Redistribution and use in source and binary forms, with or without - # modification, are permitted provided that the following conditions - # are met: - # * Redistributions of source code must retain the above copyright - # notice, this list of conditions and the following disclaimer. - # * Redistributions in binary form must reproduce the above copyright - # notice, this list of conditions and the following disclaimer in the - # documentation and/or other materials provided with the distribution. - # * Neither the name of NVIDIA CORPORATION nor the names of its - # contributors may be used to endorse or promote products derived - # from this software without specific prior written permission. - # - # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY - # EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR - # PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR - # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, - # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, - # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR - # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY - # OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - name: "tensorrt_llm" - backend: "tensorrtllm" - max_batch_size: 128 - - model_transaction_policy { - decoupled: true - } - - dynamic_batching { - preferred_batch_size: [ 128 ] - max_queue_delay_microseconds: 10 - } - - input [ - { - name: "input_ids" - data_type: TYPE_INT32 - dims: [ -1 ] - allow_ragged_batch: true - }, - { - name: "input_lengths" - data_type: TYPE_INT32 - dims: [ 1 ] - reshape: { shape: [ ] } - }, - { - name: "request_output_len" - data_type: TYPE_INT32 - dims: [ 1 ] - reshape: { shape: [ ] } - }, - { - name: "draft_input_ids" - data_type: TYPE_INT32 - dims: [ -1 ] - optional: true - allow_ragged_batch: true - }, - { - name: "decoder_input_ids" - data_type: TYPE_INT32 - dims: [ -1 ] - optional: true - allow_ragged_batch: true - }, - { - name: "decoder_input_lengths" - data_type: TYPE_INT32 - dims: [ 1 ] - optional: true - reshape: { shape: [ ] } - }, - { - name: "draft_logits" - data_type: TYPE_FP32 - dims: [ -1, -1 ] - optional: true - allow_ragged_batch: true - }, - { - name: "draft_acceptance_threshold" - data_type: TYPE_FP32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "end_id" - data_type: TYPE_INT32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "pad_id" - data_type: TYPE_INT32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "stop_words_list" - data_type: TYPE_INT32 - dims: [ 2, -1 ] - optional: true - allow_ragged_batch: true - }, - { - name: "bad_words_list" - data_type: TYPE_INT32 - dims: [ 2, -1 ] - optional: true - allow_ragged_batch: true - }, - { - name: "embedding_bias" - data_type: TYPE_FP32 - dims: [ -1 ] - optional: true - allow_ragged_batch: true - }, - { - name: "beam_width" - data_type: TYPE_INT32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "temperature" - data_type: TYPE_FP32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "runtime_top_k" - data_type: TYPE_INT32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "runtime_top_p" - data_type: TYPE_FP32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "runtime_top_p_min" - data_type: TYPE_FP32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "runtime_top_p_decay" - data_type: TYPE_FP32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "runtime_top_p_reset_ids" - data_type: TYPE_INT32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "len_penalty" - data_type: TYPE_FP32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "early_stopping" - data_type: TYPE_BOOL - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "repetition_penalty" - data_type: TYPE_FP32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "min_length" - data_type: TYPE_INT32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "beam_search_diversity_rate" - data_type: TYPE_FP32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "presence_penalty" - data_type: TYPE_FP32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "frequency_penalty" - data_type: TYPE_FP32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "random_seed" - data_type: TYPE_UINT64 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "return_log_probs" - data_type: TYPE_BOOL - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "return_context_logits" - data_type: TYPE_BOOL - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "return_generation_logits" - data_type: TYPE_BOOL - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "stop" - data_type: TYPE_BOOL - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "streaming" - data_type: TYPE_BOOL - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - { - name: "prompt_embedding_table" - data_type: TYPE_FP16 - dims: [ -1, -1 ] - optional: true - allow_ragged_batch: true - }, - { - name: "prompt_vocab_size" - data_type: TYPE_INT32 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - # the unique task ID for the given LoRA. - # To perform inference with a specific LoRA for the first time `lora_task_id` `lora_weights` and `lora_config` must all be given. - # The LoRA will be cached, so that subsequent requests for the same task only require `lora_task_id`. - # If the cache is full the oldest LoRA will be evicted to make space for new ones. An error is returned if `lora_task_id` is not cached. - { - name: "lora_task_id" - data_type: TYPE_UINT64 - dims: [ 1 ] - reshape: { shape: [ ] } - optional: true - }, - # weights for a lora adapter shape [ num_lora_modules_layers, D x Hi + Ho x D ] - # where the last dimension holds the in / out adapter weights for the associated module (e.g. attn_qkv) and model layer - # each of the in / out tensors are first flattened and then concatenated together in the format above. - # D=adapter_size (R value), Hi=hidden_size_in, Ho=hidden_size_out. - { - name: "lora_weights" - data_type: TYPE_FP16 - dims: [ -1, -1 ] - optional: true - allow_ragged_batch: true - }, - # module identifier (same size a first dimension of lora_weights) - # See LoraModule::ModuleType for model id mapping - # - # "attn_qkv": 0 # compbined qkv adapter - # "attn_q": 1 # q adapter - # "attn_k": 2 # k adapter - # "attn_v": 3 # v adapter - # "attn_dense": 4 # adapter for the dense layer in attention - # "mlp_h_to_4h": 5 # for llama2 adapter for gated mlp layer after attention / RMSNorm: up projection - # "mlp_4h_to_h": 6 # for llama2 adapter for gated mlp layer after attention / RMSNorm: down projection - # "mlp_gate": 7 # for llama2 adapter for gated mlp later after attention / RMSNorm: gate - # - # last dim holds [ module_id, layer_idx, adapter_size (D aka R value) ] - { - name: "lora_config" - data_type: TYPE_INT32 - dims: [ -1, 3 ] - optional: true - allow_ragged_batch: true - } - ] - output [ - { - name: "output_ids" - data_type: TYPE_INT32 - dims: [ -1, -1 ] - }, - { - name: "sequence_length" - data_type: TYPE_INT32 - dims: [ -1 ] - }, - { - name: "cum_log_probs" - data_type: TYPE_FP32 - dims: [ -1 ] - }, - { - name: "output_log_probs" - data_type: TYPE_FP32 - dims: [ -1, -1 ] - }, - { - name: "context_logits" - data_type: TYPE_FP32 - dims: [ -1, -1 ] - }, - { - name: "generation_logits" - data_type: TYPE_FP32 - dims: [ -1, -1, -1 ] - }, - { - name: "batch_index" - data_type: TYPE_INT32 - dims: [ 1 ] - } - ] - instance_group [ - { - count: 4 - kind : KIND_CPU - } - ] - parameters: { - key: "max_beam_width" - value: { - string_value: "1" - } - } - parameters: { - key: "FORCE_CPU_ONLY_INPUT_TENSORS" - value: { - string_value: "no" - } - } - parameters: { - key: "gpt_model_type" - value: { - string_value: "inflight_fused_batching" - } - } - parameters: { - key: "gpt_model_path" - value: { - string_value: "/opt/all_models/inflight_batcher_llm/tensorrt_llm/1" - } - } - parameters: { - key: "encoder_model_path" - value: { - string_value: "${encoder_engine_dir}" - } - } - -
- parameters: { - key: "max_tokens_in_paged_kv_cache" - value: { - string_value: "" - } - } - parameters: { - key: "max_attention_window_size" - value: { - string_value: "${max_attention_window_size}" - } - } - parameters: { - key: "sink_token_length" - value: { - string_value: "${sink_token_length}" - } - } - parameters: { - key: "batch_scheduler_policy" - value: { - string_value: "guaranteed_completion" - } - } - parameters: { - key: "kv_cache_free_gpu_mem_fraction" - value: { - string_value: "0.2" - } - } - parameters: { - key: "kv_cache_host_memory_bytes" - value: { - string_value: "${kv_cache_host_memory_bytes}" - } - } - parameters: { - key: "kv_cache_onboard_blocks" - value: { - string_value: "${kv_cache_onboard_blocks}" - } - } - # enable_trt_overlap is deprecated and doesn't have any effect on the runtime - # parameters: { - # key: "enable_trt_overlap" - # value: { - # string_value: "${enable_trt_overlap}" - # } - # } - parameters: { - key: "exclude_input_in_output" - value: { - string_value: "${exclude_input_in_output}" - } - } - parameters: { - key: "cancellation_check_period_ms" - value: { - string_value: "${cancellation_check_period_ms}" - } - } - parameters: { - key: "stats_check_period_ms" - value: { - string_value: "${stats_check_period_ms}" - } - } - parameters: { - key: "iter_stats_max_iterations" - value: { - string_value: "${iter_stats_max_iterations}" - } - } - parameters: { - key: "request_stats_max_iterations" - value: { - string_value: "${request_stats_max_iterations}" - } - } - parameters: { - key: "enable_kv_cache_reuse" - value: { - string_value: "${enable_kv_cache_reuse}" - } - } - parameters: { - key: "normalize_log_probs" - value: { - string_value: "${normalize_log_probs}" - } - } - parameters: { - key: "enable_chunked_context" - value: { - string_value: "${enable_chunked_context}" - } - } - parameters: { - key: "gpu_device_ids" - value: { - string_value: "${gpu_device_ids}" - } - } - parameters: { - key: "lora_cache_optimal_adapter_size" - value: { - string_value: "${lora_cache_optimal_adapter_size}" - } - } - parameters: { - key: "lora_cache_max_adapter_size" - value: { - string_value: "${lora_cache_max_adapter_size}" - } - } - parameters: { - key: "lora_cache_gpu_memory_fraction" - value: { - string_value: "${lora_cache_gpu_memory_fraction}" - } - } - parameters: { - key: "lora_cache_host_memory_bytes" - value: { - string_value: "${lora_cache_host_memory_bytes}" - } - } - parameters: { - key: "decoding_mode" - value: { - string_value: "${decoding_mode}" - } - } - parameters: { - key: "executor_worker_path" - value: { - string_value: "/opt/tritonserver/backends/tensorrtllm/trtllmExecutorWorker" - } - } - parameters: { - key: "medusa_choices" - value: { - string_value: "${medusa_choices}" - } - } - parameters: { - key: "gpu_weights_percent" - value: { - string_value: "${gpu_weights_percent}" - } - } \ No newline at end of file +### 1. Launch the SDK container + +```bash +export RELEASE="26.03" +docker run -it --net=host --gpus '"device=0"' nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk +``` + +### 2. Run GenAI-Perf + +```bash +export INPUT_SEQUENCE_LENGTH=128 +export OUTPUT_SEQUENCE_LENGTH=128 +export CONCURRENCY=25 + +genai-perf profile \ + -m tensorrt_llm \ + --service-kind triton \ + --backend tensorrtllm \ + --random-seed 123 \ + --synthetic-input-tokens-mean $INPUT_SEQUENCE_LENGTH \ + --synthetic-input-tokens-stddev 0 \ + --streaming \ + --output-tokens-mean $OUTPUT_SEQUENCE_LENGTH \ + --output-tokens-stddev 0 \ + --output-tokens-mean-deterministic \ + --concurrency $CONCURRENCY \ + --tokenizer deepseek-ai/DeepSeek-V4-Flash \ + --measurement-interval 4000 \ + --url localhost:8001 +``` + +More details on performance benchmarking with GenAI-Perf can be found +[here](https://github.com/triton-inference-server/perf_analyzer/blob/main/genai-perf/README.md). + +## References + +- [TensorRT-LLM User Guide](trtllm_user_guide.md) +- [TensorRT-LLM Backend README](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/README.md) +- [LLM API guide](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/llmapi.md) +- [LLM API reference](https://nvidia.github.io/TensorRT-LLM/llm-api/) From 5f756ce4b676f297fbd20524263c9d953f893069 Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Mon, 3 Aug 2026 11:51:42 -0700 Subject: [PATCH 2/7] docs: Fix LLM guide after testing on 8x B200 Validated the guide end-to-end on an 8x B200 node. Testing found several steps that could not work as written; this commit fixes them. - Replace DeepSeek-V4-Flash with models that run on released containers. V4 needs TensorRT-LLM 1.3 (not GA); 26.03/26.07 ship 1.2.x, which do not register DeepseekV4ForCausalLM. Feature nvidia/DeepSeek-R1-0528-FP4-V2 (NVFP4 + FP8 KV, 385 GB) for 8x B200, Qwen3-8B for a single GPU. - Document the openai/tensorrt_llm version conflict in 26.03-26.07 that makes the backend fail to load (ImportError: PartReasoningText) and the `pip install -U openai` workaround. - Pin the TensorRT-LLM clone to the tag matching the container instead of cloning the default branch. - Replace the GenAI-Perf section. `--service-kind` no longer exists, and `--backend tensorrtllm` targets the legacy inflight_batcher_llm input schema, so it cannot drive the LLM API backend. Use benchmark_core_model.py --test-llmapi instead. - Explain that streaming needs model_transaction_policy in config.pbtxt; triton_config.decoupled in model.yaml is ignored because the launch script passes --disable-auto-complete-config. - Note that `generate` is a raw completion endpoint and does not apply the chat template, with a worked DeepSeek-R1 example. - Update the container tag to 26.07 and refresh the sample server logs. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- docs/getting_started/llm.md | 301 ++++++++++++++++++++++++++++-------- 1 file changed, 239 insertions(+), 62 deletions(-) diff --git a/docs/getting_started/llm.md b/docs/getting_started/llm.md index ccf586035a..6976b20ed4 100644 --- a/docs/getting_started/llm.md +++ b/docs/getting_started/llm.md @@ -29,9 +29,8 @@ # Deploying an LLM with Triton and TRT-LLM This guide walks through serving a Hugging Face LLM with Triton Inference Server -using the TensorRT-LLM PyTorch backend (LLM API), and shows how to use GenAI-Perf -to benchmark throughput and latency. The PyTorch backend serves any Hugging Face -model directly — no TensorRT engine compilation required. +using the TensorRT-LLM PyTorch backend (LLM API). The PyTorch backend serves +supported Hugging Face models directly — no TensorRT engine compilation required. > [!NOTE] > The legacy TensorRT engine-build workflow (`convert_checkpoint.py` + @@ -41,13 +40,16 @@ model directly — no TensorRT engine compilation required. > [TensorRT-LLM Backend README](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/README.md) > for the full set of configuration and deployment options. -This guide uses [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) -as the example model, but you can serve any Hugging Face model by changing a -single line in `model.yaml`. +This guide uses [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B) as the +example model. You can serve any model listed in the TensorRT-LLM +[support matrix](https://nvidia.github.io/TensorRT-LLM/models/supported-models.html) +by changing a single line in `model.yaml`. - [Serve the model with Triton](#serve-the-model-with-triton) - [Send an inference request](#send-an-inference-request) -- [Benchmark with GenAI-Perf](#benchmark-with-genai-perf) +- [Streaming responses](#streaming-responses) +- [Multi-GPU models](#multi-gpu-models) +- [Benchmark](#benchmark) - [References](#references) ## Serve the model with Triton @@ -55,47 +57,76 @@ single line in `model.yaml`. ### 1. Launch the container ```bash +export RELEASE=26.07 docker run --rm -it --net host --shm-size=2g --ulimit memlock=-1 --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ - nvcr.io/nvidia/tritonserver:26.03-trtllm-python-py3 bash + nvcr.io/nvidia/tritonserver:${RELEASE}-trtllm-python-py3 bash ``` -Replace `26.03` with the latest tag from -[NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/tags). -For gated models, set your token first: `export HF_TOKEN=hf_...` - -### 2. Configure your model +Check [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/tags) +for the latest `-trtllm-python-py3` tag. For gated models, set your token first: +`export HF_TOKEN=hf_...` + +> [!IMPORTANT] +> The `26.03` and `26.07` containers (verified; the tags in between are likely +> affected too) ship `openai==1.107.3`, which is too old for the bundled +> `tensorrt_llm` package — TensorRT-LLM declares `openai` as a dependency with no +> lower bound. Without the fix below, loading the model fails with: +> +> ``` +> ImportError: cannot import name 'PartReasoningText' from +> 'openai.types.responses.response_content_part_added_event' +> ``` +> +> Upgrade the package inside the container before starting the server: +> +> ```bash +> pip install -U openai +> ``` + +### 2. Get the model repository + +The Triton model repository for the LLM API backend lives in the TensorRT-LLM +repo. Clone the tag that matches the `tensorrt_llm` version inside your +container, so that the `model.py` you run matches the library it imports: ```bash -git clone https://github.com/NVIDIA/TensorRT-LLM.git +python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)" # e.g. 1.2.1 +git clone --depth 1 --branch v1.2.1 https://github.com/NVIDIA/TensorRT-LLM.git ``` +> [!NOTE] +> Prefer the matching tag over the default branch. `main` tracks the next release +> (currently `1.3.0rc*`); it happens to work against a `1.2.1` container today, +> but nothing guarantees that, since its `model.py` is developed against the +> unreleased library. Pinning keeps the guide reproducible. + +### 3. Configure your model + Edit `TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml` -and set `model:` to any Hugging Face model ID or local path: +and set `model:` to a Hugging Face model ID or a local path: ```yaml -model: deepseek-ai/DeepSeek-V4-Flash +model: Qwen/Qwen3-8B +backend: "pytorch" +tensor_parallel_size: 1 +pipeline_parallel_size: 1 + +triton_config: + max_batch_size: 0 + decoupled: False ``` -All keys in `model.yaml` map directly to the +All keys outside `triton_config` map directly to the [`LLM()` constructor arguments](https://nvidia.github.io/TensorRT-LLM/llm-api/). This is where you configure KV cache, quantization, and parallelism. -DeepSeek-V4-Flash is a Mixture-of-Experts model that runs on a single multi-GPU -node (for example 8x B200) — set the parallelism to match your hardware: -```yaml -model: deepseek-ai/DeepSeek-V4-Flash -tensor_parallel_size: 8 -``` +### 4. Launch the server -For a quick single-GPU trial, swap in a smaller model such as -[`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B). - -### 3. Launch the server - -Run the launch script from the parent of `TensorRT-LLM/` (running it from inside -the cloned folder causes `ModuleNotFoundError: No module named -'tensorrt_llm.bindings'`): +Run the launch script from the parent of `TensorRT-LLM/`. Running it from inside +the cloned folder makes Python import the source tree instead of the installed +package and fails with `ModuleNotFoundError: No module named +'tensorrt_llm.bindings'`: ```bash python3 TensorRT-LLM/triton_backend/scripts/launch_triton_server.py \ @@ -105,11 +136,17 @@ python3 TensorRT-LLM/triton_backend/scripts/launch_triton_server.py \ You should see the following logs once the server is ready: ``` -I0503 22:01:25.210518 1175 grpc_server.cc:2463] Started GRPCInferenceService at 0.0.0.0:8001 -I0503 22:01:25.211612 1175 http_server.cc:4692] Started HTTPService at 0.0.0.0:8000 -I0503 22:01:25.254914 1175 http_server.cc:362] Started Metrics Service at 0.0.0.0:8002 +I0803 04:19:38.396545 3606543 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001" +I0803 04:19:38.396742 3606543 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000" +I0803 04:19:38.437685 3606543 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002" ``` +> [!NOTE] +> `launch_triton_server.py` starts Triton as a background process and returns +> immediately. It is designed for an interactive shell. If you wrap it in a +> script, a batch job, or `srun`, keep the parent process alive or the server is +> killed when your script exits. + To stop Triton Server inside the container, run `pkill tritonserver`. ## Send an inference request @@ -119,41 +156,180 @@ curl -X POST localhost:8000/v2/models/tensorrt_llm/generate \ -d '{"text_input": "How do I count to nine in French?", "sampling_param_max_tokens": 256}' | jq ``` -## Benchmark with GenAI-Perf +Sampling options are passed as `sampling_param_*` inputs — see the `input` +section of +`TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/config.pbtxt` for the +full list. Add `"sampling_param_exclude_input_from_output": true` to get only the +generated text back instead of prompt + completion. + +> [!NOTE] +> This endpoint performs raw text completion and does **not** apply the model's +> chat template. For instruct and reasoning models you must format the prompt +> yourself — see +> [Example: DeepSeek-R1 in NVFP4 on 8x B200](#example-deepseek-r1-in-nvfp4-on-8x-b200). + +## Streaming responses + +Streaming requires Triton's decoupled transaction policy. Setting +`decoupled: True` in `model.yaml` alone is **not** enough: the launch script +passes `--disable-auto-complete-config`, which skips the `auto_complete_config()` +hook where `model.yaml`'s `triton_config` is applied. The server then streams +while Triton core still treats the model as non-decoupled, and the request hangs +with `Streaming is only supported in decoupled mode.` in the server log. -### 1. Launch the SDK container +To enable streaming, also append the policy to +`TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/config.pbtxt`: + +``` +model_transaction_policy { + decoupled: True +} +``` + +Then restart the server and use the `generate_stream` endpoint: ```bash -export RELEASE="26.03" -docker run -it --net=host --gpus '"device=0"' nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk +curl -N -X POST localhost:8000/v2/models/tensorrt_llm/generate_stream \ + -d '{"text_input": "Count to three:", "sampling_param_max_tokens": 10, "streaming": true}' ``` -### 2. Run GenAI-Perf +``` +data: {"model_name":"tensorrt_llm","model_version":"1","text_output":" 1"} +data: {"model_name":"tensorrt_llm","model_version":"1","text_output":" 1,"} +data: {"model_name":"tensorrt_llm","model_version":"1","text_output":" 1, 2"} +``` + +> [!NOTE] +> Incremental, token-by-token events require the model repository from +> TensorRT-LLM 1.3 or newer. With the `v1.2.x` `model.py`, a `streaming: true` +> request is accepted but the server emits a single event containing the full +> response. + +> [!NOTE] +> A decoupled model cannot be used over the non-streaming HTTP `generate` +> endpoint — Triton returns +> `[501] HTTP end point doesn't support models with decoupled transaction policy`. +> Keep `decoupled: False` unless you need streaming. + +## Multi-GPU models + +Larger models run across the GPUs of a single node by setting the parallelism in +`model.yaml`. The LLM API launches its own worker processes, so no extra +`--world_size` argument is needed on `launch_triton_server.py`: + +```yaml +model: +tensor_parallel_size: 8 +``` + +For Mixture-of-Experts models, also set the expert parallelism. A bare +`tensor_parallel_size` is often not sufficient — check the model's row in the +[support matrix](https://nvidia.github.io/TensorRT-LLM/models/supported-models.html) +and its example README for required settings. + +### Example: DeepSeek-R1 in NVFP4 on 8x B200 + +[`nvidia/DeepSeek-R1-0528-FP4-V2`](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-V2) +is a 671B-parameter MoE quantized to NVFP4 with an FP8 KV cache. Quantization +brings the checkpoint from 642 GB (FP8) down to 385 GB, so it fits comfortably on +one 8x B200 node with room left for the KV cache. NVFP4 requires Blackwell +(`SM100+`). + +```yaml +model: nvidia/DeepSeek-R1-0528-FP4-V2 +backend: "pytorch" +tensor_parallel_size: 8 +moe_expert_parallel_size: 8 +max_seq_len: 4096 +max_num_tokens: 8192 +kv_cache_config: + free_gpu_memory_fraction: 0.7 + +triton_config: + max_batch_size: 0 + decoupled: False +``` + +The quantization format is read from `hf_quant_config.json` in the checkpoint +(`quant_algo: NVFP4`, `kv_cache_quant_algo: FP8`) — you do not set it in +`model.yaml`. + +Loading 385 GB takes several minutes on first start (about 6 minutes from a +warm local cache). Watch for `Started HTTPService` before sending requests. Once +resident, this configuration uses roughly 145 GiB of each B200's 183 GiB, +leaving headroom for the KV cache. + +> [!IMPORTANT] +> `generate` is a **raw completion** endpoint — it does not apply the model's +> chat template. Sending a bare question to an instruct or reasoning model makes +> it continue the text rather than answer it. DeepSeek-R1 replies to +> `"How do I count to nine in French?"` with a list of scraped page titles. +> Format the prompt yourself using the model's template: +> +> ```bash +> curl -X POST localhost:8000/v2/models/tensorrt_llm/generate -d '{ +> "text_input": "<｜begin▁of▁sentence｜><｜User｜>How do I count to nine in French?<｜Assistant｜>", +> "sampling_param_max_tokens": 512, +> "sampling_param_exclude_input_from_output": true }' +> ``` +> +> R1 then emits its reasoning trace in a `` block before the answer: +> +> ``` +> +> Okay, the user is asking how to count to nine in French. That seems +> straightforward—they probably need the French numbers from one to nine. +> ... +> ``` +> +> Each model family uses a different template — read `chat_template` in the +> checkpoint's `tokenizer_config.json`. + +> [!NOTE] +> DeepSeek-V4 (`DeepSeek-V4-Flash` / `-Pro`) is **not** usable with these +> containers. It requires TensorRT-LLM 1.3 or newer, which is not yet GA and is +> not in `26.07` or earlier — those ship TensorRT-LLM 1.2.x, which does not +> register `DeepseekV4ForCausalLM`. Loading it fails with +> `The checkpoint you are trying to load has model type 'deepseek_v4' but +> Transformers does not recognize this architecture.` The same applies to GLM-5.x +> (`glm_moe_dsa`). Use DeepSeek-R1 or GLM-4.7 until a container ships 1.3. + +## Benchmark + +The LLM API backend ships its own benchmarking client. Install the Triton client +in the server container and run it against the model, with `decoupled: False`: ```bash -export INPUT_SEQUENCE_LENGTH=128 -export OUTPUT_SEQUENCE_LENGTH=128 -export CONCURRENCY=25 - -genai-perf profile \ - -m tensorrt_llm \ - --service-kind triton \ - --backend tensorrtllm \ - --random-seed 123 \ - --synthetic-input-tokens-mean $INPUT_SEQUENCE_LENGTH \ - --synthetic-input-tokens-stddev 0 \ - --streaming \ - --output-tokens-mean $OUTPUT_SEQUENCE_LENGTH \ - --output-tokens-stddev 0 \ - --output-tokens-mean-deterministic \ - --concurrency $CONCURRENCY \ - --tokenizer deepseek-ai/DeepSeek-V4-Flash \ - --measurement-interval 4000 \ - --url localhost:8001 -``` - -More details on performance benchmarking with GenAI-Perf can be found -[here](https://github.com/triton-inference-server/perf_analyzer/blob/main/genai-perf/README.md). +pip install "tritonclient[grpc,http]" + +python3 TensorRT-LLM/triton_backend/tools/inflight_batcher_llm/benchmark_core_model.py \ + --max-input-len 500 \ + --tensorrt-llm-model-name tensorrt_llm \ + --test-llmapi \ + dataset --dataset TensorRT-LLM/triton_backend/tools/dataset/mini_cnn_eval.json \ + --tokenizer-dir Qwen/Qwen3-8B +``` + +``` +Tokenizer: Tokens per word = 1.324 +[INFO] Warm up for benchmarking. +[INFO] Start benchmarking on 37 prompts. +[INFO] Total Latency: 853.993 ms +``` + +> [!NOTE] +> GenAI-Perf's `--backend tensorrtllm` mode targets the legacy +> `inflight_batcher_llm` model, whose inputs are named `max_tokens` and `stream`. +> The LLM API backend names them `sampling_param_max_tokens` and `streaming`, so +> GenAI-Perf fails with +> `Failed to init manager inputs: The input or output 'max_tokens' is not found in +> the model configuration`. Use the client above until GenAI-Perf adds LLM API +> support. + +> [!NOTE] +> The shipped `model.yaml` sets `max_batch_size: 0`, so the backend serves +> requests without batching. Concurrency sweeps will not show throughput scaling +> until batching support lands. ## References @@ -161,3 +337,4 @@ More details on performance benchmarking with GenAI-Perf can be found - [TensorRT-LLM Backend README](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/README.md) - [LLM API guide](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/llmapi.md) - [LLM API reference](https://nvidia.github.io/TensorRT-LLM/llm-api/) +- [TensorRT-LLM supported models](https://nvidia.github.io/TensorRT-LLM/models/supported-models.html) From a13738a8d2f23a95368fb4c9c3c6d69ea5b27b93 Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Mon, 3 Aug 2026 12:00:25 -0700 Subject: [PATCH 3/7] docs: Make DeepSeek-R1 NVFP4 the primary example The guide featured Qwen3-8B and relegated DeepSeek-R1 NVFP4 to a later section. Lead with the NVFP4 model instead, since that is the configuration validated on 8x B200, and keep Qwen3-8B as the single-GPU alternative. - Intro and the step 3 model.yaml now use nvidia/DeepSeek-R1-0528-FP4-V2 with tensor_parallel_size / moe_expert_parallel_size of 8. - Move the NVFP4 note (format comes from hf_quant_config.json, Blackwell only) next to the config it applies to. - Replace the duplicated multi-GPU example with a sizing table carrying the measured numbers: ~6 min to ready, ~145 GiB per GPU, 81.83 GiB KV cache. - Label the benchmark sample output as Qwen3-8B on one GPU, and take the tokenizer argument from model.yaml rather than hardcoding it. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- docs/getting_started/llm.md | 96 ++++++++++++++++++++----------------- 1 file changed, 52 insertions(+), 44 deletions(-) diff --git a/docs/getting_started/llm.md b/docs/getting_started/llm.md index 6976b20ed4..d4c43d7513 100644 --- a/docs/getting_started/llm.md +++ b/docs/getting_started/llm.md @@ -40,8 +40,13 @@ supported Hugging Face models directly — no TensorRT engine compilation requir > [TensorRT-LLM Backend README](https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/README.md) > for the full set of configuration and deployment options. -This guide uses [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B) as the -example model. You can serve any model listed in the TensorRT-LLM +This guide uses +[`nvidia/DeepSeek-R1-0528-FP4-V2`](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-V2) +as the example model — a 671B-parameter Mixture-of-Experts model quantized to +NVFP4 with an FP8 KV cache, served across one 8x B200 node. For a quick +single-GPU trial, use a smaller model such as +[`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B) with +`tensor_parallel_size: 1`. You can serve any model listed in the TensorRT-LLM [support matrix](https://nvidia.github.io/TensorRT-LLM/models/supported-models.html) by changing a single line in `model.yaml`. @@ -107,10 +112,14 @@ Edit `TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml` and set `model:` to a Hugging Face model ID or a local path: ```yaml -model: Qwen/Qwen3-8B +model: nvidia/DeepSeek-R1-0528-FP4-V2 backend: "pytorch" -tensor_parallel_size: 1 -pipeline_parallel_size: 1 +tensor_parallel_size: 8 +moe_expert_parallel_size: 8 +max_seq_len: 4096 +max_num_tokens: 8192 +kv_cache_config: + free_gpu_memory_fraction: 0.7 triton_config: max_batch_size: 0 @@ -121,6 +130,23 @@ All keys outside `triton_config` map directly to the [`LLM()` constructor arguments](https://nvidia.github.io/TensorRT-LLM/llm-api/). This is where you configure KV cache, quantization, and parallelism. +The NVFP4 quantization format is read from `hf_quant_config.json` in the +checkpoint (`quant_algo: NVFP4`, `kv_cache_quant_algo: FP8`) — you do not declare +it in `model.yaml`. NVFP4 requires Blackwell (`SM100+`) GPUs. + +To run on a single GPU instead, swap in a smaller model and drop the parallelism: + +```yaml +model: Qwen/Qwen3-8B +backend: "pytorch" +tensor_parallel_size: 1 +pipeline_parallel_size: 1 + +triton_config: + max_batch_size: 0 + decoupled: False +``` + ### 4. Launch the server Run the launch script from the parent of `TensorRT-LLM/`. Running it from inside @@ -213,51 +239,31 @@ data: {"model_name":"tensorrt_llm","model_version":"1","text_output":" 1, 2"} ## Multi-GPU models -Larger models run across the GPUs of a single node by setting the parallelism in -`model.yaml`. The LLM API launches its own worker processes, so no extra -`--world_size` argument is needed on `launch_triton_server.py`: - -```yaml -model: -tensor_parallel_size: 8 -``` +Models run across the GPUs of a single node by setting the parallelism in +`model.yaml`, as the example config above does. The LLM API launches its own +worker processes, so no extra `--world_size` argument is needed on +`launch_triton_server.py`. -For Mixture-of-Experts models, also set the expert parallelism. A bare -`tensor_parallel_size` is often not sufficient — check the model's row in the +For Mixture-of-Experts models, set the expert parallelism alongside the tensor +parallelism. A bare `tensor_parallel_size` is often not sufficient — check the +model's row in the [support matrix](https://nvidia.github.io/TensorRT-LLM/models/supported-models.html) and its example README for required settings. -### Example: DeepSeek-R1 in NVFP4 on 8x B200 - -[`nvidia/DeepSeek-R1-0528-FP4-V2`](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-V2) -is a 671B-parameter MoE quantized to NVFP4 with an FP8 KV cache. Quantization -brings the checkpoint from 642 GB (FP8) down to 385 GB, so it fits comfortably on -one 8x B200 node with room left for the KV cache. NVFP4 requires Blackwell -(`SM100+`). - -```yaml -model: nvidia/DeepSeek-R1-0528-FP4-V2 -backend: "pytorch" -tensor_parallel_size: 8 -moe_expert_parallel_size: 8 -max_seq_len: 4096 -max_num_tokens: 8192 -kv_cache_config: - free_gpu_memory_fraction: 0.7 +### Sizing: DeepSeek-R1 NVFP4 on 8x B200 -triton_config: - max_batch_size: 0 - decoupled: False -``` +NVFP4 quantization brings the DeepSeek-R1 checkpoint from 642 GB (FP8) down to +385 GB, so it fits on one 8x B200 node with room to spare for the KV cache. On +that node, the configuration above measured: -The quantization format is read from `hf_quant_config.json` in the checkpoint -(`quant_algo: NVFP4`, `kv_cache_quant_algo: FP8`) — you do not set it in -`model.yaml`. +| | | +| --- | --- | +| Time to `Started HTTPService` | ~6 min (warm local cache) | +| GPU memory in use | ~145 GiB of each B200's 183 GiB | +| Paged KV cache | 81.83 GiB (2,500,608 tokens) | -Loading 385 GB takes several minutes on first start (about 6 minutes from a -warm local cache). Watch for `Started HTTPService` before sending requests. Once -resident, this configuration uses roughly 145 GiB of each B200's 183 GiB, -leaving headroom for the KV cache. +Loading 163 shards takes several minutes on first start. Watch for +`Started HTTPService` before sending requests. > [!IMPORTANT] > `generate` is a **raw completion** endpoint — it does not apply the model's @@ -307,9 +313,11 @@ python3 TensorRT-LLM/triton_backend/tools/inflight_batcher_llm/benchmark_core_mo --tensorrt-llm-model-name tensorrt_llm \ --test-llmapi \ dataset --dataset TensorRT-LLM/triton_backend/tools/dataset/mini_cnn_eval.json \ - --tokenizer-dir Qwen/Qwen3-8B + --tokenizer-dir ``` +Sample output (Qwen3-8B on a single GPU): + ``` Tokenizer: Tokens per word = 1.324 [INFO] Warm up for benchmarking. From 1f759af849fc4fdac5a5012dcd5902c4e3121d89 Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Mon, 3 Aug 2026 12:10:49 -0700 Subject: [PATCH 4/7] docs: Tighten the LLM getting-started guide Cut the guide down to the DeepSeek-R1 NVFP4 path only. - Shorten the intro to name the example model. - Reduce the openai version conflict to a one-line hint. - Drop the tag-pinning rationale, the NVFP4 format explanation, the launch_triton_server.py background-process note, and the chat template cross-reference. - Remove the Qwen3-8B single-GPU config, the DeepSeek-V4 and GLM notes, and the separate multi-GPU section; fold its measured sizing into step 4. - Show the working chat-templated request directly instead of a plain prompt plus an explanation of why it misbehaves. - Replace the GenAI-Perf incompatibility note with just the command that works. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- docs/getting_started/llm.md | 203 +++++++----------------------------- 1 file changed, 39 insertions(+), 164 deletions(-) diff --git a/docs/getting_started/llm.md b/docs/getting_started/llm.md index d4c43d7513..ebdbcb6696 100644 --- a/docs/getting_started/llm.md +++ b/docs/getting_started/llm.md @@ -42,18 +42,11 @@ supported Hugging Face models directly — no TensorRT engine compilation requir This guide uses [`nvidia/DeepSeek-R1-0528-FP4-V2`](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-V2) -as the example model — a 671B-parameter Mixture-of-Experts model quantized to -NVFP4 with an FP8 KV cache, served across one 8x B200 node. For a quick -single-GPU trial, use a smaller model such as -[`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B) with -`tensor_parallel_size: 1`. You can serve any model listed in the TensorRT-LLM -[support matrix](https://nvidia.github.io/TensorRT-LLM/models/supported-models.html) -by changing a single line in `model.yaml`. +as the example model. - [Serve the model with Triton](#serve-the-model-with-triton) - [Send an inference request](#send-an-inference-request) - [Streaming responses](#streaming-responses) -- [Multi-GPU models](#multi-gpu-models) - [Benchmark](#benchmark) - [References](#references) @@ -72,40 +65,20 @@ Check [NGC](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver/t for the latest `-trtllm-python-py3` tag. For gated models, set your token first: `export HF_TOKEN=hf_...` -> [!IMPORTANT] -> The `26.03` and `26.07` containers (verified; the tags in between are likely -> affected too) ship `openai==1.107.3`, which is too old for the bundled -> `tensorrt_llm` package — TensorRT-LLM declares `openai` as a dependency with no -> lower bound. Without the fix below, loading the model fails with: -> -> ``` -> ImportError: cannot import name 'PartReasoningText' from -> 'openai.types.responses.response_content_part_added_event' -> ``` -> -> Upgrade the package inside the container before starting the server: -> -> ```bash -> pip install -U openai -> ``` +If the server later fails with `ImportError: cannot import name +'PartReasoningText'`, the container's `openai` package is too old — run +`pip install -U openai` and try again. ### 2. Get the model repository The Triton model repository for the LLM API backend lives in the TensorRT-LLM -repo. Clone the tag that matches the `tensorrt_llm` version inside your -container, so that the `model.py` you run matches the library it imports: +repo. Clone the tag matching the `tensorrt_llm` version in your container: ```bash python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)" # e.g. 1.2.1 git clone --depth 1 --branch v1.2.1 https://github.com/NVIDIA/TensorRT-LLM.git ``` -> [!NOTE] -> Prefer the matching tag over the default branch. `main` tracks the next release -> (currently `1.3.0rc*`); it happens to work against a `1.2.1` container today, -> but nothing guarantees that, since its `model.py` is developed against the -> unreleased library. Pinning keeps the guide reproducible. - ### 3. Configure your model Edit `TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml` @@ -130,23 +103,6 @@ All keys outside `triton_config` map directly to the [`LLM()` constructor arguments](https://nvidia.github.io/TensorRT-LLM/llm-api/). This is where you configure KV cache, quantization, and parallelism. -The NVFP4 quantization format is read from `hf_quant_config.json` in the -checkpoint (`quant_algo: NVFP4`, `kv_cache_quant_algo: FP8`) — you do not declare -it in `model.yaml`. NVFP4 requires Blackwell (`SM100+`) GPUs. - -To run on a single GPU instead, swap in a smaller model and drop the parallelism: - -```yaml -model: Qwen/Qwen3-8B -backend: "pytorch" -tensor_parallel_size: 1 -pipeline_parallel_size: 1 - -triton_config: - max_batch_size: 0 - decoupled: False -``` - ### 4. Launch the server Run the launch script from the parent of `TensorRT-LLM/`. Running it from inside @@ -162,37 +118,41 @@ python3 TensorRT-LLM/triton_backend/scripts/launch_triton_server.py \ You should see the following logs once the server is ready: ``` -I0803 04:19:38.396545 3606543 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001" -I0803 04:19:38.396742 3606543 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000" -I0803 04:19:38.437685 3606543 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002" +I0803 18:43:44.778509 1525575 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001" +I0803 18:43:44.778681 1525575 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000" +I0803 18:43:44.819624 1525575 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002" ``` -> [!NOTE] -> `launch_triton_server.py` starts Triton as a background process and returns -> immediately. It is designed for an interactive shell. If you wrap it in a -> script, a batch job, or `srun`, keep the parent process alive or the server is -> killed when your script exits. - -To stop Triton Server inside the container, run `pkill tritonserver`. +On an 8x B200 node this configuration takes about 6 minutes to become ready and +uses roughly 145 GiB of each GPU's 183 GiB, with 81.83 GiB left for the paged KV +cache (2,500,608 tokens). ## Send an inference request +`generate` is a raw completion endpoint and does not apply the model's chat +template, so format the prompt using the template from the checkpoint's +`tokenizer_config.json`: + ```bash -curl -X POST localhost:8000/v2/models/tensorrt_llm/generate \ - -d '{"text_input": "How do I count to nine in French?", "sampling_param_max_tokens": 256}' | jq +curl -X POST localhost:8000/v2/models/tensorrt_llm/generate -d '{ + "text_input": "<｜begin▁of▁sentence｜><｜User｜>How do I count to nine in French?<｜Assistant｜>", + "sampling_param_max_tokens": 512, + "sampling_param_exclude_input_from_output": true }' | jq +``` + +DeepSeek-R1 emits its reasoning in a `` block before the answer: + +``` + +Okay, the user is asking how to count to nine in French. That seems +straightforward—they probably need the French numbers from one to nine. +... ``` Sampling options are passed as `sampling_param_*` inputs — see the `input` section of `TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/config.pbtxt` for the -full list. Add `"sampling_param_exclude_input_from_output": true` to get only the -generated text back instead of prompt + completion. - -> [!NOTE] -> This endpoint performs raw text completion and does **not** apply the model's -> chat template. For instruct and reasoning models you must format the prompt -> yourself — see -> [Example: DeepSeek-R1 in NVFP4 on 8x B200](#example-deepseek-r1-in-nvfp4-on-8x-b200). +full list. ## Streaming responses @@ -219,91 +179,18 @@ curl -N -X POST localhost:8000/v2/models/tensorrt_llm/generate_stream \ -d '{"text_input": "Count to three:", "sampling_param_max_tokens": 10, "streaming": true}' ``` -``` -data: {"model_name":"tensorrt_llm","model_version":"1","text_output":" 1"} -data: {"model_name":"tensorrt_llm","model_version":"1","text_output":" 1,"} -data: {"model_name":"tensorrt_llm","model_version":"1","text_output":" 1, 2"} -``` - > [!NOTE] -> Incremental, token-by-token events require the model repository from -> TensorRT-LLM 1.3 or newer. With the `v1.2.x` `model.py`, a `streaming: true` -> request is accepted but the server emits a single event containing the full -> response. - -> [!NOTE] -> A decoupled model cannot be used over the non-streaming HTTP `generate` -> endpoint — Triton returns -> `[501] HTTP end point doesn't support models with decoupled transaction policy`. -> Keep `decoupled: False` unless you need streaming. - -## Multi-GPU models - -Models run across the GPUs of a single node by setting the parallelism in -`model.yaml`, as the example config above does. The LLM API launches its own -worker processes, so no extra `--world_size` argument is needed on -`launch_triton_server.py`. - -For Mixture-of-Experts models, set the expert parallelism alongside the tensor -parallelism. A bare `tensor_parallel_size` is often not sufficient — check the -model's row in the -[support matrix](https://nvidia.github.io/TensorRT-LLM/models/supported-models.html) -and its example README for required settings. - -### Sizing: DeepSeek-R1 NVFP4 on 8x B200 - -NVFP4 quantization brings the DeepSeek-R1 checkpoint from 642 GB (FP8) down to -385 GB, so it fits on one 8x B200 node with room to spare for the KV cache. On -that node, the configuration above measured: - -| | | -| --- | --- | -| Time to `Started HTTPService` | ~6 min (warm local cache) | -| GPU memory in use | ~145 GiB of each B200's 183 GiB | -| Paged KV cache | 81.83 GiB (2,500,608 tokens) | - -Loading 163 shards takes several minutes on first start. Watch for -`Started HTTPService` before sending requests. - -> [!IMPORTANT] -> `generate` is a **raw completion** endpoint — it does not apply the model's -> chat template. Sending a bare question to an instruct or reasoning model makes -> it continue the text rather than answer it. DeepSeek-R1 replies to -> `"How do I count to nine in French?"` with a list of scraped page titles. -> Format the prompt yourself using the model's template: -> -> ```bash -> curl -X POST localhost:8000/v2/models/tensorrt_llm/generate -d '{ -> "text_input": "<｜begin▁of▁sentence｜><｜User｜>How do I count to nine in French?<｜Assistant｜>", -> "sampling_param_max_tokens": 512, -> "sampling_param_exclude_input_from_output": true }' -> ``` -> -> R1 then emits its reasoning trace in a `` block before the answer: -> -> ``` -> -> Okay, the user is asking how to count to nine in French. That seems -> straightforward—they probably need the French numbers from one to nine. -> ... -> ``` -> -> Each model family uses a different template — read `chat_template` in the -> checkpoint's `tokenizer_config.json`. - -> [!NOTE] -> DeepSeek-V4 (`DeepSeek-V4-Flash` / `-Pro`) is **not** usable with these -> containers. It requires TensorRT-LLM 1.3 or newer, which is not yet GA and is -> not in `26.07` or earlier — those ship TensorRT-LLM 1.2.x, which does not -> register `DeepseekV4ForCausalLM`. Loading it fails with -> `The checkpoint you are trying to load has model type 'deepseek_v4' but -> Transformers does not recognize this architecture.` The same applies to GLM-5.x -> (`glm_moe_dsa`). Use DeepSeek-R1 or GLM-4.7 until a container ships 1.3. +> Token-by-token events require the model repository from TensorRT-LLM 1.3 or +> newer. With `v1.2.x`, a `streaming: true` request is accepted but the server +> emits a single event containing the full response. A decoupled model also +> cannot be used over the non-streaming `generate` endpoint, which returns +> `[501] HTTP end point doesn't support models with decoupled transaction +> policy` — keep `decoupled: False` unless you need streaming. ## Benchmark -The LLM API backend ships its own benchmarking client. Install the Triton client -in the server container and run it against the model, with `decoupled: False`: +Install the Triton client in the server container and run the backend's +benchmarking client against the model, with `decoupled: False`: ```bash pip install "tritonclient[grpc,http]" @@ -313,27 +200,15 @@ python3 TensorRT-LLM/triton_backend/tools/inflight_batcher_llm/benchmark_core_mo --tensorrt-llm-model-name tensorrt_llm \ --test-llmapi \ dataset --dataset TensorRT-LLM/triton_backend/tools/dataset/mini_cnn_eval.json \ - --tokenizer-dir + --tokenizer-dir nvidia/DeepSeek-R1-0528-FP4-V2 ``` -Sample output (Qwen3-8B on a single GPU): - ``` -Tokenizer: Tokens per word = 1.324 [INFO] Warm up for benchmarking. [INFO] Start benchmarking on 37 prompts. -[INFO] Total Latency: 853.993 ms +[INFO] Total Latency: ``` -> [!NOTE] -> GenAI-Perf's `--backend tensorrtllm` mode targets the legacy -> `inflight_batcher_llm` model, whose inputs are named `max_tokens` and `stream`. -> The LLM API backend names them `sampling_param_max_tokens` and `streaming`, so -> GenAI-Perf fails with -> `Failed to init manager inputs: The input or output 'max_tokens' is not found in -> the model configuration`. Use the client above until GenAI-Perf adds LLM API -> support. - > [!NOTE] > The shipped `model.yaml` sets `max_batch_size: 0`, so the backend serves > requests without batching. Concurrency sweeps will not show throughput scaling From ed5dd00ae64d7e2a0263abb2f002980ac200c2d1 Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Thu, 6 Aug 2026 11:13:35 -0700 Subject: [PATCH 5/7] docs: Derive the TensorRT-LLM clone tag from the container The guide pinned the container tag and the TensorRT-LLM clone tag separately, so updating one without the other silently mismatched the model repository against the installed library. Read the version from the running container and clone that tag, leaving RELEASE as the only version to edit. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- docs/getting_started/llm.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/getting_started/llm.md b/docs/getting_started/llm.md index ebdbcb6696..3674770cd1 100644 --- a/docs/getting_started/llm.md +++ b/docs/getting_started/llm.md @@ -75,10 +75,13 @@ The Triton model repository for the LLM API backend lives in the TensorRT-LLM repo. Clone the tag matching the `tensorrt_llm` version in your container: ```bash -python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)" # e.g. 1.2.1 -git clone --depth 1 --branch v1.2.1 https://github.com/NVIDIA/TensorRT-LLM.git +TRTLLM_VERSION=$(python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)") +git clone --depth 1 --branch "v${TRTLLM_VERSION}" https://github.com/NVIDIA/TensorRT-LLM.git ``` +Deriving the tag this way keeps the model repository in step with the container, +so changing `RELEASE` above needs no second edit here. + ### 3. Configure your model Edit `TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/1/model.yaml` From f4b2779bf17634a927fb28abec67ac0fcd25f2dd Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Thu, 6 Aug 2026 11:19:56 -0700 Subject: [PATCH 6/7] docs: Drop the streaming and batching caveats Condense the streaming section to the config.pbtxt change itself and trim the batching note to the fact, without the projection about future throughput behaviour. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- docs/getting_started/llm.md | 22 ++++------------------ 1 file changed, 4 insertions(+), 18 deletions(-) diff --git a/docs/getting_started/llm.md b/docs/getting_started/llm.md index 3674770cd1..517b959fbf 100644 --- a/docs/getting_started/llm.md +++ b/docs/getting_started/llm.md @@ -159,14 +159,9 @@ full list. ## Streaming responses -Streaming requires Triton's decoupled transaction policy. Setting -`decoupled: True` in `model.yaml` alone is **not** enough: the launch script -passes `--disable-auto-complete-config`, which skips the `auto_complete_config()` -hook where `model.yaml`'s `triton_config` is applied. The server then streams -while Triton core still treats the model as non-decoupled, and the request hangs -with `Streaming is only supported in decoupled mode.` in the server log. - -To enable streaming, also append the policy to +Streaming requires Triton's decoupled transaction policy, which must be set in +`config.pbtxt` — setting `decoupled: True` in `model.yaml` alone has no effect. +Append the policy to `TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/config.pbtxt`: ``` @@ -182,14 +177,6 @@ curl -N -X POST localhost:8000/v2/models/tensorrt_llm/generate_stream \ -d '{"text_input": "Count to three:", "sampling_param_max_tokens": 10, "streaming": true}' ``` -> [!NOTE] -> Token-by-token events require the model repository from TensorRT-LLM 1.3 or -> newer. With `v1.2.x`, a `streaming: true` request is accepted but the server -> emits a single event containing the full response. A decoupled model also -> cannot be used over the non-streaming `generate` endpoint, which returns -> `[501] HTTP end point doesn't support models with decoupled transaction -> policy` — keep `decoupled: False` unless you need streaming. - ## Benchmark Install the Triton client in the server container and run the backend's @@ -214,8 +201,7 @@ python3 TensorRT-LLM/triton_backend/tools/inflight_batcher_llm/benchmark_core_mo > [!NOTE] > The shipped `model.yaml` sets `max_batch_size: 0`, so the backend serves -> requests without batching. Concurrency sweeps will not show throughput scaling -> until batching support lands. +> requests without batching. ## References From 69bf57edc033743d54606d11ea60322bb0b29f0f Mon Sep 17 00:00:00 2001 From: Faradawn Yang <73060648+faradawn@users.noreply.github.com> Date: Thu, 6 Aug 2026 12:02:27 -0700 Subject: [PATCH 7/7] docs: Fix two broken commands found by verification on 8x B200 A subagent ran the guide verbatim on an 8x B200 node (only substituting srun/pyxis for docker run). Two commands failed as written. - The derived clone tag did not work. `import tensorrt_llm` prints its banner to stdout, so the command substitution captured two lines and git failed with "Remote branch v[TensorRT-LLM] TensorRT LLM version: 1.2.1 1.2.1 not found in upstream origin". Go back to printing the version and passing the tag explicitly, which is verified to work. - Streaming did not work. Editing only config.pbtxt leaves model.py's own guard reading triton_config.decoupled from model.yaml, so the request fails with "Streaming is only supported in decoupled mode." Setting decoupled: True in model.yaml is what actually enables it. Also correct three measurements against the verification run: load takes a few minutes rather than about six, memory is ~143 GiB of each GPU's 179 GiB, and the benchmark reports 38 prompts at 1694.09 ms. Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com> --- docs/getting_started/llm.md | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/docs/getting_started/llm.md b/docs/getting_started/llm.md index 517b959fbf..602476df49 100644 --- a/docs/getting_started/llm.md +++ b/docs/getting_started/llm.md @@ -75,12 +75,13 @@ The Triton model repository for the LLM API backend lives in the TensorRT-LLM repo. Clone the tag matching the `tensorrt_llm` version in your container: ```bash -TRTLLM_VERSION=$(python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)") -git clone --depth 1 --branch "v${TRTLLM_VERSION}" https://github.com/NVIDIA/TensorRT-LLM.git +python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)" # e.g. 1.2.1 +git clone --depth 1 --branch v1.2.1 https://github.com/NVIDIA/TensorRT-LLM.git ``` -Deriving the tag this way keeps the model repository in step with the container, -so changing `RELEASE` above needs no second edit here. +Use the version the first command reports as the tag for the second. If you +change `RELEASE` above, re-run both so the model repository stays in step with +the library in the container. ### 3. Configure your model @@ -126,8 +127,8 @@ I0803 18:43:44.778681 1525575 http_server.cc:4961] "Started HTTPService at 0.0.0 I0803 18:43:44.819624 1525575 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002" ``` -On an 8x B200 node this configuration takes about 6 minutes to become ready and -uses roughly 145 GiB of each GPU's 183 GiB, with 81.83 GiB left for the paged KV +On an 8x B200 node this configuration takes a few minutes to become ready and +uses roughly 143 GiB of each GPU's 179 GiB, with 81.83 GiB left for the paged KV cache (2,500,608 tokens). ## Send an inference request @@ -159,15 +160,12 @@ full list. ## Streaming responses -Streaming requires Triton's decoupled transaction policy, which must be set in -`config.pbtxt` — setting `decoupled: True` in `model.yaml` alone has no effect. -Append the policy to -`TensorRT-LLM/triton_backend/all_models/llmapi/tensorrt_llm/config.pbtxt`: +Streaming requires decoupled mode. Set it in `model.yaml`: -``` -model_transaction_policy { +```yaml +triton_config: + max_batch_size: 0 decoupled: True -} ``` Then restart the server and use the `generate_stream` endpoint: @@ -195,8 +193,8 @@ python3 TensorRT-LLM/triton_backend/tools/inflight_batcher_llm/benchmark_core_mo ``` [INFO] Warm up for benchmarking. -[INFO] Start benchmarking on 37 prompts. -[INFO] Total Latency: +[INFO] Start benchmarking on 38 prompts. +[INFO] Total Latency: 1694.09 ms ``` > [!NOTE]