Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Original file line number Diff line number Diff line change
Expand Up @@ -414,13 +414,13 @@ Note that, `max_batch_size` and `max_num_tokens` can easily affect the performan

Generally, you should make sure that `max_batch_size` is not too low to bottleneck the throughput, and `max_num_tokens` needs to be large enough so that it covers the max input sequence length of the samples in dataset, as mentioned in below section "WIP: Chunked context support on DeepSeek models".

For more details on `max_batch_size` and `max_num_tokens`, refer to [Tuning Max Batch Size and Max Num Tokens](../legacy/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md).
For more details on `max_batch_size` and `max_num_tokens`, refer to the performance tuning guidance in the developer documentation.

### MLA chunked context

MLA currently supports the chunked context feature on both Hopper and Blackwell GPUs. You can use `--enable_chunked_context` to enable it. This feature is primarily designed to reduce TPOT (Time Per Output Token). The default chunk size is set to `max_num_tokens`. If you want to achieve a lower TPOT, you can appropriately reduce the chunk size. However, please note that this will also decrease overall throughput. Therefore, a trade-off needs to be considered.

For more details on `max_num_tokens`, refer to [Tuning Max Batch Size and Max Num Tokens](../legacy/performance/performance-tuning-guide/tuning-max-batch-size-and-max-num-tokens.md).
For more details on `max_num_tokens`, refer to the performance tuning guidance in the developer documentation.

### Out of memory issues

Expand Down
2 changes: 1 addition & 1 deletion docs/source/blogs/XQA-kernel.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

XQA kernel provides optimization for [MQA](https://arxiv.org/abs/1911.02150) and [GQA](https://arxiv.org/abs/2305.13245v3) during generation phase. It also provides optimization for beam search. Using tensor cores for acceleration, reducing data loading and conversion, it delivers increased throughput within the same latency budget. Increased throughput allows serving greater number of user requests while providing the same experience.

Support matrix and usage flags are described in [docs/source/legacy/advanced/gpt-attention](../legacy/advanced/gpt-attention.md#xqa-optimization).
Support matrix and usage flags are described in the [Attention](../features/attention.md) documentation.

**Increased Throughput:**
Looking at the Throughput-Latency curves below, we see that the enabling of XQA optimization increases throughput. Higher throughput equates to serving more users, and we can see that TPOT on the Y-axis flattens out when XQA gets enabled.
Expand Down
7 changes: 0 additions & 7 deletions docs/source/commands/trtllm-build.rst

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@ Note:
This feature is optimized for PCIe-based GPU topologies and may affect model accuracy. Please evaluate precision impact for your specific workload.
```


TRT-LLM supports `low-precision-allreduce`, a communication optimization that accelerates AllReduce operations in PCIe-based GPU environments. This feature quantizes FP16/BF16 data to FP8 during network transmission, reducing communication volume and improving performance.
TensorRT LLM supports `low-precision-allreduce`, a communication optimization that accelerates AllReduce operations in PCIe-based GPU environments. This feature quantizes FP16/BF16 data to FP8 during network transmission, reducing communication volume and improving performance. It is available on the PyTorch backend through the `allreduce_strategy` field of `LlmArgs`.

## Algorithm

Expand Down Expand Up @@ -35,20 +34,16 @@ Low-Precision-AllReduce is specifically designed for the topology shown above, w

## Usage

The Low-Precision-AllReduce algorithm can be enabled in two ways:
Enable the Low-Precision-AllReduce algorithm by setting the `allreduce_strategy` field in `LlmArgs`:

1. **Direct specification** in your code:
```
AllReduce allreduce(mapping=mapping, strategy=AllReduceStrategy.LOWPRECISION);
```
```python
from tensorrt_llm import LLM

2. Enable by LlmArgs
```
Set allreduce_strategy field in LlmArgs.
Candidates of strategies are "AUTO", "NCCL", "UB", "MINLATENCY", "ONESHOT", "TWOSHOT", "LOWPRECISION" and "MNNVL".
If no strategy is set, AUTO will be set.
llm = LLM(model="<hf_model>", allreduce_strategy="LOWPRECISION")
```

The candidate strategies are `"AUTO"`, `"NCCL"`, `"UB"`, `"MINLATENCY"`, `"ONESHOT"`, `"TWOSHOT"`, `"LOWPRECISION"` and `"MNNVL"`. If no strategy is set, `AUTO` is used.

## Performance and Accuracy Considerations

Low-Precision-AllReduce reduces communication volume by using FP8 data format for transmission. This optimization:
Expand All @@ -58,4 +53,4 @@ Low-Precision-AllReduce reduces communication volume by using FP8 data format fo

Users should evaluate the precision impact on their specific models and workloads.

**Note**: When compiling TensorRT-LLM without enabling the `ENABLE_FP8` option, setting Low Precision allreduce will not take effect.
**Note**: When TensorRT LLM is built without the `ENABLE_FP8` option, setting the low-precision AllReduce strategy will not take effect.
6 changes: 3 additions & 3 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ Welcome to TensorRT LLM's Documentation!
features/overlap-scheduler.md
features/paged-attention-ifb-scheduler.md
features/parallel-strategy.md
features/lowprecision-allreduce.md
features/quantization.md
features/sampling.md
features/additional-outputs.md
Expand Down Expand Up @@ -127,10 +128,9 @@ Welcome to TensorRT LLM's Documentation!

.. toctree::
:maxdepth: 2
:caption: Use TensorRT Engine
:hidden:
:caption: Migration

legacy/tensorrt_quickstart.md
legacy/tensorrt-backend-removal.md

Indices and tables
==================
Expand Down
86 changes: 0 additions & 86 deletions docs/source/legacy/advanced/disaggregated-service.md

This file was deleted.

Loading
Loading