Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,6 @@ images.
- [Install Triton Inference Server without Docker containers](docs/customization_guide/build.md#building-without-docker)
- [Build a custom Triton Inference Server Docker container](docs/customization_guide/compose.md)
- [Build Triton Inference Server from source](docs/customization_guide/build.md#building-on-unsupported-platforms)
- [Build Triton Inference Server for Windows 10](docs/customization_guide/build.md#building-for-windows-10)
- Examples for deploying Triton Inference Server with Kubernetes and Helm on [GCP](deploy/gcp/README.md),
[AWS](deploy/aws/README.md), and [NVIDIA FleetCommand](deploy/fleetcommand/README.md)
- [Secure Deployment Considerations](docs/customization_guide/deploy.md)
Expand Down
11 changes: 5 additions & 6 deletions docs/client_guide/in_process.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
..
.. Copyright 2024-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
.. Copyright 2024-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
..
.. Redistribution and use in source and binary forms, with or without
.. modification, are permitted provided that the following conditions
Expand Down Expand Up @@ -35,11 +35,10 @@ allows Triton to be linked directly into a C/C++/java/python application. This A
is called the "Triton Server API" or just "Server API" for short. The
API is implemented in the Triton shared library which is built from
source contained in the `core
repository <https://github.com/triton-inference-server/core>`__. On Linux
this library is libtritonserver.so and on Windows it is
tritonserver.dll. In the Triton Docker image the shared library is
found in /opt/tritonserver/lib. The header file that defines and
documents the Server API is
repository <https://github.com/triton-inference-server/core>`__. On Linux this
library is libtritonserver.so. In the Triton Docker image the shared library is
found in /opt/tritonserver/lib. The header file that defines and documents the
Server API is
`tritonserver.h <https://github.com/triton-inference-server/core/blob/main/include/triton/core/tritonserver.h>`__.
`Java bindings for In-Process Triton Server API <../customization_guide/inprocess_java_api.html#java-bindings-for-in-process-triton-server-api>`__
are built on top of `tritonserver.h` and can be used for Java applications that
Expand Down
113 changes: 11 additions & 102 deletions docs/customization_guide/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,6 @@ to build Triton on a platform that is not listed here.

* [Ubuntu 22.04, x86-64](#building-for-ubuntu-2204)

* [Windows 10, x86-64](#building-for-windows-10)

If you are developing or debugging Triton, see [Development and
Incremental Builds](#development-and-incremental-builds) for information
on how to perform incremental build.
Expand Down Expand Up @@ -261,90 +259,6 @@ For a given version of Triton you can attempt to build with
non-supported versions of TensorRT but you may have build or execution
issues since non-supported versions are not tested.

## Building for Windows 10

For Windows 10, build.py supports both a Docker build and a non-Docker
build in a similar way as described for [Ubuntu](#building-for-ubuntu-2204). The primary
difference is that the minimal/base image used as the base of
Dockerfile.buildbase image can be built from the provided
[Dockerfile.win10.min](https://github.com/triton-inference-server/server/blob/main/Dockerfile.win10.min)
file as described in [Windows 10 "Min" Image](#windows-10-min-image). When running build.py
use the --image flag to specify the tag that you assigned to this
image. For example, --image=base,win10-py3-min.

### Windows and Docker

Depending on your version of Windows 10 and your version of Docker you
may need to perform these additional steps before any of the following
step.

* Set your Docker to work with "Windows containers". Right click on
the whale icon in the lower-right status area and select "Switch to
Windows containers".

### Windows 10 "Min" Image

The "min" container describes the base dependencies needed to perform
the Windows build. The Windows min container is
[Dockerfile.win10.min](https://github.com/triton-inference-server/server/blob/main/Dockerfile.win10.min).

Before building the min container you must download the appropriate
cuDNN and TensorRT versions and place them in the same directory as
Dockerfile.win10.min.

* For cuDNN the CUDNN_VERSION and CUDNN_ZIP arguments defined in
Dockerfile.win10.min indicate the version of cuDNN that your should
download from https://developer.nvidia.com/rdp/cudnn-download.

* For TensorRT the TENSORRT_VERSION and TENSORRT_ZIP arguments defined
in Dockerfile.win10.min indicate the version of TensorRT that your
should download from
https://developer.nvidia.com/nvidia-tensorrt-download.

After downloading the zip files for cuDNN and TensorRT, you build the
min container using the following command.

```bash
$ docker build -t win10-py3-min -f Dockerfile.win10.min .
```

### Build Triton Server

Triton is built using the build.py script. The build system must have
Docker, Python3 (plus pip installed *docker* module) and git installed
so that it can execute build.py and perform a docker build. By
default, build.py does not enable any of Triton's optional features
and so you must enable them explicitly. The following build.py
invocation builds all features and backends available on windows.

```bash
python build.py --cmake-dir=<path/to/repo>/build --build-dir=/tmp/citritonbuild --no-container-pull --image=base,win10-py3-min --enable-logging --enable-stats --enable-tracing --enable-gpu --endpoint=grpc --endpoint=http --repo-tag=common:<container tag> --repo-tag=core:<container tag> --repo-tag=backend:<container tag> --repo-tag=thirdparty:<container tag> --backend=ensemble --backend=tensorrt:<container tag> --backend=onnxruntime:<container tag> --backend=openvino:<container tag> --backend=python:<container tag>
```

If you are building on *main* branch then `<container tag>` will
default to "main". If you are building on a release branch then
`<container tag>` will default to the branch name. For example, if you
are building on the r24.12 branch, `<container tag>` will default to
r24.12. Therefore, you typically do not need to provide `<container
tag>` at all (nor the preceding colon). You can use a different
`<container tag>` for a component to instead use the corresponding
branch/tag in the build. For example, if you have a branch called
"mybranch" in the
[onnxruntime_backend](https://github.com/triton-inference-server/onnxruntime_backend)
repo that you want to use in the build, you would specify
--backend=onnxruntime:mybranch.

### Extract Build Artifacts

When build.py completes, a Docker image called *tritonserver* will
contain the built Triton Server executable, libraries and other
artifacts. Windows containers do not support GPU access so you likely
want to extract the necessary files from the tritonserver image and
run them directly on your host system. All the Triton artifacts can be
found in /opt/tritonserver directory of the tritonserver image. Your
host system will need to install the CUDA, cuDNN, TensorRT and other
dependencies that were used for the build.

## Building on Unsupported Platforms

Building for an unsupported OS and/or hardware platform is
Expand Down Expand Up @@ -409,8 +323,8 @@ and cmake_build or the equivalent commands to perform a build.

If you are [building without Docker](#building-without-docker) use the
CMake invocation steps in cmake_build to invoke CMake to set-up a
build environment where you can invoke make/msbuild.exe to incremental
build the Triton core, a backend, or a repository agent.
build environment where you can invoke make to incremental build the Triton
core, a backend, or a repository agent.

### Development Builds With Docker

Expand All @@ -423,8 +337,7 @@ agents.

To perform an incremental build within the *tritonserver_buildbase*
container, map your source into the container and then run the
appropriate CMake and `make` (or `msbuild.exe`) steps from cmake_build
within the container.
appropriate CMake and `make` steps from cmake_build within the container.

#### Development Build of Triton Core

Expand All @@ -450,10 +363,9 @@ CMakeLists.txt file and source:
$ cmake <options> /server
```

Then you can change directory into the build directory and run `make`
(or `msbuild.exe`) as shown in cmake_build. As you make changes to the
source on your host system, you can perform incremental builds by
re-running `make` (or `msbuild.exe`).
Then you can change directory into the build directory and run `make` as shown
in cmake_build. As you make changes to the source on your host system, you can
perform incremental builds by re-running `make`.

#### Development Build of Backend or Repository Agent

Expand All @@ -467,10 +379,8 @@ incremental builds to test those changes. Your source code is in
/home/me/tritonserver_backend. Run the *tritonserver_buildbase*
container and map your TensorRT backend source directory into the
container at /tensorrt_backend. Note that some backends will use
Docker as part of their build, and so the host's Docker registry must
be made available within the *tritonserver_buildbase* by mounting
docker.sock (on Windows use
-v\\.\pipe\docker_engine:\\.\pipe\docker_engine).
Docker as part of their build, and so the host's Docker registry must be made
available within the *tritonserver_buildbase* by mounting docker.sock.

```
$ docker run -it --rm -v/var/run/docker.sock:/var/run/docker.sock -v/home/me/tensorrt_backend:/tensorrt_backend tritonserver_buildbase bash
Expand All @@ -487,10 +397,9 @@ CMakeLists.txt file and source:
$ cmake <options> /tensorrt_backend
```

Then you can change directory into the build directory and run `make`
(or `msbuild.exe`) as shown in cmake_build. As you make changes to the
source on your host system, you can perform incremental builds by
re-running `make` (or `msbuild.exe`).
Then you can change directory into the build directory and run `make` as shown
in cmake_build. As you make changes to the source on your host system, you can
perform incremental builds by re-running `make`.

### Building with Debug Symbols

Expand Down
12 changes: 0 additions & 12 deletions docs/customization_guide/deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,18 +279,6 @@ Directory where cache shared libraries are found.
> must be access controlled. Adding untrusted files
> can lead to arbitrarty code execution.

##### `backend-config=<backend>,additional-dependency-dirs=<string>`

This is an optional Windows feature that enables Triton to search custom
dependency directories when loading a specific backend. The user can input
these directories as a string of semicolon-separated paths (including a
trailing semicolon). These directories are programmatically prepended to
the process's PATH and are removed when the backend is loaded successfully.
Windows will search PATH last in its search sequence, so be cautious that
no untrusted files of same name exist in a location of higher search priority
(e.g., System32). It is still recommended to add backend-specific dependencies
to their corresponding backend folder when possible.

# GRPC server options
Triton Inference Server's gRPC inference handlers internally use states to manage inference requests and response queues. Each state consists of one inference request and one response queue. The response queue within a state can hold multiple response objects. These states remain allocated for reuse to optimize performance by minimizing dynamic allocations.

Expand Down
4 changes: 2 additions & 2 deletions docs/protocol/extension_shared_memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ given Triton will return an error.

> [!NOTE]
>
> Shared memory is not supported on Windows. On Jetson, only system shared
> memory is supported (CUDA shared memory is not).
> On Jetson, only system shared memory is supported (CUDA shared memory is
> not).

## HTTP/REST

Expand Down
2 changes: 0 additions & 2 deletions docs/user_guide/trace.md
Original file line number Diff line number Diff line change
Expand Up @@ -641,8 +641,6 @@ spans are properly ended.

### Limitations

- OpenTelemetry trace mode is not supported on Windows systems.

- Triton supports only
[OTLP/HTTP Exporter](https://opentelemetry.io/docs/specs/otlp/#otlphttp)
and allows specification of only url for this exporter through
Expand Down
Loading