Triton Server segfaults when Model Analyzer reloads models in Ensemble

[corentin87]

Description

I am using Model Analyzer to profile an ensemble model on a remote Triton Server
running in explicit model control mode. Profiling works fine when
reload_model_disable: true is set (concurrency sweep only, no model reloads).

However, when I want to extend the search space (no using default config.pbtxt) and Model Analyzer is allowed to reload models (the default behavior), Triton Server crashes with a segmentation fault when it tries to reload some backend composing model.

Environment

• Triton Server version: customised image on top of 26.02 including opencv and DALI python dependencies.
• Model Analyzer nvcr.io/nvidia/tritonserver:26.02-py3-sdk
• GPU: NVIDIA RTX A2000
• Setup: Model Analyzer and Triton Server running in separate containers.

Triton started with
tritonserver --model-repository /model_repository_path/ --model-control-mode explicit.
Model Analyzer started with
model-analyzer profile -f /model_analyzer/config.yml --override-output-model-repository --triton-launch-mode remote -v

Ensemble Architecture

The ensemble model (ensemble_model) has three composing models:

Model	Backend	Instance Kind
detect_preprocessing	Python	KIND_CPU
rt-detr	TensorRT	KIND_GPU
detect_postprocessing_reid	Python	KIND_CPU

Model Analyzer Config

model_repository: /model_repository_path/ 
output_model_repository_path: /outputs
export_path : /results

run_config_search_mode: quick  # default for ensemble
run_config_search_min_model_batch_size: 8
run_config_search_max_model_batch_size: 8
run_config_search_max_concurrency: 16
reload_model_disable: true  

profile_models:
  - ensemble_model

ensemble_composing_models:
  rt-detr:
    model_config_parameters:
      instance_group:
        - kind: KIND_GPU
          count: [1]  # Powers of 2 sequence
  detect_postprocessing_reid:
    model_config_parameters:
      instance_group:
        - kind: KIND_CPU
          count: [1]  # Subset of powers of 2
  detect_preprocessing:
    model_config_parameters:
      instance_group:
        - kind: KIND_CPU
          count: [1, 2]

cpu_only_composing_models:
  - detect_postprocessing_reid

perf_analyzer_flags:
  shape:
   - raw_image:1280,1280,3

What Works

With reload_model_disable: true, Model Analyzer successfully profiles all concurrency levels (1, 2, 4, 8, 16) against the already-loaded model configuration. Detect_preprocessing instance count is set to 2.
All perf_analyzer runs complete without issues and results are generated correctly.

What Crashes

Model Analyzer log

11:11:22 [Model Analyzer] DEBUG: 
{'always_report_gpu_metrics': False,
 'batch_sizes': [1],
 'bls_composing_models': [],
 'checkpoint_directory': '/workspace/checkpoints',
 'client_max_retries': 50,
 'client_protocol': 'grpc',
 'collect_cpu_metrics': False,
 'concurrency': [],
 'concurrency_sweep_disable': False,
 'config_file': '/model_analyzer/config_linear_reid-no_pre-python_dm-yes.yml',
 'constraints': {},
 'cpu_only_composing_models': ['detect_postprocessing_reid'],
 'dcgm_disable': False,
 'duration_seconds': 3,
 'early_exit_enable': False,
 'ensemble_composing_models': [{'model_name': 'rt-detr', 'cpu_only': False, 'model_config_parameters': {'instance_group': [[{'kind': ['KIND_GPU'], 'count': [1]}]]}},
                               {'model_name': 'detect_postprocessing_reid', 'cpu_only': False, 'model_config_parameters': {'instance_group': [[{'kind': ['KIND_CPU'], 'count': [1]}]]}},
                               {'model_name': 'detect_preprocessing', 'cpu_only': False, 'model_config_parameters': {'instance_group': [[{'kind': ['KIND_CPU'], 'count': [1, 2]}]]}}],
 'export_path': '/model_analyzer/linear_reid-no_pre-python_dm-yes_1280x1280/results',
 'filename_model_gpu': 'metrics-model-gpu.csv',
 'filename_model_inference': 'metrics-model-inference.csv',
 'filename_server_only': 'metrics-server-only.csv',
 'genai_perf_flags': {},
 'gpu_output_fields': ['model_name',
                       'gpu_uuid',
                       'batch_size',
                       'concurrency',
                       'model_config_path',
                       'instance_group',
                       'satisfies_constraints',
                       'gpu_used_memory',
                       'gpu_utilization',
                       'gpu_power_usage'],
 'gpus': ['all'],
 'inference_output_fields': ['model_name',
                             'batch_size',
                             'concurrency',
                             'model_config_path',
                             'instance_group',
                             'max_batch_size',
                             'satisfies_constraints',
                             'perf_throughput',
                             'perf_latency_p99'],
 'latency_budget': None,
 'max_percentage_of_search_space': 10,
 'min_percentage_of_search_space': 5,
 'min_throughput': None,
 'model_repository': '/models/ensemble_model_repository/linear_reid-no_pre-python_dm-yes',
 'model_type': 'generic',
 'monitoring_interval': 1.0,
 'num_configs_per_model': 3,
 'num_top_model_configs': 0,
 'objectives': {'perf_throughput': 10},
 'optuna_early_exit_threshold': 10,
 'optuna_max_trials': 200,
 'optuna_min_trials': 20,
 'output_model_repository_path': '/model_analyzer/linear_reid-no_pre-python_dm-yes_1280x1280/outputs',
 'override_output_model_repository': True,
 'perf_analyzer_cpu_util': 480.0,
 'perf_analyzer_flags': {'shape': ['raw_image:1280,1280,3']},
 'perf_analyzer_max_auto_adjusts': 10,
 'perf_analyzer_path': 'perf_analyzer',
 'perf_analyzer_timeout': 600,
 'perf_output': False,
 'perf_output_path': None,
 'plots': [{'name': 'throughput_v_latency', 'title': 'Throughput vs. Latency', 'x_axis': 'perf_latency_p99', 'y_axis': 'perf_throughput', 'monotonic': True},
           {'name': 'gpu_mem_v_latency', 'title': 'GPU Memory vs. Latency', 'x_axis': 'perf_latency_p99', 'y_axis': 'gpu_used_memory', 'monotonic': False}],
 'profile_models': [{'model_name': 'ensemble_model', 'cpu_only': False, 'objectives': {'perf_throughput': 10}, 'parameters': {'batch_sizes': [1], 'concurrency': [], 'request_rate': []}, 'weighting': 1, 'perf_analyzer_flags': {'shape': ['raw_image:1280,1280,3']}}],
 'reload_model_disable': False,
 'request_rate': [],
 'request_rate_search_enable': False,
 'run_config_profile_models_concurrently_enable': False,
 'run_config_search_disable': False,
 'run_config_search_max_binary_search_steps': 5,
 'run_config_search_max_concurrency': 4,
 'run_config_search_max_instance_count': 5,
 'run_config_search_max_model_batch_size': 8,
 'run_config_search_max_request_rate': 8192,
 'run_config_search_min_concurrency': 1,
 'run_config_search_min_instance_count': 1,
 'run_config_search_min_model_batch_size': 8,
 'run_config_search_min_request_rate': 16,
 'run_config_search_mode': 'quick',
 'server_output_fields': ['model_name',
                          'gpu_uuid',
                          'gpu_used_memory',
                          'gpu_utilization',
                          'gpu_power_usage'],
 'skip_detailed_reports': False,
 'skip_summary_reports': False,
 'triton_docker_args': {},
 'triton_docker_image': 'nvcr.io/nvidia/tritonserver:26.02-py3',
 'triton_docker_labels': {},
 'triton_docker_mounts': [],
 'triton_docker_shm_size': None,
 'triton_grpc_endpoint': 'localhost:8001',
 'triton_http_endpoint': 'localhost:8000',
 'triton_install_path': '/opt/tritonserver',
 'triton_launch_mode': 'remote',
 'triton_metrics_url': 'http://localhost:8002/metrics',
 'triton_output_path': None,
 'triton_server_environment': {},
 'triton_server_flags': {},
 'triton_server_path': 'tritonserver',
 'use_concurrency_formula': False,
 'weighting': None}
11:11:22 [Model Analyzer] Initializing GPUDevice handles
11:11:24 [Model Analyzer] Using GPU 0 NVIDIA RTX A2000 with UUID GPU-5b2f14e0-a8c2-32a0-fa32-063b50f7e799
11:11:24 [Model Analyzer] WARNING: Overriding the output model repo path "/model_analyzer/linear_reid-no_pre-python_dm-yes_1280x1280/outputs"
11:11:24 [Model Analyzer] Using remote Triton Server
11:11:24 [Model Analyzer] No checkpoint file found, starting a fresh run.
11:11:24 [Model Analyzer] Profiling server only metrics...
11:11:28 [Model Analyzer] 
11:11:28 [Model Analyzer] Starting quick mode search to find optimal configs
11:11:28 [Model Analyzer] 
11:11:28 [Model Analyzer] Creating model config: detect_preprocessing_config_default
11:11:28 [Model Analyzer] 
11:11:28 [Model Analyzer] Creating model config: rt-detr_config_default
11:11:28 [Model Analyzer] 
11:11:28 [Model Analyzer] Creating model config: detect_postprocessing_reid_config_default
11:11:28 [Model Analyzer] 
11:11:28 [Model Analyzer] Creating ensemble model config: ensemble_model_config_default
11:11:31 [Model Analyzer] DEBUG: Model detect_preprocessing_config_default loaded
11:11:34 [Model Analyzer] DEBUG: Model rt-detr_config_default loaded
11:11:37 [Model Analyzer] DEBUG: Model detect_postprocessing_reid_config_default loaded
11:11:38 [Model Analyzer] DEBUG: Model ensemble_model_config_default loaded
11:11:38 [Model Analyzer] Profiling ensemble_model_config_default: concurrency=16
11:11:38 [Model Analyzer] DEBUG: Running ['perf_analyzer', '-m', 'ensemble_model', '-b', '1', '-u', 'localhost:8001', '-i', 'grpc', '-f', 'ensemble_model-results.csv', '--verbose-csv', '--concurrency-range', '16', '--shape', 'raw_image:1280,1280,3', '--measurement-mode', 'count_windows', '--collect-metrics', '--metrics-url', 'http://localhost:8002/metrics', '--metrics-interval', '1000']
11:11:45 [Model Analyzer] DEBUG: Reading PA results from ensemble_model-results.csv
11:11:45 [Model Analyzer] DEBUG: Measurement for [0, 0, 0, 0, 0, 0]: throughput = 81.9757, latency = 331.348 (best throughput: 81.9757, best_latency: 331.348)
11:11:45 [Model Analyzer] Saved checkpoint to /workspace/checkpoints/0.ckpt
11:11:45 [Model Analyzer] Creating model config: detect_preprocessing_config_0
11:11:45 [Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_CPU'}]
11:11:45 [Model Analyzer]   Setting max_batch_size to 8
11:11:45 [Model Analyzer]   Enabling dynamic_batching
11:11:45 [Model Analyzer] 
11:11:45 [Model Analyzer] Creating model config: rt-detr_config_0
11:11:45 [Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_GPU'}]
11:11:45 [Model Analyzer]   Setting max_batch_size to 8
11:11:45 [Model Analyzer]   Enabling dynamic_batching
11:11:45 [Model Analyzer] 
11:11:45 [Model Analyzer] Creating model config: detect_postprocessing_reid_config_0
11:11:45 [Model Analyzer]   Setting instance_group to [{'count': 1, 'kind': 'KIND_CPU'}]
11:11:45 [Model Analyzer]   Setting max_batch_size to 8
11:11:45 [Model Analyzer]   Enabling dynamic_batching
11:11:45 [Model Analyzer] 
11:11:45 [Model Analyzer] Creating ensemble model config: ensemble_model_config_0
11:11:45 [Model Analyzer]   Setting max_batch_size to 1
11:11:48 [Model Analyzer] DEBUG: Model detect_preprocessing_config_0 loaded
11:11:50 [Model Analyzer] DEBUG: Model rt-detr_config_0 loaded
11:11:51 [Model Analyzer] DEBUG: Model detect_postprocessing_reid_config_0 loaded
11:11:52 [Model Analyzer] DEBUG: Model ensemble_model_config_0 loaded
11:11:52 [Model Analyzer] Profiling ensemble_model_config_0: concurrency=4
11:11:52 [Model Analyzer] DEBUG: Running ['perf_analyzer', '-m', 'ensemble_model', '-b', '1', '-u', 'localhost:8001', '-i', 'grpc', '-f', 'ensemble_model-results.csv', '--verbose-csv', '--concurrency-range', '4', '--shape', 'raw_image:1280,1280,3', '--measurement-mode', 'count_windows', '--collect-metrics', '--metrics-url', 'http://localhost:8002/metrics', '--metrics-interval', '1000']
11:11:56 [Model Analyzer] ERROR: Expected output file not found: ensemble_model-results.csv
11:11:56 [Model Analyzer] ERROR: perf_analyzer returned success but did not create expected output files
11:11:56 [Model Analyzer] ERROR: perf_analyzer output:
11:11:56 [Model Analyzer] ERROR: *** Measurement Settings ***
  Batch size: 1
  Service Kind: TRITON
  Using "count_windows" mode for stabilization
  Stabilizing using average latency and throughput
  Minimum number of samples in each window: 50
  Using synchronous calls for inference

Request concurrency: 4
Failed to maintain requested inference load. Worker thread(s) failed to generate concurrent requests.
terminate called without an active exception

11:11:56 [Model Analyzer] DEBUG: _auto_adjust_parameters called. returncode=0, output_length=424, has_failed_msg=False, has_larger_window_msg=False
11:11:56 [Model Analyzer] Running perf_analyzer failed with exit status 0:
*** Measurement Settings ***
  Batch size: 1
  Service Kind: TRITON
  Using "count_windows" mode for stabilization
  Stabilizing using average latency and throughput
  Minimum number of samples in each window: 50
  Using synchronous calls for inference

Request concurrency: 4
Failed to maintain requested inference load. Worker thread(s) failed to generate concurrent requests.
terminate called without an active exception

11:11:56 [Model Analyzer] DEBUG: Measurement for [0, 0, 0, 0, 0, 0]: None.
11:11:56 [Model Analyzer] No changes made to analyzer data, no checkpoint saved.
11:11:56 [Model Analyzer] DEBUG: Stepping back: [0, 0, 0, 0, 0, 0]->[0, 0, 0, 0, 0, 0]
11:11:56 [Model Analyzer] 
11:11:56 [Model Analyzer] Done with quick mode search. Gathering concurrency sweep measurements for reports
11:11:56 [Model Analyzer] 
11:11:56 [Model Analyzer] WARNING: Requested top 3 configs, but found only 1. Showing all available configs for this model.
11:11:56 [Model Analyzer] DEBUG: Failed to connect to Triton server (attempt 1/50): [StatusCode.UNAVAILABLE] failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:8001: recvmsg:Connection reset by peer
11:12:06 [Model Analyzer] DEBUG: Failed to connect to Triton server (attempt 11/50): [StatusCode.UNAVAILABLE] failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:8001: Socket closed

Triton log

tritonserver --model-repository /models/ensemble_model_repository/linear_reid-no_pre-python_dm-yes/ --model-control-mode explicit
I0414 11:10:57.810810 4757 pinned_memory_manager.cc:277] "Pinned memory pool is created at '0x7fb0d4000000' with size 268435456"
I0414 11:10:57.810875 4757 cuda_memory_manager.cc:107] "CUDA memory pool is created on device 0 with size 67108864"
I0414 11:10:57.812630 4757 server.cc:620] 
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I0414 11:10:57.812648 4757 server.cc:647] 
+---------+------+--------+
| Backend | Path | Config |
+---------+------+--------+
+---------+------+--------+

I0414 11:10:57.812677 4757 server.cc:690] 
+-------+---------+--------+
| Model | Version | Status |
+-------+---------+--------+
+-------+---------+--------+

I0414 11:10:57.864000 4757 metrics.cc:889] "Collecting metrics for GPU 0: NVIDIA RTX A2000"
I0414 11:10:57.866090 4757 metrics.cc:782] "Collecting CPU metrics"
I0414 11:10:57.866164 4757 tritonserver.cc:2598] 
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| Option                           | Value                                                                                                                                  |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| server_id                        | triton                                                                                                                                 |
| server_version                   | 2.66.0                                                                                                                                 |
| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory  |
|                                  | cuda_shared_memory binary_tensor_data parameters statistics trace logging                                                              |
| model_repository_path[0]         | /models/ensemble_model_repository/linear_reid-no_pre-python_dm-yes/                                                                    |
| model_control_mode               | MODE_EXPLICIT                                                                                                                          |
| strict_model_config              | 0                                                                                                                                      |
| model_config_name                |                                                                                                                                        |
| rate_limit                       | OFF                                                                                                                                    |
| pinned_memory_pool_byte_size     | 268435456                                                                                                                              |
| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                               |
| min_supported_compute_capability | 6.0                                                                                                                                    |
| strict_readiness                 | 1                                                                                                                                      |
| exit_timeout                     | 30                                                                                                                                     |
| cache_enabled                    | 0                                                                                                                                      |
+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------+

I0414 11:10:57.867130 4757 grpc_server.cc:2562] "Started GRPCInferenceService at 0.0.0.0:8001"
I0414 11:10:57.867313 4757 http_server.cc:4823] "Started HTTPService at 0.0.0.0:8000"
I0414 11:10:57.908119 4757 http_server.cc:358] "Started Metrics Service at 0.0.0.0:8002"
I0414 11:11:29.880790 4757 model_lifecycle.cc:473] "loading: detect_preprocessing:1"
I0414 11:11:31.162103 4757 python_be.cc:2289] "TRITONBACKEND_ModelInstanceInitialize: detect_preprocessing_0_0 (CPU device 0)"
I0414 11:11:31.162126 4757 python_be.cc:2289] "TRITONBACKEND_ModelInstanceInitialize: detect_preprocessing_0_1 (CPU device 0)"
I0414 11:11:31.375067 4757 model_lifecycle.cc:849] "successfully loaded 'detect_preprocessing'"
I0414 11:11:32.406459 4757 model_lifecycle.cc:473] "loading: rt-detr:1"
I0414 11:11:32.436626 4757 tensorrt.cc:65] "TRITONBACKEND_Initialize: tensorrt"
I0414 11:11:32.436650 4757 tensorrt.cc:75] "Triton TRITONBACKEND API version: 1.19"
I0414 11:11:32.436653 4757 tensorrt.cc:81] "'tensorrt' TRITONBACKEND API version: 1.19"
I0414 11:11:32.436657 4757 tensorrt.cc:105] "backend configuration:\n{\"cmdline\":{\"auto-complete-config\":\"true\",\"backend-directory\":\"/opt/tritonserver/backends\",\"min-compute-capability\":\"6.000000\",\"default-max-batch-size\":\"4\"}}"
I0414 11:11:32.436876 4757 tensorrt.cc:231] "TRITONBACKEND_ModelInitialize: rt-detr (version 1)"
I0414 11:11:33.237485 4757 logging.cc:46] "Loaded engine size: 85 MiB"
I0414 11:11:33.316951 4757 tensorrt.cc:297] "TRITONBACKEND_ModelInstanceInitialize: rt-detr_0 (GPU device 0)"
I0414 11:11:34.124103 4757 logging.cc:46] "Loaded engine size: 85 MiB"
I0414 11:11:34.176409 4757 logging.cc:46] "[MS] Running engine with multi stream info"
I0414 11:11:34.176432 4757 logging.cc:46] "[MS] Number of aux streams is 2"
I0414 11:11:34.176437 4757 logging.cc:46] "[MS] Number of total worker streams is 3"
I0414 11:11:34.176441 4757 logging.cc:46] "[MS] The main stream provided by execute/enqueue calls is the first worker stream"
I0414 11:11:34.312526 4757 logging.cc:46] "[MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +122, now: CPU 0, GPU 203 (MiB)"
I0414 11:11:34.312847 4757 instance_state.cc:186] "Created instance rt-detr_0 on GPU 0 with stream priority 0 and optimization profile default[0];"
I0414 11:11:34.313234 4757 model_lifecycle.cc:849] "successfully loaded 'rt-detr'"
I0414 11:11:35.340378 4757 model_lifecycle.cc:473] "loading: detect_postprocessing_reid:1"
I0414 11:11:37.273044 4757 python_be.cc:2289] "TRITONBACKEND_ModelInstanceInitialize: detect_postprocessing_reid_0_0 (CPU device 0)"
I0414 11:11:37.879570 4757 model_lifecycle.cc:849] "successfully loaded 'detect_postprocessing_reid'"
I0414 11:11:38.956398 4757 model_lifecycle.cc:473] "loading: ensemble_model:1"
I0414 11:11:38.957007 4757 model_lifecycle.cc:849] "successfully loaded 'ensemble_model'"
I0414 11:11:47.127454 4757 model_lifecycle.cc:473] "loading: detect_preprocessing:1"
I0414 11:11:48.361996 4757 model_lifecycle.cc:849] "successfully loaded 'detect_preprocessing'"
I0414 11:11:48.365733 4757 model_lifecycle.cc:473] "loading: ensemble_model:1"
I0414 11:11:48.365916 4757 model_lifecycle.cc:849] "successfully loaded 'ensemble_model'"
I0414 11:11:48.365971 4757 model_lifecycle.cc:636] "successfully unloaded 'ensemble_model' version 1"
I0414 11:11:49.387009 4757 model_lifecycle.cc:473] "loading: rt-detr:1"
I0414 11:11:49.394695 4757 tensorrt.cc:297] "TRITONBACKEND_ModelInstanceInitialize: rt-detr_0_0 (GPU device 0)"
I0414 11:11:50.197362 4757 logging.cc:46] "Loaded engine size: 85 MiB"
I0414 11:11:50.249329 4757 logging.cc:46] "[MS] Running engine with multi stream info"
I0414 11:11:50.249352 4757 logging.cc:46] "[MS] Number of aux streams is 2"
I0414 11:11:50.249357 4757 logging.cc:46] "[MS] Number of total worker streams is 3"
I0414 11:11:50.249361 4757 logging.cc:46] "[MS] The main stream provided by execute/enqueue calls is the first worker stream"
I0414 11:11:50.383189 4757 logging.cc:46] "[MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +122, now: CPU 1, GPU 407 (MiB)"
I0414 11:11:50.383490 4757 instance_state.cc:186] "Created instance rt-detr_0_0 on GPU 0 with stream priority 0 and optimization profile default[0];"
I0414 11:11:50.383824 4757 tensorrt.cc:353] "TRITONBACKEND_ModelInstanceFinalize: delete instance state"
I0414 11:11:50.403376 4757 model_lifecycle.cc:849] "successfully loaded 'rt-detr'"
I0414 11:11:50.406847 4757 model_lifecycle.cc:473] "loading: ensemble_model:1"
I0414 11:11:50.406954 4757 model_lifecycle.cc:849] "successfully loaded 'ensemble_model'"
I0414 11:11:50.407061 4757 model_lifecycle.cc:636] "successfully unloaded 'ensemble_model' version 1"
I0414 11:11:51.434856 4757 model_lifecycle.cc:473] "loading: detect_postprocessing_reid:1"
I0414 11:11:51.435246 4757 model_lifecycle.cc:849] "successfully loaded 'detect_postprocessing_reid'"
I0414 11:11:51.438403 4757 model_lifecycle.cc:473] "loading: ensemble_model:1"
I0414 11:11:51.438952 4757 model_lifecycle.cc:849] "successfully loaded 'ensemble_model'"
I0414 11:11:51.439076 4757 model_lifecycle.cc:636] "successfully unloaded 'ensemble_model' version 1"
I0414 11:11:52.502198 4757 model_lifecycle.cc:473] "loading: ensemble_model:1"
I0414 11:11:52.502426 4757 model_lifecycle.cc:849] "successfully loaded 'ensemble_model'"
I0414 11:11:52.502538 4757 model_lifecycle.cc:636] "successfully unloaded 'ensemble_model' version 1"
Signal (11) received.
 0# 0x0000561917CEF2F8 in tritonserver
 1# 0x00007FB11C6D6330 in /usr/lib/x86_64-linux-gnu/libc.so.6
 2# 0x00007FB0F8228BCA in /opt/tritonserver/backends/tensorrt/libtriton_tensorrt.so
 3# 0x00007FB0F8229882 in /opt/tritonserver/backends/tensorrt/libtriton_tensorrt.so
 4# 0x00007FB0F823AF86 in /opt/tritonserver/backends/tensorrt/libtriton_tensorrt.so
 5# TRITONBACKEND_ModelInstanceExecute in /opt/tritonserver/backends/tensorrt/libtriton_tensorrt.so
 6# 0x00007FB11D1CFE57 in /opt/tritonserver/bin/../lib/libtritonserver.so
 7# 0x00007FB11D1D054B in /opt/tritonserver/bin/../lib/libtritonserver.so
 8# 0x00007FB11D2F3D75 in /opt/tritonserver/bin/../lib/libtritonserver.so
 9# 0x00007FB11D1D4373 in /opt/tritonserver/bin/../lib/libtritonserver.so
10# 0x00007FB11C98FDB4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
11# 0x00007FB11C72DAA4 in /usr/lib/x86_64-linux-gnu/libc.so.6
12# __clone in /usr/lib/x86_64-linux-gnu/libc.so.6

Segmentation fault (core dumped)
root@cc26b416edc0:/etc/rocketboots# I0414 11:11:53.403979 4836 pb_stub.cc:2075]  Non-graceful termination detected. 
I0414 11:11:53.508003 4945 pb_stub.cc:2075]  Non-graceful termination detected. 

Is this bug related to how fast Triton server can reload the models?

thanks for your help

[corentin87]

I solved my problem temporary by running MA on my ensemble using a custom Triton image with the same entry point than the official Triton image (the one in the test above has different entry point which is why I had to use remote).
So now I can use triton_launch_mode=docker instead of remote.

However, it would still be interested to understand what's crashing with a remote Triton server.