Intermittent server crash (`boost::interprocess::lock_exception` / `std::bad_alloc` / SIGSEGV) in Python backend under concurrent BLS load — 25.02 (2.55.0)

[CarlosNacher]

Description
Under sustained concurrent load from both BLS pipelines, tritonserver crashes intermittently and is only recovered by restart: always in docker-compose. Two related crash signatures observed, both originating in libtriton_python.so → TRITONBACKEND_ModelInstanceExecute:

Signature A:

terminate called after throwing an instance of 'boost::interprocess::lock_exception'
  what():  boost::interprocess::lock_exception
Signal (6) received.
...
Signal (11) received.
 0# 0x... in tritonserver
 ...
 12# TRITONBACKEND_ModelInstanceExecute in /opt/tritonserver/backends/python/libtriton_python.so
 13-16# ... in libtritonserver.so
 17-19# ... in libstdc++.so.6 / libc.so.6 / __clone

Signature B (also seen):

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Signal (6) received.
Signal (11) received.

Signature C (most recent, after raising shm-default-byte-size to 64MB):

Signal (11) received.
 0# 0x... in tritonserver
 1# 0x... in libc.so.6
 2-5# 0x... in libtriton_python.so
 6# TRITONBACKEND_ModelInstanceExecute in libtriton_python.so
 7-10# ... in libtritonserver.so
 11-13# ... in libstdc++.so.6 / libc.so.6 / __clone

(no exception message logged this time — a raw SIGSEGV, same call path).

Immediate precursor, seen before nearly every crash (correlated to the second in the majority of occurrences we captured):

E<timestamp> model.py:109/110] "Ensemble error: in ensemble 'ensemble-station0X', Stub process '<child_model>_0_N' is not healthy."

In our logs this precursor names one specific Python-backend child model (f1pixdet_rayas-discriminator, instance_group { count: 4, kind: KIND_CPU }) in the large majority of occurrences, though we can't rule out other models triggering the same underlying condition. The gap between a fresh request's Decoding input log line and the "is not healthy" report is consistently ~1-1.3 seconds, not the stub-timeout-seconds default (30s) — i.e. the request that surfaces the error is not the one that hung; it's simply the first request routed to an instance that a background health-check already found unresponsive at some earlier, unlogged point.

We captured a burst where this happened repeatedly, resulting in a rapid crash-loop: 4 container restarts in ~11 minutes, with an ~18-19 minute stretch during which not a single request was traced to completion (confirmed via --trace-config output, not just log inspection).

We cross-checked this against Triton's own request tracing (JSON traces, REQUEST_START/COMPUTE_START/COMPUTE_END/REQUEST_END events) for the ~20 minutes immediately preceding two of these crashes. Across ~1700+ fully-traced calls per child model in that window:

• 0 requests with COMPUTE_START but no matching COMPUTE_END (i.e., nothing hangs mid-inference).
• 0 requests with REQUEST_START but no matching REQUEST_END.
• All COMPUTE durations bounded (tens to a few hundred ms across all child models, including the flagged f1pixdet_rayas-discriminator, which averaged ~92ms with a ~130ms max).
• The very last request traced before the trace stream goes silent (i.e., immediately before the process dies) completes and closes normally.

In other words: nothing at the level Triton's own request tracing can see is slow, stuck, or abnormal right up to the instant of the crash. This strongly suggests the failure originates below that layer — in the shared-memory/IPC bookkeeping between the main process and a Python-backend stub — rather than in any user model code taking too long.

Triton Information

• Triton Server version: 2.55.0
• NGC container: nvcr.io/nvidia/tritonserver:25.02-py3 (build 143749457)
• Are you using the Triton container or did you build it yourself? Official NGC container, with a small custom layer on top (extra pip packages: opencv-python-headless, scikit-learn, xgboost, a private anomalib fork, torch+cu121). No changes to Triton itself.

Environment

• Host: Docker Engine on Ubuntu, running inside WSL2 on Windows (production PC, not a dev laptop).
• GPU driver: 553.62 (CUDA 12.4 supported; container built with CUDA 12.8, running in Minor Version Compatibility mode).
• docker-compose resource limits: cpus: 16, memory: 10gb, shm_size: 4gb.
• --backend-config=python,shm-default-byte-size=16777216 (16MB) at the time of the traces below; later raised to 64MB, crash still reproduces (see below).
• --cuda-memory-pool-byte-size=0:536870912, --pinned-memory-pool-byte-size=536870912.
• --trace-config enabled (mode=triton, level=TIMESTAMPS, rate=1, log-frequency=10).
• Model topology: 2 independent Python-backend BLS models, each async_exec()-ing into its own ensemble model (config-only, no Python) that fans out to one DALI preprocessing model (GPU) and 6 individual child models (ONNX/TensorRT models, and one custom OpenCV-based Python model). Both BLS run concurrently and continuously (production line, two cameras per unit).

To Reproduce
We don't yet have a minimal standalone repro (this is a production model repository with proprietary models), but the shape is:

• 2+ concurrent BLS Python-backend models, each issuing an async_exec() BLS call per incoming request into its own ensemble.
• Each ensemble fans out to 1 DALI (GPU) + 6 child models (mixed instance_group counts, 1-4, KIND_CPU for Python Backend ones / KIND_GPU for TensorRT ones).
• Sustained concurrent request rate from 2 independent sources (roughly 1-2 req/s per source, max_batch_size: 2 with dynamic batching).
• Crash is intermittent — happens roughly every few minutes to every couple hours under this load, not deterministically reproducible on demand.

Happy to share the full docker-compose command-line flags, config.pbtxt for the affected child model, and/or Linux core dumps (tritonserver and triton_python_backend_stub, captured via WSL2's automatic crash-dump collection) if that's useful for triage.

Expected behavior
tritonserver should not abort/segfault under this load pattern; at most, an individual model instance should be marked unhealthy and recovered without taking down the whole process.

Possibly related
We found two commits to python_backend's src/shm_manager.h that look related in symptom (shared-memory accounting / bounds issues in the same code path our stack trace goes through), both not yet included in 25.02:

• fix: Verify MemoryShm::byte_size inside shared memory boundary python_backend#406 ("Verify MemoryShm::byte_size within CPU shm boundary", merged 2025-07-07)
• fix: Fix SEGFAULT in BLS Model Loading python_backend#420 ("Fix SEGFAULT in BLS Model Loading", merged 2025-11-12)

We're in the process of upgrading to a newer NGC container (2.69.0 / 26.05) and will report back here whether that resolves it. If anyone from the team can confirm whether #406/#420 are indeed expected to fix this crash signature (or point us to a different root cause), that would help us a lot.