Description
The qa/L0_lifecycle/lifecycle_test.py file contains 13 instances of time.sleep(5) that blindly wait for model load/unload operations before asserting is_model_ready(). Under CI resource contention, 5 seconds may not be sufficient for the model to reach the expected state, causing intermittent test failures.
Root cause
Fixed-duration sleeps are inherently unreliable for synchronizing with asynchronous model load/unload operations. The timing varies based on system load, I/O performance, and platform (especially Jetson/Orin). Other test files (e.g. log_format_test.py, pinned_memory_metrics_test.py) already use polling-based approaches with is_model_ready().
Proposed fix
Replace time.sleep(5) with a polling loop that uses triton_client.is_model_ready() to wait for the expected state, with a configurable timeout (default 30s). This approach:
- Is resilient to CI load variations
- Fails fast when the model reaches the expected state (typically <5s)
- Provides clear error messages when the timeout is reached
Affected test functions
test_dynamic_model_load_unload (4 instances)
test_dynamic_version_load_unload (2 instances)
test_dynamic_model_modify (2 instances)
test_multiple_model_repository_polling (2 instances)
test_apply_custom_config / test_delete_custom_config (2 instances)
- Model duplication/unload scenario (1 instance)
Not affected (intentionally unchanged)
- 4
time.sleep(5) in disabled dynamic loading negative tests (model should NOT load/unload)
- 1
time.sleep(5) for scheduler timeout wait
Description
The
qa/L0_lifecycle/lifecycle_test.pyfile contains 13 instances oftime.sleep(5)that blindly wait for model load/unload operations before assertingis_model_ready(). Under CI resource contention, 5 seconds may not be sufficient for the model to reach the expected state, causing intermittent test failures.Root cause
Fixed-duration sleeps are inherently unreliable for synchronizing with asynchronous model load/unload operations. The timing varies based on system load, I/O performance, and platform (especially Jetson/Orin). Other test files (e.g.
log_format_test.py,pinned_memory_metrics_test.py) already use polling-based approaches withis_model_ready().Proposed fix
Replace
time.sleep(5)with a polling loop that usestriton_client.is_model_ready()to wait for the expected state, with a configurable timeout (default 30s). This approach:Affected test functions
test_dynamic_model_load_unload(4 instances)test_dynamic_version_load_unload(2 instances)test_dynamic_model_modify(2 instances)test_multiple_model_repository_polling(2 instances)test_apply_custom_config/test_delete_custom_config(2 instances)Not affected (intentionally unchanged)
time.sleep(5)in disabled dynamic loading negative tests (model should NOT load/unload)time.sleep(5)for scheduler timeout wait