Skip to content

test: L0_lifecycle uses blind time.sleep(5) for model load/unload, causing flaky CI failures #8919

Description

@Moyummmm

Description

The qa/L0_lifecycle/lifecycle_test.py file contains 13 instances of time.sleep(5) that blindly wait for model load/unload operations before asserting is_model_ready(). Under CI resource contention, 5 seconds may not be sufficient for the model to reach the expected state, causing intermittent test failures.

Root cause

Fixed-duration sleeps are inherently unreliable for synchronizing with asynchronous model load/unload operations. The timing varies based on system load, I/O performance, and platform (especially Jetson/Orin). Other test files (e.g. log_format_test.py, pinned_memory_metrics_test.py) already use polling-based approaches with is_model_ready().

Proposed fix

Replace time.sleep(5) with a polling loop that uses triton_client.is_model_ready() to wait for the expected state, with a configurable timeout (default 30s). This approach:

  • Is resilient to CI load variations
  • Fails fast when the model reaches the expected state (typically <5s)
  • Provides clear error messages when the timeout is reached

Affected test functions

  • test_dynamic_model_load_unload (4 instances)
  • test_dynamic_version_load_unload (2 instances)
  • test_dynamic_model_modify (2 instances)
  • test_multiple_model_repository_polling (2 instances)
  • test_apply_custom_config / test_delete_custom_config (2 instances)
  • Model duplication/unload scenario (1 instance)

Not affected (intentionally unchanged)

  • 4 time.sleep(5) in disabled dynamic loading negative tests (model should NOT load/unload)
  • 1 time.sleep(5) for scheduler timeout wait

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions