test: replace blind time.sleep(5) with polling in L0_lifecycle

[Moyummmm]

What does the PR do?

Replace 13 instances of time.sleep(5) in qa/L0_lifecycle/lifecycle_test.py with a deterministic polling loop using a new tu.wait_for_model_state() helper in qa/common/test_util.py.

Root cause

Fixed-duration time.sleep(5) calls that wait for model load/unload are unreliable under CI resource contention. If the server takes longer than 5 seconds (e.g., under GPU/CPU contention, I/O pressure, or on Jetson/Orin), the subsequent is_model_ready() assertion fails even though the model would eventually reach the correct state.

Fix

1. qa/common/test_util.py: Added wait_for_model_state(triton_client, model_name, expected_ready, model_version, timeout_sec, interval_sec) helper. It polls is_model_ready() at 1-second intervals until the expected state is reached or the 30-second timeout expires.

2. qa/L0_lifecycle/lifecycle_test.py: Replaced time.sleep(5) in the following test functions with calls to the new helper, polling the specific model/version that changed:

   • test_dynamic_model_load_unload (4 replacements)
   • test_dynamic_version_load_unload (2 replacements)
   • test_dynamic_model_modify (2 replacements)
   • test_multiple_model_repository_polling (2 replacements)
   • test_apply_custom_config / test_delete_custom_config (2 replacements)
   • Model duplication/unload scenario (1 replacement)

3. Intentionally kept time.sleep(5) in:

   • 4 negative-test scenarios where dynamic loading is disabled (model should NOT load/unload)
   • 1 scheduler timeout wait

Checklist

• PR title reflects the change and is of format <type>: <description>
• Changes are described in the pull request.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• All template sections are filled out.

Commit Type:

• test

Related Issues:

Closes #8919

Where should the reviewer start?

• qa/common/test_util.py — new wait_for_model_state() helper
• qa/L0_lifecycle/lifecycle_test.py — replacements

Test plan:

• Local: python3 -m py_compile qa/common/test_util.py qa/L0_lifecycle/lifecycle_test.py (syntax check passes)
• CI: qa/L0_lifecycle/test.sh with standard Triton CI configuration

Caveats:

• The helper creates a new httpclient.InferenceServerClient for each polling call. This is consistent with the existing pattern in the test file where clients are created inline.
• The 4 disabled-mode time.sleep(5) calls (negative tests) and 1 scheduler timeout sleep are intentionally unchanged.

[greptile-apps]

Greptile Summary

Replaces fixed five-second lifecycle-test delays with a shared, bounded model-readiness polling helper.

• Adds wait_for_model_state with configurable timeout and polling interval.
• Updates lifecycle load, unload, repository-duplication, and configuration-reload scenarios to use the helper.
• Some configuration-reload waits poll an already-true readiness condition and therefore do not reliably synchronize with the reload.

Confidence Score: 4/5

The configuration-reload synchronization needs correction before merging because three updated tests can still proceed against stale model state and fail intermittently.

Polling version 3 for readiness cannot prove that a configuration mutation was processed when that version was already ready under the previous configuration.

Files Needing Attention: qa/L0_lifecycle/lifecycle_test.py

Important Files Changed

Filename	Overview
qa/common/test_util.py	Adds a bounded readiness polling helper; no independently actionable defect was established in the helper itself.
qa/L0_lifecycle/lifecycle_test.py	Replaces thirteen sleeps with polling, but three configuration-reload sites wait on a readiness state that was already true and can race repository polling.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Mutate model configuration] --> B{Repository poll has run?}
    B -- No --> C[Old version 3 remains ready]
    C --> D[wait_for_model_state returns immediately]
    D --> E[Assertions observe stale configuration]
    E --> F[Intermittent test failure]
    B -- Yes --> G[Reload applies new configuration]
    G --> H[Assertions observe expected state]

Loading

_{Reviews (1): Last reviewed commit: "test: replace time.sleep(5) with wait_fo..." | Re-trigger Greptile}

[greptile-apps]

Already-ready state bypasses reload

When the configuration changes between repository polling cycles, version 3 remains ready under the old configuration, so wait_for_model_state(..., True, "3") returns immediately and the subsequent label and version-policy checks race the reload, causing intermittent test failures. The same issue applies to the waits after the second configuration change and the configuration deletion.