test: capture and surface tritonserver output on server-startup failure - #126
Merged
Vinya567 merged 3 commits intoJun 10, 2026
Merged
Conversation
ScopedTritonServer previously raised a generic "Server failed to start in time." which hid the actual cause. Redirect the triton subprocess's stdout/stderr to a temp file and include the tail (plus pid, exit code, elapsed time, and timeout) in the RuntimeError raised by wait_for_server_ready. The log file is unlinked in kill_server. Pure test-harness diagnostics change; no production code path affected.
yinggeh
reviewed
Jun 9, 2026
Co-authored-by: Yingge He <157551214+yinggeh@users.noreply.github.com>
Removed unused import statement for 'os'.
yinggeh
approved these changes
Jun 9, 2026
whoisj
approved these changes
Jun 9, 2026
Vinya567
deleted the
vinyak/tri-1352-fix-ci-test-l0_triton_cli_test_trtllm-base-gpt2
branch
June 10, 2026 02:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does the PR do?
ScopedTritonServer.wait_for_server_ready()previously raised a generic"Server failed to start in time."which discarded thetritonserversubprocess output entirely. That made CI failures effectively impossible
to triage without a local repro.
This change captures the subprocess's stdout/stderr to a temporary file
and includes the tail (with pid, exit code, elapsed time, and timeout)
in the
RuntimeErrorraised on startup failure. The temp file isunlinked in
kill_server().The two failure modes are now distinguished in the error message:
timeout + log tail
This is a pure test-harness diagnostics change. No production code path
is affected; no public API change; happy-path test runs behave
identically.
Checklist
Checklist
<commit_type>: <Title>Commit Type:
Check the conventional commit type
box here and add the label to the github PR.
Related PRs:
L0_triton_cli_test_vllm--base#122 (ci: Fix L0_triton_cli_test_vllm--base) — same family of CI L0test fixes for this repo. Useful reviewer context on the failure
pattern this diagnostic surfaces.
a separate MR (tracked internally). This PR stands on its own — pure
test-harness change — and can land independently.
Where should the reviewer start?
tests/utils.py— the only file changed.ScopedTritonServer.run_server()— new temp-file redirect of thesubprocess stdout/stderr via
tempfile.NamedTemporaryFileandPopen(..., stdout=log_handle, stderr=STDOUT).ScopedTritonServer._format_server_output()— new helper thatreturns the tail (default last 50 lines) of the captured log.
ScopedTritonServer.wait_for_server_ready()— error messages nowinclude pid / exit code / elapsed / timeout / log tail.
ScopedTritonServer.kill_server()— restructured early-returninto an if/else and unconditionally unlinks the temp log at the
end, so nothing leaks on disk.
Test plan:
Re-ran the previously-failing
L0_triton_cli_test_trtllm--basematrix entries with this diagnostics change in place. The new error
output identified the actual missing binary in the test image
immediately — confirming the diagnostic surface works end-to-end on
a real failure.
Happy-path runs (server starts within timeout) behave identically;
the temp log file is unlinked by
kill_server()so nothing is lefton disk.
No new tests added — this PR changes only an existing test-harness
error path. Coverage of the harness itself is exercised every time
an L0 test runs.
CI Pipeline ID: 54134005
Caveats:
tempfile.NamedTemporaryFileunder the system temp dir;
kill_server()removes it. If a testprocess is
SIGKILL'd beforekill_server()runs, the temp file isleft for the OS to reap (standard behavior, same as any Python
tempfile).
exited with traceback" case and bounded so the
RuntimeErrorstaysreadable in CI log viewers.
Background
The previous error string discarded the actual
tritonserverstderr,so a missing binary, a missing Python module, a failed model load, and
a slow startup all produced the same useless one-liner in CI. Intent
of this PR is exactly what was asked for in review: surface the real
failure cause directly in the CI job log so future triage doesn't
require a local repro.
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)