[TRTLLM-14473][chore] Remove legacy TensorRT backend tests examples and CI plumbing#16610
Merged
Wanli-Jiang merged 1 commit intoJul 22, 2026
Conversation
Wanli-Jiang
commented
Jul 20, 2026
Wanli-Jiang
force-pushed
the
user/williamj/trtllm-14027-tests-examples
branch
from
July 20, 2026 09:01
d3a833e to
6669d31
Compare
Wanli-Jiang
marked this pull request as ready for review
July 20, 2026 09:03
Wanli-Jiang
requested review from
BowenFu,
LarryXFly,
QiJune,
kaiyux,
laikhtewari,
taylor-yb-lee,
xinhe-nv,
yiqingy0 and
yuanjingx87
July 20, 2026 09:03
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
@coderabbitai check tests/unittest/llmapi/apps, if there is any test that still uses tensorrt backend |
xinhe-nv
approved these changes
Jul 20, 2026
Collaborator
|
PR_Github #60335 [ run ] triggered by Bot. Commit: |
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: ⛔ Files ignored due to path filters (3)
📒 Files selected for processing (151)
💤 Files with no reviewable changes (110)
🚧 Files skipped from review as they are similar to previous changes (23)
WalkthroughThe PR removes legacy examples, plugins, model workflows, and integration helpers; simplifies runtime model selection; updates Jenkins backend mapping and stage parsing; migrates test matrices toward PyTorch and LLM API coverage; and revises documentation and waivers. ChangesRuntime API simplification
Integration infrastructure and coverage
CI backend selection and test matrices
Documentation and example cleanup
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
BowenFu
approved these changes
Jul 20, 2026
BowenFu
left a comment
BowenFu
left a comment
There was a problem hiding this comment.
LGTM — reviewed for over-deletion. Nothing the live PyTorch/AutoDeploy path or any scheduled test/CI stage depends on is removed: _utils.py's supports_inflight_batching is a TRT-engine-only helper with zero remaining importers (former callers all deleted here or in #16369), and the 42 dropped waives all target deleted example defs / de-scheduled TRT perf configs — no active pytorch/llmapi/disagg/moe waive was dropped, so the validator invariant holds.
Two non-blocking notes (dead code, not breakage):
runPackageSanityCheckis not actually deleted despite the PR description — it still lives inL0_Test.groovy(~:4373) and references deletedgenerate_checkpoint_config.py/run.py. Its call site is commented out so it's dead/unreachable, but the description is inaccurate.- The retained
CliFlowAccuracyTestHarnessnow dangles on deletedsummarize.py/eval_long_context.py(it's unscheduled —test_cli_flowis in no active list — so nothing breaks, but the harness is non-functional). Minor doc link rot in accuracy README + speculative-decoding.md too.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/integration/defs/examples/test_phi.py (1)
54-56: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winProfiler key doesn't match the test name.
ci_profiler.start/stopis keyed on"test_llm_torch_multi_lora_support"(the shared helper's name) rather than this test's own name, unlike the analogous new test intest_gpt.pywhich profiles under its own test name. If another Phi (or future) test reuses this helper, profiling data will collide under the same key.🔧 Proposed fix
- defs.ci_profiler.start("test_llm_torch_multi_lora_support") + defs.ci_profiler.start("test_phi_4_mini_instruct_with_bf16_lora_torch") test_llm_torch_multi_lora_support(hf_model_dir=llm_phi_model_root, llm_venv=llm_venv, num_loras=2, lora_rank=8, target_hf_modules=["qkv_proj"], target_trtllm_modules=["attn_qkv"], zero_lora_weights=True, tensor_parallel_size=1) - defs.ci_profiler.stop("test_llm_torch_multi_lora_support") + defs.ci_profiler.stop("test_phi_4_mini_instruct_with_bf16_lora_torch")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integration/defs/examples/test_phi.py` around lines 54 - 56, Update the profiler key passed to ci_profiler.start in the Phi test to use that test's own function name rather than the shared test_llm_torch_multi_lora_support helper name, and ensure the corresponding stop call uses the same key.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/source/features/auto_deploy/advanced/testing_strategy.md`:
- Around line 120-121: Update the testing strategy example around the documented
registration format so the modeling_llama entry is either a concrete fully
qualified test node ID or is explicitly identified as a command-style selector
supported by CI YAML. Keep the existing individual test-case example unchanged
and ensure the examples match the format described in the surrounding
documentation.
In `@tests/integration/defs/accuracy/test_llm_api_pytorch.py`:
- Around line 4385-4396: Remove the unconditional pytest.skip from test_tp2, or
replace it with a topology-aware skip that allows execution on validated NVLink
platforms while excluding unsupported PCIe-only topologies. Also remove the
test_tp2 entry from tests/integration/test_lists/qa/llm_function_core.txt until
the test is executable in QA.
---
Outside diff comments:
In `@tests/integration/defs/examples/test_phi.py`:
- Around line 54-56: Update the profiler key passed to ci_profiler.start in the
Phi test to use that test's own function name rather than the shared
test_llm_torch_multi_lora_support helper name, and ensure the corresponding stop
call uses the same key.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: b0e0665b-32db-490b-9b84-1f1c9fa2816c
⛔ Files ignored due to path filters (3)
examples/models/core/qwen2audio/audio/glass-breaking-151256.mp3is excluded by!**/*.mp3examples/models/core/qwenvl/pics/1.pngis excluded by!**/*.pngexamples/models/core/qwenvl/pics/demo.jpegis excluded by!**/*.jpeg
📒 Files selected for processing (149)
docs/source/developer-guide/ci-overview.mddocs/source/features/auto_deploy/advanced/testing_strategy.mdexamples/cpp_library/CMakeLists.txtexamples/cpp_library/build.shexamples/cpp_library/main.cppexamples/cpp_library/tensorrt_llm_libutils.hexamples/dora/README.mdexamples/dora/normalize_weights.pyexamples/eval_long_context.pyexamples/generate_checkpoint_config.pyexamples/generate_xgrammar_tokenizer_info.pyexamples/hf_lora_convert.pyexamples/language_adapter/README.mdexamples/llm-eval/lm-eval-harness/README.mdexamples/llm-eval/lm-eval-harness/lm_eval_tensorrt_llm.pyexamples/llm-eval/lm-eval-harness/requirements.txtexamples/mmlu.pyexamples/models/contrib/arctic/README.mdexamples/models/contrib/blip2/README.mdexamples/models/contrib/chatglm-6b/README.mdexamples/models/contrib/chatglm-6b/requirements.txtexamples/models/contrib/chatglm-6b/tokenization_chatglm.pyexamples/models/contrib/chatglm2-6b/README.mdexamples/models/contrib/chatglm2-6b/requirements.txtexamples/models/contrib/chatglm2-6b/tokenization_chatglm.pyexamples/models/contrib/chatglm3-6b-32k/README.mdexamples/models/contrib/chatglm3-6b-32k/requirements.txtexamples/models/contrib/chatglm3-6b-32k/tokenization_chatglm.pyexamples/models/contrib/internlm/.gitignoreexamples/models/contrib/internlm/README.mdexamples/models/contrib/internlm/requirements.txtexamples/models/contrib/jais/README.mdexamples/models/contrib/jais/requirements.txtexamples/models/contrib/sdxl/README.mdexamples/models/contrib/sdxl/build_sdxl_unet.pyexamples/models/contrib/sdxl/pipeline_stable_diffusion_xl.pyexamples/models/contrib/sdxl/run_sdxl.pyexamples/models/contrib/skywork/README.mdexamples/models/contrib/skywork/requirements.txtexamples/models/contrib/smaug/README.mdexamples/models/contrib/smaug/requirements.txtexamples/models/core/granite/README.mdexamples/models/core/mixtral/README.mdexamples/models/core/mixtral/requirements.txtexamples/models/core/multimodal/README.mdexamples/models/core/multimodal/__init__.pyexamples/models/core/multimodal/build_multimodal_engine.pyexamples/models/core/multimodal/eval.pyexamples/models/core/multimodal/requirements-eclair.txtexamples/models/core/multimodal/requirements-internlm-xcomposer2.txtexamples/models/core/multimodal/requirements-llava_onevision.txtexamples/models/core/multimodal/requirements-qwen2vl.txtexamples/models/core/multimodal/requirements-vila.txtexamples/models/core/multimodal/run.pyexamples/models/core/multimodal/utils.pyexamples/models/core/nemotron/README_nemotron-3.mdexamples/models/core/nemotron/requirements.txtexamples/models/core/qwen2audio/README.mdexamples/models/core/qwen2audio/requirements.txtexamples/models/core/qwen2audio/run.pyexamples/models/core/qwen2audio/run_chat.pyexamples/models/core/qwen2audio/utils.pyexamples/models/core/qwenvl/README.mdexamples/models/core/qwenvl/requirements.txtexamples/models/core/qwenvl/run.pyexamples/models/core/qwenvl/run_chat.pyexamples/models/core/qwenvl/show_pic.pyexamples/models/core/qwenvl/vit_onnx_trt.pyexamples/ngram/README.mdexamples/ngram/run_dtm_ngram.pyexamples/openai_triton/README.mdexamples/openai_triton/manual_plugin/CMakeLists.txtexamples/openai_triton/manual_plugin/README.mdexamples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.cppexamples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.hexamples/openai_triton/manual_plugin/build.pyexamples/openai_triton/manual_plugin/fmha_triton.pyexamples/openai_triton/manual_plugin/plugin.pyexamples/openai_triton/manual_plugin/run.pyexamples/openai_triton/manual_plugin/tritonPlugins.cppexamples/openai_triton/plugin_autogen/README.mdexamples/openai_triton/plugin_autogen/build_engine.pyexamples/openai_triton/plugin_autogen/kernel_config.pyexamples/openai_triton/plugin_autogen/run_engine.pyexamples/python_plugin/README.mdexamples/python_plugin/build_lookup.pyexamples/python_plugin/plugin_lib/__init__.pyexamples/python_plugin/plugin_lib/lookup_kernel.pyexamples/python_plugin/plugin_lib/lookup_plugin.pyexamples/python_plugin/run_lookup.pyexamples/run.pyexamples/sample_weight_stripping/README.mdexamples/summarize.pyexamples/utils.pyjenkins/L0_Test.groovyjenkins/scripts/cbts/blocks.pytensorrt_llm/_utils.pytensorrt_llm/models/automodel.pytests/integration/defs/accuracy/test_llm_api_pytorch.pytests/integration/defs/common.pytests/integration/defs/conftest.pytests/integration/defs/examples/run_llm_fp8_quant_llama_70b.pytests/integration/defs/examples/test_bert.pytests/integration/defs/examples/test_bindings.pytests/integration/defs/examples/test_chatglm.pytests/integration/defs/examples/test_commandr.pytests/integration/defs/examples/test_draft_target_model.pytests/integration/defs/examples/test_eagle.pytests/integration/defs/examples/test_enc_dec.pytests/integration/defs/examples/test_flux.pytests/integration/defs/examples/test_gpt.pytests/integration/defs/examples/test_gptj.pytests/integration/defs/examples/test_granite.pytests/integration/defs/examples/test_internlm.pytests/integration/defs/examples/test_llama.pytests/integration/defs/examples/test_mamba.pytests/integration/defs/examples/test_medusa.pytests/integration/defs/examples/test_mistral.pytests/integration/defs/examples/test_mixtral.pytests/integration/defs/examples/test_multimodal.pytests/integration/defs/examples/test_nemotron.pytests/integration/defs/examples/test_nemotron_nas.pytests/integration/defs/examples/test_ngram.pytests/integration/defs/examples/test_openai.pytests/integration/defs/examples/test_phi.pytests/integration/defs/examples/test_qwen.pytests/integration/defs/examples/test_qwen2audio.pytests/integration/defs/examples/test_qwenvl.pytests/integration/defs/examples/test_redrafter.pytests/integration/defs/examples/test_whisper.pytests/integration/defs/examples/validate_whisper_beam_logits.pytests/integration/defs/examples/validate_whisper_log_probs_determinism.pytests/integration/defs/perf/build.pytests/integration/defs/perf/test_perf.pytests/integration/defs/test_e2e.pytests/integration/defs/triton_server/conftest.pytests/integration/test_lists/qa/README.mdtests/integration/test_lists/qa/llm_function_core.txttests/integration/test_lists/test-db/README.mdtests/integration/test_lists/test-db/l0_a10.ymltests/integration/test_lists/test-db/l0_a100.ymltests/integration/test_lists/test-db/l0_a30.ymltests/integration/test_lists/test-db/l0_b200.ymltests/integration/test_lists/test-db/l0_dgx_h200.ymltests/integration/test_lists/test-db/l0_gb203.ymltests/integration/test_lists/test-db/l0_gh200.ymltests/integration/test_lists/test-db/l0_h100.ymltests/integration/test_lists/test-db/l0_l40s.ymltests/integration/test_lists/waives.txt
💤 Files with no reviewable changes (111)
- examples/openai_triton/README.md
- examples/llm-eval/lm-eval-harness/requirements.txt
- examples/openai_triton/manual_plugin/README.md
- examples/llm-eval/lm-eval-harness/README.md
- examples/models/contrib/chatglm3-6b-32k/requirements.txt
- tests/integration/defs/examples/test_bindings.py
- examples/models/contrib/internlm/requirements.txt
- tests/integration/defs/examples/test_gptj.py
- examples/models/contrib/skywork/README.md
- examples/python_plugin/plugin_lib/lookup_kernel.py
- examples/openai_triton/plugin_autogen/README.md
- examples/models/core/multimodal/build_multimodal_engine.py
- examples/models/contrib/jais/README.md
- examples/models/contrib/jais/requirements.txt
- examples/models/core/mixtral/README.md
- examples/models/core/qwenvl/requirements.txt
- examples/cpp_library/CMakeLists.txt
- examples/models/contrib/chatglm-6b/requirements.txt
- examples/models/core/mixtral/requirements.txt
- examples/models/contrib/chatglm2-6b/requirements.txt
- examples/models/core/multimodal/requirements-vila.txt
- examples/models/contrib/chatglm2-6b/README.md
- examples/cpp_library/build.sh
- examples/models/core/multimodal/requirements-eclair.txt
- examples/models/core/nemotron/README_nemotron-3.md
- examples/models/contrib/chatglm3-6b-32k/README.md
- examples/openai_triton/plugin_autogen/kernel_config.py
- examples/models/contrib/smaug/README.md
- examples/dora/README.md
- examples/models/contrib/internlm/README.md
- examples/models/contrib/chatglm2-6b/tokenization_chatglm.py
- examples/python_plugin/plugin_lib/init.py
- examples/models/core/qwenvl/README.md
- examples/models/core/multimodal/requirements-internlm-xcomposer2.txt
- examples/models/contrib/chatglm-6b/README.md
- tests/integration/defs/examples/test_bert.py
- examples/generate_xgrammar_tokenizer_info.py
- examples/models/core/qwen2audio/README.md
- examples/models/core/granite/README.md
- examples/sample_weight_stripping/README.md
- examples/models/core/multimodal/utils.py
- tests/integration/defs/examples/test_flux.py
- examples/models/core/qwen2audio/requirements.txt
- tests/integration/defs/examples/run_llm_fp8_quant_llama_70b.py
- examples/models/contrib/sdxl/build_sdxl_unet.py
- examples/openai_triton/manual_plugin/build.py
- examples/models/core/qwen2audio/run_chat.py
- examples/openai_triton/manual_plugin/CMakeLists.txt
- examples/models/contrib/internlm/.gitignore
- tests/integration/defs/examples/test_medusa.py
- examples/models/core/qwenvl/vit_onnx_trt.py
- examples/python_plugin/run_lookup.py
- examples/openai_triton/manual_plugin/tritonPlugins.cpp
- examples/models/contrib/arctic/README.md
- examples/models/contrib/sdxl/README.md
- tests/integration/defs/examples/test_chatglm.py
- examples/models/contrib/sdxl/run_sdxl.py
- examples/language_adapter/README.md
- tests/integration/defs/examples/test_eagle.py
- examples/models/contrib/skywork/requirements.txt
- examples/models/contrib/chatglm-6b/tokenization_chatglm.py
- examples/ngram/run_dtm_ngram.py
- examples/models/core/qwenvl/show_pic.py
- examples/models/core/nemotron/requirements.txt
- examples/models/core/multimodal/requirements-llava_onevision.txt
- examples/models/core/qwenvl/run_chat.py
- examples/cpp_library/tensorrt_llm_libutils.h
- examples/python_plugin/build_lookup.py
- examples/models/core/qwen2audio/utils.py
- examples/openai_triton/plugin_autogen/run_engine.py
- examples/python_plugin/README.md
- examples/generate_checkpoint_config.py
- tests/integration/defs/examples/test_commandr.py
- examples/run.py
- examples/models/core/multimodal/run.py
- examples/openai_triton/manual_plugin/fmha_triton.py
- examples/hf_lora_convert.py
- examples/models/contrib/chatglm3-6b-32k/tokenization_chatglm.py
- examples/models/core/multimodal/eval.py
- tests/integration/defs/examples/test_mamba.py
- examples/mmlu.py
- examples/cpp_library/main.cpp
- examples/eval_long_context.py
- examples/models/core/multimodal/requirements-qwen2vl.txt
- examples/models/contrib/sdxl/pipeline_stable_diffusion_xl.py
- tests/integration/defs/examples/test_granite.py
- examples/models/contrib/blip2/README.md
- tests/integration/defs/examples/test_internlm.py
- examples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.cpp
- examples/openai_triton/manual_plugin/run.py
- examples/dora/normalize_weights.py
- examples/models/contrib/smaug/requirements.txt
- tests/integration/defs/test_e2e.py
- tests/integration/defs/examples/test_enc_dec.py
- examples/openai_triton/manual_plugin/plugin.py
- examples/utils.py
- examples/openai_triton/plugin_autogen/build_engine.py
- tests/integration/defs/examples/test_llama.py
- examples/summarize.py
- examples/llm-eval/lm-eval-harness/lm_eval_tensorrt_llm.py
- examples/models/core/qwen2audio/run.py
- tests/integration/test_lists/test-db/l0_a30.yml
- examples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.h
- jenkins/scripts/cbts/blocks.py
- tests/integration/defs/examples/test_draft_target_model.py
- examples/models/core/qwenvl/run.py
- examples/python_plugin/plugin_lib/lookup_plugin.py
- tests/integration/test_lists/test-db/l0_dgx_h200.yml
- tests/integration/test_lists/waives.txt
- tests/integration/defs/conftest.py
- tests/integration/defs/triton_server/conftest.py
Contributor
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
scope="tests/unittest/llmapi/apps"
echo "== Files in scope =="
fd --type f . "$scope" | sort
echo
echo "== Structural outline =="
ast-grep outline "$scope" --items all --type class,function --view compact || true
echo
echo "== TensorRT/backend-related occurrences with context =="
rg -n -i -C 4 \
-e '\btensorrt\b' \
-e '\btrtllm\b' \
-e '\btrt_llm\b' \
-e '\bbackend\s*=' \
-e '\bbackend\s*:' \
-e '\bbackend\b' \
"$scope" || true
echo
echo "== Python imports/calls that may configure backend =="
ast-grep run --lang python --pattern 'LLM($$$)' "$scope" || true
ast-grep run --lang python --pattern '$OBJ = LLM($$$)' "$scope" || trueLength of output: 66941 🏁 Script executed: #!/bin/bash
set -euo pipefail
scope="tests/unittest/llmapi/apps"
echo "== Exact/near-exact legacy backend tokens in Python source =="
python3 - <<'PY'
import ast
from pathlib import Path
root = Path("tests/unittest/llmapi/apps")
for path in sorted(root.rglob("*.py")):
try:
tree = ast.parse(path.read_text())
except SyntaxError as exc:
print(f"{path}:{exc.lineno}: syntax error: {exc}")
continue
for node in ast.walk(tree):
if isinstance(node, ast.Constant) and isinstance(node.value, str):
value = node.value.lower()
if "tensorrt" in value or value in {"trt", "trtllm"}:
print(f"{path}:{node.lineno}: {node.value!r}")
PY
echo
echo "== All backend fixture parameterizations and server CLI construction =="
rg -n -i -C 3 \
-e '`@pytest`\.fixture\([^)]*params\s*=' \
-e 'pytest\.mark\.parametrize\([^)]*backend' \
-e '["'\'']--backend["'\'']' \
-e 'backend\s*==|backend\s*!=' \
"$scope" -g '*.py' || true
echo
echo "== Parent-level pytest fixtures/configuration that could supply a backend =="
fd --type f --glob 'conftest.py' tests/unittest/llmapi | sort | while read -r file; do
echo "--- $file"
rg -n -i -C 3 -e '\btensorrt\b' -e '\bbackend\b' "$file" || true
done
echo
echo "== Focused declarations for backend fixture parameters =="
rg -n -i -U -C 2 \
'`@pytest`\.fixture\([\s\S]{0,250}?params\s*=\s*\[[^\]]+\][\s\S]{0,150}?def backend\b' \
"$scope" -g '*.py' || trueLength of output: 34786
All local The only stale backend-related item is a comment in |
StanleySun639
approved these changes
Jul 20, 2026
Collaborator
|
PR_Github #60335 [ run ] completed with state
|
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #60813 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #60813 [ run ] completed with state
|
Wanli-Jiang
force-pushed
the
user/williamj/trtllm-14027-tests-examples
branch
from
July 22, 2026 03:19
1913f7a to
2f41115
Compare
Wanli-Jiang
force-pushed
the
user/williamj/trtllm-14027-tests-examples
branch
from
July 22, 2026 03:26
2f41115 to
386a95d
Compare
Collaborator
Author
|
/bot run --disable-fail-fast |
Closed
1 task
… and CI plumbing Trailing test/example/CI cleanup after the legacy TensorRT backend removal (TRTLLM-14026, PR NVIDIA#16369). CI schedules / jenkins: - Move backend-neutral tests off the retired -TensorRT- stages and delete all dead backend=tensorrt test-db buckets; flip remaining neutral buckets to pytorch so parked coverage runs again. - NEW stages A100X-PyTorch-Post-Merge-1 and L40S-PyTorch-Post-Merge-1 schedule the restored a100/l40s post-merge coverage (CI-owner review). - Retire the -TensorRT- plumbing in L0_Test.groovy (commented stage rows, backend=tensorrt mako branch, TEST_BACKEND changeMap entry, excludedBackends) and the -TensorRT- pattern in jenkins/scripts/cbts/blocks.py. - waives.txt: drop 42 waives for deleted/unscheduled tests (the validator requires waived tests to be in an active list). Legacy engine-flow example tests: - Delete the defs/examples wrappers whose every test drove the removed convert->trtllm-build->run flow: test_{bert,chatglm,commandr, draft_target_model,eagle,enc_dec,gptj,granite,internlm,llama,mamba, medusa,mistral,mixtral,multimodal,nemotron,nemotron_nas,ngram, openai,qwen,qwen2audio,qwenvl,redrafter,whisper,bindings,flux}.py, the whisper validators, and run_llm_fp8_quant_llama_70b.py. - test_gpt.py rewritten keeping only the live test_gpt_oss_20b_lora_torch; test_phi.py drops its quantization engine test; test_e2e.py drops test_gpt3_175b_1layers_build_only. - Remove the dead helper cascade in defs/common.py (convert_weights, quantize_data, prune_checkpoint, refit_model, find_tensorrt), defs/perf/build.py and test_perf.py's trtllm-build branches, and 19 orphaned example_root fixtures across defs/conftest.py and triton_server/conftest.py (verified: no dynamic getfixturevalue resolution exists for any of them). Dormant example scripts (imported deleted ModelRunner*/builder surfaces or drove engine flows; unreachable from CI): - examples/{run,summarize,utils,mmlu,eval_long_context}.py, examples/openai_triton/**, examples/python_plugin/**, examples/models/contrib/sdxl/**, examples/llm-eval/lm-eval-harness, the engine-flow multimodal/qwenvl/qwen2audio scripts, and examples/ngram/run_dtm_ngram.py (READMEs rewritten for the PyTorch flow where a live section remains). examples/ reference sweep (checked against tests, published docs, jenkins, and code): - contrib: delete arctic, blip2, chatglm-6b/2-6b/3-6b-32k, internlm, jais, skywork, smaug (README/requirements-only legacy-flow dirs; hyperclovax stays - live PyTorch README linked from release notes). - core: delete granite and mixtral; multimodal keeps the live Qwen-Image-Bench stub + eval script but drops utils.py, __init__.py and the per-model requirements-*.txt; nemotron drops the Nemo-flow README_nemotron-3.md + requirements.txt (PyTorch READMEs stay). - top level: delete generate_checkpoint_config.py, generate_xgrammar_tokenizer_info.py, hf_lora_convert.py, sample_weight_stripping/, cpp_library/, dora/, language_adapter/. - Kept deliberately: infinitebench (dependency of the retained CLI-flow suite) and longbench/opentelemetry/sparse_attention (unreferenced but document current features). Accuracy suite: - accuracy/test_cli_flow.py and CliFlowAccuracyTestHarness are KEPT (currently unscheduled; retained for possible future reuse). - Port TestQwen2_7BInstruct::test_tp2 to the PyTorch accuracy suite (skipped: TP2 hangs in the AUTO custom allreduce on PCIe-only nodes; the LMHead AllReduce ignores allreduce_strategy, so the strategy knob cannot work around it). Misc: - automodel.py isMedusa/isEagle branch and _utils.py::supports_inflight_batching (mapped to deleted TRT model classes); test-db README example refreshed to a real test id; docs fixes in ci-overview.md and the AutoDeploy testing strategy. Rider (squashed in): [https://nvbugs/6482297][fix] Parse boolean stage-config fields in test_to_stage_mapping (from PR NVIDIA#16671, same author). The stage regex only accepted numeric trailing fields, so the 47 runWithSbatch stage entries (all DGX_B200-PyTorch-*) were silently dropped and commented-out stage rows were matched; test_backend_filtering_consistency now tolerates tests legitimately listed under multiple backends. Carried here because this PR retires stages and reshuffles test-db buckets - the exact surface those consistency tests validate. legacy-files.txt/ruff baseline entries for the deleted files are left to a mechanical 'legacy_utils.py prune-files' pass after this and the python-cleanup PR merge, to avoid cross-PR conflicts. Verified: defs collection passes (4206 tests), pre-commit hooks incl. the test-list AST validator pass, no broken example links. Signed-off-by: Wanli Jiang <35160485+Wanli-Jiang@users.noreply.github.com>
Wanli-Jiang
force-pushed
the
user/williamj/trtllm-14027-tests-examples
branch
from
July 22, 2026 04:42
386a95d to
29b7da2
Compare
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #60887 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #60887 [ run ] completed with state
|
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #60932 [ run ] triggered by Bot. Commit: |
ZhanruiSunCh
approved these changes
Jul 22, 2026
ZhanruiSunCh
left a comment
ZhanruiSunCh
left a comment
Collaborator
There was a problem hiding this comment.
LGTM for infra part.
kaiyux
approved these changes
Jul 22, 2026
QiJune
approved these changes
Jul 22, 2026
Collaborator
|
PR_Github #60932 [ run ] completed with state
|
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
Author
|
/bot skip --comment "all tests passed except the one waived case" |
Wanli-Jiang
enabled auto-merge (squash)
July 22, 2026 13:55
Collaborator
|
PR_Github #60986 [ run ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #60991 [ skip ] triggered by Bot. Commit: |
Collaborator
|
PR_Github #60986 [ run ] completed with state |
Collaborator
|
PR_Github #60991 [ skip ] completed with state |
Open
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Features
Why this PR exists
Phase C removed the TensorRT engine backend itself (
trtllm-build,convert_checkpoint.pyflows,ModelRunner*,tensorrt_llm.builder).This PR removes the test wrappers, example scripts, and CI plumbing that
existed only to exercise that removed backend. Everything deleted here
falls into one of two buckets:
no longer exists (
trtllm-build,examples/run.py, deleted modelclasses), or
backend: tensorrttest-db buckets /-TensorRT-CI stages retiredhere, or referenced by nothing at all (verified by reference sweep).
Change groups
1. CI schedules / jenkins (CI-owner review requested)
-TensorRT-stages-TensorRT-Post-Mergestage rows, thebackend=tensorrtmako branch, theTEST_BACKENDchangeMap entry, andexcludedBackendsmentions removed fromjenkins/L0_Test.groovy; the("-TensorRT-", "tensorrt")pattern dropped fromjenkins/scripts/cbts/blocks.py(kept in sync with the groovy map, hence in this PR, not the infra PR)backend: tensorrtbucketsl0_a30.yml(−33),l0_dgx_h200.yml(−21), plus legacy subsets inl0_a10/h100/l40spytorchso parked coverage runs again (l0_a100,l0_b200,l0_dgx_h2004-GPU,l0_l40s)A100X-PyTorch-Post-Merge-1andL40S-PyTorch-Post-Merge-1schedule the restored a100/l40s post-merge coverage — this is the main thing CI owners should eyeballNote: the dead
runPackageSanityCheckwheel-check removal is not inthis PR — it moved to the infra PR (TRTLLM-14027) since it is package
infra, uncoupled from test schedules.
2. Legacy engine-flow integration tests (~14k lines)
tests/integration/defs/examples/wrappers — every test inthem drove the removed convert →
trtllm-build→ run flow:test_{bert, chatglm, commandr, draft_target_model, eagle, enc_dec, gptj, granite, internlm, llama, mamba, medusa, mistral, mixtral, multimodal, nemotron, nemotron_nas, ngram, openai, qwen, qwen2audio, qwenvl, redrafter, whisper, bindings, flux}.py, the two whispervalidators, and
run_llm_fp8_quant_llama_70b.py.test_gpt.py(740→84 lines, keeps the livetest_gpt_oss_20b_lora_torch),test_phi.py(drops the quantizationengine test),
test_e2e.py(dropstest_gpt3_175b_1layers_build_only—unscheduled and invokes
trtllm-build).defs/common.py:convert_weights,quantize_data,prune_checkpoint,refit_model,find_tensorrt—last consumers were the wrappers above.
defs/perf/build.py(1,524lines) and
test_perf.py'strtllm-build/build.pybranches removed;the non-
trtllm-benchbuild-script legs now raiseRuntimeErrorso afuture misconfiguration fails loudly.
*_example_rootfixtures removed acrossdefs/conftest.pyandtriton_server/conftest.py. Verified: zerousers, and no dynamic
request.getfixturevalue(f"...")resolutionexists anywhere in defs that could reach them.
3. Dormant example scripts (~6k lines)
All imported deleted
ModelRunner*/builder surfaces or drove engineflows, and were unreachable from CI:
examples/{run, summarize, utils, mmlu, eval_long_context}.pyexamples/openai_triton/**(TRT plugin demos),examples/python_plugin/**examples/models/contrib/sdxl/**,examples/llm-eval/lm-eval-harness/**build_multimodal_engine.py,run.py,eval.py),qwenvl/**,qwen2audio/**examples/ngram/run_dtm_ngram.py(READMEs rewritten for the PyTorchflow where a live section remains)
4. examples/ reference sweep (~4.5k lines)
Every path checked against tests, published docs, jenkins, and code
(three independent scan agents + manual verification; the parked CLI-flow
suite deliberately not counted as a consumer):
arctic,blip2(redirect stub),chatglm-6b/2-6b/3-6b-32k,internlm,jais,skywork,smaug—README/requirements-only legacy-flow dirs, zero references.
hyperclovaxstays (live PyTorch README linked from release notes).granite/andmixtral/deleted (legacy-flow READMEs only);multimodal/keeps the live Qwen-Image-Bench stub +qwen_image_bench_eval.py(used by live unittests) but dropsutils.py,__init__.py, and 6 per-modelrequirements-*.txt;nemotron/drops the Nemo-flowREADME_nemotron-3.md+requirements.txt(the three current PyTorch READMEs stay).generate_checkpoint_config.py(fedtrtllm-build),generate_xgrammar_tokenizer_info.py(unreferenced; the live pathcalls
_xgrammar_tokenizer_info()directly),hf_lora_convert.py(produced the removed
.npyLoRA format),sample_weight_stripping/,cpp_library/,dora/,language_adapter/.5. Small code prunes
tensorrt_llm/models/automodel.py: thedecoding_modeisMedusa/isEagle branch — it mapped to
MedusaForCausalLM/EagleForCausalLMTRT model classes deleted in Phase B, so the branchcould only raise; no caller passes
decoding_mode. PyTorch spec-decdoes not use this path.
tensorrt_llm/_utils.py::supports_inflight_batching— same story.ci-overview.md, AutoDeploytesting_strategy.md,test-db README example refreshed to a real test id.
6. Additions (the +110)
TestQwen2_7BInstruct::test_tp2ported to the PyTorch accuracy suite(
test_llm_api_pytorch.py) + itsqa/llm_function_core.txtentry.Currently skipped with reason: TP2 hangs in the AUTO custom
allreduce on PCIe-only (all-SYS topology) nodes, and the LMHead
AllReduce ignores
allreduce_strategy, so the knob cannot work aroundit. Unskip once validated on NVLink or the lm_head strategy plumbing is
fixed. (Two nvbug filings pending for the underlying defects.)
Deliberately KEPT
accuracy/test_cli_flow.py+CliFlowAccuracyTestHarnessexamples/infinitebench/examples/{longbench, opentelemetry, sparse_attention}examples/models/contrib/hyperclovax/benchmarks/prepare_dataset.pyhyperclovax,exaone)Validation
tests/integration/defscollects 4,225 tests on therebased base, zero errors (was 4,207 pre-sweep; −1 deleted dead test,
+19 upstream additions).
validator and the test-db config checks.
(TRTLLM-14027), python-cleanup (TRTLLM-14474), and cpp-cleanup
(TRTLLM-14475) branches onto
origin/mainis conflict-free and wasverified to produce the intended combined tree.
origin/mainb8604c46ce; thewaives.txtconflict was resolved as upstream'sversion minus exactly our 42 deletions (our change verified to be
pure deletion).
Known follow-ups (not in this PR)
legacy-files.txt/ ruff-baseline entries for files deleted here gostale — a mechanical
scripts/legacy_utils.py prune-files+gen-configspass runs after this and the python-cleanup PR merge(kept out of both to avoid cross-PR conflicts).
test_tp2investigation: (1) AUTO customallreduce deadlock on all-SYS PCIe topology; (2) LMHead/Linear
AllReduce constructed without
model_config.allreduce_strategy.remaining
trtllm-buildREADME sections.Dev Engineer Review
-TensorRT-stage names as forcing a TensorRT backend selection.accuracy/...SKIP cases.QA Engineer Review
Test-list changes (test-db/ and qa/)
Modified files and key entry updates:
tests/integration/test_lists/qa/README.mdaccuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_auto_dtype(instead of the CLI-flow test).tests/integration/test_lists/qa/llm_function_core.txtaccuracy/test_llm_api_pytorch.py::TestQwen2_7BInstruct::test_tp2.tests/integration/test_lists/test-db/README.mdexamples/test_llama.py::test_llm_llama_v3_1_1node_multi_gpus[...]entries withaccuracy/test_llm_api_pytorch.py::TestLlama3_1_8BInstruct::test_fp8[...]entries (varyingfp8kv).tests/integration/test_lists/test-db/l0_a10.ymltests/integration/test_lists/test-db/l0_a100.ymltests/integration/test_lists/test-db/l0_a30.ymlterms.stage: post_mergeTensorRT entries (dropping multiple TensorRT-only example coverage entries).tests/integration/test_lists/test-db/l0_b200.ymltests/integration/test_lists/test-db/l0_dgx_h200.ymltests/integration/test_lists/test-db/l0_gb203.yml,l0_gh200.ymltests/integration/test_lists/test-db/l0_h100.ymlfmhafollows directly.tests/integration/test_lists/test-db/l0_l40s.ymlfmhablock.tests/integration/test_lists/waives.txtVerdict (test-list-only): needs follow-up (CI/backend coverage mapping for newly-added tests depends on CBTS/CI data not provided here, and merge-request pipelines reportedly failed).
Test-code changes (files outside test-lists)
Changed test modules / utilities:
tests/integration/defs/accuracy/test_llm_api_pytorch.pyTestQwen2_7BInstruct.test_tp2(skipped; guarded byskip_less_device(2)).tests/integration/defs/examples/test_phi.pytest_phi_4_mini_instruct_with_bf16_lora_torch(LLM-API Torch multi-LoRA path).tests/integration/defs/test_e2e.pytest_gpt3_175b_1layers_build_only.test_qwen_e2e_cpprunner_large_new_tokens(Qwen2/DeepSeek-R1-Distill-Qwen-1.5B generation smoke).tests/integration/defs/examples/test_bert.pytests/integration/defs/examples/test_bindings.pytests/integration/defs/examples/test_chatglm.pytests/integration/defs/examples/test_commandr.pytests/integration/defs/examples/test_draft_target_model.pytests/integration/defs/examples/test_eagle.pytests/integration/defs/examples/test_enc_dec.pytests/integration/defs/examples/test_flux.pytests/integration/defs/examples/test_gptj.pytests/integration/defs/examples/test_granite.pytests/integration/defs/examples/test_internlm.pytests/integration/defs/examples/test_llama.py(module removed)tests/integration/defs/examples/test_mamba.pytests/integration/defs/examples/test_medusa.pytests/integration/defs/examples/test_phi_quantizationpath removed (via file-level edits described in diff summary)tests/integration/defs/common.py(removed plugin options + TRT helper utilities like pruning/refit/quantize discovery)tests/integration/defs/conftest.py(removed many example-root fixtures; removedplugin_gen_path)tests/integration/defs/perf/test_perf.py(removed legacy TRT backend build wiring and restricted dataset prep behavior)tests/integration/defs/triton_server/conftest.py(removed some TRT multimodal/qwen/mixtral example-root fixtures)tests/unittest/tools/test_test_to_stage_mapping.py(test logic update for backend mapping consistency)Coverage check against test-lists (from provided diff summary):
TestQwen2_7BInstruct.test_tp2tests/integration/test_lists/qa/llm_function_core.txt).tests/integration/defs/examples/test_phi.py::test_phi_4_mini_instruct_with_bf16_lora_torchtests/integration/test_lists/*diffs; follow-up needed to confirm CI selection.tests/integration/defs/test_e2e.py::test_qwen_e2e_cpprunner_large_new_tokenstests/integration/test_lists/*diffs; follow-up needed to confirm CI selection.Verdict (test-code changes): needs follow-up (new/ported tests’ CI selection is not fully verifiable from the provided test-list diffs, and merge-request pipeline failures were reported).
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.