test: Skip Torch-TensorRT QA model generation on compute capability 10.7 - #8914
Draft
mc-nv wants to merge 1 commit into
Draft
test: Skip Torch-TensorRT QA model generation on compute capability 10.7#8914mc-nv wants to merge 1 commit into
mc-nv wants to merge 1 commit into
Conversation
The TensorRT Myelin autotuner has no maxpool tactic on compute capability
10.7, so building the resnet50 Torch-TensorRT engine fails with 'Autotuner:
no tactics to implement operation' followed by 'Could not find any
implementation for node {ForeignNode[...]}'.
The generated PyTorch script runs under 'set -e', so the abort also skipped
ragged, torchvision image and custom-ops model generation and left the whole
model repository unpublished. Detect the compute capability with nvidia-smi
and skip only the Torch-TensorRT step, so the rest of the repository is still
produced. If nvidia-smi is unavailable the step runs as before.
Greptile SummaryThis PR changes Torch-TensorRT QA model generation to skip the step on compute capability 10.7.
Confidence Score: 4/5The MODEL_TYPE=igpu exclusion should be restored alongside the new compute-capability check before merging. The new condition handles the 10.7 failure but sends non-10.7 igpu invocations into Torch-TensorRT generation even though the previous code explicitly excluded that model type. Files Needing Attention: qa/common/gen_qa_model_repository Important Files Changed
Reviews (1): Last reviewed commit: "test: Skip Torch-TensorRT QA model gener..." | Re-trigger Greptile |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does the PR do?
Skips generation of the Torch-TensorRT
resnet50_libtorchQA model on GPUsreporting compute capability 10.7.
On that architecture the TensorRT Myelin autotuner has no maxpool tactic. The
fused
conv1 + bn1 + relu + maxpoolhead of resnet50 is compiled into a singleMyelin
ForeignNode, and with no kernel for its 3x3 / stride-2 maxpool theengine build fails:
This is a TensorRT architecture-enablement gap rather than a defect in the
model — the failing node is the first four layers of a stock torchvision
resnet50 in FP32. The guard is keyed on the compute capability reported by
nvidia-smiso it lifts automatically when a device with a working tactic isused, and it is a one-line revert once TensorRT ships the missing kernels.
Why it matters beyond one model
The generated PyTorch script runs under
set -e, so this failure aborted thescript partway through. Ragged, torchvision-image (resnet152 / vgg19) and
custom-ops generation never ran, and the model repository was never uploaded —
a single build failure took the whole downstream test train with it.
Checklist
<commit_type>: <Title>Commit Type:
Related PRs:
None — the change is confined to this repository.
Where should the reviewer start?
qa/common/gen_qa_model_repository, the Torch-TensorRT block in the generatedPyTorch script (~line 286). Note the block lives inside an unquoted heredoc:
\$COMPUTE_CAPis escaped so it resolves at run time inside the PyTorchcontainer, not on the CI runner at script-generation time.
Test plan:
Verified locally:
bash -npasses ongen_qa_model_repositoryand on the generated PyTorchscript rendered from the heredoc.
nvidia-smi: skips on10.7, runs on8.0,8.7,10.3,11.0and12.0.nvidia-smiabsent the capability resolves empty and generation runs asbefore, without tripping
set -e(pipeline exit status comes fromtr).Verified from the failing CI job log that
nvidia-smi --query-gpu=compute_capis available inside the PyTorch container that executes this script and returns
10.7there, so the fall-through path is not reachable on the affected runners.servermain, soit can only exercise this change once the PR is merged)
Caveats:
This removes the known blocker; it does not by itself prove the generation job
goes green on the affected architecture. Because the abort happened partway
through the script, the ragged, torchvision-image and custom-ops steps have
never executed there, and may surface their own failures on the next run.
The compute capability is hard-coded rather than derived from a TensorRT
capability query, since no such query exists for "does a tactic exist for this
op". The comparison is exact, so a future 10.7-family device is covered but an
unrelated capability is not — which is the intended conservative behavior.
Background
The Torch-TensorRT model is consumed by
L0_libtorch_torchtrt_image_models,which copies
torchtrt_model_storefrom the generated repository. That testwill need to be excluded on this architecture while the guard is in place.
Related Issues: