Skip to content

test: Skip Torch-TensorRT QA model generation on compute capability 10.7 - #8914

Draft
mc-nv wants to merge 1 commit into
mainfrom
mchornyi/TRI-1625/skip-torchtrt-resnet50-on-rubin
Draft

test: Skip Torch-TensorRT QA model generation on compute capability 10.7#8914
mc-nv wants to merge 1 commit into
mainfrom
mchornyi/TRI-1625/skip-torchtrt-resnet50-on-rubin

Conversation

@mc-nv

@mc-nv mc-nv commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

What does the PR do?

Skips generation of the Torch-TensorRT resnet50_libtorch QA model on GPUs
reporting compute capability 10.7.

On that architecture the TensorRT Myelin autotuner has no maxpool tactic. The
fused conv1 + bn1 + relu + maxpool head of resnet50 is compiled into a single
Myelin ForeignNode, and with no kernel for its 3x3 / stride-2 maxpool the
engine build fails:

ERROR: [Torch-TensorRT TorchScript Conversion Context] - Error Code: 9:
  Skipping tactic 0x0 due to exception [autotuner.cpp:3867: get_best_tactics]
  Autotuner: no tactics to implement operation:
 1533: maxpool: ... lpad: {1,1} rpad: {1,1} pad_mode: 0 strides: {2,2}
       dilations: {1,1} sizes: {3,3}
ERROR: ... IBuilder::buildSerializedNetwork: Error Code 10: Internal Error
  (Could not find any implementation for node {ForeignNode[...]})
RuntimeError: [core/conversion/conversionctx/ConversionCtx.cpp:157]
  Building serialized network failed in TensorRT

This is a TensorRT architecture-enablement gap rather than a defect in the
model — the failing node is the first four layers of a stock torchvision
resnet50 in FP32. The guard is keyed on the compute capability reported by
nvidia-smi so it lifts automatically when a device with a working tactic is
used, and it is a one-line revert once TensorRT ships the missing kernels.

Why it matters beyond one model

The generated PyTorch script runs under set -e, so this failure aborted the
script partway through. Ragged, torchvision-image (resnet152 / vgg19) and
custom-ops generation never ran, and the model repository was never uploaded —
a single build failure took the whole downstream test train with it.

Checklist

  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

  • test

Related PRs:

None — the change is confined to this repository.

Where should the reviewer start?

qa/common/gen_qa_model_repository, the Torch-TensorRT block in the generated
PyTorch script (~line 286). Note the block lives inside an unquoted heredoc:
\$COMPUTE_CAP is escaped so it resolves at run time inside the PyTorch
container, not on the CI runner at script-generation time.

Test plan:

Verified locally:

  • bash -n passes on gen_qa_model_repository and on the generated PyTorch
    script rendered from the heredoc.
  • Gate exercised against a stubbed nvidia-smi: skips on 10.7, runs on
    8.0, 8.7, 10.3, 11.0 and 12.0.
  • With nvidia-smi absent the capability resolves empty and generation runs as
    before, without tripping set -e (pipeline exit status comes from tr).

Verified from the failing CI job log that nvidia-smi --query-gpu=compute_cap
is available inside the PyTorch container that executes this script and returns
10.7 there, so the fall-through path is not reachable on the affected runners.

  • CI Pipeline ID: (none yet — the internal pipeline clones server main, so
    it can only exercise this change once the PR is merged)

Caveats:

This removes the known blocker; it does not by itself prove the generation job
goes green on the affected architecture. Because the abort happened partway
through the script, the ragged, torchvision-image and custom-ops steps have
never executed there, and may surface their own failures on the next run.

The compute capability is hard-coded rather than derived from a TensorRT
capability query, since no such query exists for "does a tactic exist for this
op". The comparison is exact, so a future 10.7-family device is covered but an
unrelated capability is not — which is the intended conservative behavior.

Background

The Torch-TensorRT model is consumed by L0_libtorch_torchtrt_image_models,
which copies torchtrt_model_store from the generated repository. That test
will need to be excluded on this architecture while the guard is in place.

Related Issues:

  • Resolves: TRI-1625

The TensorRT Myelin autotuner has no maxpool tactic on compute capability
10.7, so building the resnet50 Torch-TensorRT engine fails with 'Autotuner:
no tactics to implement operation' followed by 'Could not find any
implementation for node {ForeignNode[...]}'.

The generated PyTorch script runs under 'set -e', so the abort also skipped
ragged, torchvision image and custom-ops model generation and left the whole
model repository unpublished. Detect the compute capability with nvidia-smi
and skip only the Torch-TensorRT step, so the rest of the repository is still
produced. If nvidia-smi is unavailable the step runs as before.
@greptile-apps

greptile-apps Bot commented Jul 31, 2026

Copy link
Copy Markdown

Greptile Summary

This PR changes Torch-TensorRT QA model generation to skip the step on compute capability 10.7.

  • Queries the first visible GPU’s compute capability with nvidia-smi.
  • Skips generation when the reported capability is exactly 10.7.
  • Emits a diagnostic message when the generation step is skipped.

Confidence Score: 4/5

The MODEL_TYPE=igpu exclusion should be restored alongside the new compute-capability check before merging.

The new condition handles the 10.7 failure but sends non-10.7 igpu invocations into Torch-TensorRT generation even though the previous code explicitly excluded that model type.

Files Needing Attention: qa/common/gen_qa_model_repository

Important Files Changed

Filename Overview
qa/common/gen_qa_model_repository Adds the 10.7 architecture gate, but unintentionally removes the existing igpu model-type exclusion.

Reviews (1): Last reviewed commit: "test: Skip Torch-TensorRT QA model gener..." | Re-trigger Greptile

Comment thread qa/common/gen_qa_model_repository
@mc-nv mc-nv self-assigned this Jul 31, 2026
@mc-nv mc-nv added the testing Adding or correcting tests (test: PRs) label Jul 31, 2026
@mc-nv
mc-nv requested review from Vinya567 and yinggeh July 31, 2026 22:51
@mc-nv
mc-nv marked this pull request as draft August 4, 2026 01:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Adding or correcting tests (test: PRs)

Development

Successfully merging this pull request may close these issues.

1 participant