[cuda_xpu_alignment] DISABLED test_torch_profiler_benchmarker_reuses_inductor_helpers_hip_value0_expected_buffer_size_bytes_1024 (__main__.TestBenchmarker)#36
Draft
Stonepia wants to merge 8 commits into
Conversation
## Summary - Reverts pytorch#183243 and pytorch#183482, ending the EC2/OSDC side-by-side shadow-mode comparison (issue pytorch#183242). `_linux-build.yml` / `_linux-test.yml` go back to a single-route model gated by `use-arc`; the EC2 route again uses `runner_prefix` directly instead of the shadow-mode strip-prefix step. - Removes the `test-matrix-osdc:` callsite line from the 10 OSDC-migrated workflows that adopted the schema after the shadow-traffic PRs landed (`dtensor`, `dynamo-unittest`, `h100-cutlass-backend`, `inductor`, `inductor-micro-benchmark-x86`, `inductor-nightly`, `inductor-periodic`, `operator_benchmark`, `periodic`, `torchtitan`). Without this cleanup the reusable workflow would reject those callsites with "input not defined" once the schema goes away. - Leaves pytorch#183244 (auto-label `no-runner-experiments` on `.ci/docker/` PRs) in place: that label is a general runner-experiment opt-out in `runner_determinator.py` and is not specific to shadow traffic. After this change, we will still running 100% on OSDC with the option to dial down if needed. ## Test plan - [ ] `pull.yml`, `trunk.yml`, `periodic.yml`, and the rest of the touched workflows validate on this PR (no "input not defined" errors). - [ ] On this PR (a `pull_request` event with `use-arc=false`), the EC2 build/test jobs run and the OSDC ones are skipped — no shadow-mode dual run. - [ ] After landing, push a `ciflow/trunk/*` tag and confirm only one of EC2/OSDC runs based on `use-arc` (no parallel dual route). - [ ] After landing, confirm a direct push to `main` continues to honor `use-arc` (single route as before). Authored by Claude. Pull Request resolved: pytorch#185181 Approved by: https://github.com/jeanschmidt
…ch#185260) Authored by Claude. Pull Request resolved: pytorch#185260 Approved by: https://github.com/williamwen42
Fix a silent-corruption / out-of-bounds bug in `slow_conv_dilated3d` (CUDA and CPU) when called with non-batched 4D input and a defined bias. Pull Request resolved: pytorch#185352 Approved by: https://github.com/Skylion007
`gpus_for_rank()` in `test_store.py` has 0 references. Other tests all use `gpus_for_rank()` from `test_c10d_common.py` (same name, same function) Pull Request resolved: pytorch#185194 Approved by: https://github.com/zou3519
# Summary This is a prep commit, basically turns our old adhock sizevar vec/lane math into a more strongly typed variant and put it closer to cutedsl. It has 1 prep entry which is add_tensor_inputs. This basically mirrors what we have in mdofication wrapper but since for the packed lane shfit path we dont go through the graph replay im putting it here. This is baiscally a noop PR and all prep. I read it all :) ### Note for the reviewer I dont think this needs much review its mostly code motion / adding longer doc blocks and adding types Pull Request resolved: pytorch#185019 Approved by: https://github.com/eellison, https://github.com/liangel-02
…rch#185303) Summary: D106105718 (PyTorch DiffTrain, May 22-24) changed `torch/_native/ops/__init__.py ` from `from . import bmm_outer_product, scatter_add` to `from . import bmm_outer_product, norm, scatter_add` 2. That new norm import triggers `norm/__init__.py` which calls `register_rmsnorm_overrides()`. This registers a CuTeDSL-backed RMSNorm kernel at the CUDA dispatch key — meaning whenever `aten::_fused_rms_norm` is called on a CUDA tensor, PyTorch will check this override first. 3. The registration gate in `cutedsl_utils.py` checks `_cuda.is_built()` — but on ROCm/AMD builds, this returns True because HIP implements the CUDA API. So the override gets registered on MI350X. 4. During training, the model calls `_fused_rms_norm`. The function `_fused_rms_norm_cond` checks `torch.cuda.get_device_capability()` — MI350X is gfx950, which reports major=9, matching the major in (9, 10) check. So the cond returns True. 5. The impl function runs and lazily imports `from torch._vendor.quack.rmsnorm import _compile_rmsnorm_fwd → quack imports cutlass → cutlass.base_dsl imports cuda.bindings.driver → calls cuInit(0) → crash`. There's no NVIDIA driver on an AMD machine, so `cuInit` fails immediately with CudaDriverDependencyError. **Fix:** add a `torch.version.hip` guard in `_check_runtime_available()` so that CuTeDSL ops are never registered on ROCm builds. Test Plan: ``` import torch print("hip:", torch.version.hip) print("cuda.is_built:", torch.backends.cuda.is_built()) from torch._native.cutedsl_utils import _check_runtime_available available, version = _check_runtime_available() print("cutedsl available:", available) print("cutedsl version:", version) ``` Without fix (control): ``` hip: 7.0.51831 cuda.is_built: True cutedsl available: True ← BUG: CuTeDSL registered on AMD cutedsl version: 4.4.2 FAIL: CuTeDSL should be disabled on ROCm/HIP! ``` With fix (treatment): ``` hip: 7.0.51831 cuda.is_built: True cutedsl available: False ← FIXED: CuTeDSL blocked on AMD cutedsl version: None PASS: CuTeDSL correctly disabled on ROCm/HIP ``` Reviewed By: wdvr Differential Revision: D106426428 Pull Request resolved: pytorch#185303 Approved by: https://github.com/slayton58
# Motivation PR pytorch#185131 refactored this test file to be device-agnostic, but there are still some CUDA-specific hardcoded paths remaining. This PR updates those parts to make the test fully device-agnostic. Pull Request resolved: pytorch#185326 Approved by: https://github.com/Skylion007
[cuda_xpu_alignment] DISABLED test_torch_profiler_benchmarker_reuses_inductor_helpers_hip_value0_expected_buffer_size_bytes_1024 (__main__.TestBenchmarker)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[cuda_xpu_alignment] DISABLED test_torch_profiler_benchmarker_reuses_inductor_helpers_hip_value0_expected_buffer_size_bytes_1024 (main.TestBenchmarker)
Fixes https://github.com/intel-sandbox/agentic_xpu/issues/51
Root Cause: The TorchProfilerBenchmarker.benchmark_gpu() in torch/_inductor/runtime/benchmarking.py hardcodes torch.cuda.synchronize() and device='cuda' throughout. On XPU-only builds without CUDA, this immediately raises 'Torch not compiled with CUDA enabled'. The entire benchmarking module lacks XPU device abstraction.
Failed Tests:
test/inductor/test_benchmarking.py::TestBenchmarker::test_torch_profiler_benchmarker_reuses_inductor_helpers_hip_value0_expected_buffer_size_bytes_1024Failure Type: NEW_FAILURE
Diff stat: