test_dtensor.py (float8 TP compile): aten.abs.default NotImplemented on Float8TrainingTensor after removing allow-in-graph autograd wrappers

Commits affected: ca7488145beeda7416ceee0db43eb6f5124b2ebc
Run: https://github.com/pytorch/ao/actions/runs/33666967452

Summary

_test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py fails at inductor compile time with NotImplementedError: attempting to run aten.abs.default, this is not supported, raised from Float8TrainingTensor.__torch_dispatch__. Routing the Float8 autograd.Function classes through the standard Dynamo AutogradFunctionApply HOP (this commit) causes the backward to be retraced via make_fx, and torch.abs in the amax computation now dispatches onto a Float8TrainingTensor, which has no abs handler.

Failure details

Failed job(s) (both fail identically, deterministically):
- test (H100, mt-l-x86iamx-88-900-h100-4, --pre torch torchvision --index-url .../nightly/cu126)
- test (H100 2.13, mt-l-x86iamx-88-900-h100-4, torch==2.13.0 torchvision==0.28.0 --index-url .../cu126)
Test step: ./test/float8/test_everything_multi_gpu.sh → test/float8/test_dtensor.py FAILED (torchrun ChildFailedError, exit code 1). Preceding subtests (✅ All Tests Passed ✅) pass; the failure is isolated to the compile TP dtensor test.

Representative error:

Test _test_fp8_mlp_tensor_parallelism_compile failed with error: backend='inductor' raised:
NotImplementedError: attempting to run aten.abs.default, this is not supported

While executing %abs_1 : [num_users=1] = call_function[target=torch.abs](args = (%grad_output_reshaped,), kwargs = {})

Dispatch chain (from the original traceback):
- torchao/float8/float8_linear.py:112 backward → hp_tensor_to_float8_dynamic
- torchao/float8/float8_scaling_utils.py:56 → tensor_to_scale
- torchao/float8/float8_utils.py:106 tensor_to_scale → tensor_to_amax
- torchao/float8/float8_utils.py:65 tensor_to_amax → amax = torch.max(torch.abs(x))
- raised at torchao/float8/float8_training_tensor.py:373 __torch_dispatch__: raise NotImplementedError(f"attempting to run {func}, this is not supported")
Compile-time framing: the backward runs under torch._functorch.autograd_function.backward → torch.fx.Interpreter(bwd).run(...) during AOTAutograd make_fx tracing, i.e. the op executes against the Float8 subclass while retracing, not during normal eager backward.

Root cause

Commit ca74881 ("Remove Float8 allow-in-graph custom autograd wrappers") routes the four Float8 autograd.Function classes through the standard Dynamo AutogradFunctionApply higher-order op instead of the previous allow-in-graph trampoline. Under torch.compile + DTensor tensor parallelism, the backward is now captured and retraced by make_fx/FX Interpreter. During that retrace, tensor_to_amax's torch.abs(grad_output_reshaped) is dispatched onto a Float8TrainingTensor, whose __torch_dispatch__ only services ops present in FLOAT8_OPS_TABLE; aten.abs.default is not registered there, so it raises NotImplementedError and inductor compilation fails.

Evidence this is a regression introduced by this commit (not flaky/infra):

The immediately preceding 4xH100 run on main (parent commit a195aaf8a2da5b8496018d39e5261e93d01fb0ca, run 33632094558) and the prior 7 runs of this workflow were all green.
The failing code path — Float8 autograd backward tracing through the HOP — is exactly what this commit changed.
The failure is deterministic and reproduces identically on both the torch-nightly and torch-2.13.0 matrix legs, ruling out a per-run flake or a single-nightly toolchain change.

Classification

real regression — a deterministic compile-time failure in the exact code path modified by the triggering commit, green on the parent commit and reproducing on both matrix legs. Not intermittent (so not a flaky test) and not an environment/toolchain issue (so not infra).

Suggested fix

In Float8TrainingTensor's op table (torchao/float8/float8_ops.py / float8_training_tensor.py), the abs/amax path taken during HOP-based backward retracing needs to resolve against the underlying high-precision tensor rather than the subclass. Options:
- Have tensor_to_amax/tensor_to_scale operate on the plain (dequantized/underlying) tensor data so torch.abs/torch.max never dispatch onto Float8TrainingTensor during retrace; or
- Register aten.abs.default (and any other amax-path ops surfaced) in FLOAT8_OPS_TABLE to delegate to the underlying data.
Add a regression test that compiles the Float8 backward through the AutogradFunctionApply HOP path (the TP-compile dtensor scenario) so this is caught pre-merge.
If a fix is not immediately available, consider reverting ca74881's autograd-wrapper change or gating it until the amax dispatch under retrace is handled.

⚠️ Run Regression Tests on ROCm also failed on this commit (ca7488145beeda7416ceee0db43eb6f5124b2ebc) — https://github.com/pytorch/ao/actions/runs/33666967935

Recurrence on main — same root cause as this issue's diagnosis.

Run: https://github.com/pytorch/ao/actions/runs/33902500930 (Run 4xH100 tests)
Commit: b7ac3aacf9b6f1cb0c6bb29143329f601dfcfcb1 ("Fix cvt_packfloat compatibility with nvidia-cutlass-dsl 4.6.0+", Fix cvt_packfloat compatibility with nvidia-cutlass-dsl 4.6.0+ #4766 — unrelated to float8; the ca74881 autograd-wrapper regression remains unfixed on main and keeps breaking this workflow).

Matches the existing diagnosis exactly:

Test: _test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py (via ./test/float8/test_everything_multi_gpu.sh); preceding subtests pass (✅ All Tests Passed ✅), then torchrun ChildFailedError, exit code 1.
Error: backend='inductor' raised: NotImplementedError: attempting to run aten.abs.default, this is not supported, While executing %abs_1 = call_function[target=torch.abs](args=(%grad_output_reshaped,)).
Same dispatch chain: torch/_functorch/autograd_function.py:874 backward → torch.fx.Interpreter(bwd).run(...) (HOP AutogradFunctionApply backward retrace under make_fx) → raised at torchao/float8/float8_training_tensor.py:373 __torch_dispatch__.
Reproduces identically and deterministically on both matrix legs: --pre torch nightly cu126 and torch==2.13.0 cu126 — confirming it is not a per-run flake or single-nightly toolchain change.

No new information vs. the existing analysis; classification remains a real regression. Still needs the abs/amax path to resolve against the underlying high-precision tensor during HOP-based backward retrace (or aten.abs.default registered in FLOAT8_OPS_TABLE), or a revert/gate of ca74881.

Recurs on main at commit 3efa9eda3bd1347fa2485d16641bbc5bc2124d53 — Run 4xH100 tests, https://github.com/pytorch/ao/actions/runs/34224961671.

Same root cause and signature as the existing diagnosis (no divergence):

_test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py fails at inductor compile time with NotImplementedError: attempting to run aten.abs.default, this is not supported.
Raised from torchao/float8/float8_training_tensor.py:373 __torch_dispatch__, reached via the AutogradFunctionApply HOP backward retrace: torch/_functorch/autograd_function.py backward → torch.fx.Interpreter(bwd).run(...), executing %abs_1 = call_function[target=torch.abs](args=(%grad_output_reshaped,)) during AOTAutograd make_fx tracing.
Fails identically and deterministically on both matrix legs (torch==2.13.0 and --pre torch nightly, both cu126); torchrun exits with ChildFailedError → exit code 1. The earlier float8 subtests still pass (✅ All Tests Passed ✅).

The triggering commit 3efa9ed ("Reject unsupported wrap_with_set_grad_enabled subgraphs in PT2E prepare APIs") does not touch float8 — this is the unfixed regression from ca74881 still present on main. Confirms it remains a real regression, not flaky/infra.

Recurs on main at commit 6792133e5eb7c607d27f69756870e44e7ffed6af (run https://github.com/pytorch/ao/actions/runs/34280478407, "Run 4xH100 tests").

Same root cause and signature as this issue's diagnosis — no new information:

_test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py fails at inductor compile time with NotImplementedError: attempting to run aten.abs.default, this is not supported, raised from Float8TrainingTensor.__torch_dispatch__ (torchao/float8/float8_training_tensor.py:373).
Dispatch path is identical: AOTAutograd make_fx retraces the backward through the AutogradFunctionApply HOP → torch._functorch.autograd_function.backward → torch.fx.Interpreter(bwd).run(...), so torch.abs in the amax computation dispatches onto the Float8 subclass.
Fails deterministically on BOTH matrix legs (torch==2.13.0 ... cu126 and --pre torch ... nightly/cu126); the preceding test_everything_multi_gpu.sh subtests pass (✅ All Tests Passed ✅), then torchrun exits 1 via ChildFailedError on test/float8/test_dtensor.py.

The triggering commit 6792133 (fix(_is_mslk_available): guard against missing CUDA/ROCm runtime, #4544) does not touch float8 code — this is the same persistent regression from ca74881 still unfixed on main, not a new cause. Classification unchanged: real regression.

⚠️ Run Regression Tests on ROCm also failed on this commit (6792133e5eb7c607d27f69756870e44e7ffed6af) — https://github.com/pytorch/ao/actions/runs/34280478457

Recurs on main at commit 6f7bb3874302ebfc7bf7c15449a9c1d93d0ca549 — Run 4xH100 tests, run https://github.com/pytorch/ao/actions/runs/34342903087.

Same root cause and signature as this issue's diagnosis (no change): _test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py fails at inductor compile with NotImplementedError: attempting to run aten.abs.default, this is not supported, raised while retracing the Float8 backward:

torch.max(torch.abs(x)) in tensor_to_amax (torchao/float8/float8_utils.py:65)
dispatched onto Float8TrainingTensor → __torch_dispatch__ (torchao/float8/float8_training_tensor.py:373)
While executing %abs_1 : ... call_function[target=torch.abs](args = (%grad_output_reshaped,))

Both matrix legs fail identically and deterministically (torch==2.13.0 cu126 job 102437611971, and --pre torch nightly cu126 job 102437612207), ending in ChildFailedError / test/float8/test_dtensor.py FAILED, exit code 1.

The triggering commit 6f7bb38 ("Refactor MXTensor: align the swizzle execution...") does not touch float8 code; this is the same unresolved regression from ca74881 still present on main, not a new cause. Still a real regression — the amax torch.abs dispatch onto Float8TrainingTensor during HOP-based backward retrace remains unhandled.

⚠️ Run Regression Tests on ROCm also failed on this commit (6f7bb3874302ebfc7bf7c15449a9c1d93d0ca549) — https://github.com/pytorch/ao/actions/runs/34342903178

Recurs on main at commit 6ded493033ebba47e2631b087cb17f01992d51f0 — run https://github.com/pytorch/ao/actions/runs/34356567625 (Run 4xH100 tests). Same root cause and diagnosis as this issue, still unfixed.

_test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py fails at inductor compile time with NotImplementedError: attempting to run aten.abs.default, this is not supported, raised from Float8TrainingTensor.__torch_dispatch__. The dispatch chain is identical to the recorded one:

torchao/float8/float8_linear.py:112 backward → hp_tensor_to_float8_dynamic
torchao/float8/float8_scaling_utils.py:56 → tensor_to_scale
torchao/float8/float8_utils.py:106 tensor_to_scale → tensor_to_amax
torchao/float8/float8_utils.py:65 tensor_to_amax → amax = torch.max(torch.abs(x))
raised at torchao/float8/float8_training_tensor.py:373 __torch_dispatch__

The backward is retraced via torch._functorch.autograd_function.backward → torch.fx.Interpreter(bwd).run(...) under AOTAutograd make_fx, so torch.abs dispatches onto Float8TrainingTensor during retrace — matching the existing diagnosis (HOP-based AutogradFunctionApply backward retrace).

Both matrix legs fail identically and deterministically:

test (H100, mt-l-x86iamx-88-900-h100-4, --pre torch torchvision --index-url .../nightly/cu126)
test (H100 2.13, mt-l-x86iamx-88-900-h100-4, torch==2.13.0 torchvision==0.28.0 --index-url .../cu126)

torchrun exits with ChildFailedError; preceding float8 multi-GPU subtests pass (✅ All Tests Passed ✅), failure isolated to the compile TP dtensor test. Confirms the regression is still present on main at this newer commit.

Still failing on main at commit 22afe13b49e1dcc841ec89c616b5bfb020c357f1 — run https://github.com/pytorch/ao/actions/runs/34381027616 (Run 4xH100 tests).

Matches the existing diagnosis exactly. Both matrix legs fail identically and deterministically:

test (H100, ... --pre torch ... nightly/cu126)
test (H100 2.13, ... torch==2.13.0 ... cu126)

test/float8/test_dtensor.py::_test_fp8_mlp_tensor_parallelism_compile raises backend='inductor' raised: NotImplementedError: attempting to run aten.abs.default, this is not supported (torchrun ChildFailedError, exit 1). Preceding subtests print ✅ All Tests Passed ✅; the failure is isolated to the compile TP dtensor test. Same dispatch chain as reported: float8_linear.py:112 backward → float8_scaling_utils.py:56 hp_tensor_to_float8_dynamic → float8_utils.py:106 tensor_to_scale → float8_utils.py:65 tensor_to_amax → amax = torch.max(torch.abs(x)), dispatched onto Float8TrainingTensor during HOP-based backward retrace.

The triggering commit 22afe13 ("Make quantize_ skip already-quantized weights") does not touch float8 code; this is the same underlying regression from ca74881 continuing to fail on main, not a new cause. No divergence from the existing root-cause analysis.

Recurred on main at commit ee0d7ce304cc341fc6894df1f3b32038db0da1c3 — run https://github.com/pytorch/ao/actions/runs/34399534209 ("Run 4xH100 tests").

Matches the existing diagnosis exactly. _test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py fails at inductor compile time with:

NotImplementedError: attempting to run aten.abs.default, this is not supported
While executing %abs_1 : [num_users=1] = call_function[target=torch.abs](args = (%grad_output_reshaped,), kwargs = {})

Same dispatch chain as before: float8_linear.py:112 backward → hp_tensor_to_float8_dynamic → float8_scaling_utils.py:56 → tensor_to_scale → float8_utils.py:106 tensor_to_amax → float8_utils.py:65 amax = torch.max(torch.abs(x)), raised at float8_training_tensor.py:373 __torch_dispatch__. Backward is retraced through the AutogradFunctionApply HOP via torch._functorch/autograd_function.py:874 → torch.fx.Interpreter(bwd).run(...) during AOTAutograd make_fx, so torch.abs dispatches onto Float8TrainingTensor, which has no aten.abs.default handler.

Both matrix legs fail identically (nightly cu126 and torch==2.13.0), with a ChildFailedError / exit code 1 from torchrun; preceding subtests pass ("✅ All Tests Passed ✅"). This is the same real regression from ca74881 still present on main — the triggering commit ee0d7ce is a ruff/lint-only change (fix invalid escape sequences, W605) that does not touch torchao/float8, so it did not introduce the failure; it just re-surfaces the unfixed root cause. No new information vs. the existing diagnosis.

Recurred on main at commit 3005bc1d2407646103ff759f38beb5c59a453728 ("[CI] Enable low_bit_optim CI on intel GPU").
Run: https://github.com/pytorch/ao/actions/runs/34564632115 (Run 4xH100 tests)

Same root cause and signature as the existing diagnosis — no new information:

Both matrix legs fail identically and deterministically: test (H100 2.13, torch==2.13.0 ...) and test (H100, --pre torch ... nightly/cu126).
test/float8/test_dtensor.py → _test_fp8_mlp_tensor_parallelism_compile fails with backend='inductor' raised: NotImplementedError: attempting to run aten.abs.default, this is not supported, via tensor_to_amax → amax = torch.max(torch.abs(x)) at torchao/float8/float8_utils.py:65, raised from Float8TrainingTensor.__torch_dispatch__. Both ranks fail; torchrun exits 1 with ChildFailedError. Preceding float8 subtests still print ✅ All Tests Passed ✅, so the failure remains isolated to the TP-compile dtensor test.

The triggering commit is a CI-config change and does not touch float8; this is the same regression introduced by ca74881 continuing to break main, not a new cause. Classification unchanged: real regression.

Recurs on main at commit 5a33bcf6efb8e78acb858d93eb7a9b7c57718693 — Run 4xH100 tests, run https://github.com/pytorch/ao/actions/runs/34637189681.

Same root cause and signature as this issue's existing diagnosis (no divergence):

Test: _test_fp8_mlp_tensor_parallelism_compile (test/float8/test_dtensor.py), via ./test/float8/test_everything_multi_gpu.sh. torchrun exits 1; preceding subtests (✅ All Tests Passed ✅) pass. Failure reproduces on the torch==2.13.0 matrix leg.
Error: backend='inductor' raised: NotImplementedError: attempting to run aten.abs.default, this is not supported, While executing %abs_1 : [num_users=1] = call_function[target=torch.abs](args = (%grad_output_reshaped,), kwargs = {}).
Dispatch chain unchanged: float8_linear.py:112 backward → hp_tensor_to_float8_dynamic (float8_scaling_utils.py:56) → tensor_to_scale (float8_utils.py:106) → tensor_to_amax (float8_utils.py:65, amax = torch.max(torch.abs(x))) → raised at float8_training_tensor.py:373 __torch_dispatch__. Backward runs under torch._functorch.autograd_function.backward → fx.Interpreter during AOTAutograd make_fx retrace.

Note: the triggering commit 5a33bcf ("Add code owners for low-bit CPU kernels", #4874) is a CODEOWNERS-only change and does not touch float8 code — this is the pre-existing regression (from ca74881, removing the allow-in-graph autograd wrappers) still unfixed on main, not a new cause. Classification remains: real regression.

Recurs on main at commit 9e5ea7fde7132df0c09b2acb12c2defad90aecb9 (run https://github.com/pytorch/ao/actions/runs/34637555688, "Run 4xH100 tests").

Same root cause and signature as this issue's diagnosis — no new information:

_test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py fails at inductor compile with NotImplementedError: attempting to run aten.abs.default, this is not supported.
Raised from torchao/float8/float8_training_tensor.py:373 __torch_dispatch__, reached via torch.fx.Interpreter(bwd).run(...) (torch/_functorch/autograd_function.py:874) while AOTAutograd make_fx retraces the Float8 autograd backward through the AutogradFunctionApply HOP — exactly the path changed by ca74881.
Both matrix legs fail identically and deterministically: test (H100 2.13, torch==2.13.0) and test (H100, --pre torch nightly cu126). Preceding float8 subtests print ✅ All Tests Passed ✅; only the compile TP dtensor test fails, then torchrun exits 1 (ChildFailedError → test/float8/test_dtensor.py FAILED).

The triggering commit 9e5ea7f ("Fix MoE FP8 Triton autotune key") does not touch float8/dtensor code, so this is a continued recurrence of the pre-existing regression on main, not a new cause. Still unresolved; classification remains real regression.

⚠️ Run Regression Tests on ROCm also failed on this commit (9e5ea7fde7132df0c09b2acb12c2defad90aecb9) — https://github.com/pytorch/ao/actions/runs/34637555679

Still failing on main at commit c9c0f312bb072df6d1285985db9bd6c963050a84 (run https://github.com/pytorch/ao/actions/runs/36166246156, "Run 4xH100 tests").

Matches the existing diagnosis exactly — no divergence:

test/float8/test_dtensor.py FAILED via torchrun ChildFailedError (exit code 1); preceding subtests pass, failure isolated to _test_fp8_mlp_tensor_parallelism_compile.
backend='inductor' raised: NotImplementedError: attempting to run aten.abs.default, this is not supported, while executing %abs_1 = torch.abs(grad_output_reshaped).
Same dispatch chain: float8_linear.py:112 backward → hp_tensor_to_float8_dynamic → tensor_to_scale (float8_utils.py:106) → tensor_to_amax (float8_utils.py:65, torch.max(torch.abs(x))) → raised at float8_training_tensor.py:373 __torch_dispatch__.
Reproduces identically on both matrix legs (torch==2.13.0 and --pre torch nightly), confirming it is not flaky/infra.

Note: the run also surfaces a red "Executing the custom container implementation failed. Please contact your self hosted runner administrator." annotation, but that is just the runner reporting the non-zero exit from the test script (RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1), not a separate infra fault.

⚠️ Run Regression Tests on ROCm also failed on this commit (c9c0f312bb072df6d1285985db9bd6c963050a84) — https://github.com/pytorch/ao/actions/runs/36166246501

Recurred on main at commit cf3c03f4259fa16a837ec581ad68b78b8026c114 — workflow "Run 4xH100 tests", run https://github.com/pytorch/ao/actions/runs/36753546753.

Same root cause as this issue's diagnosis (no change): _test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py fails at inductor compile with:

Test _test_fp8_mlp_tensor_parallelism_compile failed with error: backend='inductor' raised:
NotImplementedError: attempting to run aten.abs.default, this is not supported

Fails deterministically and identically on BOTH matrix legs (H100 torch-nightly cu126 and H100 2.13 torch==2.13.0 cu126), each ending in test/float8/test_dtensor.py FAILED → torchrun ChildFailedError (exit code 1). The preceding float8 subtests pass; the failure is isolated to the TP compile dtensor test.
The triggering commit cf3c03f ("Move the CUDA and CPU test jobs to OSDC runners") only relocates CI runner pools and does not touch float8 / autograd / the affected code path, confirming this is the still-unfixed regression from ca74881 rather than anything new. The regression persists on main.

Recurs on main at commit a701b6a6058720c21f95908b7ae4a24bf0cae1b6 ("Trigger the docs build from a ciflow label, not a pull request").
Run: https://github.com/pytorch/ao/actions/runs/36798019019 (Run 4xH100 tests)

This matches the existing diagnosis exactly — it is not a new failure:

Test: _test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py (./test/float8/test_everything_multi_gpu.sh), while the preceding float8 subtests pass (✅ All Tests Passed ✅).
Error: backend='inductor' raised: NotImplementedError: attempting to run aten.abs.default, this is not supported, raised at torchao/float8/float8_training_tensor.py:373 in __torch_dispatch__.
Same dispatch path: backward retraced via torch._functorch/autograd_function.py:874 → torch.fx.Interpreter(bwd).run(...) under AOTAutograd make_fx, executing %abs_1 ... call_function[target=torch.abs](args=(%grad_output_reshaped,)) against a Float8TrainingTensor.
Deterministic on both matrix legs (H100 torch-nightly cu126 and H100 2.13 torch==2.13.0), each ending in test/float8/test_dtensor.py FAILED → ChildFailedError (exit code 1).

The triggering commit a701b6a only changes the docs-build trigger and does not touch Float8 code; it is simply the latest main commit still carrying the unfixed regression from ca74881 ("Remove Float8 allow-in-graph custom autograd wrappers"). No change to the root-cause analysis or suggested fix in this issue.

Recurs on the "Run 4xH100 tests" workflow at commit 9d0c4946b11af5e484f3ac36d61ec970c40c68cf.
Run: https://github.com/pytorch/ao/actions/runs/38192339460

Same root cause as the existing diagnosis — no change in symptom:

Test: _test_fp8_mlp_tensor_parallelism_compile (test/float8/test_dtensor.py) via ./test/float8/test_everything_multi_gpu.sh.
Error: backend='inductor' raised: NotImplementedError: attempting to run aten.abs.default, this is not supported, raised from Float8TrainingTensor.__torch_dispatch__.
Identical dispatch chain: float8_linear.py:112 backward → hp_tensor_to_float8_dynamic → float8_scaling_utils.py:56 tensor_to_scale → float8_utils.py:106 → tensor_to_amax → float8_utils.py:65 amax = torch.max(torch.abs(x)), during HOP-based backward retrace under torch.compile + DTensor TP.
Fails deterministically on both matrix legs (torch==2.13.0 cu126 and torch-nightly cu126) on ranks 0 and 1.

The triggering commit 9d0c494 ("[XPU][CI] Reclaim workspace ownership before checkout on XPU runners") is unrelated to float8/autograd; the regression (introduced by ca74881) is simply still unfixed on main. No new information vs. the existing diagnosis — this is a continued occurrence, confirming it remains unresolved.

Recurred on main at commit f7c8353e8adeefc008a32169e6f92f8271d15e2d ("[moe_training] Fused dual-axis MXFP8 quantization", #4995).

Run: https://github.com/pytorch/ao/actions/runs/38720817531 (Run 4xH100 tests)

Same root cause as this issue's diagnosis — no deviation:

_test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py fails under torch.compile + DTensor TP with backend='inductor' raised: NotImplementedError: attempting to run aten.abs.default, this is not supported.
Raised at torchao/float8/float8_training_tensor.py:373 __torch_dispatch__, reached via the HOP backward retrace: torch/_functorch/autograd_function.py:874 → torch.fx.Interpreter(bwd).run(...) → float8_linear.py:112 backward → float8_scaling_utils.py:56 hp_tensor_to_float8_dynamic → float8_utils.py:106 tensor_to_scale → float8_utils.py:65 tensor_to_amax → amax = torch.max(torch.abs(x)).
Error node: %abs_1 ... call_function[target=torch.abs](args = (%grad_output_reshaped,)).

Still deterministic and reproduces on both matrix legs:

test (H100, ... --pre torch torchvision --index-url .../nightly/cu126)
test (H100 2.13, ... torch==2.13.0 torchvision==0.28.0 --index-url .../cu126)

The triggering commit f7c8353 only touches moe_training MXFP8 quantization and does not touch the float8 autograd/amax path, so it is not the cause — this is the pre-existing regression from ca74881 continuing to break main. Classification unchanged: real regression.

⚠️ Build Linux Wheels (x86) also failed on this commit (f7c8353e8adeefc008a32169e6f92f8271d15e2d) — https://github.com/pytorch/ao/actions/runs/38720818251

Recurs on main at commit 6697dca2b50b9c1e77c761fe944bdd573441e08c (triggering commit "Fix incorrect merge base for #4982", unrelated to float8).

Run: https://github.com/pytorch/ao/actions/runs/38816940474 (Run 4xH100 tests)
Failing test: _test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py (via ./test/float8/test_everything_multi_gpu.sh), both matrix legs failing deterministically:
- test (H100, ..., --index-url .../nightly/cu126)
- test (H100 2.13, ..., torch==2.13.0 torchvision==0.28.0)
Error (unchanged): NotImplementedError: attempting to run aten.abs.default, this is not supported, raised at torchao/float8/float8_training_tensor.py:373 __torch_dispatch__, reached via the HOP backward retrace (torch._functorch/autograd_function.py:874 → torch.fx.Interpreter(bwd).run) during inductor/AOTAutograd make_fx tracing. Failing node: %abs_1 : call_function[target=torch.abs](args=(%grad_output_reshaped,)).

This matches the existing diagnosis exactly (amax-path torch.abs dispatching onto Float8TrainingTensor under the AutogradFunctionApply HOP). The preceding float8 subtests pass (✅ All Tests Passed ✅); only the TP-compile dtensor test fails. No new behavior vs. the recorded root cause — still an open regression carried on main.

⚠️ Build Linux Wheels (x86) also failed on this commit (6697dca2b50b9c1e77c761fe944bdd573441e08c) — https://github.com/pytorch/ao/actions/runs/38816941401

Recurs on main at commit 8ccef7c3e0ef232763c1d8d98c84808e7043037d — Run 4xH100 tests: https://github.com/pytorch/ao/actions/runs/39802464340

Matches the existing diagnosis exactly. ./test/float8/test_everything_multi_gpu.sh → test/float8/test_dtensor.py fails in _test_fp8_mlp_tensor_parallelism_compile with:

Test _test_fp8_mlp_tensor_parallelism_compile failed with error: backend='inductor' raised:
NotImplementedError: attempting to run aten.abs.default, this is not supported

Same dispatch chain as diagnosed here: float8_linear.py:112 backward → hp_tensor_to_float8_dynamic → float8_scaling_utils.py:56 tensor_to_scale → float8_utils.py:106 → float8_utils.py:65 tensor_to_amax (amax = torch.max(torch.abs(x))) → NotImplementedError from Float8TrainingTensor.__torch_dispatch__. Deterministic on both matrix legs (--pre nightly cu126 and torch==2.13.0 cu126), torchrun ChildFailedError on rank 0 and rank 1.

The triggering commit 8ccef7c is a README-only change and does not touch any float8/autograd code, so this is the previously-identified regression (from ca74881) still unfixed on main, not a new cause. No change to the diagnosis.

⚠️ Build Linux Wheels (x86) also failed on this commit (8ccef7c3e0ef232763c1d8d98c84808e7043037d) — https://github.com/pytorch/ao/actions/runs/39802465380

Still failing on main at commit 6d99941b3a433e27ad086f402cb4e8d2963ee52c (workflow: Run 4xH100 tests).
Run: https://github.com/pytorch/ao/actions/runs/39840767778

This matches the existing diagnosis exactly:

Test: _test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py.
Error: NotImplementedError: attempting to run aten.abs.default, this is not supported, raised at torchao/float8/float8_training_tensor.py:373 in __torch_dispatch__.
Same dispatch path: backward retraced through the AutogradFunctionApply HOP (torch/_functorch/autograd_function.py → torch.fx.Interpreter(bwd).run(...)) during AOTAutograd make_fx tracing, where torch.abs(grad_output_reshaped) dispatches onto a Float8TrainingTensor lacking an aten.abs.default handler. Reported node: %abs_1 : call_function[target=torch.abs].
Deterministic across both matrix legs: test (H100, ... nightly/cu126) and test (H100 2.13, torch==2.13.0 ... cu126) both fail identically with ChildFailedError / exit code 1.

The triggering commit 6d99941 ("Delete unused GPTQ prototype") does not touch float8 autograd/dispatch code, consistent with this being the pre-existing unfixed regression from ca74881 rather than a new cause. No new information versus the existing root-cause analysis.

⚠️ Build Linux Wheels (x86) also failed on this commit (6d99941b3a433e27ad086f402cb4e8d2963ee52c) — https://github.com/pytorch/ao/actions/runs/39840768156

Recurs on commit 9e9dca96d42cf05985aeaf0afe5546d9d002b337 ("[parq] Improve channel_bucketize efficiency, fix quantize_ collision", #5020).

Run: https://github.com/pytorch/ao/actions/runs/39854952791 (Run 4xH100 tests)

Matches the existing diagnosis exactly — no new information:

Failing test: _test_fp8_mlp_tensor_parallelism_compile in test/float8/test_dtensor.py, via ./test/float8/test_everything_multi_gpu.sh (torchrun ChildFailedError, exit code 1). The preceding float8 subtests still print ✅ All Tests Passed ✅; the failure is isolated to the compile TP dtensor test.
Error: backend='inductor' raised: NotImplementedError: attempting to run aten.abs.default, this is not supported, raised at torchao/float8/float8_training_tensor.py:373 __torch_dispatch__ while the backward is retraced through the AutogradFunctionApply HOP (torch/_functorch/autograd_function.py:874 → torch.fx.Interpreter(bwd).run(...)), executing %abs_1 = torch.abs(%grad_output_reshaped) from the amax path.
Reproduces deterministically on BOTH matrix legs — test (H100, ... nightly/cu126) and test (H100 2.13, torch==2.13.0 ... cu126).

The triggering commit 9e9dca9 is a PARQ quantization change and does not touch float8, so it is not the cause; this is the same regression introduced by ca74881 (removal of the Float8 allow-in-graph autograd wrappers) that this issue already tracks. No change to the root-cause analysis or suggested fix.

⚠️ Build Linux Wheels (x86) also failed on this commit (9e9dca96d42cf05985aeaf0afe5546d9d002b337) — https://github.com/pytorch/ao/actions/runs/39854953193

Navigation Menu

Repository navigation

test_dtensor.py (float8 TP compile): aten.abs.default NotImplemented on Float8TrainingTensor after removing allow-in-graph autograd wrappers #4864

Summary

Failure details

Root cause

Classification

Suggested fix

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

test_dtensor.py (float8 TP compile): aten.abs.default NotImplemented on Float8TrainingTensor after removing allow-in-graph autograd wrappers #4864

Description

Summary

Failure details

Root cause

Classification

Suggested fix

Activity

github-actions commented on Sep 2, 2026

github-actions commented on Sep 4, 2026

github-actions commented on Sep 8, 2026

github-actions commented on Sep 8, 2026

github-actions commented on Sep 8, 2026

github-actions commented on Sep 9, 2026

github-actions commented on Sep 9, 2026

github-actions commented on Sep 9, 2026

github-actions commented on Sep 9, 2026

github-actions commented on Sep 9, 2026

github-actions commented on Sep 11, 2026

github-actions commented on Sep 11, 2026

github-actions commented on Sep 11, 2026

github-actions commented on Sep 11, 2026

15 remaining items

github-actions commented on Sep 25, 2026

github-actions commented on Sep 25, 2026

github-actions commented on Sep 30, 2026

github-actions commented on Oct 1, 2026

github-actions commented on Oct 12, 2026

github-actions commented on Oct 15, 2026

github-actions commented on Oct 15, 2026

github-actions commented on Oct 15, 2026

github-actions commented on Oct 15, 2026

github-actions commented on Oct 22, 2026

github-actions commented on Oct 22, 2026

github-actions commented on Oct 22, 2026

github-actions commented on Oct 22, 2026

github-actions commented on Oct 22, 2026

github-actions commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions