From ada1f7e7c8b50952cd738001df26dc308df1302e Mon Sep 17 00:00:00 2001 From: Meekail Zain Date: Mon, 20 Jul 2026 16:11:50 +0000 Subject: [PATCH 1/4] Update CLAUDE.md --- CLAUDE.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index b2932a6ea..e1d7ae544 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,10 +13,13 @@ - **Python import flow**: framework selection happens in `transformer_engine/__init__.py` (driven by `NVTE_FRAMEWORK`); native `.so` resolution happens in `transformer_engine/common/__init__.py`. Start tracing from these two files for any import or framework-selection issue — function names inside may change, but the entry points are stable. - **Build orchestration**: `setup.py` + helpers in `build_tools/` + CMake. ROCm vs CUDA backend detection lives in `build_tools/utils.py` — grep there (and for `NVTE_USE_ROCM`) for current behavior. - **3rdparty submodules**: read `.gitmodules` for the current set and commit pins to determine current submodules. +- **AITER source**: the tree QoLA/TE actually builds is cloned on-demand to `3rdparty/QoLA/build/third_party/aiter` at the commit pinned in `transformer_engine/common/ck_fused_attn/qola_manifest.toml`. The in-tree `3rdparty/*/aiter` checkouts are stale — read the built tree (and check its HEAD) when reasoning about the linked ABI. ## Hipify convention The build auto-generates HIP files from CUDA sources via `hipify_torch`. Generated files are marked with `// !!! This is a file automatically generated by hipify!!!` at line 1. **Never edit generated files directly** — edit the CUDA source instead. +Hipify runs at **CMake configure time**, not as a ninja build step. After editing a hipified source (a `*.cpp` whose generated twin is `*_hip.cpp`), you must re-run `cmake .` to re-hipify before `ninja` — otherwise ninja compiles the **stale** generated mirror and your change silently has no effect (build succeeds, wrong code runs). Verify by checking the `*_hip.cpp` mtime/contents after building. The editable install loads the top-level `libtransformer_engine.so`, so `cp build/cmake/libtransformer_engine.so` over the top-level copy before testing. + File extension mapping: | CUDA source | Generated HIP file | @@ -108,3 +111,5 @@ When writing or updating memories in the project memory directory, follow these - **Framework extension won't build on ROCm**: check `build_tools/utils.py::get_frameworks()`. - **Fused-attn regression**: reproduce under multiple backend configs (`auto`, `ck`, `aotriton`, `unfused`). - **CK/AITER kernel failures**: use the `ck-debugging` skill for structured triage and isolation. +- **GPU hang or segfault in fused attention**: for hangs use `py-spy dump --pid N --native` (Python-only frames mislead — a stuck `.item()` is usually a HIP sync wait); for segfaults `rocgdb --batch -ex run --args python ...` then grep for the fault frame. Dump kernel args with `CK_FUSED_ATTN_LOG_CONFIG=1`, and bisect bwd paths with `NVTE_CK_USES_BWD_V3=0`. +- **Stale CK code after a QoLA patch that changed CK headers/codegen** (runtime `symbol not found`, blob `no template named ...`, or nullptr-call segfault): the blob cache is keyed by CK commit and the installed header tree is no-clobber, so a rebuild reuses old code. Clear `$HOME/.cache/te_ck_jit` and `transformer_engine/lib/ck_jit/3rdparty`, then rebuild. From 8e52e654e9e401e3a7fe1641110233a2ef8e7c62 Mon Sep 17 00:00:00 2001 From: Meekail Zain Date: Mon, 20 Jul 2026 16:33:46 +0000 Subject: [PATCH 2/4] Remove soon-to-be unnecessary hipify section --- CLAUDE.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e1d7ae544..747198d13 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,8 +18,6 @@ ## Hipify convention The build auto-generates HIP files from CUDA sources via `hipify_torch`. Generated files are marked with `// !!! This is a file automatically generated by hipify!!!` at line 1. **Never edit generated files directly** — edit the CUDA source instead. -Hipify runs at **CMake configure time**, not as a ninja build step. After editing a hipified source (a `*.cpp` whose generated twin is `*_hip.cpp`), you must re-run `cmake .` to re-hipify before `ninja` — otherwise ninja compiles the **stale** generated mirror and your change silently has no effect (build succeeds, wrong code runs). Verify by checking the `*_hip.cpp` mtime/contents after building. The editable install loads the top-level `libtransformer_engine.so`, so `cp build/cmake/libtransformer_engine.so` over the top-level copy before testing. - File extension mapping: | CUDA source | Generated HIP file | From 0137d72f2b5e605cf6f54ed3132be70598436848 Mon Sep 17 00:00:00 2001 From: Meekail Zain Date: Mon, 20 Jul 2026 16:46:11 +0000 Subject: [PATCH 3/4] Trim unnecessary specificity --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 747198d13..a22da37d6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -109,5 +109,5 @@ When writing or updating memories in the project memory directory, follow these - **Framework extension won't build on ROCm**: check `build_tools/utils.py::get_frameworks()`. - **Fused-attn regression**: reproduce under multiple backend configs (`auto`, `ck`, `aotriton`, `unfused`). - **CK/AITER kernel failures**: use the `ck-debugging` skill for structured triage and isolation. -- **GPU hang or segfault in fused attention**: for hangs use `py-spy dump --pid N --native` (Python-only frames mislead — a stuck `.item()` is usually a HIP sync wait); for segfaults `rocgdb --batch -ex run --args python ...` then grep for the fault frame. Dump kernel args with `CK_FUSED_ATTN_LOG_CONFIG=1`, and bisect bwd paths with `NVTE_CK_USES_BWD_V3=0`. +- **GPU hang or segfault in fused attention**: for hangs use `py-spy dump --pid N --native` (Python-only frames mislead — a stuck `.item()` is usually a HIP sync wait); for segfaults `rocgdb --batch -ex run --args python ...` then grep for the fault frame. Dump kernel args with `CK_FUSED_ATTN_LOG_CONFIG=1`. - **Stale CK code after a QoLA patch that changed CK headers/codegen** (runtime `symbol not found`, blob `no template named ...`, or nullptr-call segfault): the blob cache is keyed by CK commit and the installed header tree is no-clobber, so a rebuild reuses old code. Clear `$HOME/.cache/te_ck_jit` and `transformer_engine/lib/ck_jit/3rdparty`, then rebuild. From a89c724117654063fe2ea31035e6dcdf914b3402 Mon Sep 17 00:00:00 2001 From: Meekail Zain <34613774+Micky774@users.noreply.github.com> Date: Wed, 22 Jul 2026 12:55:44 -0400 Subject: [PATCH 4/4] Update CLAUDE.md --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index a22da37d6..b040f0a5e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,7 +13,7 @@ - **Python import flow**: framework selection happens in `transformer_engine/__init__.py` (driven by `NVTE_FRAMEWORK`); native `.so` resolution happens in `transformer_engine/common/__init__.py`. Start tracing from these two files for any import or framework-selection issue — function names inside may change, but the entry points are stable. - **Build orchestration**: `setup.py` + helpers in `build_tools/` + CMake. ROCm vs CUDA backend detection lives in `build_tools/utils.py` — grep there (and for `NVTE_USE_ROCM`) for current behavior. - **3rdparty submodules**: read `.gitmodules` for the current set and commit pins to determine current submodules. -- **AITER source**: the tree QoLA/TE actually builds is cloned on-demand to `3rdparty/QoLA/build/third_party/aiter` at the commit pinned in `transformer_engine/common/ck_fused_attn/qola_manifest.toml`. The in-tree `3rdparty/*/aiter` checkouts are stale — read the built tree (and check its HEAD) when reasoning about the linked ABI. +- **AITER source**: the tree QoLA/TE actually builds is cloned on-demand to `build/cmake/ck_fused_attn/qola/third_party/aiter/` at the commit pinned in `transformer_engine/common/ck_fused_attn/qola_manifest.toml`. The in-tree `3rdparty/*/aiter` checkouts are stale — read the built tree (and check its HEAD) when reasoning about the linked ABI. ## Hipify convention The build auto-generates HIP files from CUDA sources via `hipify_torch`. Generated files are marked with `// !!! This is a file automatically generated by hipify!!!` at line 1. **Never edit generated files directly** — edit the CUDA source instead.