diff --git a/CLAUDE.md b/CLAUDE.md index b2932a6ea..b040f0a5e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,6 +13,7 @@ - **Python import flow**: framework selection happens in `transformer_engine/__init__.py` (driven by `NVTE_FRAMEWORK`); native `.so` resolution happens in `transformer_engine/common/__init__.py`. Start tracing from these two files for any import or framework-selection issue — function names inside may change, but the entry points are stable. - **Build orchestration**: `setup.py` + helpers in `build_tools/` + CMake. ROCm vs CUDA backend detection lives in `build_tools/utils.py` — grep there (and for `NVTE_USE_ROCM`) for current behavior. - **3rdparty submodules**: read `.gitmodules` for the current set and commit pins to determine current submodules. +- **AITER source**: the tree QoLA/TE actually builds is cloned on-demand to `build/cmake/ck_fused_attn/qola/third_party/aiter/` at the commit pinned in `transformer_engine/common/ck_fused_attn/qola_manifest.toml`. The in-tree `3rdparty/*/aiter` checkouts are stale — read the built tree (and check its HEAD) when reasoning about the linked ABI. ## Hipify convention The build auto-generates HIP files from CUDA sources via `hipify_torch`. Generated files are marked with `// !!! This is a file automatically generated by hipify!!!` at line 1. **Never edit generated files directly** — edit the CUDA source instead. @@ -108,3 +109,5 @@ When writing or updating memories in the project memory directory, follow these - **Framework extension won't build on ROCm**: check `build_tools/utils.py::get_frameworks()`. - **Fused-attn regression**: reproduce under multiple backend configs (`auto`, `ck`, `aotriton`, `unfused`). - **CK/AITER kernel failures**: use the `ck-debugging` skill for structured triage and isolation. +- **GPU hang or segfault in fused attention**: for hangs use `py-spy dump --pid N --native` (Python-only frames mislead — a stuck `.item()` is usually a HIP sync wait); for segfaults `rocgdb --batch -ex run --args python ...` then grep for the fault frame. Dump kernel args with `CK_FUSED_ATTN_LOG_CONFIG=1`. +- **Stale CK code after a QoLA patch that changed CK headers/codegen** (runtime `symbol not found`, blob `no template named ...`, or nullptr-call segfault): the blob cache is keyed by CK commit and the installed header tree is no-clobber, so a rebuild reuses old code. Clear `$HOME/.cache/te_ck_jit` and `transformer_engine/lib/ck_jit/3rdparty`, then rebuild.