From ac512ba34dd6ec50e5b41db72e2f246a4140d18d Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 12 Mar 2026 19:02:41 +0000 Subject: [PATCH 1/2] Initial plan From 10703bc20daec5b36b24a3511a221bdc0df72776 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 12 Mar 2026 19:35:26 +0000 Subject: [PATCH 2/2] Add README and IntelliKit profiling results for Example 7 (GEMM all-scatter) Co-authored-by: JoseSantosAMD <87447437+JoseSantosAMD@users.noreply.github.com> --- .agents/skills/accordo/SKILL.md | 68 ++++++++++++++++++ .agents/skills/linex/SKILL.md | 98 ++++++++++++++++++++++++++ .agents/skills/metrix/SKILL.md | 76 ++++++++++++++++++++ .agents/skills/nexus/SKILL.md | 74 +++++++++++++++++++ .github/agents/skills/accordo/SKILL.md | 68 ++++++++++++++++++ .github/agents/skills/linex/SKILL.md | 98 ++++++++++++++++++++++++++ .github/agents/skills/metrix/SKILL.md | 76 ++++++++++++++++++++ .github/agents/skills/nexus/SKILL.md | 74 +++++++++++++++++++ .gitignore | 3 +- .intellikit | 1 + examples/07_gemm_all_scatter/README.md | 80 +++++++++++++++++++++ 11 files changed, 715 insertions(+), 1 deletion(-) create mode 100644 .agents/skills/accordo/SKILL.md create mode 100644 .agents/skills/linex/SKILL.md create mode 100644 .agents/skills/metrix/SKILL.md create mode 100644 .agents/skills/nexus/SKILL.md create mode 100644 .github/agents/skills/accordo/SKILL.md create mode 100644 .github/agents/skills/linex/SKILL.md create mode 100644 .github/agents/skills/metrix/SKILL.md create mode 100644 .github/agents/skills/nexus/SKILL.md create mode 160000 .intellikit create mode 100644 examples/07_gemm_all_scatter/README.md diff --git a/.agents/skills/accordo/SKILL.md b/.agents/skills/accordo/SKILL.md new file mode 100644 index 000000000..808c73ef9 --- /dev/null +++ b/.agents/skills/accordo/SKILL.md @@ -0,0 +1,68 @@ +--- +name: accordo-validation +description: Validate GPU kernel correctness by comparing reference and optimized outputs. Use when verifying that an optimized or modified kernel matches a reference implementation. +--- + +# Accordo: GPU Kernel Validation + +Capture and compare kernel outputs from reference and optimized binaries to validate correctness. Uses kernelDB for automatic kernel extraction; supports configurable tolerance and execution-time comparison. + +## When to Use + +- User has a reference and an optimized (or modified) GPU kernel and wants to check they produce the same results +- Regression testing after kernel or build changes +- Validating multiple optimization variants against one baseline + +## Instructions + +1. **Require two or more binaries:** one reference (e.g. `./app_ref`) and one or more to validate (e.g. `./app_opt`). All must expose the same kernel by name. +2. **Ensure binaries are built with debug symbols** (`-g`) so kernel arguments can be extracted. +3. **Choose execution path:** + - If an Accordo MCP server is available, call its `validate_kernel_correctness` tool, which performs capture-and-compare with the same semantics described below. + - Otherwise use the Python API from the environment where Accordo is installed. + +### Python API + +```python +from accordo import Accordo + +# Validator for the kernel to validate (binary used to extract signature) +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum") + +# Optional: set working directory if binaries expect it +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum", working_directory="./run") + +# Capture snapshots +ref = validator.capture_snapshot(binary="./app_ref") +opt = validator.capture_snapshot(binary="./app_opt") + +# Compare with tolerance (default 1e-6) +result = validator.compare_snapshots(ref, opt, tolerance=1e-6) + +if result.is_valid: + print("PASS:", result.num_arrays_validated, "arrays matched") +else: + print(result.summary()) +``` + +For multiple optimizations, capture the reference once and compare each optimized snapshot against it. + +### Snapshot and result attributes + +- **Snapshot:** `arrays`, `execution_time_ms`, `grid_size`, `block_size` +- **ValidationResult:** `is_valid`, `num_arrays_validated`, `num_mismatches`, `mismatches`, `success_rate`; use `summary()` for a human-readable report. + +## Workflow + +1. Build reference and optimized binaries with the same kernel name and `-g`. +2. Create an `Accordo(binary=ref_binary, kernel_name="...")` validator; set `working_directory` if needed. +3. Capture reference snapshot with `capture_snapshot(binary=ref_binary)`. +4. For each variant, capture with `capture_snapshot(binary=opt_binary)` and compare with `compare_snapshots(ref, opt, tolerance=...)`. +5. If `result.is_valid` is false, use `result.summary()` and `result.mismatches` to diagnose. +6. Use relative paths for binaries and working directory so the skill is portable. + +## Notes + +- kernelDB is used automatically; no separate kernelDB setup is required when using the Python API. +- Increase `tolerance` for floating-point comparisons when appropriate (e.g. 1e-4 or 1e-5 for single precision). +- Use `timeout_seconds` in `capture_snapshot` if the run may hang. diff --git a/.agents/skills/linex/SKILL.md b/.agents/skills/linex/SKILL.md new file mode 100644 index 000000000..dca5b7d6d --- /dev/null +++ b/.agents/skills/linex/SKILL.md @@ -0,0 +1,98 @@ +--- +name: linex-profiling +description: Profile GPU kernels at source-line granularity with cycle-level timing and stall analysis. Use when identifying performance hotspots at the source code level or analyzing instruction-level metrics mapped to source lines. +--- + +# Linex: Source-Level GPU Performance Profiling + +Map GPU performance metrics to your source code lines. Get cycle-level timing, stall analysis, and instruction-level metrics for each line of source code. + +## When to Use + +- User asks to profile a GPU application at source-line granularity +- Need to identify which specific lines of code are performance bottlenecks +- Analyzing stall patterns and execution bottlenecks at the source level +- Understanding cycle-level timing for each line of code +- Instruction-level analysis mapped to source lines + +## Instructions + +1. **Ensure the target runs on AMD ROCm 7.0+** with `rocprofv3` available. +2. **Kernels must be compiled with `-g`** (debug symbols) for source mapping. +3. **Choose execution path:** + - If a Linex MCP server is available, use its MCP tools: + - `profile_application` to run and profile a target application with the options below. + - `analyze_instruction_hotspots` to perform instruction-level hotspot analysis on collected profiles. + - Otherwise use the Python API from the environment where Linex is installed. + +### Python API + +```python +from linex import Linex + +profiler = Linex( + target_cu=0, # Target compute unit + shader_engine_mask="0xFFFFFFFF", # All shader engines + activity=10, # Activity counter polling +) + +profiler.profile("./my_app", kernel_filter="my_kernel") + +# Show hotspots (sorted by total_cycles) +for line in profiler.source_lines[:5]: + print(f"{line.file}:{line.line_number}") + print(f" {line.total_cycles:,} cycles ({line.stall_percent:.1f}% stalled)") + print(f" Executed {line.execution_count} times") + +# Find memory-bound lines +memory_bound = [ + l for l in profiler.source_lines + if l.stall_percent > 50 +] + +# Instruction-level analysis +for line in profiler.source_lines[:1]: + for inst in line.instructions: + print(f"{inst.isa}: {inst.latency_cycles} cycles") +``` + +### SourceLine Properties + +- `file` - Source file path +- `line_number` - Line number +- `total_cycles` - Sum of all instruction cycles +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - Total executions +- `instructions` - List of ISA instructions +- `stall_percent` - Convenience: stall_cycles / total_cycles * 100 + +### InstructionData Properties + +- `isa` - ISA instruction text +- `latency_cycles` - Total cycles for this instruction +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - How many times it ran +- `instruction_address` - Virtual address in GPU memory +- `file` - Parsed from source_location +- `line` - Parsed from source_location +- `stall_percent` - Convenience: stall_cycles / latency_cycles * 100 + +## Workflow + +1. Ensure the target binary is built with `-g` (debug symbols) for source mapping. +2. Create a `Linex()` profiler; optionally set `target_cu`, `shader_engine_mask`, or `activity`. +3. Call `profiler.profile(command, kernel_filter=...)` to run profiling. +4. Access `profiler.source_lines` (sorted by total_cycles) to find hotspots. +5. Use `line.stall_percent` to identify memory-bound or dependency-bound lines. +6. Drill down into `line.instructions` for instruction-level analysis. +7. Use relative paths for the target binary so the skill is portable. + +## Notes + +- Requires ROCm 7.0+ with `rocprofv3` support. +- Source mapping requires kernels compiled with `-g` (debug symbols). +- `source_lines` are automatically sorted by `total_cycles` (descending). +- Use `kernel_filter` to profile specific kernels by name (regex pattern). +- For Triton or other frameworks, ensure debug symbols are available in the compiled output. diff --git a/.agents/skills/metrix/SKILL.md b/.agents/skills/metrix/SKILL.md new file mode 100644 index 000000000..9a5564ece --- /dev/null +++ b/.agents/skills/metrix/SKILL.md @@ -0,0 +1,76 @@ +--- +name: metrix-profiling +description: Profile GPU kernels when performance analysis or optimization is required. Use for AMD ROCm GPU metrics, bandwidth, cache hit rates, coalescing, or kernel timing. +--- + +# Metrix: GPU Profiling + +Profile AMD GPU kernels and get human-readable metrics (bandwidth, cache, coalescing, FLOPS). Architecture is auto-detected. + +## When to Use + +- User asks to profile a GPU application or kernel +- Performance analysis, optimization, or bottleneck investigation +- Need HBM/L2/L1 bandwidth, hit rates, or compute metrics +- Need timing-only runs (fast, no hardware counters) + +## Instructions + +1. **Ensure the target runs on AMD ROCm** (e.g. `hipcc`-built binary or Python script that launches HIP/ROCm kernels). +2. **Choose execution path:** + - If a Metrix MCP server is available, use its profile tool with the same options below. + - Otherwise run the CLI or Python API from the environment where Metrix is installed. + +### CLI + +From the project or install prefix: + +```bash +# Profile with all metrics (auto-detected arch) +metrix ./my_app + +# Time only (fast, no counters) +metrix --time-only -n 10 ./my_app + +# Filter kernels by name +metrix --kernel matmul ./my_app + +# Specific metrics +metrix --metrics memory.l2_hit_rate,memory.coalescing_efficiency,compute.total_flops ./my_app + +# Save to JSON/CSV +metrix -o results.json ./my_app +``` + +Options: `--profile` (quick|memory|compute), `--metrics` (comma-separated), `--time-only`, `--kernel` (substring), `--num-replays`/`-n`, `--output`/`-o`, `--top K`, `--aggregate`, `--log`/`-l` (debug|info|warning|error), `--quiet`. + +### Python API + +```python +from metrix import Metrix + +profiler = Metrix() +results = profiler.profile("./my_app", num_replays=5) + +for kernel in results.kernels: + print(kernel.name, kernel.duration_us.avg) + for metric, stats in kernel.metrics.items(): + print(f" {metric}: {stats.avg}") +``` + +Use `metrics=[...]` for a subset; omit for all metrics. Use `cwd` when the binary expects a specific working directory. + +## Workflow + +1. Identify the executable or script to profile (e.g. `./app` or `python run_kernels.py`). +2. If only timing is needed, use `--time-only` for speed. +3. If full metrics are needed, run `metrix ./app` (or MCP equivalent); optionally restrict with `--kernel` or `--metrics`. +4. Interpret results: low L2 hit rate, low coalescing, or low HBM utilization suggest optimization targets. +5. For automation or tooling, use `-o results.json` and parse the JSON output. + +## Key Metrics (reference) + +- **Memory:** `memory.hbm_bandwidth_utilization`, `memory.l2_hit_rate`, `memory.l1_hit_rate`, `memory.coalescing_efficiency`, `memory.global_load_efficiency`, `memory.lds_bank_conflicts`, `memory.atomic_latency` +- **Compute:** `compute.total_flops`, `compute.hbm_gflops`, `compute.hbm_arithmetic_intensity`, `compute.l2_arithmetic_intensity`, `compute.l1_arithmetic_intensity` + +Use relative paths for the target binary and output files so the skill is portable across environments. diff --git a/.agents/skills/nexus/SKILL.md b/.agents/skills/nexus/SKILL.md new file mode 100644 index 000000000..ad714bc4d --- /dev/null +++ b/.agents/skills/nexus/SKILL.md @@ -0,0 +1,74 @@ +--- +name: nexus-trace +description: Extract GPU kernel assembly and HIP source from HSA packet traces. Use when analyzing what code ran on the GPU, debugging kernel dispatch, or inspecting assembly and source mapping. +--- + +# Nexus: HSA Packet Source Code Extractor + +Intercepts HSA packets from a running process and extracts, per kernel, assembly and HIP source into a structured trace (e.g. JSON). Use for kernel-level inspection and assembly/source correlation. + +## When to Use + +- User needs to see which kernels ran and their assembly or HIP source +- Debugging or analyzing GPU dispatch and code generation +- Inspecting assembly-to-source mapping for a HIP (or ROCm) application + +## Instructions + +1. **Ensure the target runs on AMD ROCm** and uses HSA (e.g. HIP application or ROCm runtime). +2. **Choose execution path:** + - If a Nexus MCP server is available, use its tools: `list_kernels` to enumerate kernels in a trace, and `extract_kernel_code` to get assembly and HIP/source mapping (signature, files, lines). See `nexus/nexus/mcp/server.py` for tool parameters and schemas. + - Otherwise use the Python API from the environment where Nexus is installed. + +### Python API (recommended when no MCP) + +```python +from nexus import Nexus + +nexus = Nexus(log_level=1) +trace = nexus.run(["python", "my_gpu_script.py"]) + +# Or run a binary: +# trace = nexus.run(["./my_hip_app"]) + +for kernel in trace: + print(kernel.name, len(kernel.assembly), "instructions") + for i, asm_line in enumerate(kernel.assembly, 1): + print(f" {i}. {asm_line}") + for line_no, hip_line in zip(kernel.lines or range(1, len(kernel.hip)+1), kernel.hip): + print(f" {line_no}: {hip_line}") + +# Access by kernel name +k = trace["vector_add(float const*, float const*, float*, int)"] +print(k.assembly, k.hip, k.signature, k.files, k.lines) + +# Save/load trace +trace.save("trace.json") +loaded = Nexus.load("trace.json") +``` + +Set `log_level` (0–4) to control verbosity. Use relative paths for the run command and output file so the skill is portable. + +### Environment-based usage (no Python API) + +When the process cannot be launched via `nexus.run()`: + +1. Set `HSA_TOOLS_LIB` to the Nexus shared library path (e.g. `build/lib/libnexus.so` or the installed path). +2. Set `NEXUS_OUTPUT_FILE` to the output JSON path. +3. Set `NEXUS_LOG_LEVEL` (0–4) if needed. +4. Run the application as usual; it will be traced and the output file will contain the kernel data. + +Optional: `NEXUS_EXTRA_SEARCH_PREFIX` (colon-separated) for HIP source search; `TRITON_DISABLE_LINE_INFO=0` for Triton kernel line info. + +## Workflow + +1. Identify the command that runs the GPU workload (e.g. `python script.py` or `./app`). +2. If using the Python API: create `Nexus(log_level=...)`, call `nexus.run([...])`, then iterate `trace` and optionally `trace.save(...)`. +3. If using the env method: set `HSA_TOOLS_LIB` and `NEXUS_OUTPUT_FILE`, then run the app; open the JSON and parse the `kernels` structure. +4. Use kernel `signature`, `assembly`, `hip`, `files`, and `lines` to analyze what ran and map assembly back to source. +5. Use relative paths for commands and output files. + +## Notes + +- Nexus is intended for research/analysis; ensure the target environment has the Nexus library and compatible ROCm/HSA stack. +- For Triton kernels, enable line info via `TRITON_DISABLE_LINE_INFO=0` when using the Python API. diff --git a/.github/agents/skills/accordo/SKILL.md b/.github/agents/skills/accordo/SKILL.md new file mode 100644 index 000000000..808c73ef9 --- /dev/null +++ b/.github/agents/skills/accordo/SKILL.md @@ -0,0 +1,68 @@ +--- +name: accordo-validation +description: Validate GPU kernel correctness by comparing reference and optimized outputs. Use when verifying that an optimized or modified kernel matches a reference implementation. +--- + +# Accordo: GPU Kernel Validation + +Capture and compare kernel outputs from reference and optimized binaries to validate correctness. Uses kernelDB for automatic kernel extraction; supports configurable tolerance and execution-time comparison. + +## When to Use + +- User has a reference and an optimized (or modified) GPU kernel and wants to check they produce the same results +- Regression testing after kernel or build changes +- Validating multiple optimization variants against one baseline + +## Instructions + +1. **Require two or more binaries:** one reference (e.g. `./app_ref`) and one or more to validate (e.g. `./app_opt`). All must expose the same kernel by name. +2. **Ensure binaries are built with debug symbols** (`-g`) so kernel arguments can be extracted. +3. **Choose execution path:** + - If an Accordo MCP server is available, call its `validate_kernel_correctness` tool, which performs capture-and-compare with the same semantics described below. + - Otherwise use the Python API from the environment where Accordo is installed. + +### Python API + +```python +from accordo import Accordo + +# Validator for the kernel to validate (binary used to extract signature) +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum") + +# Optional: set working directory if binaries expect it +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum", working_directory="./run") + +# Capture snapshots +ref = validator.capture_snapshot(binary="./app_ref") +opt = validator.capture_snapshot(binary="./app_opt") + +# Compare with tolerance (default 1e-6) +result = validator.compare_snapshots(ref, opt, tolerance=1e-6) + +if result.is_valid: + print("PASS:", result.num_arrays_validated, "arrays matched") +else: + print(result.summary()) +``` + +For multiple optimizations, capture the reference once and compare each optimized snapshot against it. + +### Snapshot and result attributes + +- **Snapshot:** `arrays`, `execution_time_ms`, `grid_size`, `block_size` +- **ValidationResult:** `is_valid`, `num_arrays_validated`, `num_mismatches`, `mismatches`, `success_rate`; use `summary()` for a human-readable report. + +## Workflow + +1. Build reference and optimized binaries with the same kernel name and `-g`. +2. Create an `Accordo(binary=ref_binary, kernel_name="...")` validator; set `working_directory` if needed. +3. Capture reference snapshot with `capture_snapshot(binary=ref_binary)`. +4. For each variant, capture with `capture_snapshot(binary=opt_binary)` and compare with `compare_snapshots(ref, opt, tolerance=...)`. +5. If `result.is_valid` is false, use `result.summary()` and `result.mismatches` to diagnose. +6. Use relative paths for binaries and working directory so the skill is portable. + +## Notes + +- kernelDB is used automatically; no separate kernelDB setup is required when using the Python API. +- Increase `tolerance` for floating-point comparisons when appropriate (e.g. 1e-4 or 1e-5 for single precision). +- Use `timeout_seconds` in `capture_snapshot` if the run may hang. diff --git a/.github/agents/skills/linex/SKILL.md b/.github/agents/skills/linex/SKILL.md new file mode 100644 index 000000000..dca5b7d6d --- /dev/null +++ b/.github/agents/skills/linex/SKILL.md @@ -0,0 +1,98 @@ +--- +name: linex-profiling +description: Profile GPU kernels at source-line granularity with cycle-level timing and stall analysis. Use when identifying performance hotspots at the source code level or analyzing instruction-level metrics mapped to source lines. +--- + +# Linex: Source-Level GPU Performance Profiling + +Map GPU performance metrics to your source code lines. Get cycle-level timing, stall analysis, and instruction-level metrics for each line of source code. + +## When to Use + +- User asks to profile a GPU application at source-line granularity +- Need to identify which specific lines of code are performance bottlenecks +- Analyzing stall patterns and execution bottlenecks at the source level +- Understanding cycle-level timing for each line of code +- Instruction-level analysis mapped to source lines + +## Instructions + +1. **Ensure the target runs on AMD ROCm 7.0+** with `rocprofv3` available. +2. **Kernels must be compiled with `-g`** (debug symbols) for source mapping. +3. **Choose execution path:** + - If a Linex MCP server is available, use its MCP tools: + - `profile_application` to run and profile a target application with the options below. + - `analyze_instruction_hotspots` to perform instruction-level hotspot analysis on collected profiles. + - Otherwise use the Python API from the environment where Linex is installed. + +### Python API + +```python +from linex import Linex + +profiler = Linex( + target_cu=0, # Target compute unit + shader_engine_mask="0xFFFFFFFF", # All shader engines + activity=10, # Activity counter polling +) + +profiler.profile("./my_app", kernel_filter="my_kernel") + +# Show hotspots (sorted by total_cycles) +for line in profiler.source_lines[:5]: + print(f"{line.file}:{line.line_number}") + print(f" {line.total_cycles:,} cycles ({line.stall_percent:.1f}% stalled)") + print(f" Executed {line.execution_count} times") + +# Find memory-bound lines +memory_bound = [ + l for l in profiler.source_lines + if l.stall_percent > 50 +] + +# Instruction-level analysis +for line in profiler.source_lines[:1]: + for inst in line.instructions: + print(f"{inst.isa}: {inst.latency_cycles} cycles") +``` + +### SourceLine Properties + +- `file` - Source file path +- `line_number` - Line number +- `total_cycles` - Sum of all instruction cycles +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - Total executions +- `instructions` - List of ISA instructions +- `stall_percent` - Convenience: stall_cycles / total_cycles * 100 + +### InstructionData Properties + +- `isa` - ISA instruction text +- `latency_cycles` - Total cycles for this instruction +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - How many times it ran +- `instruction_address` - Virtual address in GPU memory +- `file` - Parsed from source_location +- `line` - Parsed from source_location +- `stall_percent` - Convenience: stall_cycles / latency_cycles * 100 + +## Workflow + +1. Ensure the target binary is built with `-g` (debug symbols) for source mapping. +2. Create a `Linex()` profiler; optionally set `target_cu`, `shader_engine_mask`, or `activity`. +3. Call `profiler.profile(command, kernel_filter=...)` to run profiling. +4. Access `profiler.source_lines` (sorted by total_cycles) to find hotspots. +5. Use `line.stall_percent` to identify memory-bound or dependency-bound lines. +6. Drill down into `line.instructions` for instruction-level analysis. +7. Use relative paths for the target binary so the skill is portable. + +## Notes + +- Requires ROCm 7.0+ with `rocprofv3` support. +- Source mapping requires kernels compiled with `-g` (debug symbols). +- `source_lines` are automatically sorted by `total_cycles` (descending). +- Use `kernel_filter` to profile specific kernels by name (regex pattern). +- For Triton or other frameworks, ensure debug symbols are available in the compiled output. diff --git a/.github/agents/skills/metrix/SKILL.md b/.github/agents/skills/metrix/SKILL.md new file mode 100644 index 000000000..9a5564ece --- /dev/null +++ b/.github/agents/skills/metrix/SKILL.md @@ -0,0 +1,76 @@ +--- +name: metrix-profiling +description: Profile GPU kernels when performance analysis or optimization is required. Use for AMD ROCm GPU metrics, bandwidth, cache hit rates, coalescing, or kernel timing. +--- + +# Metrix: GPU Profiling + +Profile AMD GPU kernels and get human-readable metrics (bandwidth, cache, coalescing, FLOPS). Architecture is auto-detected. + +## When to Use + +- User asks to profile a GPU application or kernel +- Performance analysis, optimization, or bottleneck investigation +- Need HBM/L2/L1 bandwidth, hit rates, or compute metrics +- Need timing-only runs (fast, no hardware counters) + +## Instructions + +1. **Ensure the target runs on AMD ROCm** (e.g. `hipcc`-built binary or Python script that launches HIP/ROCm kernels). +2. **Choose execution path:** + - If a Metrix MCP server is available, use its profile tool with the same options below. + - Otherwise run the CLI or Python API from the environment where Metrix is installed. + +### CLI + +From the project or install prefix: + +```bash +# Profile with all metrics (auto-detected arch) +metrix ./my_app + +# Time only (fast, no counters) +metrix --time-only -n 10 ./my_app + +# Filter kernels by name +metrix --kernel matmul ./my_app + +# Specific metrics +metrix --metrics memory.l2_hit_rate,memory.coalescing_efficiency,compute.total_flops ./my_app + +# Save to JSON/CSV +metrix -o results.json ./my_app +``` + +Options: `--profile` (quick|memory|compute), `--metrics` (comma-separated), `--time-only`, `--kernel` (substring), `--num-replays`/`-n`, `--output`/`-o`, `--top K`, `--aggregate`, `--log`/`-l` (debug|info|warning|error), `--quiet`. + +### Python API + +```python +from metrix import Metrix + +profiler = Metrix() +results = profiler.profile("./my_app", num_replays=5) + +for kernel in results.kernels: + print(kernel.name, kernel.duration_us.avg) + for metric, stats in kernel.metrics.items(): + print(f" {metric}: {stats.avg}") +``` + +Use `metrics=[...]` for a subset; omit for all metrics. Use `cwd` when the binary expects a specific working directory. + +## Workflow + +1. Identify the executable or script to profile (e.g. `./app` or `python run_kernels.py`). +2. If only timing is needed, use `--time-only` for speed. +3. If full metrics are needed, run `metrix ./app` (or MCP equivalent); optionally restrict with `--kernel` or `--metrics`. +4. Interpret results: low L2 hit rate, low coalescing, or low HBM utilization suggest optimization targets. +5. For automation or tooling, use `-o results.json` and parse the JSON output. + +## Key Metrics (reference) + +- **Memory:** `memory.hbm_bandwidth_utilization`, `memory.l2_hit_rate`, `memory.l1_hit_rate`, `memory.coalescing_efficiency`, `memory.global_load_efficiency`, `memory.lds_bank_conflicts`, `memory.atomic_latency` +- **Compute:** `compute.total_flops`, `compute.hbm_gflops`, `compute.hbm_arithmetic_intensity`, `compute.l2_arithmetic_intensity`, `compute.l1_arithmetic_intensity` + +Use relative paths for the target binary and output files so the skill is portable across environments. diff --git a/.github/agents/skills/nexus/SKILL.md b/.github/agents/skills/nexus/SKILL.md new file mode 100644 index 000000000..ad714bc4d --- /dev/null +++ b/.github/agents/skills/nexus/SKILL.md @@ -0,0 +1,74 @@ +--- +name: nexus-trace +description: Extract GPU kernel assembly and HIP source from HSA packet traces. Use when analyzing what code ran on the GPU, debugging kernel dispatch, or inspecting assembly and source mapping. +--- + +# Nexus: HSA Packet Source Code Extractor + +Intercepts HSA packets from a running process and extracts, per kernel, assembly and HIP source into a structured trace (e.g. JSON). Use for kernel-level inspection and assembly/source correlation. + +## When to Use + +- User needs to see which kernels ran and their assembly or HIP source +- Debugging or analyzing GPU dispatch and code generation +- Inspecting assembly-to-source mapping for a HIP (or ROCm) application + +## Instructions + +1. **Ensure the target runs on AMD ROCm** and uses HSA (e.g. HIP application or ROCm runtime). +2. **Choose execution path:** + - If a Nexus MCP server is available, use its tools: `list_kernels` to enumerate kernels in a trace, and `extract_kernel_code` to get assembly and HIP/source mapping (signature, files, lines). See `nexus/nexus/mcp/server.py` for tool parameters and schemas. + - Otherwise use the Python API from the environment where Nexus is installed. + +### Python API (recommended when no MCP) + +```python +from nexus import Nexus + +nexus = Nexus(log_level=1) +trace = nexus.run(["python", "my_gpu_script.py"]) + +# Or run a binary: +# trace = nexus.run(["./my_hip_app"]) + +for kernel in trace: + print(kernel.name, len(kernel.assembly), "instructions") + for i, asm_line in enumerate(kernel.assembly, 1): + print(f" {i}. {asm_line}") + for line_no, hip_line in zip(kernel.lines or range(1, len(kernel.hip)+1), kernel.hip): + print(f" {line_no}: {hip_line}") + +# Access by kernel name +k = trace["vector_add(float const*, float const*, float*, int)"] +print(k.assembly, k.hip, k.signature, k.files, k.lines) + +# Save/load trace +trace.save("trace.json") +loaded = Nexus.load("trace.json") +``` + +Set `log_level` (0–4) to control verbosity. Use relative paths for the run command and output file so the skill is portable. + +### Environment-based usage (no Python API) + +When the process cannot be launched via `nexus.run()`: + +1. Set `HSA_TOOLS_LIB` to the Nexus shared library path (e.g. `build/lib/libnexus.so` or the installed path). +2. Set `NEXUS_OUTPUT_FILE` to the output JSON path. +3. Set `NEXUS_LOG_LEVEL` (0–4) if needed. +4. Run the application as usual; it will be traced and the output file will contain the kernel data. + +Optional: `NEXUS_EXTRA_SEARCH_PREFIX` (colon-separated) for HIP source search; `TRITON_DISABLE_LINE_INFO=0` for Triton kernel line info. + +## Workflow + +1. Identify the command that runs the GPU workload (e.g. `python script.py` or `./app`). +2. If using the Python API: create `Nexus(log_level=...)`, call `nexus.run([...])`, then iterate `trace` and optionally `trace.save(...)`. +3. If using the env method: set `HSA_TOOLS_LIB` and `NEXUS_OUTPUT_FILE`, then run the app; open the JSON and parse the `kernels` structure. +4. Use kernel `signature`, `assembly`, `hip`, `files`, and `lines` to analyze what ran and map assembly back to source. +5. Use relative paths for commands and output files. + +## Notes + +- Nexus is intended for research/analysis; ensure the target environment has the Nexus library and compatible ROCm/HSA stack. +- For Triton kernels, enable line info via `TRITON_DISABLE_LINE_INFO=0` when using the Python API. diff --git a/.gitignore b/.gitignore index d8f9754f7..c1fd3d273 100644 --- a/.gitignore +++ b/.gitignore @@ -57,4 +57,5 @@ gpucore.* logs/ *.cap hsakmt_counters.csv -core \ No newline at end of file +core +.rocprofv3/ \ No newline at end of file diff --git a/.intellikit b/.intellikit new file mode 160000 index 000000000..f69ecf13c --- /dev/null +++ b/.intellikit @@ -0,0 +1 @@ +Subproject commit f69ecf13c59fe73e58fb37ee2f6ea02fafd22444 diff --git a/examples/07_gemm_all_scatter/README.md b/examples/07_gemm_all_scatter/README.md new file mode 100644 index 000000000..8dbd197dd --- /dev/null +++ b/examples/07_gemm_all_scatter/README.md @@ -0,0 +1,80 @@ + + +# GEMM All-Scatter Benchmark + +Tile-based GEMM with all-scatter communication using Iris. Each rank computes a local tile of the output matrix and scatters results to all peer GPUs using Iris's `store` primitive, enabling communication/computation overlap. + +## Algorithm + +Each rank holds a shard of matrix B (N/world_size columns) and computes the full A×B product for its local columns. Results are scattered in-kernel via remote stores to reconstruct the global output matrix across all GPUs. + +## Usage + +```bash +# Run with torchrun (2 GPUs) +torchrun --nproc_per_node 2 examples/07_gemm_all_scatter/benchmark.py --benchmark + +# Validate correctness +torchrun --nproc_per_node 2 examples/07_gemm_all_scatter/benchmark.py --validate + +# Custom matrix dimensions +torchrun --nproc_per_node 2 examples/07_gemm_all_scatter/benchmark.py \ + --benchmark -m 8192 -n 4608 -k 36864 --datatype fp16 +``` + +## Benchmark Results + +Measured on **2× AMD MI300X (gfx942)**, FP16, default dimensions (M=8192, N=4608, K=36864): + +| Metric | Value | +|-----------------|---------------| +| Performance | ~611 TFLOPS | +| Total time | ~4.55 ms | +| GEMM kernel | ~4.41 ms | +| World size | 2 GPUs | +| Block size M | 256 | +| Block size N | 64 | +| Block size K | 64 | +| SMs used | 304 (all CUs) | + +## IntelliKit Profiling (Metrix) + +Profiled with [Metrix](https://github.com/amd/metrix) on 2× AMD MI300X (gfx942), M=8192, N=4608, K=36864, FP16, across 126 benchmark iterations. + +### Memory Profile + +| Metric | Value | +|------------------------------|-----------------| +| HBM Bandwidth Utilization | 3.2% | +| HBM Read Bandwidth | 167.4 GB/s | +| HBM Write Bandwidth | 2.5 GB/s | +| Bytes Transferred (HBM) | ~4.9 GB | +| L1 Hit Rate | 52.6% | +| L2 Hit Rate | 81.3% | +| L2 Bandwidth | 911.8 GB/s | +| Coalescing Efficiency | 25.0% | +| Global Load Efficiency | 12.5% | +| Global Store Efficiency | 6.25% | +| LDS Bank Conflicts | 0 | +| Atomic Latency | 0 | + +### Compute Profile + +| Metric | Value | +|------------------------------|-----------------| +| Total FLOPs per dispatch | ~1.39 TFLOPS | +| Throughput (HBM-normalized) | ~315,327 GFLOPS | +| HBM Arithmetic Intensity | 267.7 FLOP/Byte | +| L2 Arithmetic Intensity | 49.8 FLOP/Byte | +| L1 Arithmetic Intensity | 24.2 FLOP/Byte | +| Avg kernel duration | ~4.4 ms | + +### Analysis + +- **Compute-bound**: Low HBM bandwidth utilization (~3.2%) combined with high arithmetic intensity (~268 FLOP/Byte) confirms the kernel is compute-bound, which is optimal for GEMM workloads. +- **Good L2 reuse**: 81.3% L2 hit rate indicates effective blocking and data reuse through the L2 cache. +- **Zero LDS conflicts and atomic latency**: The blocking strategy avoids shared memory bank conflicts, and the all-scatter communication via Iris remote stores incurs no detectable atomic latency overhead. +- **Coalescing**: 25% coalescing efficiency and 12.5% global load efficiency are expected for this tiled GEMM access pattern with large tiles (BLK_M=256, BLK_N=64, BLK_K=64).