From a6c0766d9474f753d301a83368e26bd8cdd844f9 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 12 Mar 2026 17:29:45 +0000 Subject: [PATCH 1/3] Initial plan From 20e8f4cf8b51ffc2bfe0cd866e389ff821a320cc Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 12 Mar 2026 19:12:05 +0000 Subject: [PATCH 2/3] Initial plan: run Example 7 GEMM all-scatter and profile with IntelliKit Co-authored-by: JoseSantosAMD <87447437+JoseSantosAMD@users.noreply.github.com> --- .agents/skills/accordo/SKILL.md | 68 ++++++++++++++++++ .agents/skills/linex/SKILL.md | 98 ++++++++++++++++++++++++++ .agents/skills/metrix/SKILL.md | 76 ++++++++++++++++++++ .agents/skills/nexus/SKILL.md | 74 +++++++++++++++++++ .github/agents/skills/accordo/SKILL.md | 68 ++++++++++++++++++ .github/agents/skills/linex/SKILL.md | 98 ++++++++++++++++++++++++++ .github/agents/skills/metrix/SKILL.md | 76 ++++++++++++++++++++ .github/agents/skills/nexus/SKILL.md | 74 +++++++++++++++++++ .intellikit | 1 + 9 files changed, 633 insertions(+) create mode 100644 .agents/skills/accordo/SKILL.md create mode 100644 .agents/skills/linex/SKILL.md create mode 100644 .agents/skills/metrix/SKILL.md create mode 100644 .agents/skills/nexus/SKILL.md create mode 100644 .github/agents/skills/accordo/SKILL.md create mode 100644 .github/agents/skills/linex/SKILL.md create mode 100644 .github/agents/skills/metrix/SKILL.md create mode 100644 .github/agents/skills/nexus/SKILL.md create mode 160000 .intellikit diff --git a/.agents/skills/accordo/SKILL.md b/.agents/skills/accordo/SKILL.md new file mode 100644 index 000000000..808c73ef9 --- /dev/null +++ b/.agents/skills/accordo/SKILL.md @@ -0,0 +1,68 @@ +--- +name: accordo-validation +description: Validate GPU kernel correctness by comparing reference and optimized outputs. Use when verifying that an optimized or modified kernel matches a reference implementation. +--- + +# Accordo: GPU Kernel Validation + +Capture and compare kernel outputs from reference and optimized binaries to validate correctness. Uses kernelDB for automatic kernel extraction; supports configurable tolerance and execution-time comparison. + +## When to Use + +- User has a reference and an optimized (or modified) GPU kernel and wants to check they produce the same results +- Regression testing after kernel or build changes +- Validating multiple optimization variants against one baseline + +## Instructions + +1. **Require two or more binaries:** one reference (e.g. `./app_ref`) and one or more to validate (e.g. `./app_opt`). All must expose the same kernel by name. +2. **Ensure binaries are built with debug symbols** (`-g`) so kernel arguments can be extracted. +3. **Choose execution path:** + - If an Accordo MCP server is available, call its `validate_kernel_correctness` tool, which performs capture-and-compare with the same semantics described below. + - Otherwise use the Python API from the environment where Accordo is installed. + +### Python API + +```python +from accordo import Accordo + +# Validator for the kernel to validate (binary used to extract signature) +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum") + +# Optional: set working directory if binaries expect it +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum", working_directory="./run") + +# Capture snapshots +ref = validator.capture_snapshot(binary="./app_ref") +opt = validator.capture_snapshot(binary="./app_opt") + +# Compare with tolerance (default 1e-6) +result = validator.compare_snapshots(ref, opt, tolerance=1e-6) + +if result.is_valid: + print("PASS:", result.num_arrays_validated, "arrays matched") +else: + print(result.summary()) +``` + +For multiple optimizations, capture the reference once and compare each optimized snapshot against it. + +### Snapshot and result attributes + +- **Snapshot:** `arrays`, `execution_time_ms`, `grid_size`, `block_size` +- **ValidationResult:** `is_valid`, `num_arrays_validated`, `num_mismatches`, `mismatches`, `success_rate`; use `summary()` for a human-readable report. + +## Workflow + +1. Build reference and optimized binaries with the same kernel name and `-g`. +2. Create an `Accordo(binary=ref_binary, kernel_name="...")` validator; set `working_directory` if needed. +3. Capture reference snapshot with `capture_snapshot(binary=ref_binary)`. +4. For each variant, capture with `capture_snapshot(binary=opt_binary)` and compare with `compare_snapshots(ref, opt, tolerance=...)`. +5. If `result.is_valid` is false, use `result.summary()` and `result.mismatches` to diagnose. +6. Use relative paths for binaries and working directory so the skill is portable. + +## Notes + +- kernelDB is used automatically; no separate kernelDB setup is required when using the Python API. +- Increase `tolerance` for floating-point comparisons when appropriate (e.g. 1e-4 or 1e-5 for single precision). +- Use `timeout_seconds` in `capture_snapshot` if the run may hang. diff --git a/.agents/skills/linex/SKILL.md b/.agents/skills/linex/SKILL.md new file mode 100644 index 000000000..dca5b7d6d --- /dev/null +++ b/.agents/skills/linex/SKILL.md @@ -0,0 +1,98 @@ +--- +name: linex-profiling +description: Profile GPU kernels at source-line granularity with cycle-level timing and stall analysis. Use when identifying performance hotspots at the source code level or analyzing instruction-level metrics mapped to source lines. +--- + +# Linex: Source-Level GPU Performance Profiling + +Map GPU performance metrics to your source code lines. Get cycle-level timing, stall analysis, and instruction-level metrics for each line of source code. + +## When to Use + +- User asks to profile a GPU application at source-line granularity +- Need to identify which specific lines of code are performance bottlenecks +- Analyzing stall patterns and execution bottlenecks at the source level +- Understanding cycle-level timing for each line of code +- Instruction-level analysis mapped to source lines + +## Instructions + +1. **Ensure the target runs on AMD ROCm 7.0+** with `rocprofv3` available. +2. **Kernels must be compiled with `-g`** (debug symbols) for source mapping. +3. **Choose execution path:** + - If a Linex MCP server is available, use its MCP tools: + - `profile_application` to run and profile a target application with the options below. + - `analyze_instruction_hotspots` to perform instruction-level hotspot analysis on collected profiles. + - Otherwise use the Python API from the environment where Linex is installed. + +### Python API + +```python +from linex import Linex + +profiler = Linex( + target_cu=0, # Target compute unit + shader_engine_mask="0xFFFFFFFF", # All shader engines + activity=10, # Activity counter polling +) + +profiler.profile("./my_app", kernel_filter="my_kernel") + +# Show hotspots (sorted by total_cycles) +for line in profiler.source_lines[:5]: + print(f"{line.file}:{line.line_number}") + print(f" {line.total_cycles:,} cycles ({line.stall_percent:.1f}% stalled)") + print(f" Executed {line.execution_count} times") + +# Find memory-bound lines +memory_bound = [ + l for l in profiler.source_lines + if l.stall_percent > 50 +] + +# Instruction-level analysis +for line in profiler.source_lines[:1]: + for inst in line.instructions: + print(f"{inst.isa}: {inst.latency_cycles} cycles") +``` + +### SourceLine Properties + +- `file` - Source file path +- `line_number` - Line number +- `total_cycles` - Sum of all instruction cycles +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - Total executions +- `instructions` - List of ISA instructions +- `stall_percent` - Convenience: stall_cycles / total_cycles * 100 + +### InstructionData Properties + +- `isa` - ISA instruction text +- `latency_cycles` - Total cycles for this instruction +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - How many times it ran +- `instruction_address` - Virtual address in GPU memory +- `file` - Parsed from source_location +- `line` - Parsed from source_location +- `stall_percent` - Convenience: stall_cycles / latency_cycles * 100 + +## Workflow + +1. Ensure the target binary is built with `-g` (debug symbols) for source mapping. +2. Create a `Linex()` profiler; optionally set `target_cu`, `shader_engine_mask`, or `activity`. +3. Call `profiler.profile(command, kernel_filter=...)` to run profiling. +4. Access `profiler.source_lines` (sorted by total_cycles) to find hotspots. +5. Use `line.stall_percent` to identify memory-bound or dependency-bound lines. +6. Drill down into `line.instructions` for instruction-level analysis. +7. Use relative paths for the target binary so the skill is portable. + +## Notes + +- Requires ROCm 7.0+ with `rocprofv3` support. +- Source mapping requires kernels compiled with `-g` (debug symbols). +- `source_lines` are automatically sorted by `total_cycles` (descending). +- Use `kernel_filter` to profile specific kernels by name (regex pattern). +- For Triton or other frameworks, ensure debug symbols are available in the compiled output. diff --git a/.agents/skills/metrix/SKILL.md b/.agents/skills/metrix/SKILL.md new file mode 100644 index 000000000..9a5564ece --- /dev/null +++ b/.agents/skills/metrix/SKILL.md @@ -0,0 +1,76 @@ +--- +name: metrix-profiling +description: Profile GPU kernels when performance analysis or optimization is required. Use for AMD ROCm GPU metrics, bandwidth, cache hit rates, coalescing, or kernel timing. +--- + +# Metrix: GPU Profiling + +Profile AMD GPU kernels and get human-readable metrics (bandwidth, cache, coalescing, FLOPS). Architecture is auto-detected. + +## When to Use + +- User asks to profile a GPU application or kernel +- Performance analysis, optimization, or bottleneck investigation +- Need HBM/L2/L1 bandwidth, hit rates, or compute metrics +- Need timing-only runs (fast, no hardware counters) + +## Instructions + +1. **Ensure the target runs on AMD ROCm** (e.g. `hipcc`-built binary or Python script that launches HIP/ROCm kernels). +2. **Choose execution path:** + - If a Metrix MCP server is available, use its profile tool with the same options below. + - Otherwise run the CLI or Python API from the environment where Metrix is installed. + +### CLI + +From the project or install prefix: + +```bash +# Profile with all metrics (auto-detected arch) +metrix ./my_app + +# Time only (fast, no counters) +metrix --time-only -n 10 ./my_app + +# Filter kernels by name +metrix --kernel matmul ./my_app + +# Specific metrics +metrix --metrics memory.l2_hit_rate,memory.coalescing_efficiency,compute.total_flops ./my_app + +# Save to JSON/CSV +metrix -o results.json ./my_app +``` + +Options: `--profile` (quick|memory|compute), `--metrics` (comma-separated), `--time-only`, `--kernel` (substring), `--num-replays`/`-n`, `--output`/`-o`, `--top K`, `--aggregate`, `--log`/`-l` (debug|info|warning|error), `--quiet`. + +### Python API + +```python +from metrix import Metrix + +profiler = Metrix() +results = profiler.profile("./my_app", num_replays=5) + +for kernel in results.kernels: + print(kernel.name, kernel.duration_us.avg) + for metric, stats in kernel.metrics.items(): + print(f" {metric}: {stats.avg}") +``` + +Use `metrics=[...]` for a subset; omit for all metrics. Use `cwd` when the binary expects a specific working directory. + +## Workflow + +1. Identify the executable or script to profile (e.g. `./app` or `python run_kernels.py`). +2. If only timing is needed, use `--time-only` for speed. +3. If full metrics are needed, run `metrix ./app` (or MCP equivalent); optionally restrict with `--kernel` or `--metrics`. +4. Interpret results: low L2 hit rate, low coalescing, or low HBM utilization suggest optimization targets. +5. For automation or tooling, use `-o results.json` and parse the JSON output. + +## Key Metrics (reference) + +- **Memory:** `memory.hbm_bandwidth_utilization`, `memory.l2_hit_rate`, `memory.l1_hit_rate`, `memory.coalescing_efficiency`, `memory.global_load_efficiency`, `memory.lds_bank_conflicts`, `memory.atomic_latency` +- **Compute:** `compute.total_flops`, `compute.hbm_gflops`, `compute.hbm_arithmetic_intensity`, `compute.l2_arithmetic_intensity`, `compute.l1_arithmetic_intensity` + +Use relative paths for the target binary and output files so the skill is portable across environments. diff --git a/.agents/skills/nexus/SKILL.md b/.agents/skills/nexus/SKILL.md new file mode 100644 index 000000000..ad714bc4d --- /dev/null +++ b/.agents/skills/nexus/SKILL.md @@ -0,0 +1,74 @@ +--- +name: nexus-trace +description: Extract GPU kernel assembly and HIP source from HSA packet traces. Use when analyzing what code ran on the GPU, debugging kernel dispatch, or inspecting assembly and source mapping. +--- + +# Nexus: HSA Packet Source Code Extractor + +Intercepts HSA packets from a running process and extracts, per kernel, assembly and HIP source into a structured trace (e.g. JSON). Use for kernel-level inspection and assembly/source correlation. + +## When to Use + +- User needs to see which kernels ran and their assembly or HIP source +- Debugging or analyzing GPU dispatch and code generation +- Inspecting assembly-to-source mapping for a HIP (or ROCm) application + +## Instructions + +1. **Ensure the target runs on AMD ROCm** and uses HSA (e.g. HIP application or ROCm runtime). +2. **Choose execution path:** + - If a Nexus MCP server is available, use its tools: `list_kernels` to enumerate kernels in a trace, and `extract_kernel_code` to get assembly and HIP/source mapping (signature, files, lines). See `nexus/nexus/mcp/server.py` for tool parameters and schemas. + - Otherwise use the Python API from the environment where Nexus is installed. + +### Python API (recommended when no MCP) + +```python +from nexus import Nexus + +nexus = Nexus(log_level=1) +trace = nexus.run(["python", "my_gpu_script.py"]) + +# Or run a binary: +# trace = nexus.run(["./my_hip_app"]) + +for kernel in trace: + print(kernel.name, len(kernel.assembly), "instructions") + for i, asm_line in enumerate(kernel.assembly, 1): + print(f" {i}. {asm_line}") + for line_no, hip_line in zip(kernel.lines or range(1, len(kernel.hip)+1), kernel.hip): + print(f" {line_no}: {hip_line}") + +# Access by kernel name +k = trace["vector_add(float const*, float const*, float*, int)"] +print(k.assembly, k.hip, k.signature, k.files, k.lines) + +# Save/load trace +trace.save("trace.json") +loaded = Nexus.load("trace.json") +``` + +Set `log_level` (0–4) to control verbosity. Use relative paths for the run command and output file so the skill is portable. + +### Environment-based usage (no Python API) + +When the process cannot be launched via `nexus.run()`: + +1. Set `HSA_TOOLS_LIB` to the Nexus shared library path (e.g. `build/lib/libnexus.so` or the installed path). +2. Set `NEXUS_OUTPUT_FILE` to the output JSON path. +3. Set `NEXUS_LOG_LEVEL` (0–4) if needed. +4. Run the application as usual; it will be traced and the output file will contain the kernel data. + +Optional: `NEXUS_EXTRA_SEARCH_PREFIX` (colon-separated) for HIP source search; `TRITON_DISABLE_LINE_INFO=0` for Triton kernel line info. + +## Workflow + +1. Identify the command that runs the GPU workload (e.g. `python script.py` or `./app`). +2. If using the Python API: create `Nexus(log_level=...)`, call `nexus.run([...])`, then iterate `trace` and optionally `trace.save(...)`. +3. If using the env method: set `HSA_TOOLS_LIB` and `NEXUS_OUTPUT_FILE`, then run the app; open the JSON and parse the `kernels` structure. +4. Use kernel `signature`, `assembly`, `hip`, `files`, and `lines` to analyze what ran and map assembly back to source. +5. Use relative paths for commands and output files. + +## Notes + +- Nexus is intended for research/analysis; ensure the target environment has the Nexus library and compatible ROCm/HSA stack. +- For Triton kernels, enable line info via `TRITON_DISABLE_LINE_INFO=0` when using the Python API. diff --git a/.github/agents/skills/accordo/SKILL.md b/.github/agents/skills/accordo/SKILL.md new file mode 100644 index 000000000..808c73ef9 --- /dev/null +++ b/.github/agents/skills/accordo/SKILL.md @@ -0,0 +1,68 @@ +--- +name: accordo-validation +description: Validate GPU kernel correctness by comparing reference and optimized outputs. Use when verifying that an optimized or modified kernel matches a reference implementation. +--- + +# Accordo: GPU Kernel Validation + +Capture and compare kernel outputs from reference and optimized binaries to validate correctness. Uses kernelDB for automatic kernel extraction; supports configurable tolerance and execution-time comparison. + +## When to Use + +- User has a reference and an optimized (or modified) GPU kernel and wants to check they produce the same results +- Regression testing after kernel or build changes +- Validating multiple optimization variants against one baseline + +## Instructions + +1. **Require two or more binaries:** one reference (e.g. `./app_ref`) and one or more to validate (e.g. `./app_opt`). All must expose the same kernel by name. +2. **Ensure binaries are built with debug symbols** (`-g`) so kernel arguments can be extracted. +3. **Choose execution path:** + - If an Accordo MCP server is available, call its `validate_kernel_correctness` tool, which performs capture-and-compare with the same semantics described below. + - Otherwise use the Python API from the environment where Accordo is installed. + +### Python API + +```python +from accordo import Accordo + +# Validator for the kernel to validate (binary used to extract signature) +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum") + +# Optional: set working directory if binaries expect it +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum", working_directory="./run") + +# Capture snapshots +ref = validator.capture_snapshot(binary="./app_ref") +opt = validator.capture_snapshot(binary="./app_opt") + +# Compare with tolerance (default 1e-6) +result = validator.compare_snapshots(ref, opt, tolerance=1e-6) + +if result.is_valid: + print("PASS:", result.num_arrays_validated, "arrays matched") +else: + print(result.summary()) +``` + +For multiple optimizations, capture the reference once and compare each optimized snapshot against it. + +### Snapshot and result attributes + +- **Snapshot:** `arrays`, `execution_time_ms`, `grid_size`, `block_size` +- **ValidationResult:** `is_valid`, `num_arrays_validated`, `num_mismatches`, `mismatches`, `success_rate`; use `summary()` for a human-readable report. + +## Workflow + +1. Build reference and optimized binaries with the same kernel name and `-g`. +2. Create an `Accordo(binary=ref_binary, kernel_name="...")` validator; set `working_directory` if needed. +3. Capture reference snapshot with `capture_snapshot(binary=ref_binary)`. +4. For each variant, capture with `capture_snapshot(binary=opt_binary)` and compare with `compare_snapshots(ref, opt, tolerance=...)`. +5. If `result.is_valid` is false, use `result.summary()` and `result.mismatches` to diagnose. +6. Use relative paths for binaries and working directory so the skill is portable. + +## Notes + +- kernelDB is used automatically; no separate kernelDB setup is required when using the Python API. +- Increase `tolerance` for floating-point comparisons when appropriate (e.g. 1e-4 or 1e-5 for single precision). +- Use `timeout_seconds` in `capture_snapshot` if the run may hang. diff --git a/.github/agents/skills/linex/SKILL.md b/.github/agents/skills/linex/SKILL.md new file mode 100644 index 000000000..dca5b7d6d --- /dev/null +++ b/.github/agents/skills/linex/SKILL.md @@ -0,0 +1,98 @@ +--- +name: linex-profiling +description: Profile GPU kernels at source-line granularity with cycle-level timing and stall analysis. Use when identifying performance hotspots at the source code level or analyzing instruction-level metrics mapped to source lines. +--- + +# Linex: Source-Level GPU Performance Profiling + +Map GPU performance metrics to your source code lines. Get cycle-level timing, stall analysis, and instruction-level metrics for each line of source code. + +## When to Use + +- User asks to profile a GPU application at source-line granularity +- Need to identify which specific lines of code are performance bottlenecks +- Analyzing stall patterns and execution bottlenecks at the source level +- Understanding cycle-level timing for each line of code +- Instruction-level analysis mapped to source lines + +## Instructions + +1. **Ensure the target runs on AMD ROCm 7.0+** with `rocprofv3` available. +2. **Kernels must be compiled with `-g`** (debug symbols) for source mapping. +3. **Choose execution path:** + - If a Linex MCP server is available, use its MCP tools: + - `profile_application` to run and profile a target application with the options below. + - `analyze_instruction_hotspots` to perform instruction-level hotspot analysis on collected profiles. + - Otherwise use the Python API from the environment where Linex is installed. + +### Python API + +```python +from linex import Linex + +profiler = Linex( + target_cu=0, # Target compute unit + shader_engine_mask="0xFFFFFFFF", # All shader engines + activity=10, # Activity counter polling +) + +profiler.profile("./my_app", kernel_filter="my_kernel") + +# Show hotspots (sorted by total_cycles) +for line in profiler.source_lines[:5]: + print(f"{line.file}:{line.line_number}") + print(f" {line.total_cycles:,} cycles ({line.stall_percent:.1f}% stalled)") + print(f" Executed {line.execution_count} times") + +# Find memory-bound lines +memory_bound = [ + l for l in profiler.source_lines + if l.stall_percent > 50 +] + +# Instruction-level analysis +for line in profiler.source_lines[:1]: + for inst in line.instructions: + print(f"{inst.isa}: {inst.latency_cycles} cycles") +``` + +### SourceLine Properties + +- `file` - Source file path +- `line_number` - Line number +- `total_cycles` - Sum of all instruction cycles +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - Total executions +- `instructions` - List of ISA instructions +- `stall_percent` - Convenience: stall_cycles / total_cycles * 100 + +### InstructionData Properties + +- `isa` - ISA instruction text +- `latency_cycles` - Total cycles for this instruction +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - How many times it ran +- `instruction_address` - Virtual address in GPU memory +- `file` - Parsed from source_location +- `line` - Parsed from source_location +- `stall_percent` - Convenience: stall_cycles / latency_cycles * 100 + +## Workflow + +1. Ensure the target binary is built with `-g` (debug symbols) for source mapping. +2. Create a `Linex()` profiler; optionally set `target_cu`, `shader_engine_mask`, or `activity`. +3. Call `profiler.profile(command, kernel_filter=...)` to run profiling. +4. Access `profiler.source_lines` (sorted by total_cycles) to find hotspots. +5. Use `line.stall_percent` to identify memory-bound or dependency-bound lines. +6. Drill down into `line.instructions` for instruction-level analysis. +7. Use relative paths for the target binary so the skill is portable. + +## Notes + +- Requires ROCm 7.0+ with `rocprofv3` support. +- Source mapping requires kernels compiled with `-g` (debug symbols). +- `source_lines` are automatically sorted by `total_cycles` (descending). +- Use `kernel_filter` to profile specific kernels by name (regex pattern). +- For Triton or other frameworks, ensure debug symbols are available in the compiled output. diff --git a/.github/agents/skills/metrix/SKILL.md b/.github/agents/skills/metrix/SKILL.md new file mode 100644 index 000000000..9a5564ece --- /dev/null +++ b/.github/agents/skills/metrix/SKILL.md @@ -0,0 +1,76 @@ +--- +name: metrix-profiling +description: Profile GPU kernels when performance analysis or optimization is required. Use for AMD ROCm GPU metrics, bandwidth, cache hit rates, coalescing, or kernel timing. +--- + +# Metrix: GPU Profiling + +Profile AMD GPU kernels and get human-readable metrics (bandwidth, cache, coalescing, FLOPS). Architecture is auto-detected. + +## When to Use + +- User asks to profile a GPU application or kernel +- Performance analysis, optimization, or bottleneck investigation +- Need HBM/L2/L1 bandwidth, hit rates, or compute metrics +- Need timing-only runs (fast, no hardware counters) + +## Instructions + +1. **Ensure the target runs on AMD ROCm** (e.g. `hipcc`-built binary or Python script that launches HIP/ROCm kernels). +2. **Choose execution path:** + - If a Metrix MCP server is available, use its profile tool with the same options below. + - Otherwise run the CLI or Python API from the environment where Metrix is installed. + +### CLI + +From the project or install prefix: + +```bash +# Profile with all metrics (auto-detected arch) +metrix ./my_app + +# Time only (fast, no counters) +metrix --time-only -n 10 ./my_app + +# Filter kernels by name +metrix --kernel matmul ./my_app + +# Specific metrics +metrix --metrics memory.l2_hit_rate,memory.coalescing_efficiency,compute.total_flops ./my_app + +# Save to JSON/CSV +metrix -o results.json ./my_app +``` + +Options: `--profile` (quick|memory|compute), `--metrics` (comma-separated), `--time-only`, `--kernel` (substring), `--num-replays`/`-n`, `--output`/`-o`, `--top K`, `--aggregate`, `--log`/`-l` (debug|info|warning|error), `--quiet`. + +### Python API + +```python +from metrix import Metrix + +profiler = Metrix() +results = profiler.profile("./my_app", num_replays=5) + +for kernel in results.kernels: + print(kernel.name, kernel.duration_us.avg) + for metric, stats in kernel.metrics.items(): + print(f" {metric}: {stats.avg}") +``` + +Use `metrics=[...]` for a subset; omit for all metrics. Use `cwd` when the binary expects a specific working directory. + +## Workflow + +1. Identify the executable or script to profile (e.g. `./app` or `python run_kernels.py`). +2. If only timing is needed, use `--time-only` for speed. +3. If full metrics are needed, run `metrix ./app` (or MCP equivalent); optionally restrict with `--kernel` or `--metrics`. +4. Interpret results: low L2 hit rate, low coalescing, or low HBM utilization suggest optimization targets. +5. For automation or tooling, use `-o results.json` and parse the JSON output. + +## Key Metrics (reference) + +- **Memory:** `memory.hbm_bandwidth_utilization`, `memory.l2_hit_rate`, `memory.l1_hit_rate`, `memory.coalescing_efficiency`, `memory.global_load_efficiency`, `memory.lds_bank_conflicts`, `memory.atomic_latency` +- **Compute:** `compute.total_flops`, `compute.hbm_gflops`, `compute.hbm_arithmetic_intensity`, `compute.l2_arithmetic_intensity`, `compute.l1_arithmetic_intensity` + +Use relative paths for the target binary and output files so the skill is portable across environments. diff --git a/.github/agents/skills/nexus/SKILL.md b/.github/agents/skills/nexus/SKILL.md new file mode 100644 index 000000000..ad714bc4d --- /dev/null +++ b/.github/agents/skills/nexus/SKILL.md @@ -0,0 +1,74 @@ +--- +name: nexus-trace +description: Extract GPU kernel assembly and HIP source from HSA packet traces. Use when analyzing what code ran on the GPU, debugging kernel dispatch, or inspecting assembly and source mapping. +--- + +# Nexus: HSA Packet Source Code Extractor + +Intercepts HSA packets from a running process and extracts, per kernel, assembly and HIP source into a structured trace (e.g. JSON). Use for kernel-level inspection and assembly/source correlation. + +## When to Use + +- User needs to see which kernels ran and their assembly or HIP source +- Debugging or analyzing GPU dispatch and code generation +- Inspecting assembly-to-source mapping for a HIP (or ROCm) application + +## Instructions + +1. **Ensure the target runs on AMD ROCm** and uses HSA (e.g. HIP application or ROCm runtime). +2. **Choose execution path:** + - If a Nexus MCP server is available, use its tools: `list_kernels` to enumerate kernels in a trace, and `extract_kernel_code` to get assembly and HIP/source mapping (signature, files, lines). See `nexus/nexus/mcp/server.py` for tool parameters and schemas. + - Otherwise use the Python API from the environment where Nexus is installed. + +### Python API (recommended when no MCP) + +```python +from nexus import Nexus + +nexus = Nexus(log_level=1) +trace = nexus.run(["python", "my_gpu_script.py"]) + +# Or run a binary: +# trace = nexus.run(["./my_hip_app"]) + +for kernel in trace: + print(kernel.name, len(kernel.assembly), "instructions") + for i, asm_line in enumerate(kernel.assembly, 1): + print(f" {i}. {asm_line}") + for line_no, hip_line in zip(kernel.lines or range(1, len(kernel.hip)+1), kernel.hip): + print(f" {line_no}: {hip_line}") + +# Access by kernel name +k = trace["vector_add(float const*, float const*, float*, int)"] +print(k.assembly, k.hip, k.signature, k.files, k.lines) + +# Save/load trace +trace.save("trace.json") +loaded = Nexus.load("trace.json") +``` + +Set `log_level` (0–4) to control verbosity. Use relative paths for the run command and output file so the skill is portable. + +### Environment-based usage (no Python API) + +When the process cannot be launched via `nexus.run()`: + +1. Set `HSA_TOOLS_LIB` to the Nexus shared library path (e.g. `build/lib/libnexus.so` or the installed path). +2. Set `NEXUS_OUTPUT_FILE` to the output JSON path. +3. Set `NEXUS_LOG_LEVEL` (0–4) if needed. +4. Run the application as usual; it will be traced and the output file will contain the kernel data. + +Optional: `NEXUS_EXTRA_SEARCH_PREFIX` (colon-separated) for HIP source search; `TRITON_DISABLE_LINE_INFO=0` for Triton kernel line info. + +## Workflow + +1. Identify the command that runs the GPU workload (e.g. `python script.py` or `./app`). +2. If using the Python API: create `Nexus(log_level=...)`, call `nexus.run([...])`, then iterate `trace` and optionally `trace.save(...)`. +3. If using the env method: set `HSA_TOOLS_LIB` and `NEXUS_OUTPUT_FILE`, then run the app; open the JSON and parse the `kernels` structure. +4. Use kernel `signature`, `assembly`, `hip`, `files`, and `lines` to analyze what ran and map assembly back to source. +5. Use relative paths for commands and output files. + +## Notes + +- Nexus is intended for research/analysis; ensure the target environment has the Nexus library and compatible ROCm/HSA stack. +- For Triton kernels, enable line info via `TRITON_DISABLE_LINE_INFO=0` when using the Python API. diff --git a/.intellikit b/.intellikit new file mode 160000 index 000000000..f69ecf13c --- /dev/null +++ b/.intellikit @@ -0,0 +1 @@ +Subproject commit f69ecf13c59fe73e58fb37ee2f6ea02fafd22444 From 59cd591a0aba2325ef54e46b29cdb60c9db2d9f0 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Thu, 12 Mar 2026 19:22:12 +0000 Subject: [PATCH 3/3] Add Example 7 IntelliKit Metrix profiling results and update .gitignore Co-authored-by: JoseSantosAMD <87447437+JoseSantosAMD@users.noreply.github.com> --- .gitignore | 3 +- .../07_gemm_all_scatter/profiling_results.md | 131 ++++++++++++++++++ 2 files changed, 133 insertions(+), 1 deletion(-) create mode 100644 examples/07_gemm_all_scatter/profiling_results.md diff --git a/.gitignore b/.gitignore index d8f9754f7..c1fd3d273 100644 --- a/.gitignore +++ b/.gitignore @@ -57,4 +57,5 @@ gpucore.* logs/ *.cap hsakmt_counters.csv -core \ No newline at end of file +core +.rocprofv3/ \ No newline at end of file diff --git a/examples/07_gemm_all_scatter/profiling_results.md b/examples/07_gemm_all_scatter/profiling_results.md new file mode 100644 index 000000000..5c5e19acd --- /dev/null +++ b/examples/07_gemm_all_scatter/profiling_results.md @@ -0,0 +1,131 @@ +# Example 07 – GEMM All-Scatter: IntelliKit Profiling Results + +## System Configuration + +| Property | Value | +|---|---| +| GPU | AMD Instinct MI300X (gfx942) | +| Compute Units | 304 | +| HBM Capacity | 256 GB | +| GPUs Used | 2 | +| Framework | Iris (Triton-based) | + +--- + +## Benchmark Results + +### Small Problem (M=4096, N=4096, K=4096) + +| Metric | Value | +|---|---| +| Kernel | `persistent_gemm_all_scatter` | +| World Size | 2 | +| Local N per rank | 2048 | +| **Performance** | **205.8 TFLOPS** | +| Total Latency | 0.668 ms | +| GEMM Latency | 0.530 ms | +| Tile Config | BLK_M=256, BLK_N=64, BLK_K=64 | +| Total Tiles | 512 | + +### Large Problem (M=8192, N=4608, K=36864) — Default Sizes + +| Metric | Value | +|---|---| +| Kernel | `persistent_gemm_all_scatter` | +| World Size | 2 | +| Local N per rank | 2304 | +| **Performance** | **612.2 TFLOPS** | +| Total Latency | 4.546 ms | +| GEMM Latency | 4.408 ms | +| Tile Config | BLK_M=256, BLK_N=64, BLK_K=64 | +| Total Tiles | 1152 | + +--- + +## IntelliKit Metrix Profiling + +### Memory Profile – Small Problem (M=4096, N=4096, K=4096) + +| Metric | Min | Max | Avg | +|---|---|---|---| +| **Kernel Duration** | 532.4 μs | 541.9 μs | **537.1 μs** | +| HBM Bandwidth Utilization | 1.18% | 1.18% | **1.18%** | +| HBM Read Bandwidth | 53.8 GB/s | 53.9 GB/s | **53.8 GB/s** | +| HBM Write Bandwidth | 8.6 GB/s | 8.8 GB/s | **8.7 GB/s** | +| Total HBM Bytes Transferred | 266 MB | 270 MB | **268 MB** | +| L1 Hit Rate | 66.73% | 66.73% | **66.73%** | +| L2 Hit Rate | 84.19% | 84.22% | **84.20%** | +| L2 Bandwidth | 389 GB/s | 394 GB/s | **392 GB/s** | +| Coalescing Efficiency | 25.0% | 25.0% | **25.0%** | +| Global Load Efficiency | 12.54% | 12.54% | **12.54%** | +| Global Store Efficiency | 6.25% | 6.25% | **6.25%** | +| LDS Bank Conflicts | 0.0 | 0.0 | **0.0** | +| Atomic Latency | 0.0 | 0.0 | **0.0** | + +### Compute Profile – Small Problem (M=4096, N=4096, K=4096) + +| Metric | Min | Max | Avg | +|---|---|---|---| +| **Kernel Duration** | 537.2 μs | 565.3 μs | **547.6 μs** | +| Total FLOPS | 68.7 GFLOPS | 68.7 GFLOPS | **68.7 GFLOPS** | +| Compute Throughput (HBM-normalized) | 121.6 TFLOPS | 127.9 TFLOPS | **125.5 TFLOPS** | +| HBM Arithmetic Intensity | 254.3 FLOP/byte | 258.4 FLOP/byte | **256.9 FLOP/byte** | +| L2 Arithmetic Intensity | 40.9 FLOP/byte | 40.9 FLOP/byte | **40.9 FLOP/byte** | +| L1 Arithmetic Intensity | 17.0 FLOP/byte | 17.0 FLOP/byte | **17.0 FLOP/byte** | + +### Memory Profile – Large Problem (M=8192, N=4608, K=36864) + +| Metric | Min | Max | Avg | +|---|---|---|---| +| **Kernel Duration** | 4609.9 μs | 4731.7 μs | **4683.2 μs** | +| HBM Bandwidth Utilization | 3.40% | 3.43% | **3.41%** | +| HBM Read Bandwidth | 177.2 GB/s | 178.8 GB/s | **177.9 GB/s** | +| HBM Write Bandwidth | 2.72 GB/s | 2.74 GB/s | **2.72 GB/s** | +| Total HBM Bytes Transferred | 5.10 GB | 5.16 GB | **5.13 GB** | +| L1 Hit Rate | 52.65% | 52.65% | **52.65%** | +| L2 Hit Rate | 81.81% | 81.88% | **81.84%** | +| L2 Bandwidth | 975 GB/s | 994 GB/s | **985 GB/s** | +| Coalescing Efficiency | 25.0% | 25.0% | **25.0%** | +| Global Load Efficiency | 12.50% | 12.50% | **12.50%** | +| Global Store Efficiency | 6.25% | 6.25% | **6.25%** | +| LDS Bank Conflicts | 0.0 | 0.0 | **0.0** | +| Atomic Latency | 0.0 | 0.0 | **0.0** | + +### Compute Profile – Large Problem (M=8192, N=4608, K=36864) + +| Metric | Min | Max | Avg | +|---|---|---|---| +| **Kernel Duration** | 5010.5 μs | 5196.3 μs | **5076.4 μs** | +| Total FLOPS | 1.39 TFLOPS | 1.39 TFLOPS | **1.39 TFLOPS** | +| Compute Throughput (HBM-normalized) | 267.8 TFLOPS | 277.7 TFLOPS | **274.1 TFLOPS** | +| HBM Arithmetic Intensity | 271.6 FLOP/byte | 276.1 FLOP/byte | **274.5 FLOP/byte** | +| L2 Arithmetic Intensity | 49.8 FLOP/byte | 49.8 FLOP/byte | **49.8 FLOP/byte** | +| L1 Arithmetic Intensity | 24.2 FLOP/byte | 24.2 FLOP/byte | **24.2 FLOP/byte** | + +--- + +## Analysis & Observations + +### Performance +- The GEMM all-scatter kernel achieves **~612 TFLOPS** at the default large problem size on 2× MI300X GPUs, with communication overhead kept to ~3% of total time (`gemm_ms ≈ 4.41 ms`, `total_ms ≈ 4.55 ms`). +- At small problem sizes (4k³), performance is **~206 TFLOPS**, as expected for memory-bound workloads at this scale. + +### Memory Behavior +- **L2 hit rate is consistently high** (~81–84%), demonstrating effective reuse of data through the L2 cache across the K-dimension loop. +- **L1 hit rate is ~53–67%**, showing moderate reuse; this declines for larger problems as working sets exceed L1 capacity. +- **Coalescing efficiency of 25%** and **global load efficiency of ~12.5%** indicate that memory access patterns could benefit from improved tiling strategies. The 64-element BLK_N × fp16 stores translate to 128-byte transactions, but the scatter pattern for cross-rank writes contributes to suboptimal coalescing. +- **No LDS bank conflicts** and **no atomic latency** detected, confirming efficient shared-memory and atomic usage. +- **HBM bandwidth utilization is low** (1–3%), confirming this kernel is **compute-bound** at these problem sizes. + +### Compute Intensity +- **HBM arithmetic intensity of ~257–274 FLOP/byte** well above the roofline compute-bound threshold for MI300X (~200 FLOP/byte at fp16 peak), confirming **compute-bound** behavior. +- The L2 arithmetic intensity (~41–50 FLOP/byte) and L1 arithmetic intensity (~17–24 FLOP/byte) decrease up the memory hierarchy, as expected. + +### Optimization Opportunities +1. **Coalescing** (25% efficiency): BLK_N=64 with fp16 produces 128-byte writes. Increasing BLK_N or using larger tile sizes may improve coalescing for the scatter writes. +2. **Global store efficiency** (6.25%): The iris remote stores use per-element scatter semantics; batching writes or using larger BLOCK_SIZE_N would increase store granularity. +3. **L1 hit rate decline** at large sizes: Increasing `num_stages` for software pipelining could improve L1 data reuse for larger K dimensions. + +--- + +*Profiled with IntelliKit Metrix on AMD MI300X (gfx942), 2 GPUs*