Example 07 – GEMM all-scatter: IntelliKit Metrix profiling results

.agents/skills/accordo/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: accordo-validation
+description: Validate GPU kernel correctness by comparing reference and optimized outputs. Use when verifying that an optimized or modified kernel matches a reference implementation.
+---
+
+# Accordo: GPU Kernel Validation
+
+Capture and compare kernel outputs from reference and optimized binaries to validate correctness. Uses kernelDB for automatic kernel extraction; supports configurable tolerance and execution-time comparison.
+
+## When to Use
+
+- User has a reference and an optimized (or modified) GPU kernel and wants to check they produce the same results
+- Regression testing after kernel or build changes
+- Validating multiple optimization variants against one baseline
+
+## Instructions
+
+1. **Require two or more binaries:** one reference (e.g. `./app_ref`) and one or more to validate (e.g. `./app_opt`). All must expose the same kernel by name.
+2. **Ensure binaries are built with debug symbols** (`-g`) so kernel arguments can be extracted.
+3. **Choose execution path:**
+   - If an Accordo MCP server is available, call its `validate_kernel_correctness` tool, which performs capture-and-compare with the same semantics described below.
+   - Otherwise use the Python API from the environment where Accordo is installed.
+
+### Python API
+
+```python
+from accordo import Accordo
+
+# Validator for the kernel to validate (binary used to extract signature)
+validator = Accordo(binary="./app_ref", kernel_name="reduce_sum")
+
+# Optional: set working directory if binaries expect it
+validator = Accordo(binary="./app_ref", kernel_name="reduce_sum", working_directory="./run")
+
+# Capture snapshots
+ref = validator.capture_snapshot(binary="./app_ref")
+opt = validator.capture_snapshot(binary="./app_opt")
+
+# Compare with tolerance (default 1e-6)
+result = validator.compare_snapshots(ref, opt, tolerance=1e-6)
+
+if result.is_valid:
+    print("PASS:", result.num_arrays_validated, "arrays matched")
+else:
+    print(result.summary())
+```
+
+For multiple optimizations, capture the reference once and compare each optimized snapshot against it.
+
+### Snapshot and result attributes
+
+- **Snapshot:** `arrays`, `execution_time_ms`, `grid_size`, `block_size`
+- **ValidationResult:** `is_valid`, `num_arrays_validated`, `num_mismatches`, `mismatches`, `success_rate`; use `summary()` for a human-readable report.
+
+## Workflow
+
+1. Build reference and optimized binaries with the same kernel name and `-g`.
+2. Create an `Accordo(binary=ref_binary, kernel_name="...")` validator; set `working_directory` if needed.
+3. Capture reference snapshot with `capture_snapshot(binary=ref_binary)`.
+4. For each variant, capture with `capture_snapshot(binary=opt_binary)` and compare with `compare_snapshots(ref, opt, tolerance=...)`.
+5. If `result.is_valid` is false, use `result.summary()` and `result.mismatches` to diagnose.
+6. Use relative paths for binaries and working directory so the skill is portable.
+
+## Notes
+
+- kernelDB is used automatically; no separate kernelDB setup is required when using the Python API.
+- Increase `tolerance` for floating-point comparisons when appropriate (e.g. 1e-4 or 1e-5 for single precision).
+- Use `timeout_seconds` in `capture_snapshot` if the run may hang.

.agents/skills/linex/SKILL.md

@@ -0,0 +1,98 @@
+---
+name: linex-profiling
+description: Profile GPU kernels at source-line granularity with cycle-level timing and stall analysis. Use when identifying performance hotspots at the source code level or analyzing instruction-level metrics mapped to source lines.
+---
+
+# Linex: Source-Level GPU Performance Profiling
+
+Map GPU performance metrics to your source code lines. Get cycle-level timing, stall analysis, and instruction-level metrics for each line of source code.
+
+## When to Use
+
+- User asks to profile a GPU application at source-line granularity
+- Need to identify which specific lines of code are performance bottlenecks
+- Analyzing stall patterns and execution bottlenecks at the source level
+- Understanding cycle-level timing for each line of code
+- Instruction-level analysis mapped to source lines
+
+## Instructions
+
+1. **Ensure the target runs on AMD ROCm 7.0+** with `rocprofv3` available.
+2. **Kernels must be compiled with `-g`** (debug symbols) for source mapping.
+3. **Choose execution path:**
+   - If a Linex MCP server is available, use its MCP tools:
+     - `profile_application` to run and profile a target application with the options below.
+     - `analyze_instruction_hotspots` to perform instruction-level hotspot analysis on collected profiles.
+   - Otherwise use the Python API from the environment where Linex is installed.
+
+### Python API
+
+```python
+from linex import Linex
+
+profiler = Linex(
+    target_cu=0,                      # Target compute unit
+    shader_engine_mask="0xFFFFFFFF",  # All shader engines
+    activity=10,                      # Activity counter polling
+)
+
+profiler.profile("./my_app", kernel_filter="my_kernel")
+
+# Show hotspots (sorted by total_cycles)
+for line in profiler.source_lines[:5]:
+    print(f"{line.file}:{line.line_number}")
+    print(f"  {line.total_cycles:,} cycles ({line.stall_percent:.1f}% stalled)")
+    print(f"  Executed {line.execution_count} times")
+
+# Find memory-bound lines
+memory_bound = [
+    l for l in profiler.source_lines 
+    if l.stall_percent > 50
+]
+
+# Instruction-level analysis
+for line in profiler.source_lines[:1]:
+    for inst in line.instructions:
+        print(f"{inst.isa}: {inst.latency_cycles} cycles")
+```
+
+### SourceLine Properties
+
+- `file` - Source file path
+- `line_number` - Line number
+- `total_cycles` - Sum of all instruction cycles
+- `stall_cycles` - Cycles spent waiting
+- `idle_cycles` - Cycles slot was idle
+- `execution_count` - Total executions
+- `instructions` - List of ISA instructions
+- `stall_percent` - Convenience: stall_cycles / total_cycles * 100
+
+### InstructionData Properties
+
+- `isa` - ISA instruction text
+- `latency_cycles` - Total cycles for this instruction
+- `stall_cycles` - Cycles spent waiting
+- `idle_cycles` - Cycles slot was idle
+- `execution_count` - How many times it ran
+- `instruction_address` - Virtual address in GPU memory
+- `file` - Parsed from source_location
+- `line` - Parsed from source_location
+- `stall_percent` - Convenience: stall_cycles / latency_cycles * 100
+
+## Workflow
+
+1. Ensure the target binary is built with `-g` (debug symbols) for source mapping.
+2. Create a `Linex()` profiler; optionally set `target_cu`, `shader_engine_mask`, or `activity`.
+3. Call `profiler.profile(command, kernel_filter=...)` to run profiling.
+4. Access `profiler.source_lines` (sorted by total_cycles) to find hotspots.
+5. Use `line.stall_percent` to identify memory-bound or dependency-bound lines.
+6. Drill down into `line.instructions` for instruction-level analysis.
+7. Use relative paths for the target binary so the skill is portable.
+
+## Notes
+
+- Requires ROCm 7.0+ with `rocprofv3` support.
+- Source mapping requires kernels compiled with `-g` (debug symbols).
+- `source_lines` are automatically sorted by `total_cycles` (descending).
+- Use `kernel_filter` to profile specific kernels by name (regex pattern).
+- For Triton or other frameworks, ensure debug symbols are available in the compiled output.

.agents/skills/metrix/SKILL.md

@@ -0,0 +1,76 @@
+---
+name: metrix-profiling
+description: Profile GPU kernels when performance analysis or optimization is required. Use for AMD ROCm GPU metrics, bandwidth, cache hit rates, coalescing, or kernel timing.
+---
+
+# Metrix: GPU Profiling
+
+Profile AMD GPU kernels and get human-readable metrics (bandwidth, cache, coalescing, FLOPS). Architecture is auto-detected.
+
+## When to Use
+
+- User asks to profile a GPU application or kernel
+- Performance analysis, optimization, or bottleneck investigation
+- Need HBM/L2/L1 bandwidth, hit rates, or compute metrics
+- Need timing-only runs (fast, no hardware counters)
+
+## Instructions
+
+1. **Ensure the target runs on AMD ROCm** (e.g. `hipcc`-built binary or Python script that launches HIP/ROCm kernels).
+2. **Choose execution path:**
+   - If a Metrix MCP server is available, use its profile tool with the same options below.
+   - Otherwise run the CLI or Python API from the environment where Metrix is installed.
+
+### CLI
+
+From the project or install prefix:
+
+```bash
+# Profile with all metrics (auto-detected arch)
+metrix ./my_app
+
+# Time only (fast, no counters)
+metrix --time-only -n 10 ./my_app
+
+# Filter kernels by name
+metrix --kernel matmul ./my_app
+
+# Specific metrics
+metrix --metrics memory.l2_hit_rate,memory.coalescing_efficiency,compute.total_flops ./my_app
+
+# Save to JSON/CSV
+metrix -o results.json ./my_app
+```
+
+Options: `--profile` (quick|memory|compute), `--metrics` (comma-separated), `--time-only`, `--kernel` (substring), `--num-replays`/`-n`, `--output`/`-o`, `--top K`, `--aggregate`, `--log`/`-l` (debug|info|warning|error), `--quiet`.
+
+### Python API
+
+```python
+from metrix import Metrix
+
+profiler = Metrix()
+results = profiler.profile("./my_app", num_replays=5)
+
+for kernel in results.kernels:
+    print(kernel.name, kernel.duration_us.avg)
+    for metric, stats in kernel.metrics.items():
+        print(f"  {metric}: {stats.avg}")
+```
+
+Use `metrics=[...]` for a subset; omit for all metrics. Use `cwd` when the binary expects a specific working directory.
+
+## Workflow
+
+1. Identify the executable or script to profile (e.g. `./app` or `python run_kernels.py`).
+2. If only timing is needed, use `--time-only` for speed.
+3. If full metrics are needed, run `metrix ./app` (or MCP equivalent); optionally restrict with `--kernel` or `--metrics`.
+4. Interpret results: low L2 hit rate, low coalescing, or low HBM utilization suggest optimization targets.
+5. For automation or tooling, use `-o results.json` and parse the JSON output.
+
+## Key Metrics (reference)
+
+- **Memory:** `memory.hbm_bandwidth_utilization`, `memory.l2_hit_rate`, `memory.l1_hit_rate`, `memory.coalescing_efficiency`, `memory.global_load_efficiency`, `memory.lds_bank_conflicts`, `memory.atomic_latency`
+- **Compute:** `compute.total_flops`, `compute.hbm_gflops`, `compute.hbm_arithmetic_intensity`, `compute.l2_arithmetic_intensity`, `compute.l1_arithmetic_intensity`
+
+Use relative paths for the target binary and output files so the skill is portable across environments.

.agents/skills/nexus/SKILL.md

@@ -0,0 +1,74 @@
+---
+name: nexus-trace
+description: Extract GPU kernel assembly and HIP source from HSA packet traces. Use when analyzing what code ran on the GPU, debugging kernel dispatch, or inspecting assembly and source mapping.
+---
+
+# Nexus: HSA Packet Source Code Extractor
+
+Intercepts HSA packets from a running process and extracts, per kernel, assembly and HIP source into a structured trace (e.g. JSON). Use for kernel-level inspection and assembly/source correlation.
+
+## When to Use
+
+- User needs to see which kernels ran and their assembly or HIP source
+- Debugging or analyzing GPU dispatch and code generation
+- Inspecting assembly-to-source mapping for a HIP (or ROCm) application
+
+## Instructions
+
+1. **Ensure the target runs on AMD ROCm** and uses HSA (e.g. HIP application or ROCm runtime).
+2. **Choose execution path:**
+   - If a Nexus MCP server is available, use its tools: `list_kernels` to enumerate kernels in a trace, and `extract_kernel_code` to get assembly and HIP/source mapping (signature, files, lines). See `nexus/nexus/mcp/server.py` for tool parameters and schemas.
+   - Otherwise use the Python API from the environment where Nexus is installed.
+
+### Python API (recommended when no MCP)
+
+```python
+from nexus import Nexus
+
+nexus = Nexus(log_level=1)
+trace = nexus.run(["python", "my_gpu_script.py"])
+
+# Or run a binary:
+# trace = nexus.run(["./my_hip_app"])
+
+for kernel in trace:
+    print(kernel.name, len(kernel.assembly), "instructions")
+    for i, asm_line in enumerate(kernel.assembly, 1):
+        print(f"  {i}. {asm_line}")
+    for line_no, hip_line in zip(kernel.lines or range(1, len(kernel.hip)+1), kernel.hip):
+        print(f"  {line_no}: {hip_line}")
+
+# Access by kernel name
+k = trace["vector_add(float const*, float const*, float*, int)"]
+print(k.assembly, k.hip, k.signature, k.files, k.lines)
+
+# Save/load trace
+trace.save("trace.json")
+loaded = Nexus.load("trace.json")
+```
+
+Set `log_level` (0–4) to control verbosity. Use relative paths for the run command and output file so the skill is portable.
+
+### Environment-based usage (no Python API)
+
+When the process cannot be launched via `nexus.run()`:
+
+1. Set `HSA_TOOLS_LIB` to the Nexus shared library path (e.g. `build/lib/libnexus.so` or the installed path).
+2. Set `NEXUS_OUTPUT_FILE` to the output JSON path.
+3. Set `NEXUS_LOG_LEVEL` (0–4) if needed.
+4. Run the application as usual; it will be traced and the output file will contain the kernel data.
+
+Optional: `NEXUS_EXTRA_SEARCH_PREFIX` (colon-separated) for HIP source search; `TRITON_DISABLE_LINE_INFO=0` for Triton kernel line info.
+
+## Workflow
+
+1. Identify the command that runs the GPU workload (e.g. `python script.py` or `./app`).
+2. If using the Python API: create `Nexus(log_level=...)`, call `nexus.run([...])`, then iterate `trace` and optionally `trace.save(...)`.
+3. If using the env method: set `HSA_TOOLS_LIB` and `NEXUS_OUTPUT_FILE`, then run the app; open the JSON and parse the `kernels` structure.
+4. Use kernel `signature`, `assembly`, `hip`, `files`, and `lines` to analyze what ran and map assembly back to source.
+5. Use relative paths for commands and output files.
+
+## Notes
+
+- Nexus is intended for research/analysis; ensure the target environment has the Nexus library and compatible ROCm/HSA stack.
+- For Triton kernels, enable line info via `TRITON_DISABLE_LINE_INFO=0` when using the Python API.

.github/agents/skills/accordo/SKILL.md

@@ -0,0 +1,68 @@
+---
+name: accordo-validation
+description: Validate GPU kernel correctness by comparing reference and optimized outputs. Use when verifying that an optimized or modified kernel matches a reference implementation.
+---
+
+# Accordo: GPU Kernel Validation
+
+Capture and compare kernel outputs from reference and optimized binaries to validate correctness. Uses kernelDB for automatic kernel extraction; supports configurable tolerance and execution-time comparison.
+
+## When to Use
+
+- User has a reference and an optimized (or modified) GPU kernel and wants to check they produce the same results
+- Regression testing after kernel or build changes
+- Validating multiple optimization variants against one baseline
+
+## Instructions
+
+1. **Require two or more binaries:** one reference (e.g. `./app_ref`) and one or more to validate (e.g. `./app_opt`). All must expose the same kernel by name.
+2. **Ensure binaries are built with debug symbols** (`-g`) so kernel arguments can be extracted.
+3. **Choose execution path:**
+   - If an Accordo MCP server is available, call its `validate_kernel_correctness` tool, which performs capture-and-compare with the same semantics described below.
+   - Otherwise use the Python API from the environment where Accordo is installed.
+
+### Python API
+
+```python
+from accordo import Accordo
+
+# Validator for the kernel to validate (binary used to extract signature)
+validator = Accordo(binary="./app_ref", kernel_name="reduce_sum")
+
+# Optional: set working directory if binaries expect it
+validator = Accordo(binary="./app_ref", kernel_name="reduce_sum", working_directory="./run")
+
+# Capture snapshots
+ref = validator.capture_snapshot(binary="./app_ref")
+opt = validator.capture_snapshot(binary="./app_opt")
+
+# Compare with tolerance (default 1e-6)
+result = validator.compare_snapshots(ref, opt, tolerance=1e-6)
+
+if result.is_valid:
+    print("PASS:", result.num_arrays_validated, "arrays matched")
+else:
+    print(result.summary())
+```
+
+For multiple optimizations, capture the reference once and compare each optimized snapshot against it.
+
+### Snapshot and result attributes
+
+- **Snapshot:** `arrays`, `execution_time_ms`, `grid_size`, `block_size`
+- **ValidationResult:** `is_valid`, `num_arrays_validated`, `num_mismatches`, `mismatches`, `success_rate`; use `summary()` for a human-readable report.
+
+## Workflow
+
+1. Build reference and optimized binaries with the same kernel name and `-g`.
+2. Create an `Accordo(binary=ref_binary, kernel_name="...")` validator; set `working_directory` if needed.
+3. Capture reference snapshot with `capture_snapshot(binary=ref_binary)`.
+4. For each variant, capture with `capture_snapshot(binary=opt_binary)` and compare with `compare_snapshots(ref, opt, tolerance=...)`.
+5. If `result.is_valid` is false, use `result.summary()` and `result.mismatches` to diagnose.
+6. Use relative paths for binaries and working directory so the skill is portable.
+
+## Notes
+
+- kernelDB is used automatically; no separate kernelDB setup is required when using the Python API.
+- Increase `tolerance` for floating-point comparisons when appropriate (e.g. 1e-4 or 1e-5 for single precision).
+- Use `timeout_seconds` in `capture_snapshot` if the run may hang.