diff --git a/.agents/skills/accordo/SKILL.md b/.agents/skills/accordo/SKILL.md new file mode 100644 index 000000000..1917a5d24 --- /dev/null +++ b/.agents/skills/accordo/SKILL.md @@ -0,0 +1,68 @@ +--- +name: accordo-validation +description: Validate GPU kernel correctness by comparing reference and optimized outputs. Use when verifying that an optimized or modified kernel matches a reference implementation. +--- + +# Accordo: GPU Kernel Validation + +Capture and compare kernel outputs from reference and optimized binaries to validate correctness. Uses kernelDB for automatic kernel extraction; supports configurable tolerance and execution-time comparison. + +## When to Use + +- User has a reference and an optimized (or modified) GPU kernel and wants to check they produce the same results +- Regression testing after kernel or build changes +- Validating multiple optimization variants against one baseline + +## Instructions + +1. **Require two or more binaries:** one reference (e.g. `./app_ref`) and one or more to validate (e.g. `./app_opt`). All must expose the same kernel by name. +2. **Ensure binaries are built with debug symbols** (`-g`) so kernel arguments can be extracted. +3. **Choose execution path:** + - If an Accordo MCP server is available, call its `validate_kernel_correctness` tool, which performs capture-and-compare with the same semantics described below. + - Otherwise use the Python API or the `accordo validate` CLI (`accordo validate --help` for flags: `--kernel-name`, `--ref-binary`, `--opt-binary`, `--tolerance`, `--timeout`, `--working-dir`, `--kernel-args`, `--log-level`). + +### Python API + +```python +from accordo import Accordo + +# Validator for the kernel to validate (binary used to extract signature) +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum") + +# Optional: set working directory if binaries expect it +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum", working_directory="./run") + +# Capture snapshots +ref = validator.capture_snapshot(binary="./app_ref") +opt = validator.capture_snapshot(binary="./app_opt") + +# Compare with tolerance (default 1e-6) +result = validator.compare_snapshots(ref, opt, tolerance=1e-6) + +if result.is_valid: + print("PASS:", result.num_arrays_validated, "arrays matched") +else: + print(result.summary()) +``` + +For multiple optimizations, capture the reference once and compare each optimized snapshot against it. + +### Snapshot and result attributes + +- **Snapshot:** `arrays`, `execution_time_ms`, `grid_size`, `block_size` +- **ValidationResult:** `is_valid`, `num_arrays_validated`, `num_mismatches`, `mismatches`, `success_rate`; use `summary()` for a human-readable report. + +## Workflow + +1. Build reference and optimized binaries with the same kernel name and `-g`. +2. Create an `Accordo(binary=ref_binary, kernel_name="...")` validator; set `working_directory` if needed. +3. Capture reference snapshot with `capture_snapshot(binary=ref_binary)`. +4. For each variant, capture with `capture_snapshot(binary=opt_binary)` and compare with `compare_snapshots(ref, opt, tolerance=...)`. +5. If `result.is_valid` is false, use `result.summary()` and `result.mismatches` to diagnose. +6. Use relative paths for binaries and working directory so the skill is portable. + +## Notes + +- kernelDB is used automatically; no separate kernelDB setup is required when using the Python API. +- Increase `tolerance` for floating-point comparisons when appropriate (e.g. 1e-4 or 1e-5 for single precision). +- Use `timeout_seconds` in `capture_snapshot` if the run may hang. diff --git a/.agents/skills/kerncap/SKILL.md b/.agents/skills/kerncap/SKILL.md new file mode 100644 index 000000000..f7d007143 --- /dev/null +++ b/.agents/skills/kerncap/SKILL.md @@ -0,0 +1,131 @@ +--- +name: test-kerncap +description: Test local kerncap changes end-to-end by profiling an application, extracting a kernel, and validating the reproducer. Use when the user asks to test kerncap against any HIP or Triton workload, or wants to validate extraction on a real GPU application. +--- + +# Test kerncap Against an Application + +Test local kerncap changes end-to-end by extracting and validating a kernel from any application. + +## Parameters + +| Parameter | Required | Description | +|-----------|----------|-------------| +| `app_cmd` | **Yes** | Full command to run the application (binary + arguments), e.g. `$WORK/dev/llama.cpp/build/bin/llama-bench -m model.gguf -p 512 -n 32` | +| `conda_env` | No | Conda environment to activate before running commands (e.g. `llama_cpp`). If not provided, use the current environment. | +| `kernel_name` | No | Name of the kernel to extract (e.g. `mul_mat_q`). If not provided, profile the application first and select the top kernel by execution time. | + +## Paths + +| Item | Path | +|------|------| +| kerncap source | `kerncap/` (relative to IntelliKit repo root) | +| Output directory | `/tmp/kerncap-test/` | + +## Environment Setup + +If `conda_env` is provided, activate it before any other step: + +```bash +conda activate +``` + +If already in a different environment, switch explicitly. Do not assume the current shell environment is correct. + +If `conda_env` is not provided, proceed with the current environment as-is. + +## Workflow + +### Step 1: Reinstall kerncap + +Ensure the correct environment is active (if applicable), then uninstall and reinstall to pick up local changes: + +```bash +pip uninstall kerncap -y && pip install kerncap/ +``` + +### Step 2: Profile to identify target kernel + +**If `kernel_name` was provided**: Skip this step and proceed to Step 3. + +**If `kernel_name` was not provided**: Run profiling to discover the top bottleneck kernel: + +```bash +kerncap profile -- +``` + +Select the kernel with the highest total execution time from the profile output. Use its name as `kernel_name` for all subsequent steps. Tell the user which kernel was selected and why. + +**Important**: Use a sufficiently long substring from the profile output as `kernel_name` so that `kerncap extract` matches the intended kernel, not a different instantiation. For example, templated kernels like `mul_mat_q` have many instantiations differing only by template parameters; passing just `mul_mat_q` will capture the first dispatch that matches, which may not be the top-ranked one. Prefer including template parameters in the substring (e.g. `mul_mat_q<(ggml_type)39` instead of `mul_mat_q`). + +### Step 3: Extract the kernel + +```bash +kerncap extract --help +``` + +Use the help output to construct the appropriate `kerncap extract` command for the application. Key flags to determine: + +- `--cmd` — the application command (`app_cmd`) +- `--source-dir` — where the kernel source lives (ask the user if unclear) +- `--output` — `/tmp/kerncap-test/` +- `--language` — `hip` or `triton` depending on the workload +- Any additional flags (`-D` defines, `--dispatch`, etc.) + +**If extraction fails or produces errors**: Stop here and report the full error output. This indicates the local kerncap changes have a bug that needs fixing. + +**If extraction succeeds**: Inspect the output directory for expected files (metadata.json, argument dumps, source files). If the output looks reasonable, proceed to compile and run. + +### Step 4: Compile and run the reproducer + +Navigate to the output directory and build/run the reproducer: + +```bash +cd /tmp/kerncap-test/ +make run +``` + +**If `make run` fails**: Stop here and report the full compiler or runtime error output. This is the primary signal that kerncap generated an incorrect reproducer. + +**If `make run` succeeds**: Proceed to validation. + +### Step 5: Validate the reproducer + +**5a. Smoke test** — confirm baseline replay works: + +```bash +kerncap validate /tmp/kerncap-test/ +``` + +This is a smoke test only (VA-faithful captures). It confirms the replay runs without crashing but does not check numerical correctness. + +**5b. Recompile** — build a baseline HSACO from the unmodified kernel source: + +```bash +cd /tmp/kerncap-test/ +make recompile +``` + +This confirms the VFS-overlay recompile pipeline works. It produces `optimized.hsaco` from the unmodified `kernel_variant.cpp`. + +**If `make recompile` fails**: Stop here and report the error. This indicates an issue with the source finder or VFS overlay generation. + +**5c. Correctness validation** — compare recompiled HSACO against captured baseline: + +```bash +kerncap validate /tmp/kerncap-test/ --hsaco /tmp/kerncap-test//optimized.hsaco +``` + +This runs replay twice (captured HSACO vs recompiled HSACO) and compares outputs byte-for-byte. Since the kernel source is unmodified, they should match exactly. A failure here indicates a recompilation fidelity issue. + +### Step 6: Report results + +Summarize: +- Whether reinstall succeeded +- Whether profiling identified a kernel (if applicable, and which one) +- Whether extraction completed (and any warnings) +- Whether `make run` compiled and executed successfully +- Whether smoke test passed (Step 5a) +- Whether recompile succeeded (Step 5b) +- Whether correctness validation passed (Step 5c) +- Any errors or warnings encountered at each step diff --git a/.agents/skills/linex/SKILL.md b/.agents/skills/linex/SKILL.md new file mode 100644 index 000000000..dca5b7d6d --- /dev/null +++ b/.agents/skills/linex/SKILL.md @@ -0,0 +1,98 @@ +--- +name: linex-profiling +description: Profile GPU kernels at source-line granularity with cycle-level timing and stall analysis. Use when identifying performance hotspots at the source code level or analyzing instruction-level metrics mapped to source lines. +--- + +# Linex: Source-Level GPU Performance Profiling + +Map GPU performance metrics to your source code lines. Get cycle-level timing, stall analysis, and instruction-level metrics for each line of source code. + +## When to Use + +- User asks to profile a GPU application at source-line granularity +- Need to identify which specific lines of code are performance bottlenecks +- Analyzing stall patterns and execution bottlenecks at the source level +- Understanding cycle-level timing for each line of code +- Instruction-level analysis mapped to source lines + +## Instructions + +1. **Ensure the target runs on AMD ROCm 7.0+** with `rocprofv3` available. +2. **Kernels must be compiled with `-g`** (debug symbols) for source mapping. +3. **Choose execution path:** + - If a Linex MCP server is available, use its MCP tools: + - `profile_application` to run and profile a target application with the options below. + - `analyze_instruction_hotspots` to perform instruction-level hotspot analysis on collected profiles. + - Otherwise use the Python API from the environment where Linex is installed. + +### Python API + +```python +from linex import Linex + +profiler = Linex( + target_cu=0, # Target compute unit + shader_engine_mask="0xFFFFFFFF", # All shader engines + activity=10, # Activity counter polling +) + +profiler.profile("./my_app", kernel_filter="my_kernel") + +# Show hotspots (sorted by total_cycles) +for line in profiler.source_lines[:5]: + print(f"{line.file}:{line.line_number}") + print(f" {line.total_cycles:,} cycles ({line.stall_percent:.1f}% stalled)") + print(f" Executed {line.execution_count} times") + +# Find memory-bound lines +memory_bound = [ + l for l in profiler.source_lines + if l.stall_percent > 50 +] + +# Instruction-level analysis +for line in profiler.source_lines[:1]: + for inst in line.instructions: + print(f"{inst.isa}: {inst.latency_cycles} cycles") +``` + +### SourceLine Properties + +- `file` - Source file path +- `line_number` - Line number +- `total_cycles` - Sum of all instruction cycles +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - Total executions +- `instructions` - List of ISA instructions +- `stall_percent` - Convenience: stall_cycles / total_cycles * 100 + +### InstructionData Properties + +- `isa` - ISA instruction text +- `latency_cycles` - Total cycles for this instruction +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - How many times it ran +- `instruction_address` - Virtual address in GPU memory +- `file` - Parsed from source_location +- `line` - Parsed from source_location +- `stall_percent` - Convenience: stall_cycles / latency_cycles * 100 + +## Workflow + +1. Ensure the target binary is built with `-g` (debug symbols) for source mapping. +2. Create a `Linex()` profiler; optionally set `target_cu`, `shader_engine_mask`, or `activity`. +3. Call `profiler.profile(command, kernel_filter=...)` to run profiling. +4. Access `profiler.source_lines` (sorted by total_cycles) to find hotspots. +5. Use `line.stall_percent` to identify memory-bound or dependency-bound lines. +6. Drill down into `line.instructions` for instruction-level analysis. +7. Use relative paths for the target binary so the skill is portable. + +## Notes + +- Requires ROCm 7.0+ with `rocprofv3` support. +- Source mapping requires kernels compiled with `-g` (debug symbols). +- `source_lines` are automatically sorted by `total_cycles` (descending). +- Use `kernel_filter` to profile specific kernels by name (regex pattern). +- For Triton or other frameworks, ensure debug symbols are available in the compiled output. diff --git a/.agents/skills/metrix/SKILL.md b/.agents/skills/metrix/SKILL.md new file mode 100644 index 000000000..969ef6eef --- /dev/null +++ b/.agents/skills/metrix/SKILL.md @@ -0,0 +1,76 @@ +--- +name: metrix-profiling +description: Profile GPU kernels when performance analysis or optimization is required. Use for AMD ROCm GPU metrics, bandwidth, cache hit rates, coalescing, or kernel timing. +--- + +# Metrix: GPU Profiling + +Profile AMD GPU kernels and get human-readable metrics (bandwidth, cache, coalescing, FLOPS). Architecture is auto-detected. + +## When to Use + +- User asks to profile a GPU application or kernel +- Performance analysis, optimization, or bottleneck investigation +- Need HBM/L2/L1 bandwidth, hit rates, or compute metrics +- Need timing-only runs (fast, no hardware counters) + +## Instructions + +1. **Ensure the target runs on AMD ROCm** (e.g. `hipcc`-built binary or Python script that launches HIP/ROCm kernels). +2. **Choose execution path:** + - If a Metrix MCP server is available, use its profile tool with the same options below. + - Otherwise run the CLI or Python API from the environment where Metrix is installed. + +### CLI + +From the project or install prefix: + +```bash +# Profile with all metrics (auto-detected arch) +metrix ./my_app + +# Time only (fast, no counters) +metrix --time-only -n 10 ./my_app + +# Filter kernels by name +metrix --kernel matmul ./my_app + +# Specific metrics +metrix --metrics memory.l2_hit_rate,memory.coalescing_efficiency,compute.total_flops ./my_app + +# Save to JSON/CSV +metrix -o results.json ./my_app +``` + +Options: `--profile`/`-p` (run `metrix list profiles` for names: `quick`, `memory`, `memory_bandwidth`, `memory_cache`, `compute`), `--metrics`/`-m`, `--time-only`, `--kernel`/`-k` (regular expression), `--num-replays`/`-n`, `--output`/`-o`, `--top`, `--aggregate`, `--timeout`, `--no-counters`, `--log`/`-l`, `--quiet`/`-q`. Discovery: `metrix list `, `metrix info `. Note: `metrix list counters` and `metrix info counter ` are not implemented yet (CLI reports “not yet implemented”). + +### Python API + +```python +from metrix import Metrix + +profiler = Metrix() +results = profiler.profile("./my_app", num_replays=5) + +for kernel in results.kernels: + print(kernel.name, kernel.duration_us.avg) + for metric, stats in kernel.metrics.items(): + print(f" {metric}: {stats.avg}") +``` + +Use `metrics=[...]` for a subset; omit for all metrics. Use `cwd` when the binary expects a specific working directory. + +## Workflow + +1. Identify the executable or script to profile (e.g. `./app` or `python run_kernels.py`). +2. If only timing is needed, use `--time-only` for speed. +3. If full metrics are needed, run `metrix ./app` (or MCP equivalent); optionally restrict with `--kernel` or `--metrics`. +4. Interpret results: low L2 hit rate, low coalescing, or low HBM utilization suggest optimization targets. +5. For automation or tooling, use `-o results.json` and parse the JSON output. + +## Key Metrics (reference) + +- **Memory:** `memory.hbm_bandwidth_utilization`, `memory.l2_hit_rate`, `memory.l1_hit_rate`, `memory.coalescing_efficiency`, `memory.global_load_efficiency`, `memory.lds_bank_conflicts`, `memory.atomic_latency` +- **Compute:** `compute.total_flops`, `compute.hbm_gflops`, `compute.hbm_arithmetic_intensity`, `compute.l2_arithmetic_intensity`, `compute.l1_arithmetic_intensity` + +Use relative paths for the target binary and output files so the skill is portable across environments. diff --git a/.agents/skills/nexus/SKILL.md b/.agents/skills/nexus/SKILL.md new file mode 100644 index 000000000..ad714bc4d --- /dev/null +++ b/.agents/skills/nexus/SKILL.md @@ -0,0 +1,74 @@ +--- +name: nexus-trace +description: Extract GPU kernel assembly and HIP source from HSA packet traces. Use when analyzing what code ran on the GPU, debugging kernel dispatch, or inspecting assembly and source mapping. +--- + +# Nexus: HSA Packet Source Code Extractor + +Intercepts HSA packets from a running process and extracts, per kernel, assembly and HIP source into a structured trace (e.g. JSON). Use for kernel-level inspection and assembly/source correlation. + +## When to Use + +- User needs to see which kernels ran and their assembly or HIP source +- Debugging or analyzing GPU dispatch and code generation +- Inspecting assembly-to-source mapping for a HIP (or ROCm) application + +## Instructions + +1. **Ensure the target runs on AMD ROCm** and uses HSA (e.g. HIP application or ROCm runtime). +2. **Choose execution path:** + - If a Nexus MCP server is available, use its tools: `list_kernels` to enumerate kernels in a trace, and `extract_kernel_code` to get assembly and HIP/source mapping (signature, files, lines). See `nexus/nexus/mcp/server.py` for tool parameters and schemas. + - Otherwise use the Python API from the environment where Nexus is installed. + +### Python API (recommended when no MCP) + +```python +from nexus import Nexus + +nexus = Nexus(log_level=1) +trace = nexus.run(["python", "my_gpu_script.py"]) + +# Or run a binary: +# trace = nexus.run(["./my_hip_app"]) + +for kernel in trace: + print(kernel.name, len(kernel.assembly), "instructions") + for i, asm_line in enumerate(kernel.assembly, 1): + print(f" {i}. {asm_line}") + for line_no, hip_line in zip(kernel.lines or range(1, len(kernel.hip)+1), kernel.hip): + print(f" {line_no}: {hip_line}") + +# Access by kernel name +k = trace["vector_add(float const*, float const*, float*, int)"] +print(k.assembly, k.hip, k.signature, k.files, k.lines) + +# Save/load trace +trace.save("trace.json") +loaded = Nexus.load("trace.json") +``` + +Set `log_level` (0–4) to control verbosity. Use relative paths for the run command and output file so the skill is portable. + +### Environment-based usage (no Python API) + +When the process cannot be launched via `nexus.run()`: + +1. Set `HSA_TOOLS_LIB` to the Nexus shared library path (e.g. `build/lib/libnexus.so` or the installed path). +2. Set `NEXUS_OUTPUT_FILE` to the output JSON path. +3. Set `NEXUS_LOG_LEVEL` (0–4) if needed. +4. Run the application as usual; it will be traced and the output file will contain the kernel data. + +Optional: `NEXUS_EXTRA_SEARCH_PREFIX` (colon-separated) for HIP source search; `TRITON_DISABLE_LINE_INFO=0` for Triton kernel line info. + +## Workflow + +1. Identify the command that runs the GPU workload (e.g. `python script.py` or `./app`). +2. If using the Python API: create `Nexus(log_level=...)`, call `nexus.run([...])`, then iterate `trace` and optionally `trace.save(...)`. +3. If using the env method: set `HSA_TOOLS_LIB` and `NEXUS_OUTPUT_FILE`, then run the app; open the JSON and parse the `kernels` structure. +4. Use kernel `signature`, `assembly`, `hip`, `files`, and `lines` to analyze what ran and map assembly back to source. +5. Use relative paths for commands and output files. + +## Notes + +- Nexus is intended for research/analysis; ensure the target environment has the Nexus library and compatible ROCm/HSA stack. +- For Triton kernels, enable line info via `TRITON_DISABLE_LINE_INFO=0` when using the Python API. diff --git a/.github/agents/skills/accordo/SKILL.md b/.github/agents/skills/accordo/SKILL.md new file mode 100644 index 000000000..1917a5d24 --- /dev/null +++ b/.github/agents/skills/accordo/SKILL.md @@ -0,0 +1,68 @@ +--- +name: accordo-validation +description: Validate GPU kernel correctness by comparing reference and optimized outputs. Use when verifying that an optimized or modified kernel matches a reference implementation. +--- + +# Accordo: GPU Kernel Validation + +Capture and compare kernel outputs from reference and optimized binaries to validate correctness. Uses kernelDB for automatic kernel extraction; supports configurable tolerance and execution-time comparison. + +## When to Use + +- User has a reference and an optimized (or modified) GPU kernel and wants to check they produce the same results +- Regression testing after kernel or build changes +- Validating multiple optimization variants against one baseline + +## Instructions + +1. **Require two or more binaries:** one reference (e.g. `./app_ref`) and one or more to validate (e.g. `./app_opt`). All must expose the same kernel by name. +2. **Ensure binaries are built with debug symbols** (`-g`) so kernel arguments can be extracted. +3. **Choose execution path:** + - If an Accordo MCP server is available, call its `validate_kernel_correctness` tool, which performs capture-and-compare with the same semantics described below. + - Otherwise use the Python API or the `accordo validate` CLI (`accordo validate --help` for flags: `--kernel-name`, `--ref-binary`, `--opt-binary`, `--tolerance`, `--timeout`, `--working-dir`, `--kernel-args`, `--log-level`). + +### Python API + +```python +from accordo import Accordo + +# Validator for the kernel to validate (binary used to extract signature) +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum") + +# Optional: set working directory if binaries expect it +validator = Accordo(binary="./app_ref", kernel_name="reduce_sum", working_directory="./run") + +# Capture snapshots +ref = validator.capture_snapshot(binary="./app_ref") +opt = validator.capture_snapshot(binary="./app_opt") + +# Compare with tolerance (default 1e-6) +result = validator.compare_snapshots(ref, opt, tolerance=1e-6) + +if result.is_valid: + print("PASS:", result.num_arrays_validated, "arrays matched") +else: + print(result.summary()) +``` + +For multiple optimizations, capture the reference once and compare each optimized snapshot against it. + +### Snapshot and result attributes + +- **Snapshot:** `arrays`, `execution_time_ms`, `grid_size`, `block_size` +- **ValidationResult:** `is_valid`, `num_arrays_validated`, `num_mismatches`, `mismatches`, `success_rate`; use `summary()` for a human-readable report. + +## Workflow + +1. Build reference and optimized binaries with the same kernel name and `-g`. +2. Create an `Accordo(binary=ref_binary, kernel_name="...")` validator; set `working_directory` if needed. +3. Capture reference snapshot with `capture_snapshot(binary=ref_binary)`. +4. For each variant, capture with `capture_snapshot(binary=opt_binary)` and compare with `compare_snapshots(ref, opt, tolerance=...)`. +5. If `result.is_valid` is false, use `result.summary()` and `result.mismatches` to diagnose. +6. Use relative paths for binaries and working directory so the skill is portable. + +## Notes + +- kernelDB is used automatically; no separate kernelDB setup is required when using the Python API. +- Increase `tolerance` for floating-point comparisons when appropriate (e.g. 1e-4 or 1e-5 for single precision). +- Use `timeout_seconds` in `capture_snapshot` if the run may hang. diff --git a/.github/agents/skills/kerncap/SKILL.md b/.github/agents/skills/kerncap/SKILL.md new file mode 100644 index 000000000..f7d007143 --- /dev/null +++ b/.github/agents/skills/kerncap/SKILL.md @@ -0,0 +1,131 @@ +--- +name: test-kerncap +description: Test local kerncap changes end-to-end by profiling an application, extracting a kernel, and validating the reproducer. Use when the user asks to test kerncap against any HIP or Triton workload, or wants to validate extraction on a real GPU application. +--- + +# Test kerncap Against an Application + +Test local kerncap changes end-to-end by extracting and validating a kernel from any application. + +## Parameters + +| Parameter | Required | Description | +|-----------|----------|-------------| +| `app_cmd` | **Yes** | Full command to run the application (binary + arguments), e.g. `$WORK/dev/llama.cpp/build/bin/llama-bench -m model.gguf -p 512 -n 32` | +| `conda_env` | No | Conda environment to activate before running commands (e.g. `llama_cpp`). If not provided, use the current environment. | +| `kernel_name` | No | Name of the kernel to extract (e.g. `mul_mat_q`). If not provided, profile the application first and select the top kernel by execution time. | + +## Paths + +| Item | Path | +|------|------| +| kerncap source | `kerncap/` (relative to IntelliKit repo root) | +| Output directory | `/tmp/kerncap-test/` | + +## Environment Setup + +If `conda_env` is provided, activate it before any other step: + +```bash +conda activate +``` + +If already in a different environment, switch explicitly. Do not assume the current shell environment is correct. + +If `conda_env` is not provided, proceed with the current environment as-is. + +## Workflow + +### Step 1: Reinstall kerncap + +Ensure the correct environment is active (if applicable), then uninstall and reinstall to pick up local changes: + +```bash +pip uninstall kerncap -y && pip install kerncap/ +``` + +### Step 2: Profile to identify target kernel + +**If `kernel_name` was provided**: Skip this step and proceed to Step 3. + +**If `kernel_name` was not provided**: Run profiling to discover the top bottleneck kernel: + +```bash +kerncap profile -- +``` + +Select the kernel with the highest total execution time from the profile output. Use its name as `kernel_name` for all subsequent steps. Tell the user which kernel was selected and why. + +**Important**: Use a sufficiently long substring from the profile output as `kernel_name` so that `kerncap extract` matches the intended kernel, not a different instantiation. For example, templated kernels like `mul_mat_q` have many instantiations differing only by template parameters; passing just `mul_mat_q` will capture the first dispatch that matches, which may not be the top-ranked one. Prefer including template parameters in the substring (e.g. `mul_mat_q<(ggml_type)39` instead of `mul_mat_q`). + +### Step 3: Extract the kernel + +```bash +kerncap extract --help +``` + +Use the help output to construct the appropriate `kerncap extract` command for the application. Key flags to determine: + +- `--cmd` — the application command (`app_cmd`) +- `--source-dir` — where the kernel source lives (ask the user if unclear) +- `--output` — `/tmp/kerncap-test/` +- `--language` — `hip` or `triton` depending on the workload +- Any additional flags (`-D` defines, `--dispatch`, etc.) + +**If extraction fails or produces errors**: Stop here and report the full error output. This indicates the local kerncap changes have a bug that needs fixing. + +**If extraction succeeds**: Inspect the output directory for expected files (metadata.json, argument dumps, source files). If the output looks reasonable, proceed to compile and run. + +### Step 4: Compile and run the reproducer + +Navigate to the output directory and build/run the reproducer: + +```bash +cd /tmp/kerncap-test/ +make run +``` + +**If `make run` fails**: Stop here and report the full compiler or runtime error output. This is the primary signal that kerncap generated an incorrect reproducer. + +**If `make run` succeeds**: Proceed to validation. + +### Step 5: Validate the reproducer + +**5a. Smoke test** — confirm baseline replay works: + +```bash +kerncap validate /tmp/kerncap-test/ +``` + +This is a smoke test only (VA-faithful captures). It confirms the replay runs without crashing but does not check numerical correctness. + +**5b. Recompile** — build a baseline HSACO from the unmodified kernel source: + +```bash +cd /tmp/kerncap-test/ +make recompile +``` + +This confirms the VFS-overlay recompile pipeline works. It produces `optimized.hsaco` from the unmodified `kernel_variant.cpp`. + +**If `make recompile` fails**: Stop here and report the error. This indicates an issue with the source finder or VFS overlay generation. + +**5c. Correctness validation** — compare recompiled HSACO against captured baseline: + +```bash +kerncap validate /tmp/kerncap-test/ --hsaco /tmp/kerncap-test//optimized.hsaco +``` + +This runs replay twice (captured HSACO vs recompiled HSACO) and compares outputs byte-for-byte. Since the kernel source is unmodified, they should match exactly. A failure here indicates a recompilation fidelity issue. + +### Step 6: Report results + +Summarize: +- Whether reinstall succeeded +- Whether profiling identified a kernel (if applicable, and which one) +- Whether extraction completed (and any warnings) +- Whether `make run` compiled and executed successfully +- Whether smoke test passed (Step 5a) +- Whether recompile succeeded (Step 5b) +- Whether correctness validation passed (Step 5c) +- Any errors or warnings encountered at each step diff --git a/.github/agents/skills/linex/SKILL.md b/.github/agents/skills/linex/SKILL.md new file mode 100644 index 000000000..dca5b7d6d --- /dev/null +++ b/.github/agents/skills/linex/SKILL.md @@ -0,0 +1,98 @@ +--- +name: linex-profiling +description: Profile GPU kernels at source-line granularity with cycle-level timing and stall analysis. Use when identifying performance hotspots at the source code level or analyzing instruction-level metrics mapped to source lines. +--- + +# Linex: Source-Level GPU Performance Profiling + +Map GPU performance metrics to your source code lines. Get cycle-level timing, stall analysis, and instruction-level metrics for each line of source code. + +## When to Use + +- User asks to profile a GPU application at source-line granularity +- Need to identify which specific lines of code are performance bottlenecks +- Analyzing stall patterns and execution bottlenecks at the source level +- Understanding cycle-level timing for each line of code +- Instruction-level analysis mapped to source lines + +## Instructions + +1. **Ensure the target runs on AMD ROCm 7.0+** with `rocprofv3` available. +2. **Kernels must be compiled with `-g`** (debug symbols) for source mapping. +3. **Choose execution path:** + - If a Linex MCP server is available, use its MCP tools: + - `profile_application` to run and profile a target application with the options below. + - `analyze_instruction_hotspots` to perform instruction-level hotspot analysis on collected profiles. + - Otherwise use the Python API from the environment where Linex is installed. + +### Python API + +```python +from linex import Linex + +profiler = Linex( + target_cu=0, # Target compute unit + shader_engine_mask="0xFFFFFFFF", # All shader engines + activity=10, # Activity counter polling +) + +profiler.profile("./my_app", kernel_filter="my_kernel") + +# Show hotspots (sorted by total_cycles) +for line in profiler.source_lines[:5]: + print(f"{line.file}:{line.line_number}") + print(f" {line.total_cycles:,} cycles ({line.stall_percent:.1f}% stalled)") + print(f" Executed {line.execution_count} times") + +# Find memory-bound lines +memory_bound = [ + l for l in profiler.source_lines + if l.stall_percent > 50 +] + +# Instruction-level analysis +for line in profiler.source_lines[:1]: + for inst in line.instructions: + print(f"{inst.isa}: {inst.latency_cycles} cycles") +``` + +### SourceLine Properties + +- `file` - Source file path +- `line_number` - Line number +- `total_cycles` - Sum of all instruction cycles +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - Total executions +- `instructions` - List of ISA instructions +- `stall_percent` - Convenience: stall_cycles / total_cycles * 100 + +### InstructionData Properties + +- `isa` - ISA instruction text +- `latency_cycles` - Total cycles for this instruction +- `stall_cycles` - Cycles spent waiting +- `idle_cycles` - Cycles slot was idle +- `execution_count` - How many times it ran +- `instruction_address` - Virtual address in GPU memory +- `file` - Parsed from source_location +- `line` - Parsed from source_location +- `stall_percent` - Convenience: stall_cycles / latency_cycles * 100 + +## Workflow + +1. Ensure the target binary is built with `-g` (debug symbols) for source mapping. +2. Create a `Linex()` profiler; optionally set `target_cu`, `shader_engine_mask`, or `activity`. +3. Call `profiler.profile(command, kernel_filter=...)` to run profiling. +4. Access `profiler.source_lines` (sorted by total_cycles) to find hotspots. +5. Use `line.stall_percent` to identify memory-bound or dependency-bound lines. +6. Drill down into `line.instructions` for instruction-level analysis. +7. Use relative paths for the target binary so the skill is portable. + +## Notes + +- Requires ROCm 7.0+ with `rocprofv3` support. +- Source mapping requires kernels compiled with `-g` (debug symbols). +- `source_lines` are automatically sorted by `total_cycles` (descending). +- Use `kernel_filter` to profile specific kernels by name (regex pattern). +- For Triton or other frameworks, ensure debug symbols are available in the compiled output. diff --git a/.github/agents/skills/metrix/SKILL.md b/.github/agents/skills/metrix/SKILL.md new file mode 100644 index 000000000..969ef6eef --- /dev/null +++ b/.github/agents/skills/metrix/SKILL.md @@ -0,0 +1,76 @@ +--- +name: metrix-profiling +description: Profile GPU kernels when performance analysis or optimization is required. Use for AMD ROCm GPU metrics, bandwidth, cache hit rates, coalescing, or kernel timing. +--- + +# Metrix: GPU Profiling + +Profile AMD GPU kernels and get human-readable metrics (bandwidth, cache, coalescing, FLOPS). Architecture is auto-detected. + +## When to Use + +- User asks to profile a GPU application or kernel +- Performance analysis, optimization, or bottleneck investigation +- Need HBM/L2/L1 bandwidth, hit rates, or compute metrics +- Need timing-only runs (fast, no hardware counters) + +## Instructions + +1. **Ensure the target runs on AMD ROCm** (e.g. `hipcc`-built binary or Python script that launches HIP/ROCm kernels). +2. **Choose execution path:** + - If a Metrix MCP server is available, use its profile tool with the same options below. + - Otherwise run the CLI or Python API from the environment where Metrix is installed. + +### CLI + +From the project or install prefix: + +```bash +# Profile with all metrics (auto-detected arch) +metrix ./my_app + +# Time only (fast, no counters) +metrix --time-only -n 10 ./my_app + +# Filter kernels by name +metrix --kernel matmul ./my_app + +# Specific metrics +metrix --metrics memory.l2_hit_rate,memory.coalescing_efficiency,compute.total_flops ./my_app + +# Save to JSON/CSV +metrix -o results.json ./my_app +``` + +Options: `--profile`/`-p` (run `metrix list profiles` for names: `quick`, `memory`, `memory_bandwidth`, `memory_cache`, `compute`), `--metrics`/`-m`, `--time-only`, `--kernel`/`-k` (regular expression), `--num-replays`/`-n`, `--output`/`-o`, `--top`, `--aggregate`, `--timeout`, `--no-counters`, `--log`/`-l`, `--quiet`/`-q`. Discovery: `metrix list `, `metrix info `. Note: `metrix list counters` and `metrix info counter ` are not implemented yet (CLI reports “not yet implemented”). + +### Python API + +```python +from metrix import Metrix + +profiler = Metrix() +results = profiler.profile("./my_app", num_replays=5) + +for kernel in results.kernels: + print(kernel.name, kernel.duration_us.avg) + for metric, stats in kernel.metrics.items(): + print(f" {metric}: {stats.avg}") +``` + +Use `metrics=[...]` for a subset; omit for all metrics. Use `cwd` when the binary expects a specific working directory. + +## Workflow + +1. Identify the executable or script to profile (e.g. `./app` or `python run_kernels.py`). +2. If only timing is needed, use `--time-only` for speed. +3. If full metrics are needed, run `metrix ./app` (or MCP equivalent); optionally restrict with `--kernel` or `--metrics`. +4. Interpret results: low L2 hit rate, low coalescing, or low HBM utilization suggest optimization targets. +5. For automation or tooling, use `-o results.json` and parse the JSON output. + +## Key Metrics (reference) + +- **Memory:** `memory.hbm_bandwidth_utilization`, `memory.l2_hit_rate`, `memory.l1_hit_rate`, `memory.coalescing_efficiency`, `memory.global_load_efficiency`, `memory.lds_bank_conflicts`, `memory.atomic_latency` +- **Compute:** `compute.total_flops`, `compute.hbm_gflops`, `compute.hbm_arithmetic_intensity`, `compute.l2_arithmetic_intensity`, `compute.l1_arithmetic_intensity` + +Use relative paths for the target binary and output files so the skill is portable across environments. diff --git a/.github/agents/skills/nexus/SKILL.md b/.github/agents/skills/nexus/SKILL.md new file mode 100644 index 000000000..ad714bc4d --- /dev/null +++ b/.github/agents/skills/nexus/SKILL.md @@ -0,0 +1,74 @@ +--- +name: nexus-trace +description: Extract GPU kernel assembly and HIP source from HSA packet traces. Use when analyzing what code ran on the GPU, debugging kernel dispatch, or inspecting assembly and source mapping. +--- + +# Nexus: HSA Packet Source Code Extractor + +Intercepts HSA packets from a running process and extracts, per kernel, assembly and HIP source into a structured trace (e.g. JSON). Use for kernel-level inspection and assembly/source correlation. + +## When to Use + +- User needs to see which kernels ran and their assembly or HIP source +- Debugging or analyzing GPU dispatch and code generation +- Inspecting assembly-to-source mapping for a HIP (or ROCm) application + +## Instructions + +1. **Ensure the target runs on AMD ROCm** and uses HSA (e.g. HIP application or ROCm runtime). +2. **Choose execution path:** + - If a Nexus MCP server is available, use its tools: `list_kernels` to enumerate kernels in a trace, and `extract_kernel_code` to get assembly and HIP/source mapping (signature, files, lines). See `nexus/nexus/mcp/server.py` for tool parameters and schemas. + - Otherwise use the Python API from the environment where Nexus is installed. + +### Python API (recommended when no MCP) + +```python +from nexus import Nexus + +nexus = Nexus(log_level=1) +trace = nexus.run(["python", "my_gpu_script.py"]) + +# Or run a binary: +# trace = nexus.run(["./my_hip_app"]) + +for kernel in trace: + print(kernel.name, len(kernel.assembly), "instructions") + for i, asm_line in enumerate(kernel.assembly, 1): + print(f" {i}. {asm_line}") + for line_no, hip_line in zip(kernel.lines or range(1, len(kernel.hip)+1), kernel.hip): + print(f" {line_no}: {hip_line}") + +# Access by kernel name +k = trace["vector_add(float const*, float const*, float*, int)"] +print(k.assembly, k.hip, k.signature, k.files, k.lines) + +# Save/load trace +trace.save("trace.json") +loaded = Nexus.load("trace.json") +``` + +Set `log_level` (0–4) to control verbosity. Use relative paths for the run command and output file so the skill is portable. + +### Environment-based usage (no Python API) + +When the process cannot be launched via `nexus.run()`: + +1. Set `HSA_TOOLS_LIB` to the Nexus shared library path (e.g. `build/lib/libnexus.so` or the installed path). +2. Set `NEXUS_OUTPUT_FILE` to the output JSON path. +3. Set `NEXUS_LOG_LEVEL` (0–4) if needed. +4. Run the application as usual; it will be traced and the output file will contain the kernel data. + +Optional: `NEXUS_EXTRA_SEARCH_PREFIX` (colon-separated) for HIP source search; `TRITON_DISABLE_LINE_INFO=0` for Triton kernel line info. + +## Workflow + +1. Identify the command that runs the GPU workload (e.g. `python script.py` or `./app`). +2. If using the Python API: create `Nexus(log_level=...)`, call `nexus.run([...])`, then iterate `trace` and optionally `trace.save(...)`. +3. If using the env method: set `HSA_TOOLS_LIB` and `NEXUS_OUTPUT_FILE`, then run the app; open the JSON and parse the `kernels` structure. +4. Use kernel `signature`, `assembly`, `hip`, `files`, and `lines` to analyze what ran and map assembly back to source. +5. Use relative paths for commands and output files. + +## Notes + +- Nexus is intended for research/analysis; ensure the target environment has the Nexus library and compatible ROCm/HSA stack. +- For Triton kernels, enable line info via `TRITON_DISABLE_LINE_INFO=0` when using the Python API. diff --git a/.intellikit b/.intellikit new file mode 160000 index 000000000..73033fc0a --- /dev/null +++ b/.intellikit @@ -0,0 +1 @@ +Subproject commit 73033fc0a50d47607070c4232747690cafd0544b