JoseSantosAMD · JoseSantosAMD · Mar 4, 2026 · Mar 4, 2026 · Mar 4, 2026 · Mar 4, 2026
@@ -0,0 +1,19 @@
+---
+description: Build and run instructions (from skills.md)
+globs: **/build-github-coding-agent-runner.sh,**/run-github-coding-agent-runner.sh,**/start.sh,**/iris.def
+alwaysApply: false
+---
+
+# Build & Run
+
+## Build (SLURM)
+
+From the `github-runner` directory: `sbatch build-github-coding-agent-runner.sh`.
+
+- Partition: `mi3001x`, time limit: 2 hours.
+- Input: `--def=FILE` (default `iris.def`). Output: `--output=FILE` (default `github-copilot-coding-agent-runner.sif`). Same directory.
+- Submit from repo dir so `SLURM_SUBMIT_DIR` is correct. Temp/cache under build dir (`.apptainer-tmp`, `.apptainer-cache`); temp removed after success, cache kept for rebuilds.
+
+## Run
+
+After build: (1) Standalone: `./run-github-coding-agent-runner.sh --github-token=... --github-repository=... --script-dir="$(pwd)" --runner-base="$(pwd)/runner-data"`. (2) SLURM: set `GITHUB_TOKEN` and `GITHUB_REPOSITORY`, then `sbatch run-github-coding-agent-runner.sh` (script uses env and SLURM defaults). See README.md for full setup.
@@ -0,0 +1,25 @@
+---
+description: Workflow and conventions for the GitHub Actions runner (from AGENTS.md)
+alwaysApply: true
+---
+
+# GitHub Runner – Workflow and Conventions
+
+## Workflow
+
+Flow: run-github-coding-agent-runner.sh → container → start.sh → Actions listener. Two run modes: (1) Standalone: `./run-github-coding-agent-runner.sh` with required flags (--github-token, --github-repository, --script-dir, --runner-base). (2) SLURM: set GITHUB_TOKEN and GITHUB_REPOSITORY, then `sbatch run-github-coding-agent-runner.sh`; when under SLURM with no args, the script uses env and SLURM defaults. start.sh installs/configures the runner in RUNNER_HOME and starts the Actions listener.
+
+## Conventions
+
+1. **No sensitive data** – Do not hardcode tokens, passwords, or API keys. Use environment variables (e.g. export GITHUB_TOKEN before running).
+2. **No host-specific paths** – Do not add paths like /work1/amd/josantos/... Prefer SCRIPT_DIR with dirname BASH_SOURCE, or GITHUB_WORKSPACE, RUNNER_WORKDIR, RUNNER_BASE, WORK, or relative paths.
+3. **Do not edit iris.def unless the user explicitly asks.** Prefer changing start.sh or run-github-coding-agent-runner.sh for runtime behavior.
+4. **Use known writable directories** – Prefer GITHUB_WORKSPACE, RUNNER_WORKDIR, RUNNER_BASE for installs and cache. Avoid $HOME, ~, /tmp.
+
+## Directory layout (when running in runner)
+
+| Variable | Use for |
+|----------|---------|
+| GITHUB_WORKSPACE | Repo checkout; installs, cache, venv |
+| RUNNER_WORKDIR | Parent of owner/repo; job work |
+| RUNNER_BASE | Runner data root; overlay, .github-runner |
@@ -17,8 +17,12 @@ When editing scripts or config in this project:
 2. **Never use host-specific absolute paths.**  
    Do not add paths like `/work1/amd/josantos/...` or other machine-specific directories. Prefer:
    - Paths relative to the script (e.g. `SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` then `cd "${SCRIPT_DIR}"`).
-   - Environment variables (e.g. `$WORK`, `$HOME`) when a base directory is needed.
+   - Environment variables: `GITHUB_WORKSPACE`, `RUNNER_WORKDIR`, `RUNNER_BASE`, `SCRIPT_DIR`, `$WORK` when a base directory is needed. Note: `$HOME` is unreliable in containers (K8s, SLURM).
    - Relative paths from the project or script location.
 
 3. **Never edit the container definition file (e.g. `iris.def`) unless explicitly asked.**  
    Prefer changing scripts (e.g. `start.sh`, `run-github-coding-agent-runner.sh`) to install, configure, or run things at runtime. Only modify `.def` files when the user explicitly requests it.
+
+4. **Use known writable directories only.**  
+   Prefer `GITHUB_WORKSPACE`, `RUNNER_WORKDIR`, or `RUNNER_BASE` for installs, cache, and temp files.  
+   Avoid `$HOME`, `~`, and `/tmp`—they may be unwritable or limited (e.g. `/` for `nobody` in K8s, or network paths in SLURM).
@@ -188,6 +188,59 @@ scancel <job_id>
 # Resubmit: either same flags (standalone) or same env then sbatch run-github-coding-agent-runner.sh
 ```
 
+### MCP connection refused (127.0.0.1:2301)
+
+The GitHub Coding Agent (Copilot) may log `Failed to load tools from MCP server: Error: connect ECONNREFUSED 127.0.0.1:2301`. This is expected if no MCP (Model Context Protocol) server is running in the runner environment. The runner does not start an MCP server by default. Workflows still run; only MCP-based tools are unavailable. To use MCP, you would need to start and expose an MCP server in your setup (out of scope of this runner).
+
+### Copilot job stuck on "Waiting for MCP servers to be ready..."
+
+**Symptom:** The GitHub Copilot coding agent job hangs indefinitely on the "Start MCP Servers (Linux)" step, logging `Waiting for MCP servers to be ready...` and never progressing.
+
+**Cause:** The GitHub-managed `start-mcp-servers.sh` health-checks port 2301 using `curl` with no timeout. If a stale MCP server process from a previously cancelled job is still holding port 2301, `curl` connects but never gets an HTTP response — hanging forever.
+
+**How to diagnose:**
+```bash
+# Check what's listening on port 2301
+kubectl exec -n iris-ci <pod> -- sh -c 'grep " 0A " /proc/net/tcp'
+# 0x08FD = 2301. If it shows a listener, find who owns it:
+kubectl exec -n iris-ci <pod> -- sh -c 'grep -l "mcp" /proc/[0-9]*/cmdline 2>/dev/null | while read f; do pid=$(echo $f | cut -d/ -f3); echo "PID $pid: $(cat $f 2>/dev/null | tr "\0" " ")"; done'
+```
+
+**Fix:** Kill the stale MCP processes (only those from the josantos runner path), then re-run the workflow:
+```bash
+kubectl exec -n iris-ci <pod> -- sh -c '
+for pid in $(grep -l "mcp" /proc/[0-9]*/cmdline 2>/dev/null | cut -d/ -f3); do
+  cmd=$(cat /proc/$pid/cmdline 2>/dev/null | tr "\0" " ")
+  if echo "$cmd" | grep -q "josantos"; then
+    kill -9 $pid 2>/dev/null && echo "Killed PID $pid"
+  fi
+done'
+```
+
+**Prevention:** `start.sh` now kills stale MCP processes on runner shutdown via its `cleanup()` trap. This prevents orphaned processes from blocking the next job. If it recurs, check that the runner shutdown cleanly (i.e. the trap ran).
+
+**MCP config note:** Setting `{"mcpServers": {}}` in GitHub Settings → Copilot → Coding agent → MCP configuration does **not** prevent this — GitHub's action always starts `github-mcp-server` and `playwright` as built-in defaults regardless of your config.
+
+### Workflow fails with ModuleNotFoundError (e.g. torch)
+
+If a workflow step (or the Coding Agent) runs `python -c "import torch"` and gets `ModuleNotFoundError: No module named 'torch'`, the runner’s `PATH`/`PYTHONPATH` may not include your container’s Python environment. You can fix it in either place:
+
+**Option A — Set `env` in the workflow (recommended):** Add a job-level `env:` block so every step sees the container’s Python and packages. Example (adjust paths to match your image):
+
+```yaml
+jobs:
+  my-job:
+    runs-on: [self-hosted, copilot]
+    env:
+      ROCM_PATH: /opt/rocm
+      PATH: /opt/rocm/bin:$PATH
+      PYTHONPATH: /opt/triton:$PYTHONPATH
+    steps:
+      - run: python3 -c "import torch; print(torch.cuda.is_available())"
+```
+
+**Option B — Use runner-container.env:** Create `runner-container.env` from `runner-container.env.example` in the script directory and set `PATH`/`PYTHONPATH`/`ROCM_PATH` to match your image. The run script passes it into the container so all jobs get that env without editing each workflow.
+
 ## Security
 
 - **Tokens**: Never commit tokens. Use `--github-token=TOKEN` when running standalone, or set `GITHUB_TOKEN` when using `sbatch run-github-coding-agent-runner.sh`; do not put secrets in committed files.

@@ -25,4 +25,4 @@ export RUNNER_NAME=''
 export RUNNER_LABELS='copilot'
 
 # OPTIONAL: Work directory (defaults to current directory/_work)
-export RUNNER_WORKDIR=""
+export RUNNER_WORKDIR=""
@@ -7,7 +7,7 @@
 #SBATCH --error=github-coding-agent-runner-%j.err
 #SBATCH --time=8:00:00
 #SBATCH --nodes=1
-#SBATCH -p mi3008x  # MI300X partition
+#SBATCH -p mi2104x  # MI300X partition
 
 # Adjust the above SLURM parameters as needed for your system
 #
@@ -101,7 +101,7 @@ if [ -z "$RUNNER_NAME" ]; then
     RUNNER_NAME="${REPO_NAME}-runner-${CLUSTER_NAME}-$(date +%Y%m%d)-$(date +%H%M%S)"
 fi
 RUNNER_LABELS="${RUNNER_LABELS:-copilot}"
-mkdir -p "${RUNNER_WORKDIR}"
+mkdir -p "${RUNNER_WORKDIR}" "${RUNNER_WORKDIR}/.home" "${RUNNER_WORKDIR}/.pip-cache" "${RUNNER_WORKDIR}/.tmp" "${RUNNER_WORKDIR}/.cache"
 [ -n "${USE_OVERLAY}" ] && [ "${USE_OVERLAY}" != "0" ] && mkdir -p "${OVERLAY_DIR}"
 
 echo "=========================================="

@@ -22,12 +22,6 @@ export RUNNER_ALLOW_RUNASROOT=1
 # Triton kernel cache (must be writable). Default in start.sh is RUNNER_WORKDIR/.triton_cache
 # export TRITON_CACHE_DIR="${RUNNER_WORKDIR}/.triton_cache"
 
-# Pip and uv caches (writable). Workflows may override with GITHUB_WORKSPACE; set here to
-# use RUNNER_WORKDIR so pip/uv work without --cache-dir. Uncomment if your container
-# has read-only $HOME (e.g. pip’s default ~/.cache/pip would fail).
-# export PIP_CACHE_DIR="${RUNNER_WORKDIR}/.pip-cache"
-# export UV_CACHE_DIR="${RUNNER_WORKDIR}/.uv-cache"
-
 # Git config when running as root (start.sh sets a default; override if needed)
 # export GIT_CONFIG_GLOBAL="${RUNNER_WORKDIR}/.gitconfig"
 

@@ -49,6 +49,11 @@ fi
 export RUNNER_ALLOW_RUNASROOT="${RUNNER_ALLOW_RUNASROOT:-1}"
 export TRITON_CACHE_DIR="${TRITON_CACHE_DIR:-${RUNNER_WORKDIR}/.triton_cache}"
 
+# Writable HOME/TMPDIR for job steps (run-github-coding-agent-runner.sh may already create dirs on host)
+mkdir -p "${RUNNER_WORKDIR}/.home" "${RUNNER_WORKDIR}/.tmp"
+export HOME="${RUNNER_WORKDIR}/.home"
+export TMPDIR="${RUNNER_WORKDIR}/.tmp"
+
 mkdir -p "${RUNNER_HOME}"
 
 echo "=========================================="
@@ -125,6 +130,11 @@ echo "Configuring runner..."
 
 # Cleanup function
 cleanup() {
+    # Kill any stale MCP processes left over from cancelled jobs
+    pkill -f "mcp/dist/index.js" 2>/dev/null || true
+    pkill -f "mcp-server-playwright" 2>/dev/null || true
+    pkill -f "playwright-mcp" 2>/dev/null || true
+
     # Only run removal once; skip if config already removed
     if [ ! -f "${RUNNER_HOME}/.runner" ]; then
         echo "Runner config already removed or not configured. Skipping cleanup."

@@ -25,9 +25,6 @@ jobs:
       - name: Create task venv for Copilot
         run: |
           cd "$GITHUB_WORKSPACE"
-          mkdir -p "${RUNNER_WORKDIR}/.home" "${RUNNER_WORKDIR}/.tmp"
-          echo "HOME=${RUNNER_WORKDIR}/.home" >> "$GITHUB_ENV"
-          echo "TMPDIR=${RUNNER_WORKDIR}/.tmp" >> "$GITHUB_ENV"
           /opt/venv/bin/python -m venv --system-site-packages .venv
           source .venv/bin/activate
           python -m pip install --upgrade pip
@@ -36,8 +33,7 @@ jobs:
       - name: Install IntelliKit Python packages
         run: |
           source "$GITHUB_WORKSPACE/.venv/bin/activate"
-          # accordo requires kerneldb which needs libdwarf-dev; skipped until added to container
-          # pip install --no-cache-dir "git+https://github.com/AMDResearch/intellikit.git#subdirectory=accordo"
+          pip install --no-cache-dir "git+https://github.com/AMDResearch/intellikit.git#subdirectory=accordo"
           pip install --no-cache-dir "git+https://github.com/AMDResearch/intellikit.git#subdirectory=linex"
           pip install --no-cache-dir "git+https://github.com/AMDResearch/intellikit.git#subdirectory=metrix"
           pip install --no-cache-dir "git+https://github.com/AMDResearch/intellikit.git#subdirectory=nexus"
@@ -62,7 +58,6 @@ jobs:
       - name: Make venv default for subsequent steps
         run: |
           echo "$GITHUB_WORKSPACE/.venv/bin" >> "$GITHUB_PATH"
-          echo "/opt/venv/bin" >> "$GITHUB_PATH"
           echo "PYTHONPATH=$GITHUB_WORKSPACE/.venv/lib/python3.13/site-packages:$GITHUB_WORKSPACE${PYTHONPATH:+:$PYTHONPATH}" >> "$GITHUB_ENV"
 
       - name: Verify ROCm and GPU visibility