diff --git a/.github/scripts/github-runner-files/.cursor/rules/github-runner-build-run.mdc b/.github/scripts/github-runner-files/.cursor/rules/github-runner-build-run.mdc new file mode 100644 index 000000000..69a4cee8c --- /dev/null +++ b/.github/scripts/github-runner-files/.cursor/rules/github-runner-build-run.mdc @@ -0,0 +1,19 @@ +--- +description: Build and run instructions (from skills.md) +globs: **/build-github-coding-agent-runner.sh,**/run-github-coding-agent-runner.sh,**/start.sh,**/iris.def +alwaysApply: false +--- + +# Build & Run + +## Build (SLURM) + +From the `github-runner` directory: `sbatch build-github-coding-agent-runner.sh`. + +- Partition: `mi3001x`, time limit: 2 hours. +- Input: `--def=FILE` (default `iris.def`). Output: `--output=FILE` (default `github-copilot-coding-agent-runner.sif`). Same directory. +- Submit from repo dir so `SLURM_SUBMIT_DIR` is correct. Temp/cache under build dir (`.apptainer-tmp`, `.apptainer-cache`); temp removed after success, cache kept for rebuilds. + +## Run + +After build: (1) Standalone: `./run-github-coding-agent-runner.sh --github-token=... --github-repository=... --script-dir="$(pwd)" --runner-base="$(pwd)/runner-data"`. (2) SLURM: set `GITHUB_TOKEN` and `GITHUB_REPOSITORY`, then `sbatch run-github-coding-agent-runner.sh` (script uses env and SLURM defaults). See README.md for full setup. diff --git a/.github/scripts/github-runner-files/.cursor/rules/github-runner-conventions.mdc b/.github/scripts/github-runner-files/.cursor/rules/github-runner-conventions.mdc new file mode 100644 index 000000000..860cd02b0 --- /dev/null +++ b/.github/scripts/github-runner-files/.cursor/rules/github-runner-conventions.mdc @@ -0,0 +1,25 @@ +--- +description: Workflow and conventions for the GitHub Actions runner (from AGENTS.md) +alwaysApply: true +--- + +# GitHub Runner – Workflow and Conventions + +## Workflow + +Flow: run-github-coding-agent-runner.sh → container → start.sh → Actions listener. Two run modes: (1) Standalone: `./run-github-coding-agent-runner.sh` with required flags (--github-token, --github-repository, --script-dir, --runner-base). (2) SLURM: set GITHUB_TOKEN and GITHUB_REPOSITORY, then `sbatch run-github-coding-agent-runner.sh`; when under SLURM with no args, the script uses env and SLURM defaults. start.sh installs/configures the runner in RUNNER_HOME and starts the Actions listener. + +## Conventions + +1. **No sensitive data** – Do not hardcode tokens, passwords, or API keys. Use environment variables (e.g. export GITHUB_TOKEN before running). +2. **No host-specific paths** – Do not add paths like /work1/amd/josantos/... Prefer SCRIPT_DIR with dirname BASH_SOURCE, or GITHUB_WORKSPACE, RUNNER_WORKDIR, RUNNER_BASE, WORK, or relative paths. +3. **Do not edit iris.def unless the user explicitly asks.** Prefer changing start.sh or run-github-coding-agent-runner.sh for runtime behavior. +4. **Use known writable directories** – Prefer GITHUB_WORKSPACE, RUNNER_WORKDIR, RUNNER_BASE for installs and cache. Avoid $HOME, ~, /tmp. + +## Directory layout (when running in runner) + +| Variable | Use for | +|----------|---------| +| GITHUB_WORKSPACE | Repo checkout; installs, cache, venv | +| RUNNER_WORKDIR | Parent of owner/repo; job work | +| RUNNER_BASE | Runner data root; overlay, .github-runner | diff --git a/.github/scripts/github-runner-files/AGENTS.md b/.github/scripts/github-runner-files/AGENTS.md index 2c91c0a4f..d3fb3a92c 100644 --- a/.github/scripts/github-runner-files/AGENTS.md +++ b/.github/scripts/github-runner-files/AGENTS.md @@ -17,8 +17,12 @@ When editing scripts or config in this project: 2. **Never use host-specific absolute paths.** Do not add paths like `/work1/amd/josantos/...` or other machine-specific directories. Prefer: - Paths relative to the script (e.g. `SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` then `cd "${SCRIPT_DIR}"`). - - Environment variables (e.g. `$WORK`, `$HOME`) when a base directory is needed. + - Environment variables: `GITHUB_WORKSPACE`, `RUNNER_WORKDIR`, `RUNNER_BASE`, `SCRIPT_DIR`, `$WORK` when a base directory is needed. Note: `$HOME` is unreliable in containers (K8s, SLURM). - Relative paths from the project or script location. 3. **Never edit the container definition file (e.g. `iris.def`) unless explicitly asked.** Prefer changing scripts (e.g. `start.sh`, `run-github-coding-agent-runner.sh`) to install, configure, or run things at runtime. Only modify `.def` files when the user explicitly requests it. + +4. **Use known writable directories only.** + Prefer `GITHUB_WORKSPACE`, `RUNNER_WORKDIR`, or `RUNNER_BASE` for installs, cache, and temp files. + Avoid `$HOME`, `~`, and `/tmp`—they may be unwritable or limited (e.g. `/` for `nobody` in K8s, or network paths in SLURM). diff --git a/.github/scripts/github-runner-files/README.md b/.github/scripts/github-runner-files/README.md index 8850fe52d..40ae79f90 100644 --- a/.github/scripts/github-runner-files/README.md +++ b/.github/scripts/github-runner-files/README.md @@ -188,6 +188,59 @@ scancel # Resubmit: either same flags (standalone) or same env then sbatch run-github-coding-agent-runner.sh ``` +### MCP connection refused (127.0.0.1:2301) + +The GitHub Coding Agent (Copilot) may log `Failed to load tools from MCP server: Error: connect ECONNREFUSED 127.0.0.1:2301`. This is expected if no MCP (Model Context Protocol) server is running in the runner environment. The runner does not start an MCP server by default. Workflows still run; only MCP-based tools are unavailable. To use MCP, you would need to start and expose an MCP server in your setup (out of scope of this runner). + +### Copilot job stuck on "Waiting for MCP servers to be ready..." + +**Symptom:** The GitHub Copilot coding agent job hangs indefinitely on the "Start MCP Servers (Linux)" step, logging `Waiting for MCP servers to be ready...` and never progressing. + +**Cause:** The GitHub-managed `start-mcp-servers.sh` health-checks port 2301 using `curl` with no timeout. If a stale MCP server process from a previously cancelled job is still holding port 2301, `curl` connects but never gets an HTTP response — hanging forever. + +**How to diagnose:** +```bash +# Check what's listening on port 2301 +kubectl exec -n iris-ci -- sh -c 'grep " 0A " /proc/net/tcp' +# 0x08FD = 2301. If it shows a listener, find who owns it: +kubectl exec -n iris-ci -- sh -c 'grep -l "mcp" /proc/[0-9]*/cmdline 2>/dev/null | while read f; do pid=$(echo $f | cut -d/ -f3); echo "PID $pid: $(cat $f 2>/dev/null | tr "\0" " ")"; done' +``` + +**Fix:** Kill the stale MCP processes (only those from the josantos runner path), then re-run the workflow: +```bash +kubectl exec -n iris-ci -- sh -c ' +for pid in $(grep -l "mcp" /proc/[0-9]*/cmdline 2>/dev/null | cut -d/ -f3); do + cmd=$(cat /proc/$pid/cmdline 2>/dev/null | tr "\0" " ") + if echo "$cmd" | grep -q "josantos"; then + kill -9 $pid 2>/dev/null && echo "Killed PID $pid" + fi +done' +``` + +**Prevention:** `start.sh` now kills stale MCP processes on runner shutdown via its `cleanup()` trap. This prevents orphaned processes from blocking the next job. If it recurs, check that the runner shutdown cleanly (i.e. the trap ran). + +**MCP config note:** Setting `{"mcpServers": {}}` in GitHub Settings → Copilot → Coding agent → MCP configuration does **not** prevent this — GitHub's action always starts `github-mcp-server` and `playwright` as built-in defaults regardless of your config. + +### Workflow fails with ModuleNotFoundError (e.g. torch) + +If a workflow step (or the Coding Agent) runs `python -c "import torch"` and gets `ModuleNotFoundError: No module named 'torch'`, the runner’s `PATH`/`PYTHONPATH` may not include your container’s Python environment. You can fix it in either place: + +**Option A — Set `env` in the workflow (recommended):** Add a job-level `env:` block so every step sees the container’s Python and packages. Example (adjust paths to match your image): + +```yaml +jobs: + my-job: + runs-on: [self-hosted, copilot] + env: + ROCM_PATH: /opt/rocm + PATH: /opt/rocm/bin:$PATH + PYTHONPATH: /opt/triton:$PYTHONPATH + steps: + - run: python3 -c "import torch; print(torch.cuda.is_available())" +``` + +**Option B — Use runner-container.env:** Create `runner-container.env` from `runner-container.env.example` in the script directory and set `PATH`/`PYTHONPATH`/`ROCM_PATH` to match your image. The run script passes it into the container so all jobs get that env without editing each workflow. + ## Security - **Tokens**: Never commit tokens. Use `--github-token=TOKEN` when running standalone, or set `GITHUB_TOKEN` when using `sbatch run-github-coding-agent-runner.sh`; do not put secrets in committed files. diff --git a/.github/scripts/github-runner-files/env.example b/.github/scripts/github-runner-files/env.example index 76de21881..4357ec03d 100644 --- a/.github/scripts/github-runner-files/env.example +++ b/.github/scripts/github-runner-files/env.example @@ -25,4 +25,4 @@ export RUNNER_NAME='' export RUNNER_LABELS='copilot' # OPTIONAL: Work directory (defaults to current directory/_work) -export RUNNER_WORKDIR="" +export RUNNER_WORKDIR="" \ No newline at end of file diff --git a/.github/scripts/github-runner-files/run-github-coding-agent-runner.sh b/.github/scripts/github-runner-files/run-github-coding-agent-runner.sh old mode 100755 new mode 100644 index 291b4d39c..63802ec38 --- a/.github/scripts/github-runner-files/run-github-coding-agent-runner.sh +++ b/.github/scripts/github-runner-files/run-github-coding-agent-runner.sh @@ -7,7 +7,7 @@ #SBATCH --error=github-coding-agent-runner-%j.err #SBATCH --time=8:00:00 #SBATCH --nodes=1 -#SBATCH -p mi3008x # MI300X partition +#SBATCH -p mi2104x # MI300X partition # Adjust the above SLURM parameters as needed for your system # @@ -101,7 +101,7 @@ if [ -z "$RUNNER_NAME" ]; then RUNNER_NAME="${REPO_NAME}-runner-${CLUSTER_NAME}-$(date +%Y%m%d)-$(date +%H%M%S)" fi RUNNER_LABELS="${RUNNER_LABELS:-copilot}" -mkdir -p "${RUNNER_WORKDIR}" +mkdir -p "${RUNNER_WORKDIR}" "${RUNNER_WORKDIR}/.home" "${RUNNER_WORKDIR}/.pip-cache" "${RUNNER_WORKDIR}/.tmp" "${RUNNER_WORKDIR}/.cache" [ -n "${USE_OVERLAY}" ] && [ "${USE_OVERLAY}" != "0" ] && mkdir -p "${OVERLAY_DIR}" echo "==========================================" diff --git a/.github/scripts/github-runner-files/runner-container.env.example b/.github/scripts/github-runner-files/runner-container.env.example index c4304c8f1..169f44b15 100644 --- a/.github/scripts/github-runner-files/runner-container.env.example +++ b/.github/scripts/github-runner-files/runner-container.env.example @@ -22,12 +22,6 @@ export RUNNER_ALLOW_RUNASROOT=1 # Triton kernel cache (must be writable). Default in start.sh is RUNNER_WORKDIR/.triton_cache # export TRITON_CACHE_DIR="${RUNNER_WORKDIR}/.triton_cache" -# Pip and uv caches (writable). Workflows may override with GITHUB_WORKSPACE; set here to -# use RUNNER_WORKDIR so pip/uv work without --cache-dir. Uncomment if your container -# has read-only $HOME (e.g. pip’s default ~/.cache/pip would fail). -# export PIP_CACHE_DIR="${RUNNER_WORKDIR}/.pip-cache" -# export UV_CACHE_DIR="${RUNNER_WORKDIR}/.uv-cache" - # Git config when running as root (start.sh sets a default; override if needed) # export GIT_CONFIG_GLOBAL="${RUNNER_WORKDIR}/.gitconfig" diff --git a/.github/scripts/github-runner-files/start.sh b/.github/scripts/github-runner-files/start.sh index f7045fd2d..adc1aea36 100755 --- a/.github/scripts/github-runner-files/start.sh +++ b/.github/scripts/github-runner-files/start.sh @@ -49,6 +49,11 @@ fi export RUNNER_ALLOW_RUNASROOT="${RUNNER_ALLOW_RUNASROOT:-1}" export TRITON_CACHE_DIR="${TRITON_CACHE_DIR:-${RUNNER_WORKDIR}/.triton_cache}" +# Writable HOME/TMPDIR for job steps (run-github-coding-agent-runner.sh may already create dirs on host) +mkdir -p "${RUNNER_WORKDIR}/.home" "${RUNNER_WORKDIR}/.tmp" +export HOME="${RUNNER_WORKDIR}/.home" +export TMPDIR="${RUNNER_WORKDIR}/.tmp" + mkdir -p "${RUNNER_HOME}" echo "==========================================" @@ -125,6 +130,11 @@ echo "Configuring runner..." # Cleanup function cleanup() { + # Kill any stale MCP processes left over from cancelled jobs + pkill -f "mcp/dist/index.js" 2>/dev/null || true + pkill -f "mcp-server-playwright" 2>/dev/null || true + pkill -f "playwright-mcp" 2>/dev/null || true + # Only run removal once; skip if config already removed if [ ! -f "${RUNNER_HOME}/.runner" ]; then echo "Runner config already removed or not configured. Skipping cleanup." diff --git a/.github/workflows/copilot-setup-steps.yml b/.github/workflows/copilot-setup-steps.yml index 56bb10393..c1793d124 100644 --- a/.github/workflows/copilot-setup-steps.yml +++ b/.github/workflows/copilot-setup-steps.yml @@ -25,9 +25,6 @@ jobs: - name: Create task venv for Copilot run: | cd "$GITHUB_WORKSPACE" - mkdir -p "${RUNNER_WORKDIR}/.home" "${RUNNER_WORKDIR}/.tmp" - echo "HOME=${RUNNER_WORKDIR}/.home" >> "$GITHUB_ENV" - echo "TMPDIR=${RUNNER_WORKDIR}/.tmp" >> "$GITHUB_ENV" /opt/venv/bin/python -m venv --system-site-packages .venv source .venv/bin/activate python -m pip install --upgrade pip @@ -36,8 +33,7 @@ jobs: - name: Install IntelliKit Python packages run: | source "$GITHUB_WORKSPACE/.venv/bin/activate" - # accordo requires kerneldb which needs libdwarf-dev; skipped until added to container - # pip install --no-cache-dir "git+https://github.com/AMDResearch/intellikit.git#subdirectory=accordo" + pip install --no-cache-dir "git+https://github.com/AMDResearch/intellikit.git#subdirectory=accordo" pip install --no-cache-dir "git+https://github.com/AMDResearch/intellikit.git#subdirectory=linex" pip install --no-cache-dir "git+https://github.com/AMDResearch/intellikit.git#subdirectory=metrix" pip install --no-cache-dir "git+https://github.com/AMDResearch/intellikit.git#subdirectory=nexus" @@ -62,7 +58,6 @@ jobs: - name: Make venv default for subsequent steps run: | echo "$GITHUB_WORKSPACE/.venv/bin" >> "$GITHUB_PATH" - echo "/opt/venv/bin" >> "$GITHUB_PATH" echo "PYTHONPATH=$GITHUB_WORKSPACE/.venv/lib/python3.13/site-packages:$GITHUB_WORKSPACE${PYTHONPATH:+:$PYTHONPATH}" >> "$GITHUB_ENV" - name: Verify ROCm and GPU visibility