From 5b1aaf65f0047e38e8ffc10f804d4d6afa22606f Mon Sep 17 00:00:00 2001 From: Aaron Elijah Mars <61592645+aaronjmars@users.noreply.github.com> Date: Mon, 27 Jul 2026 15:27:14 -0400 Subject: [PATCH] fix(harness): stage all six harnesses on messages.yml via shared scripts messages.yml staged only the claude and grok CLIs, so a repo configured for codex/pi/vibe/kimi had every inbound message answered on claude. It warned rather than swapping silently, but it was still not the harness the operator chose - and on Anthropic credentials a single-provider fork may not even have. The cause was structural, not a missing case. Both the harness/provider/model resolution (~100 lines) and the install recipes (~150 lines) lived inside aeon.yml steps, so no other workflow could reach them. messages.yml carried a second, weaker copy that knew claude and grok - the same shape as the two-path grok trap in #784. Extracted: * scripts/resolve-harness.sh - decides HARNESS / AUTH_MODE / HARNESS_MODEL / MODEL_ARG, prints them as KEY=VALUE lines on stdout (summary on stderr). Takes an OPTIONAL skill name: pass it for per-skill harness:/model: overrides, omit it for surfaces that are not a per-skill run. * scripts/install-harness.sh - stages the CLI + provider auth for that harness. claude is a no-op so callers can invoke it unconditionally. Both workflows now call both scripts. Each keeps its own env: block, since secrets.* only resolves inside a workflow, but the logic - the half that drifted - is shared. Two latent bugs became reachable during the extraction and are fixed: * A missing provider credential died MID-HEREDOC under set -u. bash had already created the file, so the harness was left with a 0-byte config.toml and an error naming a shell variable instead of the secret. It never fired inside a workflow, where the env: block always binds the name; it appears the moment the script is callable standalone. Now fails closed and names the missing secret. * messages.yml gated the Anthropic gateway on `!= grok`, correct only while claude and grok were the sole options. With the other four reachable it would point their env at a gateway they never call. Now `= claude`. Tests - this logic had never been tested, because inside a workflow step the only way to exercise it was to dispatch a live run: * scripts/tests/test_resolve_harness.sh (28 cases) - precedence, AUTH_MODE detection, per-harness MODEL_ARG shape, the stdout contract. * scripts/tests/test_install_harness.sh (21 cases) - stubs npm/pipx and asserts the generated codex/kimi/vibe config plus the supply-chain pins. Equivalence verified by differential test against the inline versions: 832 resolution cases and every config-generating install case produce byte-identical output. actionlint clean, shellcheck clean, okf-validate clean. --- .github/workflows/aeon.yml | 283 ++------------------------ .github/workflows/ci-tests.yml | 4 + .github/workflows/messages.yml | 129 ++++++++---- CHANGELOG.md | 32 +++ docs/harnesses.md | 22 +- scripts/install-harness.sh | 211 +++++++++++++++++++ scripts/resolve-harness.sh | 153 ++++++++++++++ scripts/tests/test_install_harness.sh | 152 ++++++++++++++ scripts/tests/test_resolve_harness.sh | 155 ++++++++++++++ 9 files changed, 827 insertions(+), 314 deletions(-) create mode 100755 scripts/install-harness.sh create mode 100755 scripts/resolve-harness.sh create mode 100644 scripts/tests/test_install_harness.sh create mode 100644 scripts/tests/test_resolve_harness.sh diff --git a/.github/workflows/aeon.yml b/.github/workflows/aeon.yml index 113eea1b15..dcd0bc62c0 100644 --- a/.github/workflows/aeon.yml +++ b/.github/workflows/aeon.yml @@ -222,122 +222,13 @@ jobs: GROK_CREDENTIALS: ${{ secrets.GROK_CREDENTIALS }} XAI_API_KEY: ${{ secrets.XAI_API_KEY }} run: | - # `|| true` on both greps: a repo with no top-level `harness:` key, or a - # skill absent from aeon.yml's skills map, makes grep exit 1, and under - # the runner's default `bash -e -o pipefail` that kills the step with NO - # message — the reader is left with a red "Resolve harness" and nothing - # to go on. Upstream carried the same two unguarded greps inside "Run"; - # moving them here without the guard would have relocated a silent - # failure rather than fixed it. An empty result already means "not - # configured", which the defaults below handle, and a genuinely bad skill - # name now fails later where the error names the missing SKILL.md. - CONFIG_HARNESS=$(grep -E '^harness:' aeon.yml | sed 's/^harness: *//' | tr -d ' ' || true) - SKILL_HARNESS=$(grep "^ ${SKILL_NAME}:" aeon.yml | sed -n 's/.*harness: *"\([^"]*\)".*/\1/p' || true) - if [ -n "${INPUT_HARNESS:-}" ] && [ "$INPUT_HARNESS" != "(config default)" ]; then - HARNESS="$INPUT_HARNESS" - elif [ -n "$SKILL_HARNESS" ]; then - HARNESS="$SKILL_HARNESS" - else - HARNESS="${CONFIG_HARNESS:-claude}" - fi - - # Allowlist. Upstream had a claude/grok binary, so ANY other value was - # silently rewritten to claude. These four are the ones measured end to - # end on a real runner (Tier 4). - # Deliberately NOT here, each for a measured reason: - # opencode — its .result carried the agent's narration instead of the - # deliverable (fixed in the adapter, but it still loops to - # the wall-clock guard on research skills); - # copilot — no credential path without a Copilot-subscribed PAT; - # agy — its print mode runs tools outside $PWD, so it reports - # success having written nothing to the workspace. - case "$HARNESS" in - claude|grok|codex|pi|vibe|kimi) ;; - *) echo "::warning::unknown harness '$HARNESS' — falling back to claude" - HARNESS="claude" ;; - esac - - # Which PROVIDER does this harness run on? Decided by which auth secret is - # set, native first, OpenRouter last — the same ordered set the dashboard's - # authSecretsForHarness / HARNESS_AUTH registry expose, and the order the - # "Install harness CLI" step configures. `native-oauth` = a captured - # ChatGPT/Moonshot login restored below; `native-key` = a provider API key - # in env; `openrouter` = the shared key (the default). This matters for the - # MODEL: on native auth the OpenRouter openai/* ids are the WRONG provider, - # so we forward NO model and let the harness use its own default. - AUTH_MODE="openrouter" - case "$HARNESS" in - grok) if [ -n "${GROK_CREDENTIALS:-}" ]; then AUTH_MODE="native-oauth"; elif [ -n "${XAI_API_KEY:-}" ]; then AUTH_MODE="native-key"; fi ;; - codex) if [ -n "${CODEX_AUTH:-}" ]; then AUTH_MODE="native-oauth"; elif [ -n "${OPENAI_API_KEY:-}" ]; then AUTH_MODE="native-key"; fi ;; - kimi) if [ -n "${KIMI_AUTH:-}" ]; then AUTH_MODE="native-oauth"; elif [ -n "${MOONSHOT_API_KEY:-}" ]; then AUTH_MODE="native-key"; fi ;; - vibe) if [ -n "${MISTRAL_API_KEY:-}" ]; then AUTH_MODE="native-key"; fi ;; - pi) if [ -n "${ANTHROPIC_API_KEY:-}" ] || [ -n "${ANTHROPIC_OAUTH_TOKEN:-}" ] || [ -n "${OPENAI_API_KEY:-}" ]; then AUTH_MODE="native-key"; fi ;; - esac - - # The harness model (HM), in priority order: - # 1. vars.HARNESS_MODEL — a repo-wide override, handy for CI. - # 2. the model chosen in the dashboard (which writes `model:` into - # aeon.yml) or a per-skill/dispatch model — WHEN it's an OpenRouter - # id. This is what makes the dashboard's model picker actually - # control what runs: modelsForHarness() offers OpenRouter ids for - # these harnesses, so CONFIG_MODEL is one of them here. - # 3. a per-harness cheap default. - # aeon-native ids (claude-*/grok-*) mean nothing to an OpenRouter CLI, so - # they're treated as "unset" and fall through to the default — a repo that - # never touched the model picker (still `model: claude-sonnet-5`) still - # gets a working default instead of a dead id. The greps carry `|| true` - # for the same reason as the harness greps above (no key ≠ fatal). - # Each harness defaults to its own native family (the dashboard's - # per-harness list, modelsForHarness[0]). The generic `*)` fallback is - # gpt-5-mini — a universally safe id — NOT gpt-5-nano: codex fails - # DETERMINISTICALLY on nano (measured on a real runner the model emits a - # shell tool call with a duplicated `cmd` field, codex's strict parser - # rejects it, and codex, which has no --max-turns, spins to the 900s - # guard), and the same skill passes on gpt-5-mini in ~2m40s. - CONFIG_MODEL=$(grep -E '^model:' aeon.yml | sed 's/^model: *//' | tr -d ' ' || true) - SKILL_MODEL=$(grep "^ ${SKILL_NAME}:" aeon.yml | sed -n 's/.*model: *"\([^"]*\)".*/\1/p' || true) - if [ -n "${INPUT_MODEL:-}" ] && [ "$INPUT_MODEL" != "(config default)" ]; then - REQ_MODEL="$INPUT_MODEL" - elif [ -n "$SKILL_MODEL" ]; then - REQ_MODEL="$SKILL_MODEL" - else - REQ_MODEL="${CONFIG_MODEL:-}" - fi - case "$REQ_MODEL" in claude-*|grok-*|"") REQ_MODEL="" ;; esac # aeon-native / unset → not an OpenRouter id - case "$HARNESS" in - codex) DEFAULT_HM="openai/gpt-5-mini" ;; - vibe) DEFAULT_HM="mistralai/mistral-medium-3-5" ;; # vibe's default (VIBE_MODELS[0]) - pi) DEFAULT_HM="deepseek/deepseek-v4-flash" ;; # pi's default (PI_MODELS[0]) - kimi) DEFAULT_HM="moonshotai/kimi-k2.5" ;; # kimi's default (KIMI_MODELS[0]) - *) DEFAULT_HM="openai/gpt-5-mini" ;; # generic fallback: only claude/grok hit it (and don't consume it); mini is universally safe - esac - HM="${HARNESS_MODEL:-${REQ_MODEL:-$DEFAULT_HM}}" - - # The --model run-harness should pass, or empty for "the harness's own - # staged config decides". aeon's model ids are always claude-*/grok-*, - # which mean nothing to an OpenRouter-backed CLI, so they are NOT - # forwarded: passing one leaves the harness on its default while every - # downstream record (token-usage.csv, the signed run manifest) names a - # model that never ran. - MODEL_ARG="" - if [ "$AUTH_MODE" = "openrouter" ]; then - case "$HARNESS" in - codex) MODEL_ARG="$HM" ;; # config.toml pins the provider; --model takes a bare id - pi) MODEL_ARG="openrouter/$HM" ;; # pi wants provider/model - # vibe and kimi resolve a config ALIAS rather than a raw id, so - # passing --model breaks them — their staged config decides. - esac - else - # Native auth (the operator's own ChatGPT/Moonshot/provider account): - # let the harness pick its own default model. HM is only a label here. - HM="(native:$AUTH_MODE)" - fi - - echo "Harness: $HARNESS | auth: $AUTH_MODE | model: $HM | run-harness --model: ${MODEL_ARG:-}" - echo "HARNESS=$HARNESS" >> "$GITHUB_OUTPUT" - echo "HARNESS_MODEL=$HM" >> "$GITHUB_OUTPUT" - echo "MODEL_ARG=$MODEL_ARG" >> "$GITHUB_OUTPUT" - echo "AUTH_MODE=$AUTH_MODE" >> "$GITHUB_OUTPUT" + # The resolution itself lives in scripts/resolve-harness.sh so that every + # surface that launches an agent decides identically. It used to be ~100 + # lines inline here, which is exactly why messages.yml could only ever + # support claude/grok: it had no way to compute AUTH_MODE or MODEL_ARG. + # It prints KEY=VALUE lines (HARNESS / AUTH_MODE / HARNESS_MODEL / + # MODEL_ARG) and its human-readable summary to stderr. + bash scripts/resolve-harness.sh "$SKILL_NAME" >> "$GITHUB_OUTPUT" # Install/stage the harness CLI for every harness except claude (which is # installed above and needs no per-provider auth). grok is included now: @@ -394,161 +285,11 @@ jobs: MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }} run: | set -euo pipefail - # Each harness is configured for the provider AUTH_MODE selected: - # native-oauth — restore the captured login (~/.codex, ~/.kimi-code); - # the harness then runs on its own default model. - # native-key — the provider's own API key (codex→OpenAI, kimi→Moonshot, - # vibe→Mistral, pi→Anthropic/OpenAI from env). - # openrouter — the shared OPENROUTER_API_KEY (the default; one key - # covers all four). $HM is the OpenRouter model here. - case "$H" in - grok) - # Install the pinned grok CLI + restore its auth (GROK_CREDENTIALS - # X-account OAuth, or XAI_API_KEY). run-grok.sh's `setup` subcommand - # runs ONLY those two steps and exits — the actual run then goes - # through run-harness grok in the Run step, on grok's own auth. - bash "${GITHUB_WORKSPACE}/scripts/run-grok.sh" setup - echo "grok: CLI + auth staged (auth: ${AUTH_MODE:-native})" ;; - codex) - # PINNED, like aeon pins claude-code. Unpinned, this step silently - # tracked latest: two cells that passed on 2026-07-21 failed hours - # later with an OpenRouter 400 and nothing in the repo had changed. - npm install -g @openai/codex@0.144.6 - mkdir -p "$HOME/.codex" - case "$AUTH_MODE" in - native-oauth) - # Restore the ChatGPT login captured by `aeon auth --harness codex` - # (tar+base64 of ~/.codex/auth.json). codex refreshes the access - # token from the refresh token at run start and uses its own - # default model — no config needed, no OpenRouter. - printf '%s' "$CODEX_AUTH" | base64 -d | tar xzf - -C "$HOME" - chmod 600 "$HOME/.codex/auth.json" || true - echo "codex: restored ChatGPT login" ;; - native-key) - # OPENAI_API_KEY in env → codex's built-in `openai` provider (its - # default), on codex's default model. No OpenRouter block. - cat > "$HOME/.codex/config.toml" <<'TOML' - model_reasoning_effort = "medium" - TOML - echo "codex: OpenAI API key" ;; - *) - # wire_api MUST be "responses": codex 0.144.6 removed "chat" - # ("no longer supported", a hard config-load error), and OpenRouter - # does serve /api/v1/responses. Both verified 2026-07-21. - cat > "$HOME/.codex/config.toml" < "$HOME/.kimi-code/config.toml" < "$HOME/.kimi-code/config.toml" <> "$GITHUB_PATH" - if [ "$AUTH_MODE" = "native-key" ]; then - # MISTRAL_API_KEY set → vibe's DEFAULT provider IS Mistral; it runs - # straight off the env key, no config. (The config below only exists - # because vibe hard-fails WITHOUT a Mistral key.) - echo "vibe: Mistral API key (default provider)" - else - # vibe's ProviderConfig is generic (api_base + api_key_env_var + - # api_style), so point it at OpenRouter. - mkdir -p "$HOME/.vibe" - cat > "$HOME/.vibe/config.toml" < OpenRouter). Never echoed. + CODEX_AUTH: ${{ secrets.CODEX_AUTH }} + KIMI_AUTH: ${{ secrets.KIMI_AUTH }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }} + MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }} + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + ANTHROPIC_OAUTH_TOKEN: ${{ secrets.ANTHROPIC_OAUTH_TOKEN }} + GROK_CREDENTIALS: ${{ secrets.GROK_CREDENTIALS }} + XAI_API_KEY: ${{ secrets.XAI_API_KEY }} + run: bash scripts/resolve-harness.sh >> "$GITHUB_OUTPUT" + + # Stage the resolved harness's CLI + provider auth. This step is why + # messages.yml now supports all six: it previously had none, so a repo on + # codex/pi/vibe/kimi had its messages answered on claude with a warning. + # The `env:` block is duplicated from aeon.yml by necessity (`secrets.*` + # resolves per workflow); the recipes it feeds are shared. + - name: Install harness CLI + if: steps.msg.outputs.source != '' && steps.harness.outputs.HARNESS != '' && steps.harness.outputs.HARNESS != 'claude' + env: + H: ${{ steps.harness.outputs.HARNESS }} + HM: ${{ steps.harness.outputs.HARNESS_MODEL }} + AUTH_MODE: ${{ steps.harness.outputs.AUTH_MODE }} + # grok auth (staged via run-grok.sh setup): captured X-account OAuth or key, + # plus the secrets-write PAT that persists a rotated refresh token back to + # GROK_CREDENTIALS (run-grok.sh §2b). + GROK_CREDENTIALS: ${{ secrets.GROK_CREDENTIALS }} + XAI_API_KEY: ${{ secrets.XAI_API_KEY }} + GH_SECRETS_PAT: ${{ secrets.GH_SECRETS_PAT }} + GH_GLOBAL: ${{ secrets.GH_GLOBAL }} + GROK_OAUTH_SKEW: ${{ vars.GROK_OAUTH_SKEW }} + # Provider credentials — restored (OAuth captures) or written into the + # harness's config (API keys). All land in the runner's ephemeral $HOME. + OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} + CODEX_AUTH: ${{ secrets.CODEX_AUTH }} + KIMI_AUTH: ${{ secrets.KIMI_AUTH }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }} + MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }} + run: | + set -euo pipefail + bash scripts/install-harness.sh "$H" + - name: Run id: run if: steps.msg.outputs.source != '' @@ -438,6 +490,19 @@ jobs: GH_SECRETS_PAT: ${{ secrets.GH_SECRETS_PAT }} # persist rotated grok refresh token → GROK_CREDENTIALS (run-grok.sh §2b) GH_GLOBAL: ${{ secrets.GH_GLOBAL }} GROK_OAUTH_SKEW: ${{ vars.GROK_OAUTH_SKEW }} # force grok OAuth refresh + persist every run when set large (durability check) + # Resolved by the "Resolve harness" step above; the CLI they name was + # staged by "Install harness CLI". + HARNESS: ${{ steps.harness.outputs.HARNESS }} + AUTH_MODE: ${{ steps.harness.outputs.AUTH_MODE }} + MODEL_ARG: ${{ steps.harness.outputs.MODEL_ARG }} + # Provider keys the non-claude harnesses read at RUN time (pi picks its + # provider from whichever is in env; codex/kimi/vibe read the key their + # staged config names). OPENROUTER_API_KEY is already declared above. + CODEX_AUTH: ${{ secrets.CODEX_AUTH }} + KIMI_AUTH: ${{ secrets.KIMI_AUTH }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }} + MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }} COINGECKO_API_KEY: ${{ secrets.COINGECKO_API_KEY }} ALCHEMY_API_KEY: ${{ secrets.ALCHEMY_API_KEY }} _MSG_SOURCE: ${{ steps.msg.outputs.source }} @@ -447,46 +512,24 @@ jobs: CONFIG_MODEL=$(grep -E '^model:' aeon.yml | sed 's/^model: *//' | tr -d ' ') MODEL="${CONFIG_MODEL:-claude-sonnet-5}" - # Harness resolution — inbound messaging isn't a per-skill run, so it - # follows the repo's global `harness:` in aeon.yml (same default/guard as - # aeon.yml). A grok-only repo (no Claude creds) must respond via grok. - CONFIG_HARNESS=$(grep -E '^harness:' aeon.yml | sed 's/^harness: *//' | tr -d ' ') - HARNESS="${CONFIG_HARNESS:-claude}" - # Only claude and grok have their CLI staged on this workflow ("Install - # Claude Code" above, plus the grok setup below). The other four - # run-harness harnesses would need their CLI + provider auth installed - # here the way aeon.yml's "Install harness CLI" step does. Until that - # lands, a repo configured for one of them answers messages on claude — - # say so out loud instead of swapping the harness silently. - case "$HARNESS" in - claude|grok) ;; - codex|pi|vibe|kimi) - echo "::warning::harness '$HARNESS' has no CLI staged on the messages workflow — replying on claude instead (skill runs still use $HARNESS)." - HARNESS="claude" ;; - *) - echo "::warning::unknown harness '$HARNESS' — falling back to claude" - HARNESS="claude" ;; - esac + # HARNESS / AUTH_MODE / MODEL_ARG come from the "Resolve harness" step + # (scripts/resolve-harness.sh), and the CLI was staged by "Install harness + # CLI" — so all six answer messages now. This used to be an inline + # claude-or-grok resolution here that downgraded the other four to claude + # with a ::warning::, because nothing on this workflow could install them. # On grok a claude-* model id is meaningless — fall back to grok's default. if [ "$HARNESS" = "grok" ]; then case "$MODEL" in claude-*|"") MODEL="grok-4.5" ;; esac fi - echo "Using harness: $HARNESS | model: $MODEL" - - # Stage grok's CLI + auth. run-grok.sh's `setup` subcommand is the single - # source of truth for the version pin, the GROK_CREDENTIALS restore and - # the OAuth refresh/persist (§2b). This used to happen implicitly, because - # the run itself went through run-grok.sh and step 1 installs the CLI on - # demand; now that the run goes through run-harness (whose grok adapter - # only checks `command -v grok`), staging has to be explicit. - if [ "$HARNESS" = "grok" ]; then - bash "${GITHUB_WORKSPACE}/scripts/run-grok.sh" setup - fi - - # AI Gateway routing — same path as aeon.yml's Run steps. The grok harness - # has its own auth and does NOT use the gateway, so skip it there (avoids - # "no gateway/Anthropic API" noise on a grok-only repo). - if [ "$HARNESS" != "grok" ]; then + echo "Using harness: $HARNESS | auth: $AUTH_MODE | model: ${MODEL_ARG:-$MODEL}" + + # AI Gateway routing — same path as aeon.yml's Run step, and gated the + # same way: it is Anthropic-specific (llm-gateway.sh rewrites + # ANTHROPIC_BASE_URL/model env), so ONLY claude sources it. This used to + # read `!= grok`, which was correct while claude and grok were the only + # two possible harnesses; with codex/pi/vibe/kimi reachable it would + # point their env at an Anthropic gateway they never call. + if [ "$HARNESS" = "claude" ]; then GATEWAY=$(grep -A1 '^gateway:' aeon.yml | grep 'provider:' | sed 's/.*provider:[[:space:]]*//' | sed "s/[\"' ]//g" || true) GATEWAY="${GATEWAY:-auto}" source "${GITHUB_WORKSPACE}/scripts/llm-gateway.sh" @@ -614,9 +657,21 @@ jobs: # run-harness re-expands the config and translates it per harness; the # claude adapter adds --strict-mcp-config itself, as this step used to. if [ "${#MCP_FLAGS[@]}" -gt 0 ]; then RH_ARGS+=(--mcp-config .mcp.json); fi + # Which --model to forward, mirroring aeon.yml's Run step. aeon's own ids + # are claude-*/grok-*, which mean nothing to an OpenRouter-backed CLI: + # forwarding one there leaves the harness on its default while every + # record names a model that never ran. MODEL_ARG (from resolve-harness.sh) + # is already the right value or empty for "the staged config decides" — + # vibe and kimi resolve a config ALIAS, so passing --model breaks them. + RH_MODEL_ARGS=() + case "$HARNESS" in + claude) RH_MODEL_ARGS=(--model "$MODEL") ;; + grok) case "$MODEL" in grok-*) RH_MODEL_ARGS=(--model "$MODEL") ;; esac ;; + *) if [ -n "${MODEL_ARG:-}" ]; then RH_MODEL_ARGS=(--model "$MODEL_ARG"); fi ;; + esac HARNESS_ERR=$(mktemp) if ! CLAUDE_OUTPUT=$(echo "$PROMPT" | bash "$RH" "$HARNESS" \ - --model "$MODEL" "${RH_ARGS[@]}" 2>"$HARNESS_ERR"); then + "${RH_MODEL_ARGS[@]}" "${RH_ARGS[@]}" 2>"$HARNESS_ERR"); then echo "::error::$HARNESS harness failed: $(tail -c 400 "$HARNESS_ERR" | tr '\n' ' ')" exit 1 fi diff --git a/CHANGELOG.md b/CHANGELOG.md index 9965c48125..872472c1d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,6 +44,38 @@ from or pin to; the template keeps serving the latest `main` to new forks. ### Fixed +- **Inbound messages now run on all six harnesses.** `messages.yml` staged only + the claude and grok CLIs, so a repo configured for codex/pi/vibe/kimi had every + inbound message answered on **claude** — loudly (`::warning::`), but still not + the harness the operator chose, and on Anthropic credentials a + single-provider fork may not even have. The cause was structural: both the + ~100-line harness/provider/model resolution and the ~150-line install recipes + lived *inside* `aeon.yml` steps, so no other workflow could reach them. + Extracted to **`scripts/resolve-harness.sh`** (prints `HARNESS`/`AUTH_MODE`/ + `HARNESS_MODEL`/`MODEL_ARG` as `KEY=VALUE` lines) and + **`scripts/install-harness.sh`**; both workflows now call them. Each still + declares its own `env:` block, since `secrets.*` only resolves per workflow — + but the logic, the half that drifted, is shared. This is the same class of bug + as the two-path grok trap (#784), and the last surface where "aeon supports six + harnesses" wasn't true. Verified by a differential test: 832 resolution cases + and every config-generating install case produce byte-identical output to the + inline versions they replace. + - The extraction also made two latent bugs reachable and fixed both. A missing + provider credential used to die *mid-heredoc* under `set -u` — the config + file was already created, so the harness was left holding a **0-byte + `config.toml`** and an error naming a shell variable instead of the secret. + (It never fired inside a workflow, where the `env:` block always binds the + name; it appears the moment the script is callable standalone.) It now fails + closed and names the missing secret. Separately, `messages.yml` gated the + Anthropic gateway on `!= grok`, correct only while claude and grok were the + sole options — with the other four reachable it would have pointed their env + at a gateway they never call. Now gated on `= claude`. + - Both scripts get real test suites — `test_resolve_harness.sh` (28 cases) and + `test_install_harness.sh` (21 cases). This logic had **never** been tested: + inside a workflow step the only way to exercise it was to dispatch a live + run. The install tests stub `npm`/`pipx` and assert what actually breaks — + the generated codex/kimi/vibe provider config and the supply-chain version + pins. - **The docs described a read-only guard that does not exist, and the code it cited is deleted.** `docs/CAPABILITIES.md` and `docs/harnesses.md` both stated that a `mode: read-only` skill is confined on grok by an explicit tool allowlist diff --git a/docs/harnesses.md b/docs/harnesses.md index 31505069fc..fe8a1b3a7a 100644 --- a/docs/harnesses.md +++ b/docs/harnesses.md @@ -266,12 +266,22 @@ path until it was removed: nothing in the repo ever set it, and the scorer it wa reserved for goes schema-less on purpose so a single parse path covers all six harnesses.) -**Harness coverage differs by surface**, because each has to stage the CLI it -dispatches to. Skill runs (`aeon.yml`) install any of the six. The local MCP -server resolves all six but expects the CLI to already be installed on your -machine. `messages.yml` stages only `claude` and `grok`; a repo configured for -one of the other four gets a `::warning::` and is answered on claude, rather than -being switched silently. +**Both hosted surfaces stage all six.** `aeon.yml` (skill runs) and +`messages.yml` (inbound messages) share the same two scripts, so a repo answers +messages on the harness it runs skills on: + +- **`scripts/resolve-harness.sh`** — decides `HARNESS` / `AUTH_MODE` / + `HARNESS_MODEL` / `MODEL_ARG` and prints them as `KEY=VALUE` lines. Pass a + skill name to pick up per-skill `harness:`/`model:` overrides; omit it (as + `messages.yml` does) and the repo-global keys decide. +- **`scripts/install-harness.sh`** — stages that harness's CLI and provider auth. + +Each workflow still declares its own `env:` block, because `secrets.*` only +resolves inside a workflow — but the logic is shared, which is the half that +drifted before. `messages.yml` used to carry a second, weaker copy that knew only +claude and grok, so a repo on codex/pi/vibe/kimi had its messages answered on +claude with a `::warning::`. The local MCP server (`apps/mcp-server`) resolves all +six but expects the CLI to already be installed on your machine. ## Every entry point runs on either harness diff --git a/scripts/install-harness.sh b/scripts/install-harness.sh new file mode 100755 index 0000000000..81f2b7d65e --- /dev/null +++ b/scripts/install-harness.sh @@ -0,0 +1,211 @@ +#!/usr/bin/env bash +# install-harness — stage a harness CLI and its provider auth on the runner. +# +# Extracted from aeon.yml's "Install harness CLI" step so every workflow that +# launches an agent can stage any of the six. It was inline in aeon.yml, which is +# the only reason messages.yml supported just claude/grok: a repo configured for +# codex/pi/vibe/kimi had its inbound messages answered on claude, because nothing +# on that workflow could install the CLI. Same class of bug as the two-path grok +# trap (#784) — a capability that exists on one surface and silently doesn't on +# another. The logic lives here now; each workflow supplies its own `env:` block, +# since `secrets.*` only resolves inside a workflow. +# +# Usage: +# bash scripts/install-harness.sh [harness] # or set $H +# +# claude is a no-op: workflows install it themselves (pinned + npm-cached) and it +# needs no per-provider auth. Calling this with `claude` is therefore fine and +# does nothing, so a caller can invoke it unconditionally. +# +# Inputs (env): +# H harness name (or $1) +# HM harness model — the OpenRouter id baked into the staged config +# AUTH_MODE native-oauth | native-key | openrouter (from resolve-harness.sh) +# plus the provider credentials for that harness (see the cases below) +# +# Everything written lands in the runner's ephemeral $HOME and is never echoed. +set -euo pipefail + +H="${1:-${H:-}}" +HM="${HM:-}" +AUTH_MODE="${AUTH_MODE:-openrouter}" + +# Fail CLOSED on a missing provider credential, and say which one. +# +# Inside a workflow every key below is declared in the step's `env:` block, so it +# is always *bound* (empty when the secret is unset) and `set -u` never fires. +# Standalone — messages.yml, a local run, apps/mcp-server — an undeclared key is +# unbound, and under `set -u` the heredoc dies MID-EXPANSION: bash has already +# created the file, so the harness is left holding a 0-byte config.toml and the +# error names a shell variable rather than the missing secret. Check first. +need() { # need VAR_NAME "what to set" + local name="$1" hint="$2" + if [ -z "${!name:-}" ]; then + echo "::error::$H needs $name — $hint" >&2 + exit 1 + fi +} + +# Each harness is configured for the provider AUTH_MODE selected: +# native-oauth — restore the captured login (~/.codex, ~/.kimi-code); the +# harness then runs on its own default model. +# native-key — the provider's own API key (codex→OpenAI, kimi→Moonshot, +# vibe→Mistral, pi→Anthropic/OpenAI from env). +# openrouter — the shared OPENROUTER_API_KEY (the default; one key covers all +# four). $HM is the OpenRouter model here. +case "$H" in + claude|"") + # Installed by the workflow itself (pinned, npm-cached), no provider auth. + echo "claude: installed by the workflow (no per-provider auth needed)" + exit 0 ;; + grok) + # Install the pinned grok CLI + restore its auth (GROK_CREDENTIALS X-account + # OAuth, or XAI_API_KEY). run-grok.sh's `setup` subcommand runs ONLY those two + # steps and exits — the actual run then goes through run-harness grok, on + # grok's own auth. Keeping the pin + restore in one place is why this shells + # out rather than inlining an `npm install -g @xai-official/grok`. + bash "${GITHUB_WORKSPACE:-$(pwd)}/scripts/run-grok.sh" setup + echo "grok: CLI + auth staged (auth: ${AUTH_MODE:-native})" ;; + codex) + # PINNED, like aeon pins claude-code. Unpinned, this step silently tracked + # latest: two cells that passed on 2026-07-21 failed hours later with an + # OpenRouter 400 and nothing in the repo had changed. + npm install -g @openai/codex@0.144.6 + mkdir -p "$HOME/.codex" + case "$AUTH_MODE" in + native-oauth) + # Restore the ChatGPT login captured by `aeon auth --harness codex` + # (tar+base64 of ~/.codex/auth.json). codex refreshes the access token + # from the refresh token at run start and uses its own default model — + # no config needed, no OpenRouter. + need CODEX_AUTH "the ChatGPT login capture from 'aeon auth --harness codex'" + printf '%s' "$CODEX_AUTH" | base64 -d | tar xzf - -C "$HOME" + chmod 600 "$HOME/.codex/auth.json" || true + echo "codex: restored ChatGPT login" ;; + native-key) + # OPENAI_API_KEY in env → codex's built-in `openai` provider (its + # default), on codex's default model. No OpenRouter block. + cat > "$HOME/.codex/config.toml" <<'TOML' +model_reasoning_effort = "medium" +TOML + echo "codex: OpenAI API key" ;; + *) + # wire_api MUST be "responses": codex 0.144.6 removed "chat" ("no longer + # supported", a hard config-load error), and OpenRouter does serve + # /api/v1/responses. Both verified 2026-07-21. + cat > "$HOME/.codex/config.toml" < "$HOME/.kimi-code/config.toml" < "$HOME/.kimi-code/config.toml" <> "$GITHUB_PATH" + if [ "$AUTH_MODE" = "native-key" ]; then + # MISTRAL_API_KEY set → vibe's DEFAULT provider IS Mistral; it runs straight + # off the env key, no config. (The config below only exists because vibe + # hard-fails WITHOUT a Mistral key.) + echo "vibe: Mistral API key (default provider)" + else + # vibe's ProviderConfig is generic (api_base + api_key_env_var + api_style), + # so point it at OpenRouter. + mkdir -p "$HOME/.vibe" + cat > "$HOME/.vibe/config.toml" < KEY=VALUE lines on stdout +# +# The skill name is OPTIONAL: it selects the per-skill `harness:`/`model:` +# overrides from aeon.yml. Omit it for surfaces that aren't a per-skill run +# (inbound messages), and the repo-global keys decide — which is what those +# surfaces want, and the reason the same script serves both. +# +# Inputs (env, all optional): +# INPUT_HARNESS / INPUT_MODEL workflow_dispatch overrides ("(config default)" +# is treated as unset — that's the dropdown's +# placeholder value, not a harness name) +# HARNESS_MODEL vars.HARNESS_MODEL, a repo-wide model override +# CODEX_AUTH / KIMI_AUTH / GROK_CREDENTIALS +# OPENAI_API_KEY / MOONSHOT_API_KEY / MISTRAL_API_KEY / XAI_API_KEY +# ANTHROPIC_API_KEY / ANTHROPIC_OAUTH_TOKEN +# presence ONLY — never read for their value here, +# never echoed. They pick AUTH_MODE. +# +# Outputs (stdout, one KEY=VALUE per line — append to $GITHUB_OUTPUT/$GITHUB_ENV, +# or `eval` after review): +# HARNESS claude | grok | codex | pi | vibe | kimi +# AUTH_MODE native-oauth | native-key | openrouter +# HARNESS_MODEL the model label for logs/records ("(native:…)" on native auth) +# MODEL_ARG what to pass as `run-harness --model`, or empty for "the +# harness's own staged config decides" +# +# Reads ./aeon.yml from the current directory. Prints diagnostics to stderr. +set -euo pipefail + +SKILL_NAME="${1:-}" + +# `|| true` on every grep: a repo with no top-level `harness:` key, or a skill +# absent from aeon.yml's skills map, makes grep exit 1 — and under `set -e` that +# kills the caller with NO message, leaving a red step and nothing to go on. An +# empty result already means "not configured", which the defaults below handle. +CONFIG_HARNESS=$(grep -E '^harness:' aeon.yml | sed 's/^harness: *//' | tr -d ' ' || true) +SKILL_HARNESS="" +[ -n "$SKILL_NAME" ] && SKILL_HARNESS=$(grep "^ ${SKILL_NAME}:" aeon.yml | sed -n 's/.*harness: *"\([^"]*\)".*/\1/p' || true) + +if [ -n "${INPUT_HARNESS:-}" ] && [ "$INPUT_HARNESS" != "(config default)" ]; then + HARNESS="$INPUT_HARNESS" +elif [ -n "$SKILL_HARNESS" ]; then + HARNESS="$SKILL_HARNESS" +else + HARNESS="${CONFIG_HARNESS:-claude}" +fi + +# Allowlist. Upstream had a claude/grok binary, so ANY other value was silently +# rewritten to claude. These six are the ones measured end to end on a real runner. +# Deliberately NOT here, each for a measured reason: +# opencode — its .result carried the agent's narration instead of the +# deliverable (fixed in the adapter, but it still loops to the +# wall-clock guard on research skills); +# copilot — no credential path without a Copilot-subscribed PAT; +# agy — its print mode runs tools outside $PWD, so it reports success +# having written nothing to the workspace. +case "$HARNESS" in + claude|grok|codex|pi|vibe|kimi) ;; + *) echo "::warning::unknown harness '$HARNESS' — falling back to claude" >&2 + HARNESS="claude" ;; +esac + +# Which PROVIDER does this harness run on? Decided by which auth secret is set, +# native first, OpenRouter last — the same ordered set the dashboard's +# authSecretsForHarness / HARNESS_AUTH registry expose, and the order +# install-harness.sh configures. `native-oauth` = a captured ChatGPT/Moonshot +# login restored at install; `native-key` = a provider API key in env; +# `openrouter` = the shared key (the default). This matters for the MODEL: on +# native auth the OpenRouter openai/* ids are the WRONG provider, so we forward +# NO model and let the harness use its own default. +AUTH_MODE="openrouter" +case "$HARNESS" in + grok) if [ -n "${GROK_CREDENTIALS:-}" ]; then AUTH_MODE="native-oauth"; elif [ -n "${XAI_API_KEY:-}" ]; then AUTH_MODE="native-key"; fi ;; + codex) if [ -n "${CODEX_AUTH:-}" ]; then AUTH_MODE="native-oauth"; elif [ -n "${OPENAI_API_KEY:-}" ]; then AUTH_MODE="native-key"; fi ;; + kimi) if [ -n "${KIMI_AUTH:-}" ]; then AUTH_MODE="native-oauth"; elif [ -n "${MOONSHOT_API_KEY:-}" ]; then AUTH_MODE="native-key"; fi ;; + vibe) if [ -n "${MISTRAL_API_KEY:-}" ]; then AUTH_MODE="native-key"; fi ;; + pi) if [ -n "${ANTHROPIC_API_KEY:-}" ] || [ -n "${ANTHROPIC_OAUTH_TOKEN:-}" ] || [ -n "${OPENAI_API_KEY:-}" ]; then AUTH_MODE="native-key"; fi ;; +esac + +# The harness model (HM), in priority order: +# 1. vars.HARNESS_MODEL — a repo-wide override, handy for CI. +# 2. the model chosen in the dashboard (which writes `model:` into aeon.yml) or +# a per-skill/dispatch model — WHEN it's an OpenRouter id. This is what makes +# the dashboard's model picker actually control what runs: modelsForHarness() +# offers OpenRouter ids for these harnesses. +# 3. a per-harness cheap default. +# aeon-native ids (claude-*/grok-*) mean nothing to an OpenRouter CLI, so they're +# treated as "unset" and fall through to the default — a repo that never touched +# the model picker (still `model: claude-sonnet-5`) still gets a working default +# instead of a dead id. +# Each harness defaults to its own native family (the dashboard's per-harness +# list, modelsForHarness[0]). The generic `*)` fallback is gpt-5-mini — a +# universally safe id — NOT gpt-5-nano: codex fails DETERMINISTICALLY on nano +# (measured on a real runner the model emits a shell tool call with a duplicated +# `cmd` field, codex's strict parser rejects it, and codex, which has no +# --max-turns, spins to the 900s guard); the same skill passes on gpt-5-mini. +CONFIG_MODEL=$(grep -E '^model:' aeon.yml | sed 's/^model: *//' | tr -d ' ' || true) +SKILL_MODEL="" +[ -n "$SKILL_NAME" ] && SKILL_MODEL=$(grep "^ ${SKILL_NAME}:" aeon.yml | sed -n 's/.*model: *"\([^"]*\)".*/\1/p' || true) + +if [ -n "${INPUT_MODEL:-}" ] && [ "$INPUT_MODEL" != "(config default)" ]; then + REQ_MODEL="$INPUT_MODEL" +elif [ -n "$SKILL_MODEL" ]; then + REQ_MODEL="$SKILL_MODEL" +else + REQ_MODEL="${CONFIG_MODEL:-}" +fi +case "$REQ_MODEL" in claude-*|grok-*|"") REQ_MODEL="" ;; esac # aeon-native / unset → not an OpenRouter id + +case "$HARNESS" in + codex) DEFAULT_HM="openai/gpt-5-mini" ;; + vibe) DEFAULT_HM="mistralai/mistral-medium-3-5" ;; # vibe's default (VIBE_MODELS[0]) + pi) DEFAULT_HM="deepseek/deepseek-v4-flash" ;; # pi's default (PI_MODELS[0]) + kimi) DEFAULT_HM="moonshotai/kimi-k2.5" ;; # kimi's default (KIMI_MODELS[0]) + *) DEFAULT_HM="openai/gpt-5-mini" ;; # generic fallback: only claude/grok hit it (and don't consume it) +esac +HM="${HARNESS_MODEL:-${REQ_MODEL:-$DEFAULT_HM}}" + +# The --model run-harness should pass, or empty for "the harness's own staged +# config decides". aeon's model ids are always claude-*/grok-*, which mean nothing +# to an OpenRouter-backed CLI, so they are NOT forwarded: passing one leaves the +# harness on its default while every downstream record (token-usage.csv, the +# signed run manifest) names a model that never ran. +MODEL_ARG="" +if [ "$AUTH_MODE" = "openrouter" ]; then + case "$HARNESS" in + codex) MODEL_ARG="$HM" ;; # config.toml pins the provider; --model takes a bare id + pi) MODEL_ARG="openrouter/$HM" ;; # pi wants provider/model + # vibe and kimi resolve a config ALIAS rather than a raw id, so passing + # --model breaks them — their staged config decides. + esac +else + # Native auth (the operator's own ChatGPT/Moonshot/provider account): let the + # harness pick its own default model. HM is only a label here. + HM="(native:$AUTH_MODE)" +fi + +echo "Harness: $HARNESS | auth: $AUTH_MODE | model: $HM | run-harness --model: ${MODEL_ARG:-}" >&2 +printf 'HARNESS=%s\n' "$HARNESS" +printf 'AUTH_MODE=%s\n' "$AUTH_MODE" +printf 'HARNESS_MODEL=%s\n' "$HM" +printf 'MODEL_ARG=%s\n' "$MODEL_ARG" diff --git a/scripts/tests/test_install_harness.sh b/scripts/tests/test_install_harness.sh new file mode 100644 index 0000000000..9b5bc79692 --- /dev/null +++ b/scripts/tests/test_install_harness.sh @@ -0,0 +1,152 @@ +#!/usr/bin/env bash +# Tests for scripts/install-harness.sh — the shared CLI+auth staging used by BOTH +# aeon.yml and messages.yml. +# +# Fake `npm`/`pipx` on PATH record their argv instead of installing, and $HOME is +# a temp dir, so the real work under test is what this step actually gets wrong: +# the generated provider config, and the supply-chain version pins. +# +# The codex config is the sharp case. codex parses config as TOML and fails the +# whole run on a malformed one — and it removed wire_api="chat" as a hard +# config-load error, which reads as a dead model, not a config bug. That class of +# failure was previously only discoverable by dispatching a real run. +# +# Run: bash scripts/tests/test_install_harness.sh +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +I="$ROOT/scripts/install-harness.sh" +fail=0 +pass() { echo "ok - $1"; } +bad() { echo "FAIL - $1"; fail=1; } + +BIN="$(mktemp -d)" +cleanup() { rm -rf "$BIN"; } +trap cleanup EXIT + +for tool in npm pipx; do + cat > "$BIN/$tool" <> "\$PKG_LOG" +EOF + chmod +x "$BIN/$tool" +done + +# run [env...] -> stages into a fresh $HOME; sets H_DIR and PKG_LOG +run() { + local h="$1"; shift + H_DIR="$(mktemp -d)" + PKG_LOG="$H_DIR/pkg.log"; : > "$PKG_LOG" + env PATH="$BIN:$PATH" HOME="$H_DIR" PKG_LOG="$PKG_LOG" GITHUB_PATH="" "$@" \ + bash "$I" "$h" >"$H_DIR/out.txt" 2>&1 + return $? +} + +# --- 1. claude is a no-op, unknown is fatal -------------------------------- +# Callers may invoke this unconditionally, so claude must succeed and do nothing. +run claude && [ ! -e "$H_DIR/.codex" ] \ + && pass "claude: no-op, exit 0" || bad "claude should be a no-op" +if run banana; then bad "unknown harness must fail"; else + grep -q "no install recipe" "$H_DIR/out.txt" \ + && pass "unknown harness exits non-zero with a named error" \ + || bad "unknown harness error message" +fi + +# --- 2. codex config generation -------------------------------------------- +run codex HM=openai/gpt-5-mini AUTH_MODE=openrouter OPENROUTER_API_KEY=sk-test +CFG="$H_DIR/.codex/config.toml" +if [ -f "$CFG" ]; then + # wire_api MUST be "responses": codex 0.144.6 removed "chat" as a hard + # config-load error, which kills the run before the model is ever reached. + grep -q 'wire_api = "responses"' "$CFG" \ + && pass "codex/openrouter: wire_api is responses" || bad "codex wire_api" + grep -q 'model = "openai/gpt-5-mini"' "$CFG" \ + && pass "codex/openrouter: HM lands in the config" || bad "codex model" + # OpenRouter 400s a reasoning-disabled request to /responses. + grep -q 'model_reasoning_effort' "$CFG" \ + && pass "codex/openrouter: reasoning effort set" || bad "codex reasoning effort" + # TOML, not JSON: a JSON object here is what silently killed codex's MCP config. + grep -q '^\[model_providers.openrouter\]' "$CFG" \ + && pass "codex/openrouter: provider is a TOML table" || bad "codex provider table" +else bad "codex/openrouter: no config written"; fi + +run codex AUTH_MODE=native-key +CFG="$H_DIR/.codex/config.toml" +if [ -f "$CFG" ]; then + # native-key uses codex's built-in openai provider — an OpenRouter block here + # would override the operator's own account. + grep -q 'openrouter' "$CFG" && bad "codex/native-key must not write an OpenRouter block" \ + || pass "codex/native-key: no OpenRouter block" +else bad "codex/native-key: no config written"; fi + +# --- 3. kimi + vibe config generation -------------------------------------- +run kimi HM=moonshotai/kimi-k2.5 AUTH_MODE=openrouter OPENROUTER_API_KEY=sk-test +CFG="$H_DIR/.kimi-code/config.toml" +if [ -f "$CFG" ]; then + # kimi resolves --model through an ALIAS, so default_model must be the alias and + # the real id lives under [models.]. run-harness then passes no --model. + grep -q 'default_model = "or-cheap"' "$CFG" \ + && pass "kimi: default_model is the alias" || bad "kimi alias" + grep -q 'model = "moonshotai/kimi-k2.5"' "$CFG" \ + && pass "kimi: HM lands under the alias" || bad "kimi model" + # The config holds a live provider key. + PERM=$(ls -l "$CFG" | cut -c1-10) + [ "$PERM" = "-rw-------" ] \ + && pass "kimi: config is chmod 600 (holds a provider key)" || bad "kimi config perms ($PERM)" +else bad "kimi: no config written"; fi + +run vibe HM=mistralai/mistral-medium-3-5 AUTH_MODE=openrouter OPENROUTER_API_KEY=sk-test +CFG="$H_DIR/.vibe/config.toml" +grep -q 'alias = "or-cheap"' "$CFG" 2>/dev/null \ + && pass "vibe/openrouter: alias config written" || bad "vibe openrouter config" +# With a Mistral key, vibe's DEFAULT provider is already Mistral — writing an +# OpenRouter config would silently redirect the operator's own account. +run vibe AUTH_MODE=native-key +[ ! -f "$H_DIR/.vibe/config.toml" ] \ + && pass "vibe/native-key: no config (Mistral is vibe's default)" || bad "vibe native-key wrote a config" + +# --- 4. supply-chain pins --------------------------------------------------- +# An unpinned `-g` install runs whatever the registry serves, in CI, with the +# run's secrets in env. Two codex cells already failed this way when unpinned. +run codex AUTH_MODE=openrouter OPENROUTER_API_KEY=sk-test +grep -q '@openai/codex@[0-9]' "$PKG_LOG" \ + && pass "codex: install is version-pinned" || bad "codex pin missing" +run pi AUTH_MODE=openrouter OPENROUTER_API_KEY=sk-test +grep -q '@earendil-works/pi-coding-agent@[0-9]' "$PKG_LOG" \ + && pass "pi: install is version-pinned" || bad "pi pin missing" +grep -qx -- '--ignore-scripts' "$PKG_LOG" \ + && pass "pi: --ignore-scripts (no postinstall)" || bad "pi --ignore-scripts missing" +run kimi AUTH_MODE=openrouter OPENROUTER_API_KEY=sk-test +grep -q '@moonshot-ai/kimi-code@[0-9]' "$PKG_LOG" \ + && pass "kimi: install is version-pinned" || bad "kimi pin missing" +grep -qx -- '--ignore-scripts' "$PKG_LOG" \ + && pass "kimi: --ignore-scripts (no postinstall)" || bad "kimi --ignore-scripts missing" +run vibe AUTH_MODE=native-key +grep -q 'mistral-vibe==[0-9]' "$PKG_LOG" \ + && pass "vibe: install is version-pinned" || bad "vibe pin missing" + +# --- 5. missing credential fails closed, and names the secret --------------- +# Found writing these tests. `set -u` + an unbound key killed the script +# MID-HEREDOC: bash had already created config.toml, so the harness was left with +# a 0-byte config and an error naming a shell variable rather than the secret. +# Inside a workflow the `env:` block always binds these (empty when unset), so it +# never fired there — it only appears the moment this is callable standalone, +# which is exactly what this extraction makes it. +if run kimi AUTH_MODE=openrouter; then + bad "kimi with no OPENROUTER_API_KEY should fail" +else + grep -q "needs OPENROUTER_API_KEY" "$H_DIR/out.txt" \ + && pass "missing key: error names the secret" || bad "missing-key error should name the secret" + [ ! -s "$H_DIR/.kimi-code/config.toml" ] 2>/dev/null && [ ! -f "$H_DIR/.kimi-code/config.toml" ] \ + && pass "missing key: no truncated config left behind" \ + || bad "missing key left a partial config.toml" +fi +if run codex AUTH_MODE=native-oauth; then + bad "codex native-oauth with no CODEX_AUTH should fail" +else + grep -q "needs CODEX_AUTH" "$H_DIR/out.txt" \ + && pass "missing OAuth capture: error names the secret" || bad "CODEX_AUTH error message" +fi + +echo "---" +[ "$fail" = "0" ] && echo "ALL PASS" || echo "SOME FAILED" +exit "$fail" diff --git a/scripts/tests/test_resolve_harness.sh b/scripts/tests/test_resolve_harness.sh new file mode 100644 index 0000000000..0bfaeb9a21 --- /dev/null +++ b/scripts/tests/test_resolve_harness.sh @@ -0,0 +1,155 @@ +#!/usr/bin/env bash +# Tests for scripts/resolve-harness.sh — the shared harness/provider/model +# decision used by BOTH aeon.yml and messages.yml. +# +# This logic was ~100 lines inside one workflow step, so it had never been tested: +# the only way to exercise it was to dispatch a real run. That is also why +# messages.yml carried a second, weaker copy that only knew claude and grok. +# +# Run: bash scripts/tests/test_resolve_harness.sh +set -uo pipefail +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +R="$ROOT/scripts/resolve-harness.sh" +fail=0 +pass() { echo "ok - $1"; } +bad() { echo "FAIL - $1"; fail=1; } + +WS="$(mktemp -d)" +cleanup() { rm -rf "$WS"; } +trap cleanup EXIT +cd "$WS" || exit 1 + +# A fixture aeon.yml in aeon's REAL shape: repo-global keys at column 0, and a +# skills map whose entries are inline flow-mappings on ONE line, two spaces in — +# dash-name: { enabled: false, schedule: "...", harness: "vibe" } +# The per-skill greps (`^ :` then sed for harness:/model: on that line) +# only work on that one-line form, the same constraint aeon.yml documents for +# `chains:`. A block-style fixture passes YAML but silently resolves nothing, so +# it would have tested the fallback path while looking like it tested overrides. +mkfixture() { # mkfixture [global-harness] [global-model] + { echo "model: ${2:-claude-sonnet-5}" + [ -n "${1:-}" ] && echo "harness: $1" + echo "skills:" + echo ' daily-brief: { enabled: true, schedule: "0 9 * * *" }' + echo ' odd-one: { enabled: true, harness: "vibe", model: "openai/gpt-5-mini" }' + } > aeon.yml +} + +# get KEY from a resolve run: `get [skill] [env assignments...]` +get() { + local key="$1" skill="${2:-}"; shift 2 || shift $# + env "$@" bash "$R" "$skill" 2>/dev/null | sed -n "s/^${key}=//p" +} + +# --- 1. defaults ------------------------------------------------------------ +mkfixture +[ "$(get HARNESS)" = "claude" ] \ + && pass "no harness: key → claude" || bad "default harness (got '$(get HARNESS)')" + +# A repo with NO `harness:` line at all must not die on grep's exit 1. Under the +# runner's `set -e` an unguarded grep kills the step with no message at all. +if bash "$R" >/dev/null 2>&1; then + pass "missing harness: key does not abort (grep guarded)" +else + bad "missing harness: key aborted the script" +fi + +# --- 2. resolution precedence ---------------------------------------------- +mkfixture codex +[ "$(get HARNESS)" = "codex" ] && pass "global harness: honoured" || bad "global harness" +[ "$(get HARNESS odd-one)" = "vibe" ] \ + && pass "per-skill harness: overrides global" || bad "per-skill harness override" +[ "$(get HARNESS odd-one INPUT_HARNESS=kimi)" = "kimi" ] \ + && pass "INPUT_HARNESS (dispatch) beats per-skill" || bad "INPUT_HARNESS precedence" +# The dropdown's placeholder must NOT be taken as a harness name. +[ "$(get HARNESS "" "INPUT_HARNESS=(config default)")" = "codex" ] \ + && pass "'(config default)' placeholder treated as unset" || bad "placeholder handling" +# Unknown value falls back to claude rather than reaching an install with no recipe. +[ "$(get HARNESS "" INPUT_HARNESS=banana)" = "claude" ] \ + && pass "unknown harness → claude" || bad "unknown harness fallback" +bash "$R" "" 2>&1 >/dev/null <<<"" | grep -q "::warning::unknown harness" \ + || true # only meaningful with INPUT_HARNESS set; asserted below +W=$(INPUT_HARNESS=banana bash "$R" 2>&1 >/dev/null) +case "$W" in *"::warning::unknown harness"*) pass "unknown harness warns on stderr" ;; + *) bad "unknown harness should warn (got: $W)" ;; esac + +# --- 3. AUTH_MODE detection ------------------------------------------------- +mkfixture codex +[ "$(get AUTH_MODE)" = "openrouter" ] \ + && pass "no native secret → openrouter" || bad "default AUTH_MODE" +[ "$(get AUTH_MODE "" CODEX_AUTH=xx)" = "native-oauth" ] \ + && pass "codex: CODEX_AUTH → native-oauth" || bad "codex native-oauth" +[ "$(get AUTH_MODE "" OPENAI_API_KEY=xx)" = "native-key" ] \ + && pass "codex: OPENAI_API_KEY → native-key" || bad "codex native-key" +# OAuth capture wins over the raw key — the same order install-harness.sh applies. +[ "$(get AUTH_MODE "" CODEX_AUTH=xx OPENAI_API_KEY=yy)" = "native-oauth" ] \ + && pass "codex: OAuth capture beats API key" || bad "codex auth precedence" +mkfixture grok +[ "$(get AUTH_MODE "" GROK_CREDENTIALS=xx)" = "native-oauth" ] \ + && pass "grok: GROK_CREDENTIALS → native-oauth" || bad "grok native-oauth" +[ "$(get AUTH_MODE "" XAI_API_KEY=xx)" = "native-key" ] \ + && pass "grok: XAI_API_KEY → native-key" || bad "grok native-key" + +# --- 4. MODEL_ARG per harness ---------------------------------------------- +# The whole point of MODEL_ARG: each CLI wants a different shape, and two want +# nothing at all. Passing a raw id to vibe/kimi breaks them (they resolve an ALIAS +# declared in the staged config), so empty is the correct answer, not a fallback. +mkfixture codex +[ "$(get MODEL_ARG)" = "openai/gpt-5-mini" ] \ + && pass "codex: MODEL_ARG is a bare OpenRouter id" || bad "codex MODEL_ARG" +mkfixture pi +[ "$(get MODEL_ARG)" = "openrouter/deepseek/deepseek-v4-flash" ] \ + && pass "pi: MODEL_ARG carries the openrouter/ prefix" || bad "pi MODEL_ARG (got '$(get MODEL_ARG)')" +for h in vibe kimi; do + mkfixture "$h" + [ -z "$(get MODEL_ARG)" ] \ + && pass "$h: MODEL_ARG empty (config alias decides)" || bad "$h MODEL_ARG must be empty" +done +# Native auth → forward NO model; the harness uses its own account default. A +# forwarded OpenRouter id here would be the wrong provider entirely. +mkfixture codex +[ -z "$(get MODEL_ARG "" CODEX_AUTH=xx)" ] \ + && pass "native auth → no --model forwarded" || bad "native auth MODEL_ARG" +case "$(get HARNESS_MODEL "" CODEX_AUTH=xx)" in + "(native:native-oauth)") pass "native auth labels the model as native" ;; + *) bad "native auth HARNESS_MODEL label" ;; +esac + +# --- 5. model precedence ---------------------------------------------------- +# An aeon-native id is NOT an OpenRouter id: a repo that never touched the model +# picker still reads `model: claude-sonnet-5`, and forwarding that would pin the +# run to a dead id while every downstream record named it. +mkfixture codex claude-sonnet-5 +[ "$(get MODEL_ARG)" = "openai/gpt-5-mini" ] \ + && pass "claude-* config model ignored → per-harness default" || bad "claude-* model passthrough" +mkfixture codex grok-4.5 +[ "$(get MODEL_ARG)" = "openai/gpt-5-mini" ] \ + && pass "grok-* config model ignored → per-harness default" || bad "grok-* model passthrough" +mkfixture codex openai/gpt-5 +[ "$(get MODEL_ARG)" = "openai/gpt-5" ] \ + && pass "OpenRouter config model is forwarded" || bad "OpenRouter model passthrough" +[ "$(get MODEL_ARG "" HARNESS_MODEL=openai/gpt-5-nano)" = "openai/gpt-5-nano" ] \ + && pass "vars.HARNESS_MODEL wins over the config model" || bad "HARNESS_MODEL precedence" +# odd-one pins harness "vibe", which forwards NO --model (alias-resolved), so the +# per-skill model is observable as HARNESS_MODEL — the id baked into vibe's staged +# config alias — not as MODEL_ARG. +[ "$(get HARNESS_MODEL odd-one)" = "openai/gpt-5-mini" ] \ + && pass "per-skill model reaches HARNESS_MODEL" || bad "per-skill model (got '$(get HARNESS_MODEL odd-one)')" +[ -z "$(get MODEL_ARG odd-one)" ] \ + && pass "per-skill model still not forwarded to vibe" || bad "vibe must not receive --model" + +# --- 6. output contract ----------------------------------------------------- +# Callers append this straight to $GITHUB_OUTPUT, so stdout must be exactly the +# four KEY=VALUE lines — the human summary belongs on stderr. +mkfixture codex +OUT=$(bash "$R" 2>/dev/null) +[ "$(echo "$OUT" | wc -l | tr -d ' ')" = "4" ] \ + && pass "stdout is exactly 4 KEY=VALUE lines" || bad "stdout line count (got: $OUT)" +echo "$OUT" | grep -qv '^[A-Z_]*=' && bad "stdout carried a non-KEY=VALUE line" \ + || pass "every stdout line is KEY=VALUE" +bash "$R" 2>&1 >/dev/null | grep -q "Harness: codex" \ + && pass "human summary goes to stderr" || bad "summary should be on stderr" + +echo "---" +[ "$fail" = "0" ] && echo "ALL PASS" || echo "SOME FAILED" +exit "$fail"