Skip to content

fix(harness): stage all six harnesses on messages.yml via shared resolve/install scripts #789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aaronjmars merged 1 commit into from
Jul 27, 2026
Merged

fix(harness): stage all six harnesses on messages.yml via shared resolve/install scripts #789

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
283 changes: 12 additions & 271 deletions .github/workflows/aeon.yml
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions .github/workflows/ci-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,10 @@ jobs:
run: bash scripts/tests/test_run_grok.sh
- name: harness-adapter grok tests
run: bash scripts/tests/test_harness_adapter_grok.sh
- name: harness resolution tests
run: bash scripts/tests/test_resolve_harness.sh
- name: harness install/config tests
run: bash scripts/tests/test_install_harness.sh
- name: state reducer tests
run: python3 scripts/tests/test_state_reduce.py
- name: health triage tests
Expand Down
129 changes: 92 additions & 37 deletions .github/workflows/messages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,58 @@ jobs:
# (and the cache key version above).
run: npm install -g @anthropic-ai/claude-code@2.1.168 --prefer-offline

# Inbound messaging isn't a per-skill run, so no skill name is passed and
# the repo-global `harness:`/`model:` keys decide — the same script aeon.yml
# uses, so a repo answers messages on the harness it runs skills on.
- name: Resolve harness
id: harness
if: steps.msg.outputs.source != ''
env:
HARNESS_MODEL: ${{ vars.HARNESS_MODEL }}
# Native-auth secrets — presence ONLY, to detect which provider each
# harness runs on (native login/key > OpenRouter). Never echoed.
CODEX_AUTH: ${{ secrets.CODEX_AUTH }}
KIMI_AUTH: ${{ secrets.KIMI_AUTH }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }}
MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
ANTHROPIC_OAUTH_TOKEN: ${{ secrets.ANTHROPIC_OAUTH_TOKEN }}
GROK_CREDENTIALS: ${{ secrets.GROK_CREDENTIALS }}
XAI_API_KEY: ${{ secrets.XAI_API_KEY }}
run: bash scripts/resolve-harness.sh >> "$GITHUB_OUTPUT"

# Stage the resolved harness's CLI + provider auth. This step is why
# messages.yml now supports all six: it previously had none, so a repo on
# codex/pi/vibe/kimi had its messages answered on claude with a warning.
# The `env:` block is duplicated from aeon.yml by necessity (`secrets.*`
# resolves per workflow); the recipes it feeds are shared.
- name: Install harness CLI
if: steps.msg.outputs.source != '' && steps.harness.outputs.HARNESS != '' && steps.harness.outputs.HARNESS != 'claude'
env:
H: ${{ steps.harness.outputs.HARNESS }}
HM: ${{ steps.harness.outputs.HARNESS_MODEL }}
AUTH_MODE: ${{ steps.harness.outputs.AUTH_MODE }}
# grok auth (staged via run-grok.sh setup): captured X-account OAuth or key,
# plus the secrets-write PAT that persists a rotated refresh token back to
# GROK_CREDENTIALS (run-grok.sh §2b).
GROK_CREDENTIALS: ${{ secrets.GROK_CREDENTIALS }}
XAI_API_KEY: ${{ secrets.XAI_API_KEY }}
GH_SECRETS_PAT: ${{ secrets.GH_SECRETS_PAT }}
GH_GLOBAL: ${{ secrets.GH_GLOBAL }}
GROK_OAUTH_SKEW: ${{ vars.GROK_OAUTH_SKEW }}
# Provider credentials — restored (OAuth captures) or written into the
# harness's config (API keys). All land in the runner's ephemeral $HOME.
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
CODEX_AUTH: ${{ secrets.CODEX_AUTH }}
KIMI_AUTH: ${{ secrets.KIMI_AUTH }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }}
MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
run: |
set -euo pipefail
bash scripts/install-harness.sh "$H"

- name: Run
id: run
if: steps.msg.outputs.source != ''
Expand Down Expand Up @@ -438,6 +490,19 @@ jobs:
GH_SECRETS_PAT: ${{ secrets.GH_SECRETS_PAT }} # persist rotated grok refresh token → GROK_CREDENTIALS (run-grok.sh §2b)
GH_GLOBAL: ${{ secrets.GH_GLOBAL }}
GROK_OAUTH_SKEW: ${{ vars.GROK_OAUTH_SKEW }} # force grok OAuth refresh + persist every run when set large (durability check)
# Resolved by the "Resolve harness" step above; the CLI they name was
# staged by "Install harness CLI".
HARNESS: ${{ steps.harness.outputs.HARNESS }}
AUTH_MODE: ${{ steps.harness.outputs.AUTH_MODE }}
MODEL_ARG: ${{ steps.harness.outputs.MODEL_ARG }}
# Provider keys the non-claude harnesses read at RUN time (pi picks its
# provider from whichever is in env; codex/kimi/vibe read the key their
# staged config names). OPENROUTER_API_KEY is already declared above.
CODEX_AUTH: ${{ secrets.CODEX_AUTH }}
KIMI_AUTH: ${{ secrets.KIMI_AUTH }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
MOONSHOT_API_KEY: ${{ secrets.MOONSHOT_API_KEY }}
MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
COINGECKO_API_KEY: ${{ secrets.COINGECKO_API_KEY }}
ALCHEMY_API_KEY: ${{ secrets.ALCHEMY_API_KEY }}
_MSG_SOURCE: ${{ steps.msg.outputs.source }}
Expand All @@ -447,46 +512,24 @@ jobs:
CONFIG_MODEL=$(grep -E '^model:' aeon.yml | sed 's/^model: *//' | tr -d ' ')
MODEL="${CONFIG_MODEL:-claude-sonnet-5}"

# Harness resolution — inbound messaging isn't a per-skill run, so it
# follows the repo's global `harness:` in aeon.yml (same default/guard as
# aeon.yml). A grok-only repo (no Claude creds) must respond via grok.
CONFIG_HARNESS=$(grep -E '^harness:' aeon.yml | sed 's/^harness: *//' | tr -d ' ')
HARNESS="${CONFIG_HARNESS:-claude}"
# Only claude and grok have their CLI staged on this workflow ("Install
# Claude Code" above, plus the grok setup below). The other four
# run-harness harnesses would need their CLI + provider auth installed
# here the way aeon.yml's "Install harness CLI" step does. Until that
# lands, a repo configured for one of them answers messages on claude —
# say so out loud instead of swapping the harness silently.
case "$HARNESS" in
claude|grok) ;;
codex|pi|vibe|kimi)
echo "::warning::harness '$HARNESS' has no CLI staged on the messages workflow — replying on claude instead (skill runs still use $HARNESS)."
HARNESS="claude" ;;
*)
echo "::warning::unknown harness '$HARNESS' — falling back to claude"
HARNESS="claude" ;;
esac
# HARNESS / AUTH_MODE / MODEL_ARG come from the "Resolve harness" step
# (scripts/resolve-harness.sh), and the CLI was staged by "Install harness
# CLI" — so all six answer messages now. This used to be an inline
# claude-or-grok resolution here that downgraded the other four to claude
# with a ::warning::, because nothing on this workflow could install them.
# On grok a claude-* model id is meaningless — fall back to grok's default.
if [ "$HARNESS" = "grok" ]; then
case "$MODEL" in claude-*|"") MODEL="grok-4.5" ;; esac
fi
echo "Using harness: $HARNESS | model: $MODEL"

# Stage grok's CLI + auth. run-grok.sh's `setup` subcommand is the single
# source of truth for the version pin, the GROK_CREDENTIALS restore and
# the OAuth refresh/persist (§2b). This used to happen implicitly, because
# the run itself went through run-grok.sh and step 1 installs the CLI on
# demand; now that the run goes through run-harness (whose grok adapter
# only checks `command -v grok`), staging has to be explicit.
if [ "$HARNESS" = "grok" ]; then
bash "${GITHUB_WORKSPACE}/scripts/run-grok.sh" setup
fi

# AI Gateway routing — same path as aeon.yml's Run steps. The grok harness
# has its own auth and does NOT use the gateway, so skip it there (avoids
# "no gateway/Anthropic API" noise on a grok-only repo).
if [ "$HARNESS" != "grok" ]; then
echo "Using harness: $HARNESS | auth: $AUTH_MODE | model: ${MODEL_ARG:-$MODEL}"

# AI Gateway routing — same path as aeon.yml's Run step, and gated the
# same way: it is Anthropic-specific (llm-gateway.sh rewrites
# ANTHROPIC_BASE_URL/model env), so ONLY claude sources it. This used to
# read `!= grok`, which was correct while claude and grok were the only
# two possible harnesses; with codex/pi/vibe/kimi reachable it would
# point their env at an Anthropic gateway they never call.
if [ "$HARNESS" = "claude" ]; then
GATEWAY=$(grep -A1 '^gateway:' aeon.yml | grep 'provider:' | sed 's/.*provider:[[:space:]]*//' | sed "s/[\"' ]//g" || true)
GATEWAY="${GATEWAY:-auto}"
source "${GITHUB_WORKSPACE}/scripts/llm-gateway.sh"
Expand Down Expand Up @@ -614,9 +657,21 @@ jobs:
# run-harness re-expands the config and translates it per harness; the
# claude adapter adds --strict-mcp-config itself, as this step used to.
if [ "${#MCP_FLAGS[@]}" -gt 0 ]; then RH_ARGS+=(--mcp-config .mcp.json); fi
# Which --model to forward, mirroring aeon.yml's Run step. aeon's own ids
# are claude-*/grok-*, which mean nothing to an OpenRouter-backed CLI:
# forwarding one there leaves the harness on its default while every
# record names a model that never ran. MODEL_ARG (from resolve-harness.sh)
# is already the right value or empty for "the staged config decides" —
# vibe and kimi resolve a config ALIAS, so passing --model breaks them.
RH_MODEL_ARGS=()
case "$HARNESS" in
claude) RH_MODEL_ARGS=(--model "$MODEL") ;;
grok) case "$MODEL" in grok-*) RH_MODEL_ARGS=(--model "$MODEL") ;; esac ;;
*) if [ -n "${MODEL_ARG:-}" ]; then RH_MODEL_ARGS=(--model "$MODEL_ARG"); fi ;;
esac
HARNESS_ERR=$(mktemp)
if ! CLAUDE_OUTPUT=$(echo "$PROMPT" | bash "$RH" "$HARNESS" \
--model "$MODEL" "${RH_ARGS[@]}" 2>"$HARNESS_ERR"); then
"${RH_MODEL_ARGS[@]}" "${RH_ARGS[@]}" 2>"$HARNESS_ERR"); then
echo "::error::$HARNESS harness failed: $(tail -c 400 "$HARNESS_ERR" | tr '\n' ' ')"
exit 1
fi
Expand Down
32 changes: 32 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,38 @@ from or pin to; the template keeps serving the latest `main` to new forks.

### Fixed

- **Inbound messages now run on all six harnesses.** `messages.yml` staged only
the claude and grok CLIs, so a repo configured for codex/pi/vibe/kimi had every
inbound message answered on **claude** — loudly (`::warning::`), but still not
the harness the operator chose, and on Anthropic credentials a
single-provider fork may not even have. The cause was structural: both the
~100-line harness/provider/model resolution and the ~150-line install recipes
lived *inside* `aeon.yml` steps, so no other workflow could reach them.
Extracted to **`scripts/resolve-harness.sh`** (prints `HARNESS`/`AUTH_MODE`/
`HARNESS_MODEL`/`MODEL_ARG` as `KEY=VALUE` lines) and
**`scripts/install-harness.sh`**; both workflows now call them. Each still
declares its own `env:` block, since `secrets.*` only resolves per workflow —
but the logic, the half that drifted, is shared. This is the same class of bug
as the two-path grok trap (#784), and the last surface where "aeon supports six
harnesses" wasn't true. Verified by a differential test: 832 resolution cases
and every config-generating install case produce byte-identical output to the
inline versions they replace.
- The extraction also made two latent bugs reachable and fixed both. A missing
provider credential used to die *mid-heredoc* under `set -u` — the config
file was already created, so the harness was left holding a **0-byte
`config.toml`** and an error naming a shell variable instead of the secret.
(It never fired inside a workflow, where the `env:` block always binds the
name; it appears the moment the script is callable standalone.) It now fails
closed and names the missing secret. Separately, `messages.yml` gated the
Anthropic gateway on `!= grok`, correct only while claude and grok were the
sole options — with the other four reachable it would have pointed their env
at a gateway they never call. Now gated on `= claude`.
- Both scripts get real test suites — `test_resolve_harness.sh` (28 cases) and
`test_install_harness.sh` (21 cases). This logic had **never** been tested:
inside a workflow step the only way to exercise it was to dispatch a live
run. The install tests stub `npm`/`pipx` and assert what actually breaks —
the generated codex/kimi/vibe provider config and the supply-chain version
pins.
- **The docs described a read-only guard that does not exist, and the code it
cited is deleted.** `docs/CAPABILITIES.md` and `docs/harnesses.md` both stated
that a `mode: read-only` skill is confined on grok by an explicit tool allowlist
Expand Down
22 changes: 16 additions & 6 deletions docs/harnesses.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,12 +266,22 @@ path until it was removed: nothing in the repo ever set it, and the scorer it wa
reserved for goes schema-less on purpose so a single parse path covers all six
harnesses.)

**Harness coverage differs by surface**, because each has to stage the CLI it
dispatches to. Skill runs (`aeon.yml`) install any of the six. The local MCP
server resolves all six but expects the CLI to already be installed on your
machine. `messages.yml` stages only `claude` and `grok`; a repo configured for
one of the other four gets a `::warning::` and is answered on claude, rather than
being switched silently.
**Both hosted surfaces stage all six.** `aeon.yml` (skill runs) and
`messages.yml` (inbound messages) share the same two scripts, so a repo answers
messages on the harness it runs skills on:

- **`scripts/resolve-harness.sh`** — decides `HARNESS` / `AUTH_MODE` /
`HARNESS_MODEL` / `MODEL_ARG` and prints them as `KEY=VALUE` lines. Pass a
skill name to pick up per-skill `harness:`/`model:` overrides; omit it (as
`messages.yml` does) and the repo-global keys decide.
- **`scripts/install-harness.sh`** — stages that harness's CLI and provider auth.

Each workflow still declares its own `env:` block, because `secrets.*` only
resolves inside a workflow — but the logic is shared, which is the half that
drifted before. `messages.yml` used to carry a second, weaker copy that knew only
claude and grok, so a repo on codex/pi/vibe/kimi had its messages answered on
claude with a `::warning::`. The local MCP server (`apps/mcp-server`) resolves all
six but expects the CLI to already be installed on your machine.

## Every entry point runs on either harness

Expand Down
Loading