Skip to content

agentcompass 0.1.3: :minimal and :ultra are efforts; ultra reads as the model's Max, minimal sits below its Low - #750

Merged
yetone merged 8 commits into
magpie-community:mainfrom
akiiwa-games:agentcompass-effort-fallback
Nov 5, 2026
Merged

yetone merged 8 commits into
magpie-community:mainfrom
akiiwa-games:agentcompass-effort-fallback

Conversation

@akiiwa-games

@akiiwa-games akiiwa-games commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What

On main, model:minimal isn't an effort. EFFORT_ORDER is low..ultra, so rankCandidates doesn't split the suffix off. It expands the entry into every effort the model lists, and sends gpt-6-astra:minimal:low … gpt-6-astra:minimal:max. An uncalibrated ultra (Astra, Sol) scores at tier + 0, the same as High.

Changes (this differs from AgentCompass)

AgentCompass's effort_value (allocator.py:593) reads the model's curve, then policy[f'effort_{name}'], then the default. It has no ultra→max step and no minimal clamp, and its EFFORT_ORDER is ('low', 'medium', 'high', 'xhigh', 'max', 'ultra'). This PR adds, plugin-side:

  • EFFORT_ORDER gains none, off and minimal below low. model:minimal is one candidate at minimal, and the warm-cache downgrade check counts minimal after low as a downgrade.
  • effortValue:
    • ultra with no row of its own reads as the model's Max, or the policy's Max for a model with no curve.
    • none/off/minimal sit below the model's own Low by the policy's Low→(effort) step, at least 0.2. They burn at the policy's minimal/Low ratio of the model's Low, at most 0.75 of it. A model without a curve (glm-5.3, deepseek) scales from the policy table's Low the same way.
  • scoring.effort_ability gains none −1.1, off −1.1 and minimal −1.0, used for models without a curve. These are plugin values, not calibrated upstream.
  • evaluateModelQuota reads burnScale, as allocator.py derives its quota scale from burn_scale, so the allocator and the quota opportunity cost price an effort the same way. effortValue and burnScale live in quota.js; allocator.js re-exports them.
  • An effort written in a candidate is held to the task's ability floor like any other. :minimal, :none and :off score below :low, so a hard task excludes them.
  • The gateway never leaves a request unrouted. On main, when every candidate is below the floor, onRequest returns undefined and the request goes upstream as agentcompass and fails; a lone gemini-3.8-flash-high on a coding task does that today, and a lone :minimal would too once minimal is an effort. AgentCompass may decline to route, but a gateway must not. selectBestRoute takes fallbackToBestExcluded: it picks an excluded candidate and marks the decision fallback: first one that only misses the floor, then one that can see the request's images, with ability breaking ties. middleware and hooks pass it on their relaxed retry, which now runs whenever the first pass finds nothing. The allocator's own result is unchanged. The README says so.

Tests

  • 9.10:

    • parsing: :minimal and :ultra are one candidate each;
    • scoring: ultra equals the model's Max, minimal is the model's Low less the policy step, and the policy step applies for a curve without Low;
    • the warm-cache downgrade: minimal after low keeps 0.0625 of 0.25.
  • 9.11, the branches the default policy can't reach:

    • ultra on glm-5.3 (no curve) scores as the policy's Max;
    • the 0.2 step floor, under a policy whose Low→minimal step is −0.06;
    • the 0.75 burn clamp, under a policy with effort_burn.minimal 0.19 (and a control at 0.1);
    • none/off parse as efforts and score from their own rows, on and off a model's curve.
  • 9.13: through middleware.onRequest: a lone gpt-6-astra:minimal routes to itself, a lone gemini-3.8-flash-high routes, none+minimal picks minimal, minimal+high still picks high, an image request goes to a model that can see it and to a non-visual model only when the list has no other kind; through hooks.onRequest, the stored decision carries fallback. selectBestRoute without the option still returns ok: false.

  • 9.12: the quota opportunity cost on claude-opus-5-5, gpt-6-astra and glm-5.3 (no curve): :ultra costs what :max costs, none/off/minimal cost less than :low, max > high > low.

  • Broken by hand, one at a time:

    Break Red
    ultra → own max 9.10
    ultra → policy max 9.11
    the minimal branch 9.10, 9.11
    the step floor Math.min(delta, -0.2) 9.11
    the burn branch 9.10, 9.11
    the burn clamp Math.min(ratio, 0.75) 9.11
    minimal out of EFFORT_ORDER 9.10, 9.11
    none/off out of EFFORT_ORDER 9.11
    the old EFFORT_ORDER 9.10, 9.11
    minimal out of the policy 9.10
    none/off out of the policy 9.11
    quota scale from the raw burn tables (5a14905) 9.12
    below-Low scaling only from the model's own Low (faa2fa2) 9.12 (glm-5.3)
    no fallback in middleware 9.13
    no fallback in hooks 9.13
    the first excluded candidate instead of the most capable 9.13
    fallback by ability alone (6ee28fe) 9.13
    no floor-only tier 9.13
    no tier for seeing the input 9.13

At ff614f0: bun test 44 pass, node --test 44 pass, bun scripts/check.mjs agentcompass ok.

Main (with #749) is merged in at 5a14905. 8.12 compares effort_ability's low..max with v0.3.8 and pins none/off/minimal separately. With #749's policy (low −0.54), the Low→minimal step is −0.46; 9.10 reads it from the policy. bun test and node --test 42 pass.

🤖 Generated with Claude Code

…ax, minimal below Low

Before this, `model:minimal` was not an effort: rankCandidates expanded it to the model's whole
effort list, so it was scored and sent as High. AgentCompass's effort_value has no such fallback
either (it reads the model's curve, then the policy table, then the default), so this differs
from upstream on purpose:

- EFFORT_ORDER gains none/off/minimal below low (upstream is low..ultra).
- effortValue: an uncalibrated ultra reads as the model's own Max (upstream: tier + 0); none,
  off and minimal sit below the model's own Low by the policy's Low->minimal step, at least 0.2,
  and burn at most 0.75 of Low.
- scoring.effort_ability gains none -1.1, off -1.1 and minimal -1.0 for models with no curve.
  These are plugin-side values, not calibrated upstream.

9.10 covers parsing, scoring, the curve-less step and the warm-cache downgrade check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@yetone yetone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 16ee124. Requesting changes for the untested branches below, and for the body's claims about which breaks turn 9.10 red.

What was run (WSL, Bun 1.4.2, Node 22): bun test 35 pass, node --test 35 pass, bun scripts/check.mjs exit 0.

Hand breaks on effortValue (allocator.js:77-99). These turn 9.10 red:

  • no ultra→own max;
  • no minimal branch;
  • no burn branch;
  • minimal removed from EFFORT_ORDER;
  • the old EFFORT_ORDER;
  • minimal removed from the policy.

These stay green:

  1. No ultra→policy max (line 98). Add a model with no curve of its own and assert :ultra gets the policy's max.
  2. No Math.min(delta, -0.2) step floor (line 90). The default policy's step is never smaller than 0.2, so no test can see it. Add a policy where it matters.
  3. No Math.min(ratio, 0.75) burn clamp (line 94). effort_burn has no minimal row, so the ratio is always 0.12/0.2 = 0.6 and the clamp can't be reached. Test it with a policy that has effort_burn.minimal, or drop the clamp.
  4. none/off removed from EFFORT_ORDER or from the policy. Nothing covers them.

The body. It says that dropping the minimal clamp or the burn clamp turns 9.10 red. For the Math.min parts above, it doesn't. Please correct it.

Order. This conflicts with #749 on the version line. After #749 lands, the Low→minimal step becomes -0.46 (policy low -0.54), so rebase onto main and run the tests again then. The merge order is 0.1.2 (#749), then this.

…t policy can't reach

9.11: `ultra` on a model with no curve reads as the policy's Max; the 0.2 Low->minimal step floor and the
0.75 burn clamp, each under a policy where it decides the value; none/off parse as efforts and score from
their own policy rows, on and off a model's curve.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@akiiwa-games

Copy link
Copy Markdown
Contributor Author

Thanks. Pushed 2c8f8c9 with 9.11, which covers the four branches the default policy can't reach:

  1. Ultra → the policy's Max: glm-5.3 has no curve, and glm-5.3:ultra scores as glm-5.3:max (High + policy max). The test also calls effortValue(policy, undefined, …, "ultra") directly for ability and burn.
  2. The step floor: with a policy where Low→minimal is −0.06 (low −0.54, minimal −0.6), minimal lands 0.2 below the model's Low (−0.7). A deeper step (−0.46) is used as given (−0.96).
  3. The burn clamp: I kept it and tested it with effort_burn.minimal. Minimal 0.19 against Low 0.2 is clamped to 0.75 × the model's Low (0.4 → 0.3). Minimal 0.1 is used as given (0.2). The clamp keeps minimal at least 25% cheaper than Low, even under a policy that prices them nearly the same.
  4. none/off: :none and :off parse as efforts and score from their own policy rows, below minimal, both on Astra's curve and on glm-5.3.

Hand breaks, each run myself:

Break Red
ultra → own max 9.10
ultra → policy max 9.11
the minimal branch 9.10, 9.11
the step floor 9.11
the burn branch 9.10, 9.11
the burn clamp 9.11
minimal out of EFFORT_ORDER 9.10, 9.11
none/off out of EFFORT_ORDER 9.11
the old EFFORT_ORDER 9.10, 9.11
minimal out of the policy 9.10
none/off out of the policy 9.11

You were right about the body: the step floor and the burn clamp never turned 9.10 red. It's corrected.

After #749 lands: I checked the merge locally. The Low→minimal step becomes −1.0 − (−0.54) = −0.46. 9.10 computes the step from the policy, so it follows. 8.12 needs a one-line change to compare low..max only. 42 pass after the merge. I'll merge main in (no rebase or force-push) once #749 is in.

bun test 36 pass, node --test 36 pass, bun scripts/check.mjs agentcompass exit 0.

@yetone

yetone commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Re-reviewed at 2c8f8c9. The code holds on its own, but it can't merge yet: #749 landed as bf959aa, and this PR now conflicts with main.

What was run at 2c8f8c9 (mote, macOS, Bun 1.4.2, Node 22.22.0): bun test 36 pass, node --test test/router.test.mjs 36 pass, bun scripts/check.mjs agentcompass exit 0.

Hand breaks. I broke each of these by hand, one at a time. Every one turned a test red:

Break Red
ultra → own max 9.10
ultra → policy max 9.11
the minimal branch 9.10, 9.11
the step floor Math.min(delta, -0.2) 9.11
the burn branch 9.10, 9.11
the burn clamp Math.min(ratio, 0.75) 9.11
minimal out of EFFORT_ORDER 9.10, 9.11
none/off out of EFFORT_ORDER 9.11
the old EFFORT_ORDER 9.10, 9.11
minimal out of the policy 9.10
none/off out of the policy 9.11
none changed to -1.05 9.11

The four gaps from the last review are covered now.

Why it doesn't merge. I merged 2c8f8c9 into a copy of main with #749 squashed in. Three files conflict:

  • package.json: the version;
  • catalog.js: effort_ability;
  • router.test.mjs: the allocator import line.

Then I resolved them the obvious way: version 0.1.3, the policy as none/off/minimal −1.1/−1.1/−1.0 plus #749's low −0.54 and medium −0.2, and the union of the imports. With that resolution, 41 pass and 8.12 fails. It deepEquals effort_ability to v0.3.8's low..max, and this PR adds none/off/minimal to it. 9.10 passes with the −0.46 step.

Please merge main in and make 8.12 compare low..max only, as you planned. Then post the run, and I'll merge at that head.

…allback

Conflicts resolved: version stays 0.1.3; effort_ability keeps none/off/minimal (-1.1/-1.1/-1.0) on top of
magpie-community#749's low..max (low -0.54, medium -0.2); the test import line takes the union. 8.12 now compares
effort_ability's low..max with AgentCompass v0.3.8 and pins none/off/minimal as the plugin's own rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@akiiwa-games

Copy link
Copy Markdown
Contributor Author

Merged main in at 5a14905 with a normal merge, no rebase or force-push.

Conflicts:

8.12 now compares effort_ability's low..max with v0.3.8 and pins none/off/minimal separately as the plugin's own rows. Hand breaks at this head: low −0.54 → −0.7 turns 8.12 red, and minimal −1.0 → −0.9 turns 8.12 red. 9.10 reads the step from the policy, so with #749's low it uses the −0.46 Low→minimal step.

Run at 5a14905 (Windows, Bun 1.4, Node 24.12): bun test 42 pass, 0 fail; node --test test/router.test.mjs 42 pass, 0 fail; bun scripts/check.mjs agentcompass exit 0. README has no raw CR.

@yetone yetone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 5a14905. The merge resolution is right: package.json is 0.1.3, effort_ability is none/off/minimal −1.1/−1.1/−1.0 on top of #749's low..max unchanged, the import line is the union, and 8.12 compares low..max with v0.3.8 and pins the plugin's three rows apart. The PR's own diff against main (git diff origin/main...5a1490532 -- packages/agentcompass) is the same patch as the two-dot diff, so nothing from main was lost or reverted in the merge.

One thing still has to change before it merges.

Quota cost still reads :ultra and :minimal as High

effortValue's two fallbacks reach burnScale in allocator.js, but evaluateModelQuota in src/quota.js (lines 908-910) works out its own effortScale straight from the tables:

const refBurn = curves?.burn?.high ?? policy?.effort_burn?.high ?? 0.7;
const effBurn = curves?.burn?.[effort] ?? policy?.effort_burn?.[effort] ?? refBurn;

Neither table has an ultra or minimal row, so both get High's burn there. That effortScale sets the subscription's opportunity cost (stepUsd, unitRatePct). So after this PR:

  • :ultra scores Max's ability but costs High's quota, so on a subscription account it beats :max on every task.
  • :minimal costs more quota than :low (High's cost, ~3× Low's on Opus), the opposite of what the PR says.

evaluateModelQuota at this head, claude-opus-5-5, one subscription account with a 7d window 40% left and 120h to reset, { hours: 1.5, difficulty: "hard" }:

effort opportunity_cost allocator burnScale
minimal 0.1448 0.171
low 0.0457 0.286
high 0.1448 1.000
max 0.412 2.900
ultra 0.1448 2.900

Before this PR :ultra was High in both places, so the two agreed. Please have evaluateModelQuota use burnScale(policy, curves, effort) (or effortValue) so the quota side and the allocator read one scale, and add a test that fails without it: :ultra's opportunity cost equals :max's, and :minimal's is below :low's. none/off go the same way.

What I ran at 5a14905

On mote (macOS, Bun 1.4.2, Node 22.22.0):

  • bun test packages/agentcompass: 42 pass, 0 fail.
  • node --test test/router.test.mjs: 42 pass, 0 fail.
  • bun scripts/check.mjs agentcompass: exit 0.
  • A probe calling burnScale, effortAbility and evaluateModelQuota for claude-opus-5-5 and gpt-6-astra at minimal/low/high/max/ultra (the table above; gpt-6-astra shows the same split: burnScale 1.929 for ultra, quota scale 1.0).

Everything else in the diff reads right to me; once quota.js follows the same scale, this is ready to merge as 0.1.3.

evaluateModelQuota worked out its own effort scale from the burn tables, so
`:ultra` and `:minimal`/`none`/`off` (no table rows) cost High's quota there
while the allocator scored them through effortValue's fallbacks. It now calls
burnScale, as AgentCompass's allocator.py does for the quota effort scale;
effortValue and burnScale move to quota.js (allocator.js re-exports them) so
the two modules share one implementation without a circular import.

Test 9.12: `:ultra` costs what `:max` costs and `:minimal`/`none`/`off` cost
less than `:low`, on claude-opus-5-5 and gpt-6-astra subscriptions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@akiiwa-games

akiiwa-games commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks, you're right. Pushed faa2fa2.

Root cause: evaluateModelQuota computed its own effort scale from the raw burn tables, so efforts with no table row (ultra, minimal, none, off) fell back to High there. The allocator, meanwhile, scored them through effortValue's fallbacks. AgentCompass's allocator.py derives the quota effort scale from burn_scale, and the port should have done the same.

Fix:

  • evaluateModelQuota now calls burnScale(policy, curves, effort).
  • effortValue and burnScale moved into quota.js. allocator.js imports and re-exports them, so existing imports still work, both modules share one implementation, and there is no circular import.
  • Side effect: the quota side now also honours a model's burn_mult when it has one, as the allocator and upstream already did.

Test: new test 9.12 checks claude-opus-5-5 on a claude subscription and gpt-6-astra on a codex subscription (7d window, 40% left, 120h to reset, { hours: 1.5, difficulty: "hard" }). It asserts three things:

  • :ultra's opportunity cost equals :max's;
  • none, off and minimal each cost less than :low;
  • max > high > low.

It fails at 5a14905 and passes at faa2fa2.

The same probe as your table, at faa2fa2:

effort opus opportunity_cost opus burnScale astra opportunity_cost astra burnScale
none / minimal 0.0299 0.171 0.1165 0.266
low 0.0457 0.286 0.1905 0.443
high 0.1448 1.000 0.4252 1.000
max 0.412 2.900 0.8214 1.929
ultra 0.412 2.900 0.8214 1.929

What I ran at faa2fa2 (Windows, Bun and Node 24.12):

  • bun test packages/agentcompass: 43 pass, 0 fail.
  • node --test test/router.test.mjs: 43 pass, 0 fail.
  • bun scripts/check.mjs agentcompass: exit 0.

package.json stays at 0.1.3.

@yetone yetone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at faa2fa2. evaluateModelQuota now reads burnScale, and 9.12 holds for the two models it checks. One more thing still has to change before this merges, and there is one behaviour change I'd like you to look at.

1. On a model with no curve, :minimal / :none / :off still burn like High

The burn fallback in effortValue (quota.js) runs only when the model's own curve has a Low row ("low" in own). A model with no curve (glm-5.3, deepseek, every model outside EFFORT_CURVES) goes on to policyTable[effort]. effort_burn has no minimal, none or off row, so it returns fallback. That fallback is the High reference, so the scale comes out at 1.0.

burnScale(DEFAULT_SCORING_POLICY, {}, effort) at faa2fa2:

effort burnScale
low 0.286
minimal / none / off 1.000
high 1.000
ultra / max 1.857

So on those models :minimal costs 3.5x :low in both the allocator and the quota opportunity cost. That is the same split the last round found for ultra, here on the models without a curve. The body says "A model without a Low row takes the policy table's value", but the policy table has no such row. Please give the no-curve case the same rule, e.g. policyTable.low * min(ratio, 0.75), or add minimal/none/off rows to effort_burn. Then add a model without a curve (glm-5.3) to 9.12 so it goes red without the fix.

2. A lone :minimal / :none candidate now routes nothing on coding tasks

Through magpie's own moejs middleware runner, on the request "Refactor the payment retry loop in billing/retry.go and add tests for the backoff" with candidateModels: ["gpt-6-astra:minimal"]:

  • main: model becomes gpt-6-astra:minimal. The effort is wrong (high), but the request goes through.
  • faa2fa2: the candidate is excluded (ability 3.15 below the coupled floor 3.60; none is 3.05). Nothing is selected, so onRequest returns undefined and the request goes upstream as model: "agentcompass". This happens on anthropic, responses and chat alike.

With ["gpt-6-astra:minimal", "gpt-6-astra:high"], minimal is never chosen for that task.

Gating minimal by the floor may be what you want. But a user who lists only :minimal had a working route before and gets a failed request now. Please either keep an explicitly listed effort reachable when it is the only candidate (or fall back to the best excluded one), or say in the body and README that :minimal is held to the same floor and can leave a request unrouted.

What I ran at faa2fa2 (merged onto main 15f4fa8)

On mote (macOS, Bun 1.4.2, Node 22):

  • bun test packages/agentcompass 43 pass; node --test test/router.test.mjs 43 pass; bun scripts/check.mjs agentcompass ok.
  • main and this PR loaded as middleware into magpie's moejs runner (internal/middleware Begin/Request). The candidates were :minimal, :ultra, :none and glm-5.3:ultra+:max, on the anthropic, responses and chat protocols. No load errors or hook failures, and each onRequest took under 1ms. This is where item 2 came from.
  • A Bun probe calling rankCandidates/selectBestRoute gave the exclusion reasons above, and burnScale on an empty curve gave the table in item 1.

Everything else in the diff reads right to me. That covers the EFFORT_ORDER change (middleware.js/hooks.js already strip none/off/minimal as efforts), the ultra fallbacks, the move of effortValue/burnScale into quota.js with no import cycle, and 9.10-9.12.

…a model without a curve

effortValue scaled minimal/none/off from the model's own Low row only. A model with no effort curve
(glm-5.3, deepseek, any model outside EFFORT_CURVES) fell through to effort_burn, which has no such rows,
so their burn came back as the High reference: :minimal cost 3.5x :low there. The policy table's Low is
now the base when the model has none. 9.12 adds glm-5.3 and fails without the fix.

README: an effort written in a candidate is held to the task's ability floor like any other; a list whose
every candidate is below the floor leaves the request unrouted, as a lone gemini-3.8-flash-high already
does on main.
@akiiwa-games

Copy link
Copy Markdown
Contributor Author

Thanks, both reproduced. Pushed ee9e07e.

1. Efforts below Low on a model with no curve

Root cause: effortValue scaled none/off/minimal from the model's own Low row only. A model with no curve fell through to effort_burn, which has no such rows, so it got the High reference.

Fix: the base is the model's own Low, else the policy table's Low. Ability is unchanged on those models, since the default table's Low plus the step equals its own none/off/minimal rows.

burnScale(DEFAULT_SCORING_POLICY, {}, effort) at ee9e07e:

effort faa2fa2 ee9e07e
low 0.286 0.286
minimal / none / off 1.000 0.171
high 1.000 1.000
ultra / max 1.857 1.857

Test: 9.12 now also checks glm-5.3 on a glm-5.3/sub subscription. At faa2fa2 it fails with glm-5.3 minimal 0.0247 vs low 0.0115; at ee9e07e it passes.

2. A lone :minimal candidate on a coding task

I went with documenting it. Through middleware.onRequest with your request:

candidates main (2b7eb23) ee9e07e
gpt-6-astra:minimal gpt-6-astra:minimal unrouted
gpt-6-astra:none gpt-6-astra:none unrouted
gemini-3.8-flash-high unrouted unrouted
gpt-6-astra:minimal, gpt-6-astra:high gpt-6-astra:minimal gpt-6-astra:high

On main, a lone candidate below the task's floor already leaves the request unrouted; gemini-3.8-flash-high shows it. Main routed :minimal only because it didn't read the suffix as an effort, and then sent it at High. This PR holds :minimal to the same floor as every other candidate.

A fallback to the best excluded candidate would change that for every list, so I kept it out of this PR. I can open it as its own PR if you want that behaviour.

The body and the README now say that an effort written in a candidate is held to the floor, and that a list whose every candidate is below the floor leaves the request unrouted. The README suggests listing a candidate that clears the floor next to a low one.

Run at ee9e07e

bun test 43 pass, node --test test/router.test.mjs 43 pass, bun scripts/check.mjs agentcompass ok (Windows, Bun 1.4.0, Node 24.12).

When every candidate was below the task's ability floor, onRequest returned undefined and the request went
upstream as the virtual model name, which fails: a lone gemini-3.8-flash-high on a coding task on main, and
a lone :minimal once minimal became an effort. AgentCompass may decline to route; a gateway must not.
selectBestRoute takes fallbackToBestExcluded, which picks the most capable excluded candidate and marks the
decision fallback; middleware and hooks use it on their relaxed retry, which now runs whenever the first
pass finds nothing. 9.13 covers both entry points and fails without the fix.

README: the floor note says the gateway falls back instead of leaving the request unrouted.
@akiiwa-games

Copy link
Copy Markdown
Contributor Author

On item 2, I changed my mind: leaving the request unrouted is the wrong behaviour for a gateway, so this PR now falls back instead of only documenting it. Pushed 6ee28fe.

AgentCompass itself may decline to route when nothing clears the floor. Magpie can't: an unrouted request goes upstream as agentcompass and fails. That's already true on main for a lone gemini-3.8-flash-high on your coding request.

Fix:

  • selectBestRoute takes fallbackToBestExcluded. When nothing is eligible, it picks the excluded candidate with the highest ability and marks the decision fallback (with its exclusion reasons).
  • middleware and hooks pass the option on their relaxed retry. That retry now runs whenever the first pass finds nothing, not only when unhealthy, context or quota constraints were set.
  • Without the option, selectBestRoute returns what it did before, so the allocator's own result is unchanged.

Through middleware.onRequest with your request:

candidates main (2b7eb23) ee9e07e 6ee28fe
gpt-6-astra:minimal gpt-6-astra:minimal (sent at High) unrouted gpt-6-astra:minimal
gemini-3.8-flash-high unrouted unrouted gemini-3.8-flash-high:high
gpt-6-astra:none, gpt-6-astra:minimal gpt-6-astra:none unrouted gpt-6-astra:minimal
gpt-6-astra:minimal, gpt-6-astra:high gpt-6-astra:minimal gpt-6-astra:high gpt-6-astra:high

Test: 9.13 covers both middleware.onRequest and hooks.onRequest. I broke each part by hand and each break turned 9.13 red:

  • the option dropped from middleware;
  • the option dropped from hooks;
  • picking the first excluded candidate instead of the most capable.

bun test 44 pass, node --test test/router.test.mjs 44 pass, bun scripts/check.mjs agentcompass ok (Windows, Bun 1.4.0, Node 24.12). The README note now says the gateway falls back.

@yetone yetone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 6ee28fe. Both earlier items are fixed, and each fix has a test that fails without it. Two things still stop the merge.

1. The fallback sends an image request to a model that can't see it, even when the list has one that can

selectBestRoute (allocator.js:830-834) falls back to the excluded candidate with the highest ability_value, whatever excluded it. task needs visual input (allocator.js:391) is a hard exclusion, not a floor miss, so it shouldn't be weighed the same way.

Chat request through middleware.onRequest with "Refactor the payment retry loop in billing/retry.go to match this screenshot and add tests for the backoff", plus one image_url part:

candidates main 39d4456 6ee28fe fallback.excluded
claude-fable-5-1:high, gemini-3.8-flash-high:low unrouted claude-fable-5-1:high ["task needs visual input"]
grok-4.7:high, gemini-3.8-flash-high:minimal unrouted grok-4.7:high ["task needs visual input", "ability 3.40 below the coupled floor 3.60"]

In both cases the list has a visual model (gemini-3.8-flash-high) that misses only the floor, but the fallback picks the non-visual one. Please have the fallback prefer candidates whose only exclusion is the ability floor (or at least ones not excluded for visual input), and use raw ability only to break ties among those. Falling back to a non-visual model is fine when it's the only kind in the list. Please add the first row to 9.13.

The candidate pool is only the user's own candidateModels, so the fallback never routes outside the list. I checked that.

2. It conflicts with main

#765 (15f4fa8) rewrote packages/agentcompass/README.md as a concise English README, and GitHub reports this PR as CONFLICTING there. Please rebase onto main and put the floor/fallback note in the new README. Right now it's a Chinese bullet only, in a section #765 rewrote. Please also say there that the fallback can override the visual/unhealthy/context/quota constraints, since the relaxed retry drops the last three.

What I ran at 6ee28fe (on mote: macOS, Bun 1.4.2, Node 22, HOME=$(mktemp -d))

  • bun test 44 pass; node --test test/router.test.mjs 44 pass; bun scripts/check.mjs agentcompass ok.
  • Broke each fix by hand. Each break turned a test red:
    • quota.js low without the policy-table fallback: 9.12 red.
    • fallbackToBestExcluded removed from middleware.js: 9.13 red.
    • The same removal from hooks.js: 9.13 red.
    • allCandidates[0] instead of the most capable candidate: 9.13 red.
    • middleware's old retry condition (unhealthy || minContext || quotas): 9.13 red.
  • The visual probe above, at this head and at main 39d4456, with selectBestRoute(..., { fallbackToBestExcluded: true }) called directly too.

The rest of the diff reads right: the policy-table Low in effortValue, the retry that now always runs, and fallback carried on the hooks route.

akiiwa-games and others added 2 commits November 6, 2026 06:56
…-fallback

Takes magpie-community#765's English README and puts the floor and fallback note in both of its languages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… see it

The fallback picked the excluded candidate with the highest ability, whatever excluded it. A request with an image
could go to a capable model that can't read it, even when the list had one that can and only missed the floor.
Prefer candidates that only miss the floor, then ones that can see the input; ability breaks ties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@akiiwa-games

Copy link
Copy Markdown
Contributor Author

Thanks, reproduced. Pushed ff614f0, with main merged in. I merged rather than rebased so the branch stays free of force-pushes.

1. The fallback ignored why a candidate was excluded

Root cause: the fallback compared ability_value only, so a visual exclusion counted for nothing.

Fix: selectBestRoute now picks in this order, with ability breaking ties inside a tier:

  1. Candidates whose only exclusion is the ability floor.
  2. Candidates that can see the request's images.
  3. Candidates that can't. These are chosen only when the list has no other kind.

Your request, with { needs_visual: true }:

candidates 6ee28fe ff614f0 fallback.excluded
claude-fable-5-1:high, gemini-3.8-flash-high:low claude-fable-5-1:high gemini-3.8-flash-high:low ["ability 2.60 below the coupled floor 3.60"]
grok-4.7:high, gemini-3.8-flash-high:minimal grok-4.7:high gemini-3.8-flash-high:minimal ["ability 2.14 below the coupled floor 3.60"]
claude-fable-5-1:high, grok-4.7:high claude-fable-5-1:high claude-fable-5-1:high ["task needs visual input"]

Test: 9.13 has both of your rows through middleware.onRequest, your first row through hooks.onRequest, and the non-visual-only list. It also has two selectBestRoute cases with a restrict_models route, one for each tier boundary. The first row failed before the fix (claude-fable-5-1:high instead of gemini-3.8-flash-high:low).

Each break below turned 9.13 red:

  • ability only, with no tiers
  • no floor-only tier
  • no tier for seeing the input
  • tier order flipped
  • onlyBelowFloor set from the floor alone, ignoring other exclusions

2. Conflict and README

Merged main (42331f6), taking #765's README whole. The Chinese bullet from this PR is gone. The new README has the floor and fallback note in its English and Chinese sections, under the gateway middleware quickstart. It says the fallback can override the visual, unhealthy, context and quota constraints, and that it only picks from the candidate list. The file has no raw CRs.

What I ran at ff614f0

Windows, Bun 1.4.0, Node 24.12, temp HOME: bun test 44 pass, node --test test/router.test.mjs 44 pass, bun scripts/check.mjs agentcompass exit 0.

@yetone
yetone merged commit 10a61c2 into magpie-community:main Nov 5, 2026
2 checks passed
@yetone

yetone commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Thanks @akiiwa-games. Every finding from the earlier rounds is fixed, and each fix goes red when I break it by hand. Merged as 10a61c2. gateway-agentcompass 0.1.3 is on the dist branch now (tgz/magpie-community-gateway-agentcompass-0.1.3.tgz), and magpie installs it from there. The package isn't on npm yet: its first npm version needs an owner to publish it.

akiiwa-games added a commit to akiiwa-games/plugins that referenced this pull request Nov 6, 2026
…cal-model

Keeps 0.1.4, the version after magpie-community#750's 0.1.3, and imports both sides' names in the test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants