Repository navigation
agentcompass 0.1.3: :minimal and :ultra are efforts; ultra reads as the model's Max, minimal sits below its Low - #750
Conversation
…ax, minimal below Low Before this, `model:minimal` was not an effort: rankCandidates expanded it to the model's whole effort list, so it was scored and sent as High. AgentCompass's effort_value has no such fallback either (it reads the model's curve, then the policy table, then the default), so this differs from upstream on purpose: - EFFORT_ORDER gains none/off/minimal below low (upstream is low..ultra). - effortValue: an uncalibrated ultra reads as the model's own Max (upstream: tier + 0); none, off and minimal sit below the model's own Low by the policy's Low->minimal step, at least 0.2, and burn at most 0.75 of Low. - scoring.effort_ability gains none -1.1, off -1.1 and minimal -1.0 for models with no curve. These are plugin-side values, not calibrated upstream. 9.10 covers parsing, scoring, the curve-less step and the warm-cache downgrade check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yetone
left a comment
There was a problem hiding this comment.
Reviewed at 16ee124. Requesting changes for the untested branches below, and for the body's claims about which breaks turn 9.10 red.
What was run (WSL, Bun 1.4.2, Node 22): bun test 35 pass, node --test 35 pass, bun scripts/check.mjs exit 0.
Hand breaks on effortValue (allocator.js:77-99). These turn 9.10 red:
- no ultra→own max;
- no minimal branch;
- no burn branch;
- minimal removed from
EFFORT_ORDER; - the old
EFFORT_ORDER; - minimal removed from the policy.
These stay green:
- No ultra→policy max (line 98). Add a model with no curve of its own and assert
:ultragets the policy's max. - No
Math.min(delta, -0.2)step floor (line 90). The default policy's step is never smaller than 0.2, so no test can see it. Add a policy where it matters. - No
Math.min(ratio, 0.75)burn clamp (line 94).effort_burnhas nominimalrow, so the ratio is always 0.12/0.2 = 0.6 and the clamp can't be reached. Test it with a policy that haseffort_burn.minimal, or drop the clamp. none/offremoved fromEFFORT_ORDERor from the policy. Nothing covers them.
The body. It says that dropping the minimal clamp or the burn clamp turns 9.10 red. For the Math.min parts above, it doesn't. Please correct it.
Order. This conflicts with #749 on the version line. After #749 lands, the Low→minimal step becomes -0.46 (policy low -0.54), so rebase onto main and run the tests again then. The merge order is 0.1.2 (#749), then this.
…t policy can't reach 9.11: `ultra` on a model with no curve reads as the policy's Max; the 0.2 Low->minimal step floor and the 0.75 burn clamp, each under a policy where it decides the value; none/off parse as efforts and score from their own policy rows, on and off a model's curve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks. Pushed 2c8f8c9 with 9.11, which covers the four branches the default policy can't reach:
Hand breaks, each run myself:
You were right about the body: the step floor and the burn clamp never turned 9.10 red. It's corrected. After #749 lands: I checked the merge locally. The Low→minimal step becomes −1.0 − (−0.54) = −0.46. 9.10 computes the step from the policy, so it follows. 8.12 needs a one-line change to compare low..max only. 42 pass after the merge. I'll merge main in (no rebase or force-push) once #749 is in.
|
|
Re-reviewed at 2c8f8c9. The code holds on its own, but it can't merge yet: #749 landed as bf959aa, and this PR now conflicts with main. What was run at 2c8f8c9 (mote, macOS, Bun 1.4.2, Node 22.22.0): Hand breaks. I broke each of these by hand, one at a time. Every one turned a test red:
The four gaps from the last review are covered now. Why it doesn't merge. I merged 2c8f8c9 into a copy of main with #749 squashed in. Three files conflict:
Then I resolved them the obvious way: version 0.1.3, the policy as none/off/minimal −1.1/−1.1/−1.0 plus #749's low −0.54 and medium −0.2, and the union of the imports. With that resolution, 41 pass and 8.12 fails. It deepEquals Please merge main in and make 8.12 compare low..max only, as you planned. Then post the run, and I'll merge at that head. |
…allback Conflicts resolved: version stays 0.1.3; effort_ability keeps none/off/minimal (-1.1/-1.1/-1.0) on top of magpie-community#749's low..max (low -0.54, medium -0.2); the test import line takes the union. 8.12 now compares effort_ability's low..max with AgentCompass v0.3.8 and pins none/off/minimal as the plugin's own rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merged main in at 5a14905 with a normal merge, no rebase or force-push. Conflicts:
8.12 now compares Run at 5a14905 (Windows, Bun 1.4, Node 24.12): |
yetone
left a comment
There was a problem hiding this comment.
Re-reviewed at 5a14905. The merge resolution is right: package.json is 0.1.3, effort_ability is none/off/minimal −1.1/−1.1/−1.0 on top of #749's low..max unchanged, the import line is the union, and 8.12 compares low..max with v0.3.8 and pins the plugin's three rows apart. The PR's own diff against main (git diff origin/main...5a1490532 -- packages/agentcompass) is the same patch as the two-dot diff, so nothing from main was lost or reverted in the merge.
One thing still has to change before it merges.
Quota cost still reads :ultra and :minimal as High
effortValue's two fallbacks reach burnScale in allocator.js, but evaluateModelQuota in src/quota.js (lines 908-910) works out its own effortScale straight from the tables:
const refBurn = curves?.burn?.high ?? policy?.effort_burn?.high ?? 0.7;
const effBurn = curves?.burn?.[effort] ?? policy?.effort_burn?.[effort] ?? refBurn;Neither table has an ultra or minimal row, so both get High's burn there. That effortScale sets the subscription's opportunity cost (stepUsd, unitRatePct). So after this PR:
:ultrascores Max's ability but costs High's quota, so on a subscription account it beats:maxon every task.:minimalcosts more quota than:low(High's cost, ~3× Low's on Opus), the opposite of what the PR says.
evaluateModelQuota at this head, claude-opus-5-5, one subscription account with a 7d window 40% left and 120h to reset, { hours: 1.5, difficulty: "hard" }:
| effort | opportunity_cost | allocator burnScale |
|---|---|---|
| minimal | 0.1448 | 0.171 |
| low | 0.0457 | 0.286 |
| high | 0.1448 | 1.000 |
| max | 0.412 | 2.900 |
| ultra | 0.1448 | 2.900 |
Before this PR :ultra was High in both places, so the two agreed. Please have evaluateModelQuota use burnScale(policy, curves, effort) (or effortValue) so the quota side and the allocator read one scale, and add a test that fails without it: :ultra's opportunity cost equals :max's, and :minimal's is below :low's. none/off go the same way.
What I ran at 5a14905
On mote (macOS, Bun 1.4.2, Node 22.22.0):
bun test packages/agentcompass: 42 pass, 0 fail.node --test test/router.test.mjs: 42 pass, 0 fail.bun scripts/check.mjs agentcompass: exit 0.- A probe calling
burnScale,effortAbilityandevaluateModelQuotafor claude-opus-5-5 and gpt-6-astra at minimal/low/high/max/ultra (the table above; gpt-6-astra shows the same split:burnScale1.929 for ultra, quota scale 1.0).
Everything else in the diff reads right to me; once quota.js follows the same scale, this is ready to merge as 0.1.3.
evaluateModelQuota worked out its own effort scale from the burn tables, so `:ultra` and `:minimal`/`none`/`off` (no table rows) cost High's quota there while the allocator scored them through effortValue's fallbacks. It now calls burnScale, as AgentCompass's allocator.py does for the quota effort scale; effortValue and burnScale move to quota.js (allocator.js re-exports them) so the two modules share one implementation without a circular import. Test 9.12: `:ultra` costs what `:max` costs and `:minimal`/`none`/`off` cost less than `:low`, on claude-opus-5-5 and gpt-6-astra subscriptions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, you're right. Pushed faa2fa2. Root cause: Fix:
Test: new test 9.12 checks claude-opus-5-5 on a claude subscription and gpt-6-astra on a codex subscription (7d window, 40% left, 120h to reset,
It fails at 5a14905 and passes at faa2fa2. The same probe as your table, at faa2fa2:
What I ran at faa2fa2 (Windows, Bun and Node 24.12):
|
yetone
left a comment
There was a problem hiding this comment.
Re-reviewed at faa2fa2. evaluateModelQuota now reads burnScale, and 9.12 holds for the two models it checks. One more thing still has to change before this merges, and there is one behaviour change I'd like you to look at.
1. On a model with no curve, :minimal / :none / :off still burn like High
The burn fallback in effortValue (quota.js) runs only when the model's own curve has a Low row ("low" in own). A model with no curve (glm-5.3, deepseek, every model outside EFFORT_CURVES) goes on to policyTable[effort]. effort_burn has no minimal, none or off row, so it returns fallback. That fallback is the High reference, so the scale comes out at 1.0.
burnScale(DEFAULT_SCORING_POLICY, {}, effort) at faa2fa2:
| effort | burnScale |
|---|---|
| low | 0.286 |
| minimal / none / off | 1.000 |
| high | 1.000 |
| ultra / max | 1.857 |
So on those models :minimal costs 3.5x :low in both the allocator and the quota opportunity cost. That is the same split the last round found for ultra, here on the models without a curve. The body says "A model without a Low row takes the policy table's value", but the policy table has no such row. Please give the no-curve case the same rule, e.g. policyTable.low * min(ratio, 0.75), or add minimal/none/off rows to effort_burn. Then add a model without a curve (glm-5.3) to 9.12 so it goes red without the fix.
2. A lone :minimal / :none candidate now routes nothing on coding tasks
Through magpie's own moejs middleware runner, on the request "Refactor the payment retry loop in billing/retry.go and add tests for the backoff" with candidateModels: ["gpt-6-astra:minimal"]:
- main:
modelbecomesgpt-6-astra:minimal. The effort is wrong (high), but the request goes through. - faa2fa2: the candidate is excluded (
ability 3.15 below the coupled floor 3.60; none is 3.05). Nothing is selected, soonRequestreturnsundefinedand the request goes upstream asmodel: "agentcompass". This happens on anthropic, responses and chat alike.
With ["gpt-6-astra:minimal", "gpt-6-astra:high"], minimal is never chosen for that task.
Gating minimal by the floor may be what you want. But a user who lists only :minimal had a working route before and gets a failed request now. Please either keep an explicitly listed effort reachable when it is the only candidate (or fall back to the best excluded one), or say in the body and README that :minimal is held to the same floor and can leave a request unrouted.
What I ran at faa2fa2 (merged onto main 15f4fa8)
On mote (macOS, Bun 1.4.2, Node 22):
bun test packages/agentcompass43 pass;node --test test/router.test.mjs43 pass;bun scripts/check.mjs agentcompassok.- main and this PR loaded as middleware into magpie's moejs runner (
internal/middlewareBegin/Request). The candidates were:minimal,:ultra,:noneandglm-5.3:ultra+:max, on the anthropic, responses and chat protocols. No load errors or hook failures, and eachonRequesttook under 1ms. This is where item 2 came from. - A Bun probe calling
rankCandidates/selectBestRoutegave the exclusion reasons above, andburnScaleon an empty curve gave the table in item 1.
Everything else in the diff reads right to me. That covers the EFFORT_ORDER change (middleware.js/hooks.js already strip none/off/minimal as efforts), the ultra fallbacks, the move of effortValue/burnScale into quota.js with no import cycle, and 9.10-9.12.
…a model without a curve effortValue scaled minimal/none/off from the model's own Low row only. A model with no effort curve (glm-5.3, deepseek, any model outside EFFORT_CURVES) fell through to effort_burn, which has no such rows, so their burn came back as the High reference: :minimal cost 3.5x :low there. The policy table's Low is now the base when the model has none. 9.12 adds glm-5.3 and fails without the fix. README: an effort written in a candidate is held to the task's ability floor like any other; a list whose every candidate is below the floor leaves the request unrouted, as a lone gemini-3.8-flash-high already does on main.
|
Thanks, both reproduced. Pushed ee9e07e. 1. Efforts below Low on a model with no curveRoot cause: Fix: the base is the model's own Low, else the policy table's Low. Ability is unchanged on those models, since the default table's Low plus the step equals its own none/off/minimal rows.
Test: 9.12 now also checks glm-5.3 on a 2. A lone
|
| candidates | main (2b7eb23) | ee9e07e |
|---|---|---|
gpt-6-astra:minimal |
gpt-6-astra:minimal |
unrouted |
gpt-6-astra:none |
gpt-6-astra:none |
unrouted |
gemini-3.8-flash-high |
unrouted | unrouted |
gpt-6-astra:minimal, gpt-6-astra:high |
gpt-6-astra:minimal |
gpt-6-astra:high |
On main, a lone candidate below the task's floor already leaves the request unrouted; gemini-3.8-flash-high shows it. Main routed :minimal only because it didn't read the suffix as an effort, and then sent it at High. This PR holds :minimal to the same floor as every other candidate.
A fallback to the best excluded candidate would change that for every list, so I kept it out of this PR. I can open it as its own PR if you want that behaviour.
The body and the README now say that an effort written in a candidate is held to the floor, and that a list whose every candidate is below the floor leaves the request unrouted. The README suggests listing a candidate that clears the floor next to a low one.
Run at ee9e07e
bun test 43 pass, node --test test/router.test.mjs 43 pass, bun scripts/check.mjs agentcompass ok (Windows, Bun 1.4.0, Node 24.12).
When every candidate was below the task's ability floor, onRequest returned undefined and the request went upstream as the virtual model name, which fails: a lone gemini-3.8-flash-high on a coding task on main, and a lone :minimal once minimal became an effort. AgentCompass may decline to route; a gateway must not. selectBestRoute takes fallbackToBestExcluded, which picks the most capable excluded candidate and marks the decision fallback; middleware and hooks use it on their relaxed retry, which now runs whenever the first pass finds nothing. 9.13 covers both entry points and fails without the fix. README: the floor note says the gateway falls back instead of leaving the request unrouted.
|
On item 2, I changed my mind: leaving the request unrouted is the wrong behaviour for a gateway, so this PR now falls back instead of only documenting it. Pushed 6ee28fe. AgentCompass itself may decline to route when nothing clears the floor. Magpie can't: an unrouted request goes upstream as Fix:
Through
Test: 9.13 covers both
|
yetone
left a comment
There was a problem hiding this comment.
Re-reviewed at 6ee28fe. Both earlier items are fixed, and each fix has a test that fails without it. Two things still stop the merge.
1. The fallback sends an image request to a model that can't see it, even when the list has one that can
selectBestRoute (allocator.js:830-834) falls back to the excluded candidate with the highest ability_value, whatever excluded it. task needs visual input (allocator.js:391) is a hard exclusion, not a floor miss, so it shouldn't be weighed the same way.
Chat request through middleware.onRequest with "Refactor the payment retry loop in billing/retry.go to match this screenshot and add tests for the backoff", plus one image_url part:
| candidates | main 39d4456 | 6ee28fe | fallback.excluded |
|---|---|---|---|
claude-fable-5-1:high, gemini-3.8-flash-high:low |
unrouted | claude-fable-5-1:high |
["task needs visual input"] |
grok-4.7:high, gemini-3.8-flash-high:minimal |
unrouted | grok-4.7:high |
["task needs visual input", "ability 3.40 below the coupled floor 3.60"] |
In both cases the list has a visual model (gemini-3.8-flash-high) that misses only the floor, but the fallback picks the non-visual one. Please have the fallback prefer candidates whose only exclusion is the ability floor (or at least ones not excluded for visual input), and use raw ability only to break ties among those. Falling back to a non-visual model is fine when it's the only kind in the list. Please add the first row to 9.13.
The candidate pool is only the user's own candidateModels, so the fallback never routes outside the list. I checked that.
2. It conflicts with main
#765 (15f4fa8) rewrote packages/agentcompass/README.md as a concise English README, and GitHub reports this PR as CONFLICTING there. Please rebase onto main and put the floor/fallback note in the new README. Right now it's a Chinese bullet only, in a section #765 rewrote. Please also say there that the fallback can override the visual/unhealthy/context/quota constraints, since the relaxed retry drops the last three.
What I ran at 6ee28fe (on mote: macOS, Bun 1.4.2, Node 22, HOME=$(mktemp -d))
bun test44 pass;node --test test/router.test.mjs44 pass;bun scripts/check.mjs agentcompassok.- Broke each fix by hand. Each break turned a test red:
- quota.js
lowwithout the policy-table fallback: 9.12 red. fallbackToBestExcludedremoved from middleware.js: 9.13 red.- The same removal from hooks.js: 9.13 red.
allCandidates[0]instead of the most capable candidate: 9.13 red.- middleware's old retry condition (
unhealthy || minContext || quotas): 9.13 red.
- quota.js
- The visual probe above, at this head and at main 39d4456, with
selectBestRoute(..., { fallbackToBestExcluded: true })called directly too.
The rest of the diff reads right: the policy-table Low in effortValue, the retry that now always runs, and fallback carried on the hooks route.
…-fallback Takes magpie-community#765's English README and puts the floor and fallback note in both of its languages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… see it The fallback picked the excluded candidate with the highest ability, whatever excluded it. A request with an image could go to a capable model that can't read it, even when the list had one that can and only missed the floor. Prefer candidates that only miss the floor, then ones that can see the input; ability breaks ties. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, reproduced. Pushed ff614f0, with main merged in. I merged rather than rebased so the branch stays free of force-pushes. 1. The fallback ignored why a candidate was excludedRoot cause: the fallback compared Fix:
Your request, with
Test: 9.13 has both of your rows through Each break below turned 9.13 red:
2. Conflict and READMEMerged main (42331f6), taking #765's README whole. The Chinese bullet from this PR is gone. The new README has the floor and fallback note in its English and Chinese sections, under the gateway middleware quickstart. It says the fallback can override the visual, unhealthy, context and quota constraints, and that it only picks from the candidate list. The file has no raw CRs. What I ran at ff614f0Windows, Bun 1.4.0, Node 24.12, temp HOME: |
|
Thanks @akiiwa-games. Every finding from the earlier rounds is fixed, and each fix goes red when I break it by hand. Merged as 10a61c2. gateway-agentcompass 0.1.3 is on the dist branch now (tgz/magpie-community-gateway-agentcompass-0.1.3.tgz), and magpie installs it from there. The package isn't on npm yet: its first npm version needs an owner to publish it. |
…cal-model Keeps 0.1.4, the version after magpie-community#750's 0.1.3, and imports both sides' names in the test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
On main,
model:minimalisn't an effort.EFFORT_ORDERis low..ultra, sorankCandidatesdoesn't split the suffix off. It expands the entry into every effort the model lists, and sendsgpt-6-astra:minimal:low…gpt-6-astra:minimal:max. An uncalibratedultra(Astra, Sol) scores at tier + 0, the same as High.Changes (this differs from AgentCompass)
AgentCompass's
effort_value(allocator.py:593) reads the model's curve, thenpolicy[f'effort_{name}'], then the default. It has no ultra→max step and no minimal clamp, and itsEFFORT_ORDERis('low', 'medium', 'high', 'xhigh', 'max', 'ultra'). This PR adds, plugin-side:EFFORT_ORDERgains none, off and minimal below low.model:minimalis one candidate at minimal, and the warm-cache downgrade check counts minimal after low as a downgrade.effortValue:ultrawith no row of its own reads as the model's Max, or the policy's Max for a model with no curve.scoring.effort_abilitygains none −1.1, off −1.1 and minimal −1.0, used for models without a curve. These are plugin values, not calibrated upstream.evaluateModelQuotareadsburnScale, asallocator.pyderives its quota scale fromburn_scale, so the allocator and the quota opportunity cost price an effort the same way.effortValueandburnScalelive inquota.js;allocator.jsre-exports them.:minimal,:noneand:offscore below:low, so a hard task excludes them.onRequestreturnsundefinedand the request goes upstream asagentcompassand fails; a lonegemini-3.8-flash-highon a coding task does that today, and a lone:minimalwould too once minimal is an effort. AgentCompass may decline to route, but a gateway must not.selectBestRoutetakesfallbackToBestExcluded: it picks an excluded candidate and marks the decisionfallback: first one that only misses the floor, then one that can see the request's images, with ability breaking ties. middleware and hooks pass it on their relaxed retry, which now runs whenever the first pass finds nothing. The allocator's own result is unchanged. The README says so.Tests
9.10:
:minimaland:ultraare one candidate each;9.11, the branches the default policy can't reach:
glm-5.3(no curve) scores as the policy's Max;effort_burn.minimal0.19 (and a control at 0.1);9.13: through
middleware.onRequest: a lonegpt-6-astra:minimalroutes to itself, a lonegemini-3.8-flash-highroutes, none+minimal picks minimal, minimal+high still picks high, an image request goes to a model that can see it and to a non-visual model only when the list has no other kind; throughhooks.onRequest, the stored decision carriesfallback.selectBestRoutewithout the option still returnsok: false.9.12: the quota opportunity cost on claude-opus-5-5, gpt-6-astra and glm-5.3 (no curve):
:ultracosts what:maxcosts, none/off/minimal cost less than:low, max > high > low.Broken by hand, one at a time:
Math.min(delta, -0.2)Math.min(ratio, 0.75)EFFORT_ORDEREFFORT_ORDEREFFORT_ORDERAt ff614f0:
bun test44 pass,node --test44 pass,bun scripts/check.mjs agentcompassok.Main (with #749) is merged in at 5a14905. 8.12 compares
effort_ability's low..max with v0.3.8 and pins none/off/minimal separately. With #749's policy (low −0.54), the Low→minimal step is −0.46; 9.10 reads it from the policy.bun testandnode --test42 pass.🤖 Generated with Claude Code