feat: call out dispatches that went below their agent's pin - #27
Merged
Conversation
AqueGen
force-pushed
the
feat/below-pin-dispatches
branch
from
August 11, 2026 07:54
09227c0 to
2f005e5
Compare
A week of another user's stats showed reviewer and implementer running on haiku, and the report scored all four as wins: they were genuinely cheaper than the session, so they padded the routed-down percentage while breaking the thing they were routed for. A pin is a ceiling and a floor, and only the ceiling was enforced anywhere. The floor is min(pin, session model), not the pin, because the pins-are-ceilings rule requires capping at the session model - reviewer running sonnet on a sonnet session is the rule being followed, so it must not be flagged. Unpinned agent types have no floor, and CLAUDE_CODE_SUBAGENT_MODEL is excluded because it forces every subagent at once: that is a deliberate machine-wide setting, not a per-dispatch judgement, and its only remedy is unsetting the variable, which the env= rows already say. Below-pin dispatches now carry the pin in their row key, group into their own section rather than the win column, and add a line under the headline. The headline itself does not move - they really are cheaper than the session - so the annotation qualifies the figure instead of redefining it and breaking comparison with older reports. On the author's own 14-day window this immediately surfaces 23 reviewer dispatches on sonnet against an opus pin, which had needed a manual read of the per-agent rows to notice. The rule is stated in the skill next to the ceiling half, and the README gains a section, including the part that matters when the cheap tier really was right: pick an agent pinned for it rather than overriding a role agent downward. 71 tests pass, three of them new - the flag firing on two tiers of undercut, the ceiling cap explicitly NOT firing, and unpinned agents having no floor.
…known Both review axes found the same bug independently. The floor fell back to the raw pin when the session tier could not be ranked, which broke the report two ways at once. A reviewer dispatched on haiku from a session on an unrecognized family was announced in the headline as going below its pin, while verdictOf called the same entry non-comparable and filed the row under "not tier-comparable" - a claim with no section under it. And a pre-0.5.3 entry with no session at all was flagged against the pin even though it may have been the correct cap on a cheap session. An unknown tier on any of the three sides now yields no verdict, which is the rule this file already applies to its own pair. The flag also moved out of a Set of formatted row keys into the per-key aggregate. Two accumulators existed only because the fact lived in a display string; s.underPin removes both, and the grouping condition s.underPin === s.n makes the collision above impossible by construction rather than by argument. The key suffix stays, and its comment now records that it PARTITIONS the keys - the property the grouping depends on, which a later edit moving it into the row text would break silently. The headline claimed "the session could afford it", which is false for the case where the session sits below the pin and the dispatch went under the cap anyway - it now claims only that nothing required going under. The README sample was impossible: its headline said two while its rows totalled four, because I took the count from a test fixture and the rows from a real report. Both docs also stated the min(pin, session) floor unconditionally, which was only true after this fix. Tests 71 to 74. The three new ones cover the unknown and absent session, the session-below-pin case, and the env exclusion - which turned out already covered by the env-override test, contrary to the review claim that deleting the guard passed everything; verified by deleting it and watching that test fail. The existing assertions were also section-scoped: reaching from a heading to the end of the report would have passed with a row in the wrong group, which is the whole thing under test.
AqueGen
force-pushed
the
feat/below-pin-dispatches
branch
from
August 11, 2026 07:56
2f005e5 to
a95b4b1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #26 - base retargets to
mainonce that merges.A week of stats from another user showed
reviewerandimplementerrunning on haiku. The report scored all four as wins, because they genuinely were cheaper than the session: they padded the routed-down percentage while breaking the thing they were routed for.A pin is a ceiling and a floor. Only the ceiling was enforced anywhere.
The rule
The floor is
min(pin, session model), not the pin. The pins-are-ceilings rule requires capping at the session model, soreviewerrunning sonnet on a sonnet session is the rule being followed and must not be flagged. Two more exclusions:CLAUDE_CODE_SUBAGENT_MODELis excluded, because it forces every subagent at once. That is a deliberate machine-wide setting rather than a per-dispatch judgement, and its only remedy is unsetting the variable - which theenv=rows already say plainly. Including it would fill the section with rows carrying advice that does not apply to them.What does not change
The headline. These dispatches really are cheaper than the session, so redefining "routed down" would break comparison with every report anyone has already run. The annotation qualifies the figure instead, and the rows move out of the win column into their own section so nothing is double-listed.
Validation on real data
On my own 14-day window this immediately surfaces
23 model-routing:reviewer (model=sonnet, pin=opus). I had spotted that same pattern earlier this week only by reading the per-agent rows by hand and comparing them against the pin table from memory - which is exactly the work a report should be doing.Tests
71 pass, three new: the flag firing on two different depths of undercut, the ceiling cap explicitly NOT firing, and unpinned agents having no floor. One existing test moved behaviour rather than breaking - the env-override row stayed in the win column once env was excluded, which is what its name asserts.