feat: report reasoning effort, and stop losing big sessions from the math - #25
Merged
Conversation
…math The report measured one of the two knobs the plugin is built on. Tier was visible, effort was not, so a mechanical errand dispatched to an unpinned agent type looked like a win at sonnet while it inherited the session level and thought as hard as the session does. Each dispatch now records the session effortLevel from the settings cascade - transcripts do not carry effort at all, so that cascade is the only observable source - and the report prints how many dispatches ran on an agent type with no pin and inherited it. Values outside the ladder are ignored so a typo cannot become a data point, and the section is omitted entirely when nothing recorded an effort. Also fixes the session-model lookup in the tokens report, which read only the first 256KB of the parent transcript. A long session pushes its first assistant message past that window - 6 of 60 transcripts here, all multi-MB - and those sessions dropped out of the routed-down math as "session unknown", taking 120M tokens with them, concentrated in the biggest sessions precisely because size is what caused it. The lookup now falls back to the tail, accepting a late-window model over no model. On the author's 14d window the denominator went from 3.06B to 3.18B with the share unchanged at 59%, so the exclusion had been hiding volume rather than flattering the figure. The not-comparable note blamed TIER_PATTERNS for both of its causes; it now names the unreadable-session case separately, since extending the tier table would never have fixed it.
Review round 1 across three independent reviewers found the effort source was both too narrow and too wide. Too narrow in two ways. CLAUDE_CODE_EFFORT_LEVEL overrides the settings cascade for the whole session, and reading only settings therefore reported a level the session did not run on - the same class of bug the model side already solved via CLAUDE_CODE_SUBAGENT_MODEL. And an unset effortLevel is not unknown: it is the documented model default, high wherever effort is supported and xhigh on Opus 4.7, so omitting the entry left the section blank on the commonest configuration of all. Both sources are now read in the order Claude Code applies them, and each entry records which rung it came from so an inferred default never reads as a setting somebody chose. Too wide in accepting max from a settings file, where it is rejected as session-only. It is accepted from the env var, which is the one place it is allowed, and ultracode is accepted nowhere. A bad value no longer falls through to the next rung either. A local file carrying a typo used to log the user-level value, replacing a missing data point with a wrong one; a file that defines the key now ends the walk whatever the value, while an unparseable file still defers, matching how the harness ignores a broken settings file wholesale. Three more fixes from the same round. The tokens footer still promised session-START attribution after the tail fallback shipped, which is the opposite bias for exactly the sessions the fallback targets; it now says so. The effort section claimed to know which agents carry no pin when it only knows the bundled ones, so a third-party agent pinning its own effort is now described as counted inherited rather than silently mislabelled. And projRoot still inlined the config-dir resolution that configDir() was extracted to remove. Tests grew from 45 to 56: a PINNED_EFFORT frontmatter sync test guarding the drift class 0.7.1 fixed for PINNED_MODELS, a ladder test pinning which levels each source accepts, every precedence rung including the project file that had no coverage at all, the env override, auto, both model defaults, the no-fall-through rule, a malformed file, and the two report lines that had no assertions. run() now defaults cwd to the temp config dir and blanks CLAUDE_CODE_EFFORT_LEVEL, so no test can be decided by a settings file or an env var belonging to whoever runs the suite.
…numbers Review round 2 found the model default being handed to models that have no effort knob at all. defaultEffortFor gated on "can TIER_PATTERNS rank this", which claude-3-5-sonnet passes happily despite predating adaptive reasoning, and a version-threshold guard would have been just as invented - it would have given Haiku 4.5 a default, when Haiku 4.5 is simply absent from the support table. Support is now that table, transcribed: Fable 5, Opus 5, Sonnet 5, Opus 4.8 and Opus 4.7 take the whole ladder, Opus 4.6 and Sonnet 4.6 take everything but xhigh, and anything unlisted supports no effort and therefore receives no level. The same table supplies a rule that was missing: a level the model does not support runs as the highest supported level at or below it, so xhigh configured on an Opus 4.6 session ran as high. The log records what ran rather than what was configured, which is the whole point of the round-1 fixes. When the session model is unknown the clamp cannot be computed and the configured level is kept as-is - the one figure here that is a configuration rather than an observation, and it says so. Two report defects both Claude reviewers caught independently. The inferred- default count was taken over every entry while printing directly under a sentence about the inherited subset, so a pinned agent on the model default could print "3 of these" beneath a total of 2; it is now scoped to that subset, and the fixture was rebuilt so the two scopings give different numbers - the old one could not fail. And two comments the round-1 rewrite left behind still described the settings cascade as the only observable source and promised the section is omitted when effortLevel is unset, neither true since the env and default rungs landed. The two pin tables became one AGENT_PINS with model and effort columns. They were two views of one fact - the agent frontmatter - and the report asked "is this agent pinned" through PINNED_EFFORT while the leak section asked BUNDLED, which is derived from the other table. One table, one sync test covering both columns, and the deliberate difference (Explore is cheap but not pinned) stays explicit in CHEAP_AGENTS. The ladder test that regexed the source for its own constants is gone too: it passed on text rather than behaviour, so it is replaced by cases that drive the hook - ultracode rejected from both sources, max from the env var only, the clamp, and the unsupported-model cases. Docs follow the same source. The README effort ladder promised high with no exception; it now names the Opus 4.7 default, the support list, and the clamp, and the routing skill carried the same gap. 57 tests pass.
Both reviewers found the same hole independently. clampEffort kept a configured level unchanged when the session model could not be read, on the reasoning that a configuration is better than nothing. The counter-example settles it: a Haiku 4.5 session whose transcript cannot be read, with effortLevel high in settings, logged effort "high" - a level that does not exist on that model at all. The clamp needs the model, because the same configured level means different things on different models, so an unreadable model now records nothing. That matches how every other figure in this report treats an unknown session: excluded, never guessed. Six precedence tests were only reaching the early return that hole created, so dispatchWithSettings now defaults to a supported session model and takes null explicitly. "max is accepted from the env var" was the worst of them - it had been asserting nothing about max being supported anywhere. The README still told maintainers to keep PINNED_MODELS and PINNED_EFFORT in step, one commit after they became AGENT_PINS, and still claimed unsupported levels are always recorded as what ran. Round-2 prose against round-3 code, which is the drift that paragraph exists to warn about. Also: the two identical EFFORT_SUPPORT rows are one alternation, and the test named for rankability is named for support instead - the gate that actually fires. One reviewer recommendation declined. Folding EFFORT_SUPPORT into TIER_PATTERNS would trade one duplication for another: tier is a property of the family, effort support a property of the version, so a single ordered list has to repeat the tier across every version row and add a family fallback carrying no efforts. The tables stay separate with a consistency test pinning them together - every model with effort support must be rankable - and the reason is now a comment on the table rather than a decision someone has to rediscover. 59 tests pass.
The one finding Codex could produce this round came from the documentation rather than the code, and it lands on the wording. The comments and the README said this log records the effort that ran. Four documented states override the reconstruction and none of them are visible to a PostToolUse hook: a /effort or --effort choice made inside a running session, ultracode, an organization effort cap, and the model-default hold that Fable 5, Opus 4.8 and Opus 4.7 apply on first run "even if you previously set a different level for another model" - which means a persisted setting can be overridden while this hook still reads it and reports it. The clamp still closes the one gap a hook can see, and the algorithm is unchanged. What changes is the claim: the report and the README now name all four blind spots and say to read the figure as what the visible sources resolve to, not as a measurement. 59 tests pass.
Round-4 leftovers, all non-blocking, from the two reviewers that reached a mergeable verdict. The consistency test parsed TIER_PATTERNS with a dot-star that stops at a newline, so reformatting that table across lines would have failed the test for its own parsing rather than for its subject. It also checked which models appear in the support table but never that the levels in it exist: a stray level would pass the includes check in clampEffort and then drop out of the downward walk silently, because indexOf returns -1 for a level the ladder does not carry. Both are now asserted. The README said an unsupported level is "recorded as the level that actually ran", two lines above the paragraph explaining that none of this is a measurement. It now says it records the level Claude Code falls back to, which is what the clamp actually knows. 59 tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two gaps the report had, both found by reading a real 14d run rather than the code.
Effort was invisible
The plugin is built on two knobs, and the report only ever showed one. In the live window here, 211 of 489 dispatches went to
general-purpose- a type with no frontmatter pin, so every one of them inherited the session effort (high) even where the tier was correctly dropped to sonnet. Cheap model, expensive thinking, and the report scored it as a clean win.Each dispatch now records the session
effortLevel. Transcripts carry no effort field at all (verified - zero structured matches), so the settings cascade is the only observable source; the report says so instead of implying per-dispatch precision.Levels outside the ladder are ignored, so a typo never becomes a data point, and the section is omitted when no entry recorded an effort - old logs and machines with no
effortLevelset simply do not show it.Big sessions fell out of the token math
tokensread only the first 256KB of the parent transcript to find the session model. A long session pushes its first assistant message past that window - measured here at 6 of 60 transcripts, every one multi-MB - and those sessions were reported as(session unknown): 58 agents and 123.1M tokens excluded, concentrated in the largest sessions precisely because size is what caused it.The lookup now falls back to the tail, taking a late-window model over none. Before and after on the same 14d window:
The share did not move, which is the point: the excluded volume had been routing down at the same rate, so this recovers a denominator rather than improving a number.
The not-comparable note also blamed
TIER_PATTERNSfor both of its causes. Extending the tier table would never have fixed this one, so the note now names the unreadable-session case separately.Tests
6 new cases, 45 pass / 0 fail: effort from the user file, project file outranking it, an out-of-ladder value ignored, only unpinned agents counted as inheriting, the section omitted when unrecorded, and the tail fallback with a >256KB model-free head. The effort tests pin both a temp
CLAUDE_CONFIG_DIRand a temp cwd - without the temp cwd the result would depend on the directory the suite runs from.