fix(tuner): wire the research-scout benchmark feeder into model_routing - #347
Closed
Nibbler1250 wants to merge 1 commit into
Closed
fix(tuner): wire the research-scout benchmark feeder into model_routing#347Nibbler1250 wants to merge 1 commit into
Nibbler1250 wants to merge 1 commit into
Conversation
Nibbler1250
force-pushed
the
feat/wire-model-routing-benchmark-scout
branch
from
August 3, 2026 14:18
8746ef4 to
b25ebde
Compare
The model_routing subject exposes an injectable benchmarkProvider (the "research scout" seam) but its default is `async () => []`, and at composition (registerWisecronSubjects) only the dispatchReader was wired — the benchmarkProvider was left at the empty default. So the benchmark reroute had no external evidence and could never surface a new-model reroute: the benchmark-side twin of the obs=0 dispatchReader gap. The scout itself (cache-first getModelBenchmarks over the Artificial Analysis free tier + enrichWithAnthropicCoding) was already built and tested; it was simply never injected. - add makeBenchmarkProvider() beside makeModeDispatchReader in observation-readers.ts (the runtime seam): cache-first fetch, ensures the cache dir exists, enriches Claude coding gaps from the Anthropic seed, graceful [] on missing key/outage, fetchImpl/nowMs injectable for tests. - inject it at ModelRoutingSubject construction; cache path + TTL are config-overridable (benchmark_cache_path / benchmark_ttl_ms). - the subject stays pure (never fetches the web itself); the runtime composition injects the feeder, per the governance rule in the subject. - tests: fetch+enrich, cache-dir creation, cache-first reuse, empty-key short-circuit (all hermetic).
Nibbler1250
force-pushed
the
feat/wire-model-routing-benchmark-scout
branch
from
August 3, 2026 14:19
b25ebde to
c36a0d2
Compare
Collaborator
Author
|
Closing this. On review, it wires the injectable Rather than ship a node without a live consumer, I'd rather wait until the proactive evidence pass is actually driven by the scheduler (the Subject 2/3 → 3/3 staging), at which point wiring the scout feeder is a natural part of that change. Closing to keep the queue honest — happy to revisit then. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ModelRoutingSubjectexposes an injectablebenchmarkProvider(the "researchscout" seam) whose default is
async () => []. At composition inregisterWisecronSubjects, only thedispatchReaderwas wired — thebenchmarkProviderwas left at the empty default.Consequence: the benchmark reroute has no external evidence and can never
surface a new-model reroute. It is the benchmark-side twin of the obs=0
dispatchReadergap that was already closed on the cost side. The scout itself(
getModelBenchmarks— cache-first over the Artificial Analysis free tier — plusenrichWithAnthropicCoding) was already built and tested; it was simply neverinjected, so the whole proactive routing face sat inert.
Fix
makeBenchmarkProvider()next tomakeModeDispatchReaderinobservation-readers.ts(the runtime seam): cache-first fetch, ensures thecache dir exists (the cache writer uses
writeFileSyncwithoutmkdir),enriches Claude coding-index gaps from the Anthropic seed, and is graceful —
a missing key or benchmark outage yields
[], never a throw, so the proactiveloop is never stalled.
fetchImpl/nowMsstay injectable for tests.ModelRoutingSubjectconstruction. Cache path and TTL areconfig-overridable via
benchmark_cache_path/benchmark_ttl_ms.Why it's safe
runtime composition injects the feeder, exactly as the subject's own comment
prescribes. The subject remains pure and unit-testable with an injected
provider.
[](samecontract as the cost-signal reader); a stale cache is preferred over nothing.
not hand-add Opus 5's figure (the seed file explicitly forbids guessing).
Once wired, the live scout covers new models from the AA free tier directly.
Tests
New
observation-readers.test.ts(all hermetic — mockedfetchImpl):fetch + Anthropic enrichment, cache-dir creation, cache-first reuse (no second
fetch), and empty-key short-circuit (no fetch,
[]).Full
src/tuner/__tests__/wisecron/suite: 403 pass / 0 fail (run with the AAkey unset to match CI). Biome clean on the touched files.