Skip to content

fix(tuner): wire the research-scout benchmark feeder into model_routing - #347

Closed
Nibbler1250 wants to merge 1 commit into
TerrysPOV:mainfrom
Nibbler1250:feat/wire-model-routing-benchmark-scout
Closed

fix(tuner): wire the research-scout benchmark feeder into model_routing#347
Nibbler1250 wants to merge 1 commit into
TerrysPOV:mainfrom
Nibbler1250:feat/wire-model-routing-benchmark-scout

Conversation

@Nibbler1250

Copy link
Copy Markdown
Collaborator

Problem

ModelRoutingSubject exposes an injectable benchmarkProvider (the "research
scout" seam) whose default is async () => []. At composition in
registerWisecronSubjects, only the dispatchReader was wired — the
benchmarkProvider was left at the empty default.

Consequence: the benchmark reroute has no external evidence and can never
surface a new-model reroute. It is the benchmark-side twin of the obs=0
dispatchReader gap that was already closed on the cost side. The scout itself
(getModelBenchmarks — cache-first over the Artificial Analysis free tier — plus
enrichWithAnthropicCoding) was already built and tested; it was simply never
injected, so the whole proactive routing face sat inert.

Fix

  • Add makeBenchmarkProvider() next to makeModeDispatchReader in
    observation-readers.ts (the runtime seam): cache-first fetch, ensures the
    cache dir exists (the cache writer uses writeFileSync without mkdir),
    enriches Claude coding-index gaps from the Anthropic seed, and is graceful —
    a missing key or benchmark outage yields [], never a throw, so the proactive
    loop is never stalled. fetchImpl/nowMs stay injectable for tests.
  • Inject it at ModelRoutingSubject construction. Cache path and TTL are
    config-overridable via benchmark_cache_path / benchmark_ttl_ms.

Why it's safe

  • Governance preserved. The subject still never fetches the web itself — the
    runtime composition injects the feeder, exactly as the subject's own comment
    prescribes. The subject remains pure and unit-testable with an injected
    provider.
  • Graceful degradation. No API key or a fetch failure returns [] (same
    contract as the cost-signal reader); a stale cache is preferred over nothing.
  • No fabricated data. The Anthropic SWE-bench seed is left untouched — I did
    not hand-add Opus 5's figure (the seed file explicitly forbids guessing).
    Once wired, the live scout covers new models from the AA free tier directly.

Tests

New observation-readers.test.ts (all hermetic — mocked fetchImpl):
fetch + Anthropic enrichment, cache-dir creation, cache-first reuse (no second
fetch), and empty-key short-circuit (no fetch, []).

Full src/tuner/__tests__/wisecron/ suite: 403 pass / 0 fail (run with the AA
key unset to match CI). Biome clean on the touched files.

@Nibbler1250
Nibbler1250 force-pushed the feat/wire-model-routing-benchmark-scout branch from 8746ef4 to b25ebde Compare August 3, 2026 14:18
The model_routing subject exposes an injectable benchmarkProvider (the
"research scout" seam) but its default is `async () => []`, and at
composition (registerWisecronSubjects) only the dispatchReader was wired —
the benchmarkProvider was left at the empty default. So the benchmark
reroute had no external evidence and could never surface a new-model
reroute: the benchmark-side twin of the obs=0 dispatchReader gap.

The scout itself (cache-first getModelBenchmarks over the Artificial
Analysis free tier + enrichWithAnthropicCoding) was already built and
tested; it was simply never injected.

- add makeBenchmarkProvider() beside makeModeDispatchReader in
  observation-readers.ts (the runtime seam): cache-first fetch, ensures the
  cache dir exists, enriches Claude coding gaps from the Anthropic seed,
  graceful [] on missing key/outage, fetchImpl/nowMs injectable for tests.
- inject it at ModelRoutingSubject construction; cache path + TTL are
  config-overridable (benchmark_cache_path / benchmark_ttl_ms).
- the subject stays pure (never fetches the web itself); the runtime
  composition injects the feeder, per the governance rule in the subject.
- tests: fetch+enrich, cache-dir creation, cache-first reuse, empty-key
  short-circuit (all hermetic).
@Nibbler1250
Nibbler1250 force-pushed the feat/wire-model-routing-benchmark-scout branch from b25ebde to c36a0d2 Compare August 3, 2026 14:19
@Nibbler1250

Copy link
Copy Markdown
Collaborator Author

Closing this. On review, it wires the injectable benchmarkProvider seam, but its only consumer — proposeEvidencePatch on the proactive EvidenceDrivenSubject face — isn't driven by any scheduler cycle yet (both ProposalEngine.runCycle and the legacy engine call the reactive collectObservations → detectProblems → proposeChange path). So injecting the feeder is a no-op today, and the title oversells it as a behavioural fix.

Rather than ship a node without a live consumer, I'd rather wait until the proactive evidence pass is actually driven by the scheduler (the Subject 2/3 → 3/3 staging), at which point wiring the scout feeder is a natural part of that change. Closing to keep the queue honest — happy to revisit then.

@Nibbler1250 Nibbler1250 closed this Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant