fix(backend): deduplicate AUTO empty-pool warnings per family

[RaviTharuma]

OmniRoute Version

3.8.49 (official image diegosouzapw/omniroute:3.8.49)

Installation Method

Docker / Docker Compose

Operating System

Linux

OS Version

Linux (single-replica orchestrated deploy)

Node.js Version

Image default (official Docker image)

Provider(s) Involved

ZAI / family-combo path via auto/* (example label: auto/zai). Any auto-combo family that has zero connected models for the process.

Model(s) Involved

N/A — failure mode is empty pool resolution, not a specific model ID

Client Tool

N/A (observed as continuous process log noise; may be internal tick rather than end-user chat)

Description

When an auto/<family> combo has no connected models, OmniRoute emits a warn on every resolve — and something in the running process resolves that label about every 1–2 minutes forever.

Live evidence on official 3.8.49, single replica, one Node process, ~6h of process logs:

Observation	Value
Message	[AUTO] auto/zai matched no connected models; returning an empty pool.
Cadence	~every 1–2 minutes, continuously
Rate sample	~120 hits in 2 hours (same process)
Level	warn
Outcome	Empty pool returned; no connected models for that family

Source (stock tree):

// open-sse/services/autoCombo/virtualFactory.ts ~538–540
log.warn("AUTO", /* auto/<family> matched no connected models; returning an empty pool */);

What is solid (observed / code):

1. Family combo with zero connected models → empty pool.
2. That condition is logged at warn on every resolve, not once per process / per label.
3. On a real 3.8.49 process with no ZAI connections (or none selected into the pool), the message is chronic, not a one-shot at boot.

Suspicion (not proven — do not treat as root cause):

Something other than an end-user chat request may be invoking auto/zai on a timer — e.g. dashboard refresh, connection health tick, catalog poll, or another internal scheduler. We have not traced the caller stack in production; we only know the warn fires continuously without matching user traffic for that family.

Even if the caller is legitimate, warn-every-resolve for a stable configuration state is wrong. An empty pool for an unconfigured family is either:

• Expected/steady-state → log once (or at debug), or
• Misconfiguration that needs attention → log once per process per label at warn, then silence / debug.

It should not be a 1–2 minute metronome that fills operator logs and competes with real faults for attention on the same event loop already under catalog/compression pressure (#9685 / #10117 / #10052).

Steps to Reproduce

1. Run official image diegosouzapw/omniroute:3.8.49 as a single process.
2. Ensure the ZAI (or other family used by auto/zai) provider has no connected models selected into the auto pool — empty family is the point.
3. Leave the process running with normal dashboard / health / catalog activity (no special client required for the log to appear in our environment).
4. Tail logs for the AUTO tag:

   # count empty-pool warns over ~2h
   # expect: message about every 1–2 minutes when the family has zero connections

5. Observe repeated:

   [AUTO] auto/zai matched no connected models; returning an empty pool.
   

6. Optionally confirm in source that virtualFactory.ts uses log.warn("AUTO", ...) on the empty-pool path with no once-per-label guard.

Expected Behavior

1. Empty auto-family pool is still a valid resolution result (return empty pool / fail the request path as designed).
2. Logging for that stable condition is not warn-on-every-resolve:
   • Prefer: once per process per label (fingerprint auto/<family> + “empty”) at warn, then debug / silence;
   • Or: debug-level only for the steady empty state;
   • Or: warn only when emptiness changes (had models → now empty).
3. Optional product improvement: skip scheduling health / catalog probes for auto families that currently have zero connections (reduces useless resolves). Marked as optional — the logging fix alone is enough for the storm.
4. Real AUTO errors (all candidates failed, auth, upstream 5xx) remain visible.

Actual Behavior

Item	Actual on 3.8.49
Empty family resolve	Returns empty pool (OK)
Log level	warn
Dedup / once-per-label	None observed
Live cadence (no ZAI connections)	~1–2 min forever
Sample volume	~120 / 2h single process
Operator UX	Chronic noise; real errors compete with a steady metronome
Caller identity	Not proven — see Suspicion

Test Impact

Needs a new unit test

Error Logs / Output

image: diegosouzapw/omniroute:3.8.49
file: open-sse/services/autoCombo/virtualFactory.ts ~538–540

repeated (sanitized):
  [AUTO] auto/zai matched no connected models; returning an empty pool.

rate (single replica, ~2h window):
  ~120 identical warns
  ~1 every 1–2 minutes continuously

no user chat for that family is required for the storm to continue in our sample

Screenshots

N/A

Additional Context

Suspicion: dashboard / health tick / catalog poll may invoke auto/zai even when no user request needs it. This is a hypothesis from cadence + empty config, not a stack-traced caller. Please treat as Suspicion until confirmed.

Related noise / loop load

• Header-budget warn-per-stream and default debugMode issues already filed (log volume class).
• Event-loop pressure issues (fix(ops): /healthz (and other HTTP) times out while event loop is busy (k8s probes flap) #10052 / perf(compression): countTextTokens blocks the event loop on large Codex payloads (worker wedge) #10117 / fix(api): GET /v1/models pegs event loop — getModelsDevPricing re-reads per model #9685): every periodic resolve is more main-thread work for a no-op empty pool.

Ask

1. Log once per process per label (or debug) for empty-pool steady state.
2. Do not warn on every resolve of a known-empty family.
3. Optional: skip health scheduling for families with zero connections.
4. If there is an internal timer resolving unused auto labels, document or gate it.

No secrets, hostnames, account IDs, or private network details.

Validation Plan

• Unit: resolve the same empty auto family 100 times → ≤ 1 warn (or debug-only after first fingerprint).
• Unit: adding a connection then removing it can warn again once when emptiness reappears.
• Integration: process with no ZAI connections does not emit linear warn growth over a 10-minute idle window after the first empty resolve.
• Optional: health scheduler does not register ticks for families with zero connections (if that path is confirmed).

[diegosouzapw]

Triaged: this is tracked by an open pull request — #10344 — so we are routing it through the PR review workflow instead of re-analyzing or re-implementing it here. That keeps the contributor's authorship intact.

This issue stays open until that PR lands.

Note for reviewers: the PR rate-limits the warning to once per 60s per label, which still emits roughly 60 warnings per hour for a permanently empty family. Worth deciding during review whether once-per-process or warn-on-state-change is preferred.