fix(backend): avoid models.dev sync catalog stalls on supported releases

[RaviTharuma]

OmniRoute Version

3.8.49 (official image diegosouzapw/omniroute:3.8.49)

Installation Method

Docker / Docker Compose

Operating System

Linux

OS Version

Linux (Kubernetes single replica)

Node.js Version

Image default (official Docker image)

Provider(s) Involved

models.dev sync (affects all clients of GET /v1/models)

Model(s) Involved

N/A (row count / N+1 shape matters; we observed 190 models_dev_pricing entries)

Client Tool

Desktop / coding-agent clients that periodically call GET /v1/models (e.g. catalog refresh)

Description

Hourly models.dev periodic sync is a self-DoS on 3.8.49. Each successful sync invalidates pricing/catalog memoization; the next client catalog fetch rebuilds with the unmemoized N+1 path; the event loop sticks; the dashboard dies — on a timer.

Sequence (stock code):

1. startPeriodicSync / periodic timer (src/lib/modelsDevSync.ts)
2. Sync succeeds → saveModelsDevPricing(...)
3. saveModelsDevPricing calls invalidateDbCache("pricing") (same file ~222–236)
4. Catalog memo dropped via dropCatalogCacheIfStateChanged() in src/app/api/v1/models/catalogCache.ts
5. Next Desktop/Codex/curl GET /v1/models runs unmemoized getModelsDevPricing per model on 3.8.49 (fix(api): GET /v1/models pegs event loop — getModelsDevPricing re-reads per model #9685)
6. Event loop stuck → /dashboard 502/timeout / 0-byte public responses; /healthz multi-second or timeout

Observed settings shape:

Key	Value
modelsDevSyncEnabled	true
modelsDevSyncInterval	3600000 (1 hour — schema minimum in settingsSchemas.ts)
models_dev_pricing rows	190

After we set modelsDevSyncEnabled=false, deleted the 190 rows, and restarted:

• UI stayed in the 10–50 ms range
• /healthz 6–13 ms
• No more hourly "OmniRoute is down" episodes tied to the sync cycle

This is not random flakiness. It is a scheduled invalidation of a path that is known to be CPU-hostile on the image operators are forced to run until a SemVer with #10055 exists (#10296).

Steps to Reproduce

1. Run diegosouzapw/omniroute:3.8.49 single process.
2. Enable models.dev sync with interval 1h (modelsDevSyncInterval=3600000) and allow an initial sync to populate models_dev_pricing (order of hundreds of provider keys / pricing rows is enough; we had 190).
3. Confirm catalog has been built once and is serving from memo.
4. Either wait for the periodic timer or invoke the same write path the timer uses (saveModelsDevPricing → invalidateDbCache("pricing")).
5. Immediately have a client call GET /v1/models (Desktop/Codex catalog refresh is enough).
6. Concurrently time /dashboard and /healthz:

   curl -sS -o /dev/null -w 'models %{time_total}s\n' --max-time 120 \
     'http://127.0.0.1:20128/v1/models' -H "Authorization: Bearer $KEY"
   curl -sS -o /dev/null -w 'healthz %{time_total}s http=%{http_code}\n' --max-time 15 \
     'http://127.0.0.1:20128/healthz'
   curl -sS -o /dev/null -w 'dashboard %{time_total}s http=%{http_code}\n' --max-time 25 \
     'http://127.0.0.1:20128/dashboard'

7. Observe multi-second stalls / timeouts on dashboard and health during the rebuild.
8. Disable sync, delete pricing rows, restart; observe stable 10–50 ms UI.

Expected Behavior

1. Do not invalidate the live catalog on pricing sync until the new pricing data is memoized / safe to serve — or dual-buffer so readers never hit a cold unmemoized rebuild on the request thread.
2. Never run sync + full catalog rebuild on the request/event-loop thread that serves /dashboard and /healthz.
3. Default / minimum interval must not be 1 hour if each cycle can retrigger a 50s+ CPU loop on stable images; either raise the cost floor awareness, gate sync behind cheaper rebuilds, or refuse to ship intervals that schedule known multi-minute self-DoS on 3.8.49-class builds.
4. After a sync, the next GET /v1/models should not be a multi-tens-of-seconds event-loop stall for all other HTTP.

Actual Behavior

Step	What happens on 3.8.49
Periodic sync (1h)	Writes models_dev_pricing, invalidateDbCache("pricing")
Catalog memo	Dropped
Next GET /v1/models	Full rebuild + unmemoized per-model pricing (#9685)
Event loop	Stuck for seconds–tens of seconds
Dashboard	Dies every cycle (timeout / 502 / 0 bytes)
After disable + delete 190 rows + restart	UI stable 10–50 ms

The gateway schedules its own outage window.

Test Impact

Needs a new integration test

Error Logs / Output

image: diegosouzapw/omniroute:3.8.49
modelsDevSyncEnabled=true
modelsDevSyncInterval=3600000
models_dev_pricing rows=190

chain:
  startPeriodicSync
    -> syncModelsDev
      -> saveModelsDevPricing
        -> invalidateDbCache("pricing")
          -> dropCatalogCacheIfStateChanged()
            -> next GET /v1/models (unmemoized N+1 on 3.8.49)
              -> event loop stuck
                -> dashboard dead

after:
  modelsDevSyncEnabled=false
  DELETE 190 models_dev_pricing rows
  restart
  -> /healthz 6-13ms, /dashboard 10-50ms class, stable

Additional Context

Related: #9685 (N+1), #10052 (healthz), #10055 (memoize merged, not in 3.8.49 image), #10296 (no SemVer with fix), dashboard catch-22 (settings unreachable during the cycle).

Ask (concrete):

1. Do not drop the live catalog on pricing sync until the new pricing is memoized and a warm entry can be published atomically.
2. Never couple sync write + full catalog rebuild to the request thread.
3. Revisit 1h as a default/minimum while stable images still pay multi-tens-of-seconds rebuilds.
4. Until fixed: document that enabling models.dev sync on 3.8.49 is an intentional periodic self-DoS under real catalog sizes.

No secrets, hostnames, tokens, or private combo names.

Validation Plan

• Integration: after saveModelsDevPricing, concurrent /healthz and /dashboard stay fast; next /v1/models does not pin the loop for tens of seconds.
• Assert invalidation strategy (dual-buffer / post-memoize publish) with unit tests around invalidateDbCache("pricing") vs catalog memo.
• Soak with 1h timer accelerated (e.g. short interval in test) over many cycles without UI multi-second freezes on a 3.8.49-equivalent builder cost (or with memoize + large synthetic builder delay).

[diegosouzapw]

Triaged: this is tracked by an open pull request — #10299 — so we are routing it through the PR review workflow instead of re-analyzing or re-implementing it here. That keeps the contributor's authorship intact.

This issue stays open until that PR lands.

Housekeeping: #10299 was retargeted to release/v3.8.50 (the active development branch) so it can be merged into the current cycle.

Note for reviewers: that PR is a recovery mitigation (an env kill switch) and does not by itself replace the memoization fix, so this issue should not be closed on the PR alone.