U4: Worker refresh + warmup#11
Conversation
Shorten the scan/health/exam cycle from 1h to 15min so dead/new models are discovered faster, and add a lightweight warmup pinger that hits passing models every 2min to keep upstream connections hot and surface failures between full cycles. The warmup uses its own Redis lock (worker:warmup-leader) so it can run alongside the main worker on any replica without fighting over the existing worker:leader key. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the background worker scheduling to run more frequently and adds a separate “warmup” loop to ping recently passing models on a shorter cadence, aiming to keep upstream connections warm and detect provider issues sooner.
Changes:
- Reduce the main worker cycle interval from 1 hour to 15 minutes.
- Add a new warmup worker that pings passing (non-cooldown) models every 2 minutes under its own Redis leader lock.
- Wire warmup startup into the existing worker startup flow.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
src/lib/worker/warmup.ts |
Adds the new warmup worker cycle, leader election, model selection query, and ping logic. |
src/lib/worker/index.ts |
Shortens the main worker’s schedule and starts the warmup loop alongside it. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const next = new Date(Date.now() + 15 * 60 * 1000).toISOString(); | ||
| await setState("next_run", next); |
There was a problem hiding this comment.
The worker cycle interval was shortened to 15 minutes, but src/lib/worker/leader.ts still uses a 55-minute LEADER_TTL_SEC. If the leader process dies, the lock can block all replicas from running the cycle for up to ~55 minutes. Please reduce the leader TTL to slightly longer than the new cycle window (and keep renew cadence consistent) so failover works as intended.
| headers: { | ||
| "Content-Type": "application/json", | ||
| Authorization: `Bearer ${apiKey || "dummy"}`, | ||
| }, | ||
| body: JSON.stringify({ |
There was a problem hiding this comment.
pingModel doesn’t include the OpenRouter-specific headers (HTTP-Referer / X-Title) that the rest of the worker code uses. This can cause warmup pings to be rejected/rate-limited and then incorrectly counted as failures. Consider sharing the same header builder logic used in health/benchmark (and avoid sending an Authorization header when there is no key).
| const res = await fetch(url, { | ||
| method: "POST", | ||
| headers: { | ||
| "Content-Type": "application/json", | ||
| Authorization: `Bearer ${apiKey || "dummy"}`, |
There was a problem hiding this comment.
The warmup fetch returns based on res.ok without consuming/cancelling the response body. With Node/undici this can prevent the underlying connection from being reused (or leak resources) across repeated pings. Please ensure the response body is fully read or explicitly cancelled before returning.
| recordOutcome(m.provider, m.model_id, true, latency); | ||
| } else { | ||
| failed++; | ||
| recordOutcome(m.provider, m.model_id, false, latency); |
There was a problem hiding this comment.
Warmup results are written into the live EMA via recordOutcome for both success and failure. Because warmup uses a much shorter timeout than the normal request/health paths, transient slowness can disproportionately penalize models/providers and affect routing decisions. Consider either (a) using timeouts/headers consistent with the real request path, or (b) recording warmup outcomes in a separate channel / only recording successes.
| recordOutcome(m.provider, m.model_id, false, latency); | |
| // Warmup uses a shorter timeout than the normal request path, so do | |
| // not let warmup-only failures penalize the live EMA used for routing. |
| export async function runWarmupCycle(): Promise<void> { | ||
| if (isWarming) return; | ||
|
|
||
| const isLeader = await acquireWarmupLeader(); | ||
| if (!isLeader) return; | ||
|
|
There was a problem hiding this comment.
This PR introduces a new warmup worker (leader election + SQL selection + concurrent fetch pings) but there are no tests covering its core behaviors (leader lock handling, model selection query filters, and ping error handling). Since the worker package already has Vitest coverage for similar modules, adding unit tests with mocked Redis/SQL/fetch would help prevent regressions.
Summary
Test plan
🤖 Generated with Claude Code