Bound media request queues, filmstrip cache keys, and export job sizes

[badbread]

What changed

The media routes could queue work without limit. Every expensive media path (clip and low-bitrate transcodes, filmstrip frame extraction, the live camera still proxy) waits on a shared work semaphore, and those waits were open-ended: once the permits were gone, a request parked indefinitely, held its connection, and never answered. The media router also carried neither a request timeout nor a rate limit, unlike the JSON routes.

Alongside that, two paths let one request ask for an unbounded amount of work: the filmstrip cache key included an unquantized width, and POST /export capped neither the time range nor the camera list.

Bounded waits on the media router

• New services/api/src/media_limits.rs. One helper (acquire_bounded) wraps every media semaphore acquire with a per-call-site wait budget. A saturated queue now answers 503 with Retry-After instead of hanging. Applied to the clip transcode, the clip thumbnail, the low-bitrate segment transcode, filmstrip extraction, and the still proxy. Budgets: 15 s for the transcodes (which run in seconds in practice), 5 s for a single-frame extraction and for a still (a scrub thumbnail that arrives after 5 s is off screen; a wall tile polls again in about a second).
• The media router gets its own, much larger rate-limit bucket, reusing the existing rate_limit::RateLimiter and keyed exactly as the JSON bucket is: burst 1200, 240/s sustained. The busiest real client (a 16-tile low-bandwidth wall at 1 Hz, plus segment fetches, plus a scrub burst) sits near 30 to 40 req/s, so that is roughly a 6x margin, while a runaway loop is still stopped well before it can queue work.
• The media router also gets a 3-minute time-to-response bound. tower_http's TimeoutLayer bounds only the handler future, not the response body (body deadlines are a separate ResponseBodyTimeoutLayer that is deliberately not used), so segment downloads, export archives, and the open-ended live stream.mp4 cannot be cut: they return headers immediately and stream afterwards. The one route that legitimately produces for minutes, the on-demand DB-vs-disk size verification walk, is mounted outside the timeout for that reason.

Filmstrip cache keys

Widths clamp and then snap to 80/160/320/480/640 before entering the cache key, so one instant produces five renditions instead of 593. The buckets are chosen so every width the shipped clients ask for lands on itself (160 for the Android and desktop scrub lists, 320 for the Android playback wall, 480 for the iOS scrub still, the desktop preview frame, and the THUMB_PREGEN_WIDTH default), so no warm or pre-generated cache entry is invalidated. The frame route's stale "width is unused" doc comment and dead_code attribute are corrected: the width has been honoured since it became part of the cache key.

Export job sizes

• POST /export sorts and de-duplicates its camera list. The RBAC scope filter never did (for an admin it returns the list verbatim), so a body repeating one camera N times ran N sequential full-range encodes, all writing the same output file with -y, all counted as one job against EXPORT_MAX_CONCURRENT. The misleading "filter still runs to dedup" comment is gone.
• The distinct-camera count is capped at 50, matching what /export/batch already caps clips at: both routes fan out to one sequential encode per unit of work, so they get the same ceiling.
• New EXPORT_MAX_RANGE_SECONDS (default 86400, a full day of one camera) caps the window with a clear 400. The same cap applies per clip in a batch export.
• Export ffmpeg children get kill_on_drop(true) (as the filmstrip and backup children already had) plus a wall-clock budget derived from the range: 4x realtime, floored at 10 minutes, capped at 24 hours. A wedged child now fails the job with a stored error instead of leaving it Running forever, which used to consume an EXPORT_MAX_CONCURRENT slot permanently until the api was restarted.
• The export byte-budget sweeper now counts in-flight jobs' bytes against the budget (it still never deletes them), so the on-disk total no longer sits above EXPORT_CACHE_MAX_BYTES for as long as jobs keep running.

Live still proxy

GET /cameras/:id/frame.jpg goes behind a new frame-proxy semaphore. A camera that is disabled, rejected by go2rtc, or whose last fetch failed is latched for 10 seconds and served by a single quick attempt, so a wall of tiles pointed at a camera that is down fails in about a second per poll instead of roughly 17. A successful fetch clears the latch immediately, so a camera that comes back is served normally on the next poll.

Operator-visible changes

Three env keys are now documented and forwarded by compose. All have defaults; nothing is required.

Key	Default	Status
EXPORT_MAX_RANGE_SECONDS	86400	new
FRAME_PROXY_MAX_CONCURRENCY	one per core, clamped 8 to 32	new
EXPORT_MAX_CONCURRENT	2	already existed, was undocumented and not forwarded

Updated: .env.example, docker-compose.yml, docs-site/docs/configuration/environment-reference.md. docs/AI-INSTALL.md needs no change: it does not enumerate optional tuning keys (same as the existing EXPORT_CACHE_MAX_BYTES, SEGMENT_LOW_CACHE_MAX_BYTES, and THUMB_* keys). Two docs/DECISIONS.md entries record the bounded-wait choice and the quantizer plus export caps, with the alternatives rejected and revisit triggers.

Behaviour a client can notice: under genuine saturation a scrub thumbnail, clip, or wall tile now fails fast with 503 plus Retry-After instead of eventually arriving. That is the intended trade, and the clients already treat a failed media fetch as "keep the placeholder and poll again". An export naming more than 50 distinct cameras, or a window longer than a day, is refused with a 400 that says why.

How it was tested

Full gate on the build box against this branch: cargo fmt --all -- --check, cargo clippy --all-targets -- -D warnings, and cargo test --workspace against a throwaway Postgres 16, all green.

New tests:

• media_limits: a zero-permit semaphore must take the 503 path inside its budget (and not park), and a free semaphore hands out and returns a permit.
• filmstrip: the width quantizer keeps 160/320/480 exact, snaps to the nearest bucket, breaks ties deterministically downward, and collapses the whole legal input range onto exactly the five buckets.
• export: the range, distinct-camera, dedup, empty and inverted-range validation; the derived ffmpeg budget is floored, scaled and capped; a child that outlives its budget is killed, reaped, and reported as TimedOut.
• services/api/tests/export_limits.rs (new integration test, the real handler through the shared harness): a duplicated camera list produces one job entry per distinct camera, and an over-range request is 400 with no job allocated, while a window exactly at the cap is still accepted.

Not verified by running: nothing here touches the clients, so no Android, iOS or Flutter build was needed. The saturation behaviour itself is covered by the unit test on the helper rather than by driving a real server to saturation.