[P1] Track translation workers explicitly during shutdown

[smereczynski]

Priority: P1 (High)

Category: Reliability / Background Processing

Issue

TranslationBackgroundService acquires a semaphore permit and then starts untracked Task.Run workers. During shutdown, cancellation can happen between permit acquisition and worker execution. In that case the worker may never reach its finally block and the permit is never released, causing shutdown waits and possible job loss.

Location

• src/Chat.Web/Services/TranslationBackgroundService.cs

Risk

• Graceful shutdown can hang until timeout
• In-flight jobs may be dropped or requeued inconsistently
• Semaphore state can become detached from actual worker state
• Service behavior becomes brittle during app recycle or deployment

Proposed Fix

Replace untracked Task.Run fire-and-forget workers with explicit worker tracking or a bounded worker loop that owns semaphore lifetime deterministically.

Suggested implementation direction:

• Maintain a concurrent set of active worker tasks and await them during shutdown
• Acquire and release permits inside the tracked worker lifecycle only
• Avoid passing the host stopping token to Task.Run as the only worker coordination mechanism
• Add tests that simulate shutdown while jobs are being dequeued

Effort Estimate

2-3 hours

Acceptance Criteria

• No semaphore permit can be lost during shutdown
• Background service awaits tracked in-flight translation workers before stopping
• Jobs are either completed or requeued predictably during shutdown
• Tests cover cancellation between dequeue, permit acquisition, and worker start
• Service logs clearly show shutdown behavior and remaining in-flight work

Reference

Static review finding from architecture and reliability review of src on 2026-05-02