Skip to content

Replace threading.Lock with asyncio.Lock in forecaster - #26

Merged
GrowlyX merged 1 commit into
breezycourses:mainfrom
philyuchkoff:feat/forecaster-worker-pool
Sep 4, 2026
Merged

Replace threading.Lock with asyncio.Lock in forecaster#26
GrowlyX merged 1 commit into
breezycourses:mainfrom
philyuchkoff:feat/forecaster-worker-pool

Conversation

@philyuchkoff

Copy link
Copy Markdown
Contributor

Problem

Torch is re-entrant but produces nondeterministic garbage under memory pressure when called concurrently from multiple threads. The previous threading.Lock serialised inference correctly, but at the cost of blocking the entire thread — health checks, queue handling, and other requests all piled up behind a single forecast.

Solution

  • TimesFMModel.forecast() is now async def forecast() using asyncio.Lock instead of threading.Lock
  • server.py: endpoint is now async def and uses await model.forecast()
  • Same correctness guarantees: serialisation at the model level preserves safe Torch behaviour
  • rest of the server (healthz, readyz, /v1/model, the async lifespan) is never behind the lock

What was not done

  • Not a worker pool — this project benefits from preserving a single model, and several workers would mean multiple model instances and complexity not needed
  • No changes to the Go controller (the forecaster as a separate process keeps the concern separate)

Verification

All 31 Python tests pass (uv run --extra dev pytest -q).

Torch is re-entrant but produces nondeterministic garbage under memory
pressure when called concurrently from multiple threads. The previous
threading.Lock serialised inference correctly, but at the cost of
blocking the entire thread -- health checks, queue handling, and other
requests all piled up behind a single forecast.

asyncio.Lock is still serialising at the model level, so correctness
is preserved. The rest of the server (healthz, readyz, /v1/model, the
async lifespan) is never behind the lock, and multiple prediction
requests queue cooperatively without blocking the event loop.

Test suite: 31 passed.
@GrowlyX
GrowlyX merged commit 12ffe88 into breezycourses:main Sep 4, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants