Skip to content

perf(forecaster): scale PD training fetch and window features#48

Merged
tder311 merged 2 commits into
mainfrom
feat/forecaster/pd-fetch-scale
Jul 9, 2026
Merged

perf(forecaster): scale PD training fetch and window features#48
tder311 merged 2 commits into
mainfrom
feat/forecaster/pd-fetch-scale

Conversation

@tder311

@tder311 tder311 commented Jul 9, 2026

Copy link
Copy Markdown
Owner

Why

predispatch_price now holds 519k+ rows (#46) and grows ~8k/week. The training path pulled every stored run-row into pandas and computed window features via per-group Python lambdas — flagged as a scaling time bomb in #45's review (this path also runs inside the API via /api/forecast/retrain).

What

  • Two-stage SQL fetch in _fetch_predispatch_history: stage 1 selects run_datetimes with ≥1 interval inside the LEAD_BUCKETS lead envelope (computed from the buckets, not hardcoded); stage 2 fetches those runs' full row sets — unbounded by the window edge so boundary-day window aggregates are complete (an improvement over the old single query, which silently truncated the boundary day).
  • Vectorized predispatch_window_features: indicator-sum groupby + block-id longest-run; no Python-level per-group lambdas. Locked pd_window tests pass unchanged; outputs verified identical on a 500k synthetic frame.

Evidence

  • Benchmark (500k rows): 3.91s → 0.13s (~30×).
  • Live smoke test (local DB, read-only): 30-day window → 123 qualifying runs; rows 68,745 (edge-truncated) → 95,865 (complete windows) after the edge fix; features compute cleanly.
  • Tests: test_forecaster.py 40 passed; full unit suite green.

🤖 Generated with Claude Code

tder311 and others added 2 commits July 9, 2026 20:48
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tder311
tder311 marked this pull request as ready for review July 9, 2026 23:52
@tder311
tder311 merged commit 4474275 into main Jul 9, 2026
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant