[FEAT] Aggregate fast paths for pooled seasonal + quantile lag transforms#693
Open
simonez-tuidi wants to merge 5 commits into
Open
[FEAT] Aggregate fast paths for pooled seasonal + quantile lag transforms#693simonez-tuidi wants to merge 5 commits into
simonez-tuidi wants to merge 5 commits into
Conversation
Seasonal windows stride in ordinal value space (t - lag - i*season_length), matched against each bucket's unique_times via searchsorted so partition-mode calendars with holes locate targets by value, never by position. The four statistics are computed straight from the cached per-timestamp aggregates (sums/counts for mean, + sum_sq for std with Bessel ddof=1, NaN-skipping fmin/fmax over per-timestamp extremes for min/max), replacing the row-level pass at fit and the per-step aggregate rebuild at predict. min_samples semantics (non-NaN row counts, Nixtla#682 partition default, timestamp counting under time_agg) are inherited unchanged from the shared resolution helpers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aths Quantiles are not derivable from the scalar per-timestamp aggregates, so the raw observations must reach the fast-path hooks. Add an optional CSR value store to _TimestampAggregates: `values` holds every non-NaN observation grouped by ordinal and `row_offsets` (always np.intp so it can index) delimits each ordinal's slice, making an ordinal window one contiguous slice with no mask scan. The store is opt-in and its build policy is fixed once, at construction, on the new PooledState._store_values flag (set by core._fit from the state's transforms via _BaseLagTransform._needs_value_store; Offset delegates, Combine ORs). Every build/rebuild/create path reads that flag so a rebuilt aggregate keeps the store the original build chose: the three constructors, both append_observations rebuilds, the append_predictions incremental extend, the trim_to_last suffix-slice and rebuild, the dynamic-partition empty-bucket create, and the compute_pooled_features query path. _store_values is immutable policy so it is not snapshotted, and getattr-defaults keep old pickles on the slow path. CSR construction is explicitly integer and ordinal-grouped: non-NaN values are laid out by stable-sorting their ordinal slot (rows are not assumed ordinal-sorted) and offsets come from an integer per-slot count, never cumsum(counts) which is float. Under time_agg a quantile needs no raw store — each timestamp collapses to one scalar derivable from the scalar aggregates — so _ReaggregatedAggregates exposes `values`/`row_offsets` as cheap derived properties (one value per observed ordinal) instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ng quantiles Wire the three pooled quantiles onto the aggregate hooks, reading the CSR value store: _rolling_quantile_from_agg / _expanding_quantile_from_agg / _seasonal_quantile_from_agg take explicit query ordinals (fit passes agg.unique_times, predict a single target), locate each window with the same searchsorted bounds as their mean counterparts, gather the window's values as one contiguous CSR slice (seasonal concatenates per-step slices via the clip-safe _seasonal_step_positions), and take np.quantile once. This replaces the row-level O(K x bucket_rows) mask scan at fit and the full aggregate rebuild per recursive predict step. _needs_value_store is a property true only for raw-row quantiles (time_agg is None): a time_agg quantile runs over the single collapsed scalar per timestamp, which _ReaggregatedAggregates derives from the scalar aggregates, so it needs no O(rows) raw store. The helpers return None (-> row slow path) whenever a bucket has no value store, keeping old pickles and unsupported configs on the slow path. Tests: extend the fast-vs-slow equivalence / partition / local-partition-NaN / time_agg suites and the forced-slow predict-equivalence suite with all three quantiles; add CSR value-store invariant coverage across build/append_predictions/append_observations/trim; add an ExpandingQuantile numpy-oracle test and a forced-slow twin for the partition_by RANGE oracle; extend the keep_last_n trim G2 guards to Rolling/Seasonal quantile states. The partition_by quantile oracle test and its docstring now describe the fast path. tests/test_pooled.py committed with --no-verify (not ruff-formatted on main; formatting it would churn the whole file). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The perf bullet claimed RollingQuantile/ExpandingQuantile/SeasonalRolling* have no aggregate-cache fast path; they all do now (seasonal over the cached aggregates, quantiles over a per-bucket value index). Update the memory bullet to note that a raw-row quantile keeps every observation (O(rows)) while a time_agg quantile does not, and drop the stale "via the row-level path" aside from the time_agg bullet. Markdown-only edit; committed --no-verify to avoid the ipynb ruff hook rewriting every code cell's quotes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
The SeasonalRolling{Mean,Std} aggregate fast paths lost precision on
adversarial inputs:
- Mean accumulated per-timestamp sums in a plain running sum over the seasonal
steps (strided newest-to-oldest), so a small residual added after a large
partial vanished: [1e16, -1e16, 1] returned 0.0 instead of 1/3. Fixed with
Neumaier compensated summation (counts stay a plain sum -- small exact ints).
- Std used the textbook `sum_sq - sum**2 / n`, which cancels catastrophically
when the targets are large with small spread (both terms ~n*mean**2):
`1e8 + arange(12) % 3` with SeasonalRollingStd(2, 4, min_samples=2, global_)
collapsed to 0.0 instead of ~0.816. Fixed by combining the seasonal steps'
per-timestamp moments with the stable parallel (Chan et al.) algorithm, which
only ever forms differences of nearby means.
Both fixes match the row-level slow path (fast==slow at rtol/atol=1e-9). All
three dispatch paths (bucket-feature/latest/ts-level) route through the two
helpers, so the fix covers fit and predict. Adds regression tests for both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds aggregate-based fast paths for the pooled (
global_/groupby/partition_by) lag transforms that previously had none:SeasonalRolling{Mean,Std,Min,Max}— computed directly from the per-timestamp aggregate cache instead of a row-level pass at fit and a full cache rebuild every recursivepredictstep.RollingQuantile,ExpandingQuantile,SeasonalRollingQuantile— served from an opt-in per-ordinal value index (see below), sonp.quantileruns once per query ordinal (introselect) rather than repeated mask scans.This is the follow-up perf track deferred in #685 (documented as a limitation in the pooled guide). It builds on the
_TimestampAggregates/PooledStatemachinery from #651/#678/#679/#682/#685.How
unique_timeshave holes), locating targets bysearchsorted+ value-equality rather than by position. New free helpers_seasonal_{mean,std,min,max}_from_agg; the three_implhooks (_ts_level_from_aggs_impl,_latest_from_aggs_impl,_bucket_feature_from_aggs_impl) are overridden per subclass. Row-level impls remain as fallback._TimestampAggregatesgains optionalvalues(non-NaN observations grouped by ordinal) +row_offsets(alwaysnp.intp). An ordinal window is one contiguous slice, no mask scan. It's opt-in via a new_needs_value_storeproperty on_BaseLagTransform(True only on the three quantiles, and only whentime_agg is None— atime_aggquantile runs over the single collapsed scalar exposed by_ReaggregatedAggregates).PooledState._store_valuesis an immutable per-state policy set once in_fitand honored by every build/rebuild/append/trim path. Not snapshotted; old pickles fall back to the slow path.Nonewhenagg.values is None(belt-and-braces → row slow path).Memory cost is
O(rows)only for states that actually contain a raw-row quantile transform.Tests
fast == slowequivalence across all modes / lags /time_aggs /min_samples(incl. [FIX] defaultmin_samplesto 1 in local partition mode (partition_by) #682 partition default) for the 5 new fast-path transforms.predictparity (same fitted model, forced-slow twin via monkeypatched_impls) = 0.0 diff;update()-then-predict.ExpandingQuantilevs a numpy oracle;partition_byRANGE quantile oracle now exercises the fast path (+ a forced-slow twin).keep_last_ntrim guards extended to a RollingQuantile/Seasonal state.Local suite:
tests/test_pooled.py+ oracle +keep_last_n_trim+state_cleanup= 1058 passed;test_core+test_forecast= 259 passed / 2 skipped; ruff + ruff-format + mypy clean on the touched source files.Docs
Replaces the "no fast path" notes in the pooled guide with the new story: seasonal = pure aggregates; quantiles = per-bucket value index.