Skip to content

Add opt-in winsorization for revenue/price outliers - #34

Merged
BasLinders merged 1 commit into
mainfrom
winsorize-revenue-outliers
Aug 21, 2026
Merged

Add opt-in winsorization for revenue/price outliers#34
BasLinders merged 1 commit into
mainfrom
winsorize-revenue-outliers

Conversation

@BasLinders

@BasLinders BasLinders commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Summary

  • New prox.winsorize_series() in prox/data_manager.py: same technique as first-order-engine's ContinuousMetricEngine.winsorize_series (https://github.com/BasLinders/first-order-engine/blob/main/foe/continuous/operations.py#L110) — caps values at mean ± N standard deviations, or at a percentile band — adapted to return a pd.Series instead of a JSON-serializable list, to match this codebase's DataFrame-in/DataFrame-out convention (PRoX doesn't depend on foe outside the optional BigQuery extra, so this isn't imported from there).
  • Wired into main.py as a new opt-in "2. Handle Outliers" step (existing sections renumbered 3→6 to make room), applied to the revenue/price column on both raw_df and df_ready right after the data is loaded/cached and before any filtering, sampling, or analysis reads that column — caps values in place rather than dropping the rows they came from.
  • Off by default, with a method choice (Standard Deviation / Percentile) and a slider for the threshold, plus a summary of how many values were capped and to what range.

Why

Average Order Value, revenue trend, and category revenue breakdown in Business Insights all read from the same price/revenue column, so a handful of extreme values (e.g. one bulk order) can dilute those stats with no way to address it short of editing the source data. Scoped to the revenue/price column specifically — winsorizing case duration/lead time directly would mean mutating event timestamps, which is a separate, more invasive change.

Test plan

  • pytest tests/ — 135 passed (6 new tests: percentile capping of a single injected outlier, std-based bounds match mean ± N·std, NaN positions preserved, no-outlier case leaves values unchanged, empty/all-NaN series return zero bounds)
  • Manually injected an extreme outlier (999999) into generated mock data and confirmed both methods cap it correctly (percentile bounds stayed tight and sane; std bounds showed the technique's known sensitivity to a single huge outlier inflating its own reference std — same behavior as the upstream first-order-engine implementation, not a regression)
  • Ran the full pipeline end-to-end with winsorized price fed into run_full_analysis and confirmed Average Order Value reflects the capped values
  • Manual click-through of the new checkbox/method/slider controls in a running Streamlit session — not run in this environment (no display)

@BasLinders BasLinders self-assigned this Aug 21, 2026
@BasLinders BasLinders added the enhancement New feature or request label Aug 21, 2026
Business reports (Average Order Value, revenue trend, category
revenue breakdown) all read from the same price/revenue column, so a
handful of extreme values (e.g. one bulk order) can dilute those
stats without any way to address it short of manually editing the
source data.

Adds prox.winsorize_series() - same technique as first-order-engine's
ContinuousMetricEngine.winsorize_series (mean +/- N std devs, or a
percentile band), adapted to return a Series instead of a JSON list to
match this codebase's DataFrame-in/DataFrame-out convention. Wired
into main.py as a new opt-in "2. Handle Outliers" step, applied to
both raw_df and df_ready right after the data is loaded and cached,
before any filtering/sampling/analysis reads the price column -
capping values rather than dropping the rows they came from.
@BasLinders
BasLinders force-pushed the winsorize-revenue-outliers branch from 174b8c5 to 7545b7c Compare August 21, 2026 11:55
@BasLinders
BasLinders marked this pull request as ready for review August 21, 2026 11:57
@BasLinders
BasLinders merged commit f830c2d into main Aug 21, 2026
1 check passed
@BasLinders
BasLinders deleted the winsorize-revenue-outliers branch August 21, 2026 11:58
BasLinders added a commit that referenced this pull request Sep 8, 2026
Add opt-in winsorization for revenue/price outliers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant