feat: v0.7.0 — statistical diagnostics + global/panel + classical/multivariate models

.claude/skills/anofox-forecast-models/SKILL.md

@@ -1,13 +1,14 @@
 ---
 name: anofox-forecast-models
 description: >
-  Forecasting models and the `ts_forecast_by` API surface of the
-  anofox_forecast DuckDB extension. Covers 33 models (baseline,
-  exponential smoothing, state-space, ARIMA, Theta, multi-seasonal,
-  intermittent-demand, distributional Laplace with three variants),
-  parameter surfaces (MAP + STRUCT), model selection guidance, and
-  common workflow gotchas. Use when picking a model or writing
-  `ts_forecast_by` / `ts_forecast_agg` calls.
+  Forecasting models and the `ts_forecast_by` / `ts_forecast_var_by` API surface of the
+  anofox_forecast DuckDB extension. Covers 36 models (baseline,
+  exponential smoothing, state-space ARIMA + Kalman, classical GARCH,
+  Theta, multi-seasonal, intermittent-demand, distributional Laplace with
+  three variants, panel/global GlobalETS/GlobalTheta/GlobalCroston, and
+  multivariate VAR via ts_forecast_var_by), parameter surfaces (MAP + STRUCT),
+  model selection guidance, and common workflow gotchas. Use when picking a
+  model or writing `ts_forecast_by` / `ts_forecast_agg` / `ts_forecast_var_by` calls.
 version: 0.15.3
 user-invocable: false
 ---
@@ -16,7 +17,7 @@ user-invocable: false
 
 **Extension:** `anofox_forecast` v0.15.3 (Rust crate `anofox-forecast` v0.15.3) | **DuckDB:** v1.4.5 LTS / v1.5.4+ | **Dual naming:** `ts_*` and `anofox_fcst_ts_*`
 
-33 forecasting models exposed by SQL via three call surfaces (table macro, aggregate, scalar).
+36 forecasting models exposed by SQL via three call surfaces (table macro, aggregate, scalar) + `ts_forecast_var_by` for multivariate VAR.
 
 ## Critical gotchas
 
@@ -61,6 +62,98 @@ ts_forecast_by(
 ) → TABLE(group_col, forecast_step INT, ds, yhat DOUBLE, yhat_lower, yhat_upper, model_name)
 ```
 
+## `ts_forecast_panel_by` (panel / global models)
+
+**Fit-once-emit-many** panel API — pools parameter optimization across all series, then predicts per-series. Use when individual series are short but collectively form a large, homogeneous panel. **Three panel methods:**
+
+```sql
+ts_forecast_panel_by(
+    source VARCHAR,       -- table name (quoted string)
+    group_col COLUMN,     -- series identifier (unquoted)
+    date_col COLUMN,      -- date / timestamp (unquoted)
+    target_col COLUMN,    -- value to forecast (unquoted)
+    method VARCHAR,       -- 'GlobalETS' | 'GlobalTheta' | 'GlobalCroston'
+    horizon INTEGER,
+    frequency VARCHAR,    -- '1d', '1mo', ...
+    params MAP{}          -- optional; see below
+) → TABLE(group_col, forecast_step INT, date_col TIMESTAMP, yhat DOUBLE, model_name)
+```
+
+### Panel methods
+
+| Method | Best for | Key params |
+|---|---|---|
+| `'GlobalETS'` | Many related series, shared seasonal dynamics | `seasonal_period` (0=non-seasonal default), `model_pool` ('Reduced' default \| 'Complete') |
+| `'GlobalTheta'` | Trended panels, minimal config | none (seasonal_period ignored) |
+| `'GlobalCroston'` | Intermittent/spare-parts panels (many zeros) | `croston_variant` ('Classic' default \| 'SBA') |
+
+### Critical panel gotchas
+
+1. **Series dropped below 10 obs:** Short series after alignment emit `DROPPED: too_short` rows — not errors. Check `model_name` in the result.
+2. **Minimum 3 series after drop:** Fewer than 3 kept series → `InvalidInputException`. Ensure your panel has enough history.
+3. **Point forecasts only (v1):** `yhat_lower`/`yhat_upper` are not populated. Use `ts_conformal_by` separately if intervals needed.
+4. **TABLE arg must be a subselect:** The underlying `_ts_forecast_panel_native` uses the subselect pattern internally. Pass only a table name (quoted string) to `ts_forecast_panel_by` — do NOT pass a CTE or subquery as `source`.
+5. **GlobalCroston with all-zero panel fails:** Ensure at least one series has ≥ 2 non-zero demand events in the aligned window.
+6. **GlobalETS `seasonal_period=0`** → non-seasonal (Reduced pool, `ANN`/`AAdN`/`MNN`/`MAdN` candidates only). Period=1 has the same effect.
+
+### Panel quick examples
+
+```sql
+-- GlobalETS weekly seasonal panel
+SELECT * FROM ts_forecast_panel_by('sales', product_id, ds, y, 'GlobalETS', 14, '1d',
+    MAP {'seasonal_period': '7'});
+
+-- GlobalTheta trended panel (no config needed)
+SELECT * FROM ts_forecast_panel_by('sales', product_id, ds, y, 'GlobalTheta', 14, '1d');
+
+-- GlobalCroston SBA for spare-parts panel
+SELECT * FROM ts_forecast_panel_by('spares', item_id, ds, qty, 'GlobalCroston', 6, '1d',
+    MAP {'croston_variant': 'SBA'});
+```
+
+---
+
+## `ts_forecast_var_by` (multivariate VAR)
+
+**Dedicated multivariate function** — distinct from `ts_forecast_by`. Fits a VAR(p) model
+across K variables simultaneously (cross-variable dynamics). Returns long format.
+
+**v1 constraints:** Single-panel only (no `group_col`). Named param `p` for lag order (`order` is a SQL reserved word). Point forecasts only (no intervals).
+
+```sql
+ts_forecast_var_by(
+    source     VARCHAR,     -- source table name (quoted string)
+    date_col   VARCHAR,     -- date column name (quoted string)
+    value_cols VARCHAR[],   -- array of value column names ['y1', 'y2', ...]
+    horizon    INTEGER,     -- periods to forecast
+    frequency  VARCHAR,     -- time step between observations
+    p          INTEGER,     -- lag order (named param, default: 1)
+    params     MAP          -- reserved for future use (default: MAP{})
+) → TABLE(variable VARCHAR, forecast_step BIGINT, <date_col>, forecast_value DOUBLE)
+```
+
+**Output:** `k_vars × horizon` rows in long format. One row per (variable, forecast_step).
+
+```sql
+-- VAR(1) — 2-variable system, 14-step ahead
+SELECT * REPLACE(ROUND(forecast_value, 6) AS forecast_value)
+FROM ts_forecast_var_by('var_src', 'ds', ['y1', 'y2'], 14, '1d')
+ORDER BY variable, forecast_step;
+-- Returns 28 rows: y1 × 14 + y2 × 14
+
+-- VAR(2) — higher lag order
+SELECT * FROM ts_forecast_var_by('var_src', 'ds', ['y1', 'y2'], 14, '1d', p:=2);
+```
+
+**Pitfalls:**
+- Use `p:=2` NOT `order:=2` (ORDER is a SQL reserved word).
+- All value columns must have the same valid observation count after null imputation.
+- Minimum obs: n > k×p+1 (n=obs, k=variables, p=lag order).
+- Non-stationary series → unstable coefficient matrix; difference first with `ts_diff_by`.
+- **Benchmark:** VAR(1) on synthetic VAR(1) data — MAE ratio vs statsmodels = 1.000 (exact match, PASS).
+
+---
+
 ## `ts_forecast_agg` (aggregate)
 
 For custom `GROUP BY` shapes.
@@ -77,7 +170,7 @@ FROM sales GROUP BY product_id;
 
 Access fields: `(fcst).point_forecast`, `(fcst).lower_90`, etc.
 
-## Model catalogue (33)
+## Model catalogue (36)
 
 ### Automatic selection (6)
 
@@ -121,12 +214,48 @@ Access fields: `(fcst).point_forecast`, `(fcst).lower_90`, etc.
 | `DynamicOptimizedTheta` | `seasonal_period` |
 | `AutoTheta` | `seasonal_period` (listed above) |
 
-### State-space / ARIMA (2 — `AutoETS`/`AutoARIMA` counted above)
+### Classical volatility (1)
+
+| Model | Required | Optional | Important |
+|---|---|---|---|
+| `GARCH` | — | `garch_p` (default 1), `garch_q` (default 1) | **`yhat` is VOLATILITY (std-dev = sqrt(variance)), NOT variance.** Use on returns (first differences), not raw price levels. Min obs: p+q+10. |
+
+```sql
+-- GARCH(1,1) — volatility forecast on returns (40 obs minimum > 12)
+SELECT asset_id, forecast_step, ds, yhat AS conditional_volatility, model_name
+FROM ts_forecast_by('returns', asset_id, ds, y, 'GARCH', 7, '1d');
+
+-- GARCH(1,1) with explicit params
+FROM ts_forecast_by('returns', asset_id, ds, y, 'GARCH', 7, '1d',
+    params := MAP{'garch_p':'1','garch_q':'1'})
+```
+
+**Critical:** `yhat` is σ (std-dev), not σ². Square for variance: `yhat * yhat`.
+**Critical:** Use on returns (LN differences of prices), not raw levels — non-stationary levels cause α+β→1 divergence.
+**Benchmark:** ratio vs arch package = 0.897 on M4 Daily (PASS).
+
+### State-space / ARIMA (3 — `AutoETS`/`AutoARIMA` counted above)
 
 | Model | Required | Optional |
 |---|---|---|
 | `ETS` | — | `seasonal_period`, `model` (`'AAA'`, `'AAN'`, …) |
 | `ARIMA` | `p`, `d`, `q` | `P`, `D`, `Q`, `s` |
+| `Kalman` | — | `kalman_model` (`'local_level'` default \| `'local_linear_trend'`) |
+
+**Kalman details:**
+- `local_level` (default): random walk + noise; h-step forecast is flat at filtered level.
+- `local_linear_trend`: level + trend; h-step forecast grows/shrinks linearly.
+- Uses fixed variance params (obs_var=1.0, level_var=0.1), NOT MLE-estimated.
+- Benchmark: ratio vs statsmodels UnobservedComponents = 1.000 (local_level) / 0.992 (llt), both PASS.
+
+```sql
+-- Kalman local_level (default)
+SELECT * FROM ts_forecast_by('sales', product_id, ds, y, 'Kalman', 14, '1d');
+
+-- Kalman local_linear_trend
+SELECT * FROM ts_forecast_by('sales', product_id, ds, y, 'Kalman', 14, '1d',
+    params := MAP{'kalman_model': 'local_linear_trend'});
+```
 
 ### Multi-seasonal (3 — `Auto*` counted above)
 

.planning/MILESTONES.md

@@ -0,0 +1,13 @@
+# Milestones
+
+## v0.7.0 Close the Crate→Extension Gap (Diagnostics + Model Coverage) (Shipped: 2026-08-22)
+
+**Phases completed:** 3 phases, 9 plans, 10 tasks
+
+**Key accomplishments:**
+
+- GlobalETS fit-once-emit-many panel architecture proven end-to-end: Rust FFI PanelForecastResult → C++ ragged-alignment Finalize → ts_forecast_panel_by SQL macro returning per-series forecasts for a 3-series ragged panel
+- Committed M4 Daily benchmark proving behavioral parity: GlobalETS (+1.8%), GlobalTheta (-0.7%), GlobalCroston (-6.9%) vs statsforecast references — all within the D-Area4 tolerance standard on 500-series subset.
+- New ts_forecast_var_by macro backed by a VAR(p) FFI export and _ts_forecast_var_native C++ table function, delivering true multivariate cross-variable forecasting in long-format SQL output (CLAS-03).
+
+---

.planning/PROJECT.md

@@ -0,0 +1,104 @@
+# anofox-forecast — Milestone: Close the Crate→Extension Gap (Diagnostics + Model Coverage)
+
+## What This Is
+
+`anofox-forecast` is a DuckDB extension that exposes SQL-native time-series forecasting, backed by the `anofox-forecast` Rust crate (v0.15.3) via an FFI boundary. It already surfaces 36 forecasting models, 117 features, cross-validation, conformal prediction intervals, seasonality/period/changepoint/peak detection, and data-prep utilities as SQL functions and `ts_*_by` macros.
+
+This milestone extends that SQL surface to reach crate capabilities that are currently unreachable from SQL: statistical **diagnostics & validation**, and additional **forecasting models** (global/panel and classical). It is a brownfield capability-exposure milestone, not a rewrite — the delivery pattern is the established one: Rust FFI export → C++ table/scalar/aggregate function → `ts_*_by` SQL macro → runnable example → docs.
+
+## Core Value
+
+SQL users can validate whether a series/model is statistically sound (stationarity, residual adequacy, demand regime) and can reach the crate's higher-coverage models (global + classical) — all without leaving DuckDB.
+
+## Requirements
+
+### Validated
+
+<!-- Inferred from existing codebase (brownfield); relied upon and shipped. -->
+
+- ✓ `ts_forecast_by` + 36 model strings (baselines, ETS/Holt-Winters, Theta, ARIMA, MFLES, MSTL, TBATS, intermittent, Laplace) — existing
+- ✓ 117-feature extraction (`ts_features*`), tsfresh-compatible — existing
+- ✓ Cross-validation + backtest (`ts_cv_folds_by`, `ts_cv_forecast_by`, leakage check) — existing
+- ✓ Conformal prediction intervals (split/adaptive/asymmetric/per-step, learn+apply) + bootstrap — existing
+- ✓ Period/seasonality detection (~15 methods), MSTL decomposition, changepoints (PELT + BOCPD), peaks — existing
+- ✓ 12 accuracy metrics, data-quality scoring, gap/null/differencing data prep — existing
+- ✓ FFI + native-table-function + SQL-macro exposure pattern; DuckDB GROUP BY parallelism (no custom threading) — existing
+- ✓ Stationarity tests: `ts_adf(_by)`, `ts_kpss(_by)`, combined `ts_stationarity(_by)` four-way verdict — v0.7.0 (statsmodels-cross-checked)
+- ✓ Residual diagnostics: `ts_ljung_box_by`, `ts_durbin_watson_by`, `ts_jarque_bera_by`, combined `ts_residual_diagnostics_by` — v0.7.0
+- ✓ Global/panel models: `ts_forecast_panel_by` (GlobalETS/GlobalTheta/GlobalCroston, cross-series learning) — v0.7.0 (statsforecast M4 parity)
+- ✓ Classical models: `ts_forecast_by` methods `'GARCH'` (conditional volatility) and `'Kalman'` (state-space) — v0.7.0
+- ✓ Multivariate: `ts_forecast_var_by` (VAR, N value columns → per-variable long-format forecasts) — v0.7.0 (statsmodels VAR parity)
+- ✓ Milestone DoD upheld for every new function: runnable verified `examples/*.sql`, committed benchmark parity, `docs/api/` + `docs/reference/models/`, statsmodels/arch/R cross-checks
+
+### Active
+
+<!-- Next milestone. -->
+
+- [ ] (none yet — define via `/gsd-new-milestone`)
+
+Deferred from v0.7.0 (candidates for a future milestone):
+- Intermittent-demand classification (ADI/CV² taxonomy) — INTER-01 descoped; user has a more advanced approach TBD
+- Prediction intervals for the new global/panel + GARCH/Kalman/VAR surfaces (route through the existing conformal path)
+- VAR automatic lag-order selection (AIC/BIC); per-panel VAR; GARCH advanced coefficient overrides beyond p/q
+
+### Out of Scope
+
+- Anomaly detection (Mahalanobis/Parade/ZBank) — deferred to a later milestone despite `anomaly` feature being compiled in; not selected for v1
+- Hierarchical reconciliation (MinTrace/BottomUp/TopDown/MiddleOut) — large standalone capability, own milestone
+- Forecastability / triage (AMI, GCMI, transfer entropy, Lyapunov, STI, `run_triage`) — requires enabling the `forecastability` crate feature; deferred
+- Multicollinearity / VIF — deferred with the exogenous-regression track
+- Power transforms (Box-Cox / Yeo-Johnson) and scaling/rolling/EWM transforms — deferred (pairs with global-regression-fe work later)
+- Ensemble / AutoEnsemble — not selected for v1
+- Extra conformal methods (IDR, QRA, CQR, EnbPI, binned) and extra changepoint algorithms (Binseg/BottomUp/Dynp/Window/KernelCpd) — existing coverage sufficient for now
+- Outlier detection, model persistence (save/load), feature selection — deferred
+
+## Context
+
+- **Delivery pattern (established):** new capability = Rust FFI `#[no_mangle] pub extern "C"` export in `crates/anofox-fcst-ffi` → C++ table/scalar/aggregate function in `src/` → registration in `src/anofox_forecast_extension.cpp` → user-facing `ts_*_by` macro in `src/macros/ts_macros.cpp` → `examples/*.sql` → `docs/`.
+- **Crate features currently enabled:** `anomaly`, `serde`, and default `postprocess` (→ `distributional`). NOT enabled: `forecastability`, `seasonal-detection`, `parallel`. The diagnostics and models in this milestone live under already-enabled features (`crate::validation`, `crate::models::*`), so no new feature flags are required for v1 scope.
+- **Global models** are panel/batch forecasters (`GlobalETS`/`GlobalTheta`/`GlobalCroston`, `crate::batch`) — they cross-learn across series, so the SQL surface must accept a grouped panel, not a single series. This differs from the per-series `ts_forecast_by` dispatch and needs design attention.
+- **VAR** is multivariate — output/interface shape differs from univariate models; may warrant its own function rather than a `method` string on `ts_forecast_by`.
+- **Diagnostics** operate on residuals or a raw series and return scalar/struct verdicts — natural fit for scalar functions + `_by` macros, mirroring the metrics functions.
+- Verified reference: docs SQL examples must be run through the built extension, not eyeballed (established rule from PR #230).
+- **Shipped v0.7.0** (2026-08-22): +17.9k LOC across 57 commits / 111 files. Extension now surfaces 36 forecasting models (incl. GARCH, Kalman, panel Global*, multivariate VAR via `ts_forecast_var_by`) plus 7 statistical-diagnostic functions. `arch` added to `benchmark/.venv` (comparison group) for GARCH parity. New non-globbed C++ sources (`diagnostics.cpp`, `ts_forecast_panel_native.cpp`, `ts_forecast_var_native.cpp`) are explicitly listed in CMakeLists.
+- **Panel/table-in macro convention (v0.7.0 lesson):** table-in macros must wrap `query_table(...)` in a subselect `(SELECT ... FROM query_table(...))`; a bare TABLE arg silently fails to register.
+
+## Constraints
+
+- **Tech stack**: DuckDB v1.4.3+ extension; Rust 1.86+ core via FFI; C++17. No new languages.
+- **Architecture**: Parallelism stays at the DuckDB GROUP BY / scalar-function layer — no custom threading or table-in/table-out (established project rule).
+- **Dependencies**: Stay on `anofox-forecast` 0.15.3 unless a required capability is missing; global-model steady-state ARIMA optimization tracked separately (awaiting 0.5.4-class improvements).
+- **Compatibility**: Must build and load across Linux/macOS/Windows and WASM; OpenSSL stays statically linked; verify clean-machine load (not just green CI).
+- **Verification**: Every new SQL function must be exercised by a runnable example against the built extension before it counts as done.
+
+## Key Decisions
+
+| Decision | Rationale | Outcome |
+|----------|-----------|---------|
+| Scope milestone to diagnostics + model coverage (defer anomaly, reconciliation, triage) | Both chosen themes reuse already-enabled crate features and the existing exposure pattern; lower risk than new-feature-flag work | ✓ Good — v0.7.0 shipped all diagnostics + 6 new models via the existing pattern with no new crate feature flags |
+| Expose diagnostics as scalar functions + `ts_*_by` macros | Mirrors existing metrics surface; returns scalar/struct verdicts per series | ✓ Good — 7 diagnostic functions shipped, statsmodels-cross-checked |
+| Global/panel models need a panel-aware SQL surface | GlobalETS/Theta/Croston cross-learn across series; per-series `ts_forecast_by` dispatch is insufficient | ✓ Good — `ts_forecast_panel_by` fit-once-emit-many native table function delivered |
+| VAR is a dedicated multivariate function (`ts_forecast_var_by`), not a `method` string | Multivariate I/O shape (N cols → N forecasts) differs from univariate `ts_forecast_by` | ✓ Good — long-format `{variable, forecast_date, forecast_value}` surface delivered |
+| Definition of done = example + benchmark parity + docs + reference cross-check | User requires all four validation signals for every item | ✓ Good — upheld for all 13 requirements |
+| ForecastOptions FFI ABI extended additively for GARCH/Kalman params | Backward-compatible with existing univariate methods; avoids a parallel options struct | ✓ Good — integration-checker confirmed no ABI breakage across pre-milestone methods |
+| Autonomous code-review + fix loop after each phase | Happy-path verifiers miss edge cases (spurious intervals, WASM free UB, overflow) | ⚠️ Revisit — caught real bugs, but each phase needed 2–3 fix iterations; consider tightening executor guidance to prevent recurrence |
+
+## Evolution
+
+This document evolves at phase transitions and milestone boundaries.
+
+**After each phase transition** (via `/gsd-transition`):
+1. Requirements invalidated? → Move to Out of Scope with reason
+2. Requirements validated? → Move to Validated with phase reference
+3. New requirements emerged? → Add to Active
+4. Decisions to log? → Add to Key Decisions
+5. "What This Is" still accurate? → Update if drifted
+
+**After each milestone** (via `/gsd-complete-milestone`):
+1. Full review of all sections
+2. Core Value check — still the right priority?
+3. Audit Out of Scope — reasons still valid?
+4. Update Context with current state
+
+---
+*Last updated: 2026-08-22 after v0.7.0 milestone*