Skip to content

Milestones v0.8.0 + v0.9.0 — Ensemble Forecasting + WASM Runtime Verification (Phases 4–8) - #256

Merged
sipemu merged 86 commits into
mainfrom
milestone/v0.8.0-ensemble-forecasting
Sep 2, 2026
Merged

Milestones v0.8.0 + v0.9.0 — Ensemble Forecasting + WASM Runtime Verification (Phases 4–8)#256
sipemu merged 86 commits into
mainfrom
milestone/v0.8.0-ensemble-forecasting

Conversation

@sipemu

@sipemu sipemu commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Milestones v0.8.0 + v0.9.0 — Ensemble Forecasting + WASM Runtime Verification (Phases 4–8)

main currently has neither milestone, so this PR ships both to main in one merge (86 commits): the v0.8.0 ensemble surface (Phases 4–6) and the v0.9.0 WASM runtime verification + CI gating (Phases 7–8). The v0.9.0 summary is below; the original v0.8.0 summary follows unchanged.


Milestone v0.9.0 — WASM Runtime Verification

Proves the built anofox_forecast .wasm actually loads and runs in DuckDB-Wasm — not just that it compiles and links — and gates that in CI so WASM regressions fail the build. No new SQL functions or crate changes; work lives in test/wasm/, .github/workflows/, vcpkg.json, and README.md. Ports the CI/harness pattern from anofox-statistics PR #131.

Phase 7 — WASM Node harness + local green ✅ verified

  • test/wasm/run.mjs + sqllogic.mjs — boots DuckDB-Wasm (eh bundle, pthreadWorker=null), serves the built .wasm over localhost, FORCE INSTALL + LOADs it, and runs a sqllogictest subset with per-file db.open() catalog isolation and a COLUMNS(*)::VARCHAR wrap so DECIMAL rendering matches native (WASM-01, WASM-02).
  • DEP-01 — pins @duckdb/duckdb-wasm@1.33.1-dev64.0 (engine v1.5.5) + web-worker@1.2.0, with a documented strings | grep version-verification procedure in test/wasm/README.md.
  • DEP-02vcpkg.json declares openssl as { "platform": "!wasm32" }. Build-confirmed: a fresh local wasm_eh build compiled zero OpenSSL for the wasm target (mbedtls is used instead); native builds still statically link it.
  • Curated green subset: 396/396 assertions across 8 files pass against the built .wasm.

Phase 8 — CI gating + dedicated workflow + badge ◆ implemented

  • CI-01wasm-runtime-test gating job in MainDistributionPipeline.yml: needs: the wasm build, downloads the anofox_forecast-v1.5.5-extension-wasm_eh artifact, runs the curated harness, fails the build on any WASM load/runtime error.
  • CI-02 — dedicated .github/workflows/WasmTest.yml (workflow_run-triggered) so WASM status is independently observable.
  • CI-03 — README WASM status badge wired to WasmTest.yml.
  • Gate mechanism proven locally: curated run exit 0 → a deliberately broken curated assertion → exit 1 (red) → reverted → exit 0.

Honest scope notes

  • WASM-03 (full 66-file suite green) is deferred. The full --all run is red (184 failures / 23 files), but these were confirmed against a fresh HEAD build to be pre-existing test/sql test-suite debt — stale references to removed/renamed API (ts_backtest_auto_by, ts_hydrate_features_by, …), DATE + BIGINT bugs, and cascade failures — not WASM or artifact-drift issues. They're masked natively because the unittest runner skips these files on an unsatisfied require json; DuckDB-Wasm auto-loads json and exposes them. CI gates on the curated subset, not --all. Making the full suite green is tracked as separate test-triage.
  • CI-01 live negative-control (observing green → red → green in GitHub Actions + the badge flip) is a post-merge operator step; the gate's core mechanism is already proven locally.
  • CI hygiene: excluded windows_amd64 on the v1.4.5 LTS line only — the v1.4-andium ci-tools vcpkg baseline pins an OpenSSL dependency whose upstream download now 404s (not fixable here; same class anofox-statistics documented). The v1.5.5 windows_amd64 build uses a newer baseline, passes, and still ships.

Tag: v0.9.0. Includes the full .planning/ audit trail (repo convention).


Milestone v0.8.0 — Ensemble Forecasting

Exposes the anofox-forecast crate's ensemble surface to SQL: combine multiple models per series (automatically or by explicit member list), with six combination methods, distribution-free conformal prediction intervals, and member/weight introspection — all via the established FFI → C++ → macro → example → docs pattern.

8/8 requirements delivered and verified against the built extension · 3 phases · 6 plans · ~13.9k LOC / 57 files.

New SQL surface

Function Purpose Requirement
ts_forecast_by(..., 'AutoEnsemble', ..., {top_k, combination_method, seasonal_period}) Auto-fit ARIMA/ETS/Theta, combine top-K ENS-01
six combination_methods: mean,median,weighted_mse,inverse_aic,stacking,horizon_adaptive Blend members COMB-01..04
ts_forecast_ensemble_by('table', grp, ds, y, members VARCHAR[], ...) Explicit user-named member ensemble (+ build_forecaster factory: 26-member allowlist, 10 blocked with clear errors) ENS-02
ensemble conformal intervals via ts_cv_folds_by + ts_conformal_calibrate/apply Distribution-free lower/upper bounds (reuses existing path — no new interval code) EPI-01
ts_ensemble_inspect_by / ts_auto_ensemble_inspect_by Selected members + combination weights + MSE score/rank, long format INSP-01

Verification

  • Every example run against the built extension (PR docs: fix README ToC + getting-started SQL examples (#224, #225, #226) #230 rule): Mean cross-check diff=0.0, all six methods produce finite forecasts, all error paths raise clear member-naming errors, conformal lower ≤ point ≤ upper per step, all INSP-01 DoD assertions 0 failures.
  • 233 Rust unit tests green. Code review across the milestone fixed 5 warnings (negative top_k clamp, exog param path, NULL-interpolation/min-length guards, SMA window) + info items.
  • Cross-phase integration checked: shared parse_combination_method + build_forecaster back all three surfaces consistently; no regression on pre-milestone models.

Known tech debt (documented in milestones/v0.8.0-MILESTONE-AUDIT.md)

  • ts_cv_forecast_by('AutoEnsemble') segfaults — pre-existing crate/CV-native bug (ts_cv_forecast_native.cpp:380-388 never parses the ensemble params). EPI-01 ships via a documented manual per-fold _ts_forecast_scalar workaround. Recommended follow-up: wire ensemble-param parsing into the CV native.
  • AutoEnsemble non-Mean combination weights return NULL — crate 0.15.3 exposes no inner-ensemble weight accessor (documented; upstream enhancement).
  • build_forecaster SeasonalWindowAverage n_seasons=2 hardcoded (TODO ENS-03).

Tag: v0.8.0. Includes the full .planning/ audit trail (repo convention).

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

sipemu and others added 30 commits August 30, 2026 21:51
- Add AutoEnsemble enum variant to ModelType (with doc comment)
- Add from_str exact-match ("AutoEnsemble") and lowercase fallback
  ("autoensemble" | "auto_ensemble") arms
- Add name() arm returning "AutoEnsemble"
- Add ensemble_top_k: usize and ensemble_method: Option<String> to
  ForecastOptions and ForecastOptionsExog (additive, after kalman_model)
- Thread new fields through ForecastOptionsExog::Default and
  From<ForecastOptions> for ForecastOptionsExog
- Add AutoEnsemble dispatch arm in forecast() and forecast_with_model()
- Add AutoEnsemble to both CI-skip guards and calculate_fitted_values
  guard (point-forecast-only in Phase 4; intervals deferred to EPI-01)
- Add parse_combination_method() helper (all 6 methods; custom rejected)
- Add forecast_auto_ensemble() helper mirroring forecast_kalman() pattern
- Kalman tests still pass (no regression)
… regen header

- Append ensemble_top_k: c_int and ensemble_method: [c_char; 32] at END of
  ForecastOptions (FFI) and ForecastOptionsExog (FFI) in types.rs (ABI-safe;
  never middle-insert — callers zero-init via memset so existing GARCH/Kalman
  behavior is unchanged)
- Extend Default impls for both structs with ensemble_top_k: 0, ensemble_method: [0; 32]
- Thread ensemble_top_k and ensemble_method through all three core-option
  construction sites in lib.rs:
    - anofox_ts_forecast (~line 3452): parse ensemble_method from CStr
    - anofox_ts_forecast_with_exog (~line 3743): same pattern for exog path
    - build_core_options (~line 4152): same pattern for aggregation path
- Regenerate anofox_fcst_ffi.h via `make header` (cbindgen); both structs
  now expose ensemble_top_k and ensemble_method to C++ callers
…eck example

ts_forecast_scalar.cpp — 6 additive sites mirroring kalman_model pattern:
  - TsForecastScalarBindData: add ensemble_top_k (int64_t) and ensemble_method (string)
  - Copy(): forward both new fields
  - ValidateParams valid_keys: add "top_k", "combination_method"
  - Error message: append top_k, combination_method to valid-params list
  - Local var decls in TsForecastScalarExecute: read from bind_data
  - MAP/STRUCT params parse: ParseInt64Param("top_k"), ParseStringParam("combination_method")
  - opts building: ensemble_top_k cast + guarded strncpy for ensemble_method

ts_forecast_native.cpp — 4 additive sites (same field names):
  - TsForecastNativeBindData: add ensemble_top_k and ensemble_method
  - ValidateParamKeys valid_keys + error message: same additions
  - Bind params parse: ParseInt64FromParams, ParseStringFromParams
  - opts building in Finalize: same guarded strncpy pattern

examples/forecasting/autoensemble.sql — new runnable example:
  - Synthetic 60-obs linear series (all three Auto* members converge reliably)
  - Step 1: AutoEnsemble(mean, top_k=3) → ae_mean
  - Steps 2–3: independent AutoARIMA/AutoETS/AutoTheta → manual arithmetic mean
  - Step 4: cross-check — every row shows abs(ensemble_yhat - manual_mean) < 1e-6
  - Assertion query fails loudly if any row mismatches (DoD, PR#230 rule)
  - Section 3: six-method smoke test (all six combination_method values return
    finite non-NULL yhat)

Build: extension rebuilt via ninja; example ran against built extension —
cross-check all 5 steps pass (diff=0.0, match=true); yhat_lower/upper=NULL.
…y to autoensemble.sql

- Section 3 upgraded from display-only to UNION-ALL assertion: all six
  combination_method strings (mean, median, weighted_mse, inverse_aic,
  stacking, horizon_adaptive) tagged with ok=isfinite(yhat) AND IS NOT NULL;
  failure query returns 0 rows on success (COMB-01..04)
- Section 4 added: skewed series (exp growth + spikes at obs 10/30/50)
  runs Mean and Median combinations separately; comparison shows delta
  1.45-2.69 per step; assertion confirms at least one step has delta > 1e-6
  (COMB-01 demonstrability)
- Verified against built extension: all assertions return 0-row failure sets
Add docs/reference/models/ensemble/autoensemble.md (method-string dispatch,
top_k/combination_method/seasonal_period params, six combination methods,
Mean cross-check, NULL-interval note, fewer-than-top_k behavior) and an
AutoEnsemble section in docs/api/07-forecasting.md. Every SQL snippet
verified against the built extension (PR #230).

Reconciled by orchestrator: the executor reported a commit (a37f8c9) and
SUMMARY that were never actually written; the on-disk work was verified
correct (example + doc snippets run clean against the built extension) and
is committed here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
… mis-report)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
Mirror the .max(0) pattern already used for the window field at all three
ensemble_top_k cast sites. Negative c_int values like -1 previously wrapped
to usize::MAX; they now clamp to 0, which the core's top_k==0 guard
converts to the default of 3.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
forecast_with_exog's else-branch routed AutoEnsemble through
forecast_with_model, which hard-codes top_k=3 and method=None.
Dispatch AutoEnsemble separately in that branch, reading the caller's
ensemble_top_k and ensemble_method from ForecastOptionsExog.
All other models continue to go through forecast_with_model unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
…utoensemble.md (IN-01)

Add a note to the Rust doc comment on parse_combination_method that
'stacking' maps to CombinationMethod::Stacking { folds: 2 } with a
fixed 2-fold second-half in-sample holdout not user-configurable in v1.
Mirror the same note in the combination_method SQL parameter table.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
…e (IN-02)

Adds five inline tests in forecast::tests:
- parse_combination_method_rejects_unknown: 'custom' and unknown strings -> Err
- parse_combination_method_accepts_canonical_strings: all six methods -> Ok
- parse_combination_method_accepts_aliases: representative aliases -> Ok
- parse_combination_method_none_and_empty_give_mean: None/''/'' -> Ok
- forecast_auto_ensemble_basic: 60-pt series, top_k=0, horizon=5 -> Ok

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
…semble to core

- build_forecaster: compiler-exhaustive match over all 36 ModelType variants;
  10 blocked variants (GARCH, Laplace, ARIMA, MFLES, AutoMFLES, MSTL, AutoMSTL,
  TBATS, AutoTBATS, AutoEnsemble) return Err(InvalidParameter) naming the member
- forecast_explicit_ensemble: builds member Vec, calls Ensemble::new(members)
  .with_method(combination_method); reuses parse_combination_method from Phase 4;
  returns model_name = 'Ensemble' (point-only, intervals Phase 6 EPI-01)
- Ensemble implements Forecaster (model.rs:575); extract_forecast works directly
…header

- anofox_ts_forecast_ensemble: null-delimited member buffer + explicit
  members_buf_len (no over-read); catch_unwind panic safety; InvalidParameter
  → ErrorCode::InvalidInput path (same as forecast); NULL lower/upper bounds
  (point-only, EPI-01 deferred)
- Export forecast_explicit_ensemble as pub from anofox-fcst-core lib.rs
- make header: anofox_ts_forecast_ensemble + members_buf_len in generated
  anofox_fcst_ffi.h (verified: 1 fn + 2 param occurrences)
…fied tracer

- _ts_forecast_ensemble_native: ScalarFunction (per _ts_forecast_scalar precedent);
  takes LIST(VARCHAR) members, builds null-delimited buffer, calls
  anofox_ts_forecast_ensemble; yhat_lower/upper emit NULL (EPI-01 deferred)
- CMakeLists.txt: src/table_functions/ts_forecast_ensemble_native.cpp added to
  EXTENSION_SOURCES after ts_forecast_var_native.cpp (Phase 5: ENS-02)
- extension.cpp: #include ts_forecast_ensemble_native.hpp + Register* call
- ts_macros.cpp: ts_forecast_ensemble_by macro (per-series, GROUP BY, unnest shape)
- ensemble_explicit_tracer.sql: mismatch_count=0, non_null_intervals=0,
  model_name='Ensemble'; diff=0.0 on all 5 steps (exact match, no 1e-6 tolerance
  needed)
sipemu and others added 28 commits August 31, 2026 20:41
Archive ROADMAP/REQUIREMENTS/AUDIT + phase dirs to milestones/v0.8.0-*;
reorganize ROADMAP.md (v0.8.0 shipped, collapsed); evolve PROJECT.md
(5 requirements → Validated, v0.8.0 decisions, current state); append
v0.8.0 retrospective. MILESTONES.md + STATE.md updated by milestone.complete.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
…ate)

The ensemble build_forecaster/forecast_explicit_ensemble (Phase 5) and the
introspection FFI exports (Phase 6) were not rustfmt-clean; CI's Rust Tests
job gates on cargo fmt --check. Formatting-only — cargo test still 240+12
green, no logic change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
…g safety doc)

The Rust Tests CI job runs fmt -> clippy -D warnings -> test. Two clippy lints
in the Phase 5 ensemble code: field_reassign_with_default in build_forecaster
(AutoARIMAConfig) and missing_safety_doc on the anofox_ts_forecast_ensemble
FFI export. Both fixed; fmt + clippy + 240/12 tests green locally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
- Add test/wasm/run.mjs: main harness (eh bundle, pthreadWorker=null,
  web-worker@1.2.0, version-agnostic localhost server, per-file db.open()
  catalog isolation, FORCE INSTALL + LOAD)
- Add test/wasm/sqllogic.mjs: minimal sqllogictest-subset parser/runner
  (statement ok/error, query, mode skip; SELECT COLUMNS(*)::VARCHAR
  DECIMAL fix; float tolerance; bool normalization)
- Add test/wasm/package.json: exact pins @duckdb/duckdb-wasm@1.33.1-dev64.0
  (engine v1.5.5) and web-worker@1.2.0
- Add test/wasm/package-lock.json: committed lockfile for reproducible installs
- Update .gitignore: add test/wasm/node_modules/
- Tracer: ts_forecast_by.test passes 90/90 assertions in DuckDB-Wasm

Note: pre-built local artifact (July 10) was stale and missing the
duckdb_signature metadata section; replaced with CI artifact from run
#33554081155 (build/wasm_eh). Emscripten not available locally.
Convert bare 'openssl' dependency string to object with platform: '!wasm32'
so Emscripten does not compile/link OpenSSL for the WASM target. Native builds
continue to statically link OpenSSL (issues #211/#215 requirements unchanged).

WASM rebuild verification: Emscripten not available locally; vcpkg guard
verified by node check (node -e 'vcpkg.json parser'); harness re-run against
CI artifact (run #33554081155) still green (90 passed, 0 failed).
Full rebuild + OpenSSL-drop confirmation deferred to CI.
- SKIP_FILES: 4 structurally WASM-infeasible files (heap overflow on
  TSFresh, UNNEST engine limit, Emscripten ___trap); each entry carries
  a one-line reason that is logged at run time
- CURATED: updated to the 8 files verified green against the current
  WASM artifact (ts_forecast_by, ts_metrics, ts_decomposition,
  ts_diagnostics, ts_conformal, ts_cv_folds, ts_aggregate_hierarchy,
  ts_periods) — 396/396 assertions pass
- --all baseline: 2255 passed, 188 failed (23 files); 39 of 66 files
  fully green; 23 failing files track artifact-API drift and pre-existing
  test bugs, documented in SUMMARY.md
- Purpose, requirements, quick-start, running modes table
- DEP-01: strings-based procedure to read embedded DuckDB version from
  the duckdb_signature WASM custom section; version-mapping table; bump
  checklist
- Full-suite status table (66 files, 39 pass, 23 fail, 4 skipped)
- Skip-list rationale and known-failure categories for the 23 failing
  files (artifact-API drift vs. pre-existing test bugs)
- Architecture notes: pthreadWorker=null, COLUMNS(*)::VARCHAR wrap,
  per-file db.open() catalog isolation
…ifact drift); DEP-02 build-confirmed

Local HEAD wasm_eh build (emsdk + wasm32-unknown-emscripten + binaryen 123) reproduced
the same 23 failing files, disproving the stale-artifact theory. Failures are pre-existing
test-suite debt (removed/renamed API refs, DATE+BIGINT, cascades) masked natively by the
unittest require-json skip. DEP-02 openssl !wasm32 guard confirmed at build time (zero
openssl compiled for wasm). Phase 7 harness accepted; full-suite green deferred as
out-of-scope test triage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LkRFReMJRg33mDS4zo55Tt
…ase 8 framing

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LkRFReMJRg33mDS4zo55Tt
- Ports the dedicated WASM workflow from anofox-statistics PR #131
- Triggers via workflow_run on "Main Extension Distribution Pipeline" (completed)
- Downloads anofox_forecast-v1.5.5-extension-wasm_eh artifact via cross-run
  github-token download (run-id: github.event.workflow_run.id)
- Runs curated harness: node test/wasm/run.mjs --ext "$EXT" (no --all)
- Permissions: actions read + contents read (read-only, T-08-01 mitigated)
- Powers the README WASM badge; badge file name WasmTest.yml matches exactly
…line.yml (CI-01)

- New job wasm-runtime-test needs duckdb-latest-build; if-guarded on success
- Downloads anofox_forecast-v1.5.5-extension-wasm_eh (same-run, no run-id/github-token)
- Runs curated harness: node test/wasm/run.mjs --ext "$EXT" (no --all, no --file)
- No continue-on-error: harness exit non-zero -> step fails -> pipeline turns red
- checkout step omits submodules (harness only reads test/ + downloaded .wasm)
- No existing job modified or removed (duckdb-lts-build/deploy, latest-build/deploy,
  smoke-tests, build-and-test-rust all unchanged)
- Inserts WASM badge into the existing header badge block (p align=center)
- Badge href: https://github.com/DataZooDE/anofox-forecast/actions/workflows/WasmTest.yml
- Badge img src: .../WasmTest.yml/badge.svg?branch=main (branch=main keeps dev branches
  from flipping the visible badge; file name WasmTest.yml matches Task 1 exactly)
- Existing License, DuckDB, Build, Tests badges unchanged
…en locally (green→red→green); live-CI push remains (human)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LkRFReMJRg33mDS4zo55Tt
…ferred_human)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LkRFReMJRg33mDS4zo55Tt
…line)

The v1.4-andium ci-tools vcpkg baseline pins an OpenSSL dependency whose upstream
download URL now 404s ('Download failed, halting portfile'), failing the LTS
windows_amd64 build + deploy. Not fixable here (vcpkg comes from extension-ci-tools);
same class of issue anofox-statistics documented and excluded. Scoped to the LTS line
only — the v1.5.5 (variegata) windows_amd64 build uses a newer, working baseline and
still ships, honoring the cross-platform Windows requirement on current DuckDB.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LkRFReMJRg33mDS4zo55Tt
@sipemu sipemu changed the title Milestone v0.8.0 — Ensemble Forecasting (AutoEnsemble + explicit-member ensembles, intervals, introspection) Milestones v0.8.0 + v0.9.0 — Ensemble Forecasting + WASM Runtime Verification (Phases 4–8) Sep 2, 2026
@sipemu
sipemu merged commit a9328e1 into main Sep 2, 2026
91 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant