Milestones v0.8.0 + v0.9.0 — Ensemble Forecasting + WASM Runtime Verification (Phases 4–8) - #256
Merged
Merged
Conversation
- Add AutoEnsemble enum variant to ModelType (with doc comment)
- Add from_str exact-match ("AutoEnsemble") and lowercase fallback
("autoensemble" | "auto_ensemble") arms
- Add name() arm returning "AutoEnsemble"
- Add ensemble_top_k: usize and ensemble_method: Option<String> to
ForecastOptions and ForecastOptionsExog (additive, after kalman_model)
- Thread new fields through ForecastOptionsExog::Default and
From<ForecastOptions> for ForecastOptionsExog
- Add AutoEnsemble dispatch arm in forecast() and forecast_with_model()
- Add AutoEnsemble to both CI-skip guards and calculate_fitted_values
guard (point-forecast-only in Phase 4; intervals deferred to EPI-01)
- Add parse_combination_method() helper (all 6 methods; custom rejected)
- Add forecast_auto_ensemble() helper mirroring forecast_kalman() pattern
- Kalman tests still pass (no regression)
… regen header
- Append ensemble_top_k: c_int and ensemble_method: [c_char; 32] at END of
ForecastOptions (FFI) and ForecastOptionsExog (FFI) in types.rs (ABI-safe;
never middle-insert — callers zero-init via memset so existing GARCH/Kalman
behavior is unchanged)
- Extend Default impls for both structs with ensemble_top_k: 0, ensemble_method: [0; 32]
- Thread ensemble_top_k and ensemble_method through all three core-option
construction sites in lib.rs:
- anofox_ts_forecast (~line 3452): parse ensemble_method from CStr
- anofox_ts_forecast_with_exog (~line 3743): same pattern for exog path
- build_core_options (~line 4152): same pattern for aggregation path
- Regenerate anofox_fcst_ffi.h via `make header` (cbindgen); both structs
now expose ensemble_top_k and ensemble_method to C++ callers
…eck example
ts_forecast_scalar.cpp — 6 additive sites mirroring kalman_model pattern:
- TsForecastScalarBindData: add ensemble_top_k (int64_t) and ensemble_method (string)
- Copy(): forward both new fields
- ValidateParams valid_keys: add "top_k", "combination_method"
- Error message: append top_k, combination_method to valid-params list
- Local var decls in TsForecastScalarExecute: read from bind_data
- MAP/STRUCT params parse: ParseInt64Param("top_k"), ParseStringParam("combination_method")
- opts building: ensemble_top_k cast + guarded strncpy for ensemble_method
ts_forecast_native.cpp — 4 additive sites (same field names):
- TsForecastNativeBindData: add ensemble_top_k and ensemble_method
- ValidateParamKeys valid_keys + error message: same additions
- Bind params parse: ParseInt64FromParams, ParseStringFromParams
- opts building in Finalize: same guarded strncpy pattern
examples/forecasting/autoensemble.sql — new runnable example:
- Synthetic 60-obs linear series (all three Auto* members converge reliably)
- Step 1: AutoEnsemble(mean, top_k=3) → ae_mean
- Steps 2–3: independent AutoARIMA/AutoETS/AutoTheta → manual arithmetic mean
- Step 4: cross-check — every row shows abs(ensemble_yhat - manual_mean) < 1e-6
- Assertion query fails loudly if any row mismatches (DoD, PR#230 rule)
- Section 3: six-method smoke test (all six combination_method values return
finite non-NULL yhat)
Build: extension rebuilt via ninja; example ran against built extension —
cross-check all 5 steps pass (diff=0.0, match=true); yhat_lower/upper=NULL.
…y to autoensemble.sql - Section 3 upgraded from display-only to UNION-ALL assertion: all six combination_method strings (mean, median, weighted_mse, inverse_aic, stacking, horizon_adaptive) tagged with ok=isfinite(yhat) AND IS NOT NULL; failure query returns 0 rows on success (COMB-01..04) - Section 4 added: skewed series (exp growth + spikes at obs 10/30/50) runs Mean and Median combinations separately; comparison shows delta 1.45-2.69 per step; assertion confirms at least one step has delta > 1e-6 (COMB-01 demonstrability) - Verified against built extension: all assertions return 0-row failure sets
Add docs/reference/models/ensemble/autoensemble.md (method-string dispatch, top_k/combination_method/seasonal_period params, six combination methods, Mean cross-check, NULL-interval note, fewer-than-top_k behavior) and an AutoEnsemble section in docs/api/07-forecasting.md. Every SQL snippet verified against the built extension (PR #230). Reconciled by orchestrator: the executor reported a commit (a37f8c9) and SUMMARY that were never actually written; the on-disk work was verified correct (example + doc snippets run clean against the built extension) and is committed here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
… mis-report) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
Mirror the .max(0) pattern already used for the window field at all three ensemble_top_k cast sites. Negative c_int values like -1 previously wrapped to usize::MAX; they now clamp to 0, which the core's top_k==0 guard converts to the default of 3. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
forecast_with_exog's else-branch routed AutoEnsemble through forecast_with_model, which hard-codes top_k=3 and method=None. Dispatch AutoEnsemble separately in that branch, reading the caller's ensemble_top_k and ensemble_method from ForecastOptionsExog. All other models continue to go through forecast_with_model unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
…utoensemble.md (IN-01)
Add a note to the Rust doc comment on parse_combination_method that
'stacking' maps to CombinationMethod::Stacking { folds: 2 } with a
fixed 2-fold second-half in-sample holdout not user-configurable in v1.
Mirror the same note in the combination_method SQL parameter table.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
…e (IN-02) Adds five inline tests in forecast::tests: - parse_combination_method_rejects_unknown: 'custom' and unknown strings -> Err - parse_combination_method_accepts_canonical_strings: all six methods -> Ok - parse_combination_method_accepts_aliases: representative aliases -> Ok - parse_combination_method_none_and_empty_give_mean: None/''/'' -> Ok - forecast_auto_ensemble_basic: 60-pt series, top_k=0, horizon=5 -> Ok Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
…semble to core - build_forecaster: compiler-exhaustive match over all 36 ModelType variants; 10 blocked variants (GARCH, Laplace, ARIMA, MFLES, AutoMFLES, MSTL, AutoMSTL, TBATS, AutoTBATS, AutoEnsemble) return Err(InvalidParameter) naming the member - forecast_explicit_ensemble: builds member Vec, calls Ensemble::new(members) .with_method(combination_method); reuses parse_combination_method from Phase 4; returns model_name = 'Ensemble' (point-only, intervals Phase 6 EPI-01) - Ensemble implements Forecaster (model.rs:575); extract_forecast works directly
…header - anofox_ts_forecast_ensemble: null-delimited member buffer + explicit members_buf_len (no over-read); catch_unwind panic safety; InvalidParameter → ErrorCode::InvalidInput path (same as forecast); NULL lower/upper bounds (point-only, EPI-01 deferred) - Export forecast_explicit_ensemble as pub from anofox-fcst-core lib.rs - make header: anofox_ts_forecast_ensemble + members_buf_len in generated anofox_fcst_ffi.h (verified: 1 fn + 2 param occurrences)
…fied tracer - _ts_forecast_ensemble_native: ScalarFunction (per _ts_forecast_scalar precedent); takes LIST(VARCHAR) members, builds null-delimited buffer, calls anofox_ts_forecast_ensemble; yhat_lower/upper emit NULL (EPI-01 deferred) - CMakeLists.txt: src/table_functions/ts_forecast_ensemble_native.cpp added to EXTENSION_SOURCES after ts_forecast_var_native.cpp (Phase 5: ENS-02) - extension.cpp: #include ts_forecast_ensemble_native.hpp + Register* call - ts_macros.cpp: ts_forecast_ensemble_by macro (per-series, GROUP BY, unnest shape) - ensemble_explicit_tracer.sql: mismatch_count=0, non_null_intervals=0, model_name='Ensemble'; diff=0.0 on all 5 steps (exact match, no 1e-6 tolerance needed)
Archive ROADMAP/REQUIREMENTS/AUDIT + phase dirs to milestones/v0.8.0-*; reorganize ROADMAP.md (v0.8.0 shipped, collapsed); evolve PROJECT.md (5 requirements → Validated, v0.8.0 decisions, current state); append v0.8.0 retrospective. MILESTONES.md + STATE.md updated by milestone.complete. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
…ate) The ensemble build_forecaster/forecast_explicit_ensemble (Phase 5) and the introspection FFI exports (Phase 6) were not rustfmt-clean; CI's Rust Tests job gates on cargo fmt --check. Formatting-only — cargo test still 240+12 green, no logic change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
…g safety doc) The Rust Tests CI job runs fmt -> clippy -D warnings -> test. Two clippy lints in the Phase 5 ensemble code: field_reassign_with_default in build_forecaster (AutoARIMAConfig) and missing_safety_doc on the anofox_ts_forecast_ensemble FFI export. Both fixed; fmt + clippy + 240/12 tests green locally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KS9bB2RQf1JFLM8oRoVV8a
- Add test/wasm/run.mjs: main harness (eh bundle, pthreadWorker=null, web-worker@1.2.0, version-agnostic localhost server, per-file db.open() catalog isolation, FORCE INSTALL + LOAD) - Add test/wasm/sqllogic.mjs: minimal sqllogictest-subset parser/runner (statement ok/error, query, mode skip; SELECT COLUMNS(*)::VARCHAR DECIMAL fix; float tolerance; bool normalization) - Add test/wasm/package.json: exact pins @duckdb/duckdb-wasm@1.33.1-dev64.0 (engine v1.5.5) and web-worker@1.2.0 - Add test/wasm/package-lock.json: committed lockfile for reproducible installs - Update .gitignore: add test/wasm/node_modules/ - Tracer: ts_forecast_by.test passes 90/90 assertions in DuckDB-Wasm Note: pre-built local artifact (July 10) was stale and missing the duckdb_signature metadata section; replaced with CI artifact from run #33554081155 (build/wasm_eh). Emscripten not available locally.
Convert bare 'openssl' dependency string to object with platform: '!wasm32' so Emscripten does not compile/link OpenSSL for the WASM target. Native builds continue to statically link OpenSSL (issues #211/#215 requirements unchanged). WASM rebuild verification: Emscripten not available locally; vcpkg guard verified by node check (node -e 'vcpkg.json parser'); harness re-run against CI artifact (run #33554081155) still green (90 passed, 0 failed). Full rebuild + OpenSSL-drop confirmation deferred to CI.
- SKIP_FILES: 4 structurally WASM-infeasible files (heap overflow on TSFresh, UNNEST engine limit, Emscripten ___trap); each entry carries a one-line reason that is logged at run time - CURATED: updated to the 8 files verified green against the current WASM artifact (ts_forecast_by, ts_metrics, ts_decomposition, ts_diagnostics, ts_conformal, ts_cv_folds, ts_aggregate_hierarchy, ts_periods) — 396/396 assertions pass - --all baseline: 2255 passed, 188 failed (23 files); 39 of 66 files fully green; 23 failing files track artifact-API drift and pre-existing test bugs, documented in SUMMARY.md
- Purpose, requirements, quick-start, running modes table - DEP-01: strings-based procedure to read embedded DuckDB version from the duckdb_signature WASM custom section; version-mapping table; bump checklist - Full-suite status table (66 files, 39 pass, 23 fail, 4 skipped) - Skip-list rationale and known-failure categories for the 23 failing files (artifact-API drift vs. pre-existing test bugs) - Architecture notes: pthreadWorker=null, COLUMNS(*)::VARCHAR wrap, per-file db.open() catalog isolation
…ifact drift); DEP-02 build-confirmed Local HEAD wasm_eh build (emsdk + wasm32-unknown-emscripten + binaryen 123) reproduced the same 23 failing files, disproving the stale-artifact theory. Failures are pre-existing test-suite debt (removed/renamed API refs, DATE+BIGINT, cascades) masked natively by the unittest require-json skip. DEP-02 openssl !wasm32 guard confirmed at build time (zero openssl compiled for wasm). Phase 7 harness accepted; full-suite green deferred as out-of-scope test triage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LkRFReMJRg33mDS4zo55Tt
…ase 8 framing Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LkRFReMJRg33mDS4zo55Tt
- Ports the dedicated WASM workflow from anofox-statistics PR #131 - Triggers via workflow_run on "Main Extension Distribution Pipeline" (completed) - Downloads anofox_forecast-v1.5.5-extension-wasm_eh artifact via cross-run github-token download (run-id: github.event.workflow_run.id) - Runs curated harness: node test/wasm/run.mjs --ext "$EXT" (no --all) - Permissions: actions read + contents read (read-only, T-08-01 mitigated) - Powers the README WASM badge; badge file name WasmTest.yml matches exactly
…line.yml (CI-01) - New job wasm-runtime-test needs duckdb-latest-build; if-guarded on success - Downloads anofox_forecast-v1.5.5-extension-wasm_eh (same-run, no run-id/github-token) - Runs curated harness: node test/wasm/run.mjs --ext "$EXT" (no --all, no --file) - No continue-on-error: harness exit non-zero -> step fails -> pipeline turns red - checkout step omits submodules (harness only reads test/ + downloaded .wasm) - No existing job modified or removed (duckdb-lts-build/deploy, latest-build/deploy, smoke-tests, build-and-test-rust all unchanged)
- Inserts WASM badge into the existing header badge block (p align=center) - Badge href: https://github.com/DataZooDE/anofox-forecast/actions/workflows/WasmTest.yml - Badge img src: .../WasmTest.yml/badge.svg?branch=main (branch=main keeps dev branches from flipping the visible badge; file name WasmTest.yml matches Task 1 exactly) - Existing License, DuckDB, Build, Tests badges unchanged
…en locally (green→red→green); live-CI push remains (human) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LkRFReMJRg33mDS4zo55Tt
…ferred_human) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LkRFReMJRg33mDS4zo55Tt
…line)
The v1.4-andium ci-tools vcpkg baseline pins an OpenSSL dependency whose upstream
download URL now 404s ('Download failed, halting portfile'), failing the LTS
windows_amd64 build + deploy. Not fixable here (vcpkg comes from extension-ci-tools);
same class of issue anofox-statistics documented and excluded. Scoped to the LTS line
only — the v1.5.5 (variegata) windows_amd64 build uses a newer, working baseline and
still ships, honoring the cross-platform Windows requirement on current DuckDB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LkRFReMJRg33mDS4zo55Tt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestones v0.8.0 + v0.9.0 — Ensemble Forecasting + WASM Runtime Verification (Phases 4–8)
maincurrently has neither milestone, so this PR ships both tomainin one merge (86 commits): the v0.8.0 ensemble surface (Phases 4–6) and the v0.9.0 WASM runtime verification + CI gating (Phases 7–8). The v0.9.0 summary is below; the original v0.8.0 summary follows unchanged.Milestone v0.9.0 — WASM Runtime Verification
Proves the built
anofox_forecast.wasmactually loads and runs in DuckDB-Wasm — not just that it compiles and links — and gates that in CI so WASM regressions fail the build. No new SQL functions or crate changes; work lives intest/wasm/,.github/workflows/,vcpkg.json, andREADME.md. Ports the CI/harness pattern from anofox-statistics PR #131.Phase 7 — WASM Node harness + local green ✅ verified
test/wasm/run.mjs+sqllogic.mjs— boots DuckDB-Wasm (ehbundle,pthreadWorker=null), serves the built.wasmover localhost,FORCE INSTALL+LOADs it, and runs a sqllogictest subset with per-filedb.open()catalog isolation and aCOLUMNS(*)::VARCHARwrap so DECIMAL rendering matches native (WASM-01, WASM-02).@duckdb/duckdb-wasm@1.33.1-dev64.0(engine v1.5.5) +web-worker@1.2.0, with a documentedstrings | grepversion-verification procedure intest/wasm/README.md.vcpkg.jsondeclaresopensslas{ "platform": "!wasm32" }. Build-confirmed: a fresh localwasm_ehbuild compiled zero OpenSSL for the wasm target (mbedtls is used instead); native builds still statically link it..wasm.Phase 8 — CI gating + dedicated workflow + badge ◆ implemented
wasm-runtime-testgating job inMainDistributionPipeline.yml:needs:the wasm build, downloads theanofox_forecast-v1.5.5-extension-wasm_ehartifact, runs the curated harness, fails the build on any WASM load/runtime error..github/workflows/WasmTest.yml(workflow_run-triggered) so WASM status is independently observable.WasmTest.yml.Honest scope notes
--allrun is red (184 failures / 23 files), but these were confirmed against a fresh HEAD build to be pre-existingtest/sqltest-suite debt — stale references to removed/renamed API (ts_backtest_auto_by,ts_hydrate_features_by, …),DATE + BIGINTbugs, and cascade failures — not WASM or artifact-drift issues. They're masked natively because theunittestrunner skips these files on an unsatisfiedrequire json; DuckDB-Wasm auto-loads json and exposes them. CI gates on the curated subset, not--all. Making the full suite green is tracked as separate test-triage.windows_amd64on the v1.4.5 LTS line only — thev1.4-andiumci-tools vcpkg baseline pins an OpenSSL dependency whose upstream download now 404s (not fixable here; same class anofox-statistics documented). The v1.5.5windows_amd64build uses a newer baseline, passes, and still ships.Tag:
v0.9.0. Includes the full.planning/audit trail (repo convention).Milestone v0.8.0 — Ensemble Forecasting
Exposes the
anofox-forecastcrate's ensemble surface to SQL: combine multiple models per series (automatically or by explicit member list), with six combination methods, distribution-free conformal prediction intervals, and member/weight introspection — all via the established FFI → C++ → macro → example → docs pattern.8/8 requirements delivered and verified against the built extension · 3 phases · 6 plans · ~13.9k LOC / 57 files.
New SQL surface
ts_forecast_by(..., 'AutoEnsemble', ..., {top_k, combination_method, seasonal_period})combination_methods:mean,median,weighted_mse,inverse_aic,stacking,horizon_adaptivets_forecast_ensemble_by('table', grp, ds, y, members VARCHAR[], ...)build_forecasterfactory: 26-member allowlist, 10 blocked with clear errors)ts_cv_folds_by+ts_conformal_calibrate/applyts_ensemble_inspect_by/ts_auto_ensemble_inspect_byVerification
diff=0.0, all six methods produce finite forecasts, all error paths raise clear member-naming errors, conformallower ≤ point ≤ upperper step, all INSP-01 DoD assertions 0 failures.top_kclamp, exog param path, NULL-interpolation/min-length guards, SMA window) + info items.parse_combination_method+build_forecasterback all three surfaces consistently; no regression on pre-milestone models.Known tech debt (documented in
milestones/v0.8.0-MILESTONE-AUDIT.md)ts_cv_forecast_by('AutoEnsemble')segfaults — pre-existing crate/CV-native bug (ts_cv_forecast_native.cpp:380-388never parses the ensemble params). EPI-01 ships via a documented manual per-fold_ts_forecast_scalarworkaround. Recommended follow-up: wire ensemble-param parsing into the CV native.build_forecasterSeasonalWindowAveragen_seasons=2hardcoded (TODO ENS-03).Tag:
v0.8.0. Includes the full.planning/audit trail (repo convention).🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.