Skip to content

investigate: continuous AR predict warmup (cold vs warm u0) #476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Saswatsusmoy wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

investigate: continuous AR predict warmup (cold vs warm u0) #476

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3e776cf
investigate: continuous AR predict warmup harness
Saswatsusmoy Jul 17, 2026
04c9f44
investigate: full-matrix warmup analysis at #456 Lorenz scale
Saswatsusmoy Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions benchmarks/continuous_warmup/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Manifest.toml
# Latest-run convenience copies (tagged files are committed when useful)
results/summary.json
results/summary.md
# Keep FINDINGS.md + summary_full.* for the investigation PR
*.png
*.jld2
21 changes: 21 additions & 0 deletions benchmarks/continuous_warmup/Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Investigation environment only — not a Julia package.
# Continuous autoregressive predict warmup (cold vs warm u0).

[deps]
DataInterpolations = "82cc6244-b520-54b8-b5a6-8a565e85f1d0"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"
OrdinaryDiffEqTsit5 = "b1df2697-797e-41e3-8120-5422d3b24e4a"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
ReservoirComputing = "7c2d2b1e-3dd4-11ea-355a-8f6a8116e294"
SciMLBase = "0bca4576-84f4-4d90-8ffe-ffa030f20462"
Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"

[compat]
DataInterpolations = "6, 7, 8, 9"
JSON = "0.21, 1"
OrdinaryDiffEq = "6"
OrdinaryDiffEqTsit5 = "1"
SciMLBase = "2, 3"
julia = "1.10"
89 changes: 89 additions & 0 deletions benchmarks/continuous_warmup/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Continuous AR predict warmup investigation

**Branch:** `investigate/continuous-predict-warmup`
**Context:** Deferred from [#456](https://github.com/SciML/ReservoirComputing.jl/pull/456)
(Francesco: *“add the warmup as part of a separate PR… investigate that a bit further”*).

This harness **does not modify package `src/` or root `Project.toml`**.
It only measures cold vs warm reservoir state at the start of continuous
autoregressive `predict`, and probes candidate API shapes **locally**.

## Problem

| Path | Initial reservoir state for AR `predict` |
|------|------------------------------------------|
| Discrete `ESN` | Carries through `StatefulLayer` / sequential `apply` |
| `SciMLProblemReservoir` | `res.prob.u0` (often zeros) |
| `ContinuousESN` / `ContinuousESNCell` | Always `zeros(out_dims)` |

On Lorenz, PR #456 measured roughly **NRMSE ~1.5 (cold)** vs **~0.11 (warmed)**.

## Experiments

| ID | Question |
|----|----------|
| E1 | Reproduce cold vs terminal-train warm on `ContinuousESN` (Lorenz) |
| E2 | Same for hand-rolled `SciMLProblemReservoir` (eq. 5) |
| E3 | Warmup **length** sweep (last `K` teacher-forced inputs before AR) |
| E4 | Seed variants: zeros / random / train terminal / test-prefix warm / wrong state |
| E5 | Horizon curve: NRMSE vs Lyapunov time (cold vs warm) |
| E6 | Does `st` after `train!` / `collectstates` hold usable continuous state? |
| E7 | Discrete `ESN` control (same data split) |
| E8 | Washout interaction (train with washout, warm from post-washout tail) |

## Setup

```bash
cd benchmarks/continuous_warmup
julia --project=. -e 'using Pkg; Pkg.develop(path="../.."); Pkg.instantiate()'
```

## Run

Smoke (plumbing only — small N, short series, ~20s):

```bash
julia --project=. run.jl --smoke
```

**Full matrix** (forecast analysis — matches #456 Lorenz scale):

| knob | full default |
|------|----------------|
| `n_res` | 300 |
| train / predict | 5000 / 1250 |
| Wr spectral radius | 0.9 |
| Win scale / bias scale | 0.1 / 0.05 |
| solver | Tsit5 (`reltol=1e-6`) |

```bash
julia --project=. run.jl # full
julia --project=. run.jl --n-res=300
julia --project=. run.jl --only=E1,E2,E5,E7
```

Expect minutes–tens of minutes for continuous `collectstates` at N=300 / T=5000 (wall times are recorded per experiment).

Outputs land in `results/`:

- `results/summary_full.json` / `summary_smoke.json` — machine-readable
- `results/summary_full.md` / `summary_smoke.md` — tables
- `results/summary.json` — latest run copy
- `results/FINDINGS.md` — interpretation

## Design options under test (no API change yet)

Local experimental helpers in `src/predict_variants.jl`:

1. **`initial_state`** — pass reservoir `u0` into a mirrored AR loop
2. **`warmup_data`** — run teacher-forced `collectstates` on a prefix, take last column as `u0`, then AR
3. **`remake(prob; u0=…)`** — only for `SciMLProblemReservoir` public path
4. **Read `st` after train** — check whether continuous `st` already carries terminal state (expected: **no**)

Package `predict` is left untouched so results stay comparable to merged `master`.

## Relation to other work

- Not PR4 perf ([#467](https://github.com/SciML/ReservoirComputing.jl/issues/467))
- Not a user-facing API PR until API choice is agreed
- Feeds a future tracking issue + design PR
176 changes: 176 additions & 0 deletions benchmarks/continuous_warmup/results/FINDINGS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
# Findings — continuous AR predict warmup

Branch: `investigate/continuous-predict-warmup`
PR: https://github.com/SciML/ReservoirComputing.jl/pull/476
Related: #456 (merged), #397

## Reproduce

```bash
cd benchmarks/continuous_warmup
julia --project=. -e 'using Pkg; Pkg.develop(path="../.."); Pkg.instantiate()'

# plumbing only (~20s)
julia --project=. run.jl --smoke

# forecast analysis — #456-scale (~5 min on M-series laptop)
julia --project=. run.jl
# → results/summary_full.{json,md}
```

### Full-run config

| knob | value |
|------|-------|
| mode | **full** (not smoke) |
| n_res | 300 |
| train / predict | 5000 / 1250 samples |
| dt / λ_max | 0.02 / 0.9056 → predict span ≈ **22.6 t_λ** |
| Wr spectral radius | 0.9 |
| Win scale / bias scale | 0.1 / 0.05 |
| state modifiers | `NLAT2()` |
| solver | Tsit5, `reltol=1e-6`, `abstol=1e-8` |
| ridge | 1e-6 |
| seed | 17 |
| suite wall | **~275 s** (this machine) |

---

## 1. Structural (independent of HPs)

| Check | Result |
|-------|--------|
| Seeded `u0=0` vs package `predict` | **max abs diff = 0** |
| `remake(prob; u0=…)` vs seeded warm | **identical NRMSE** |
| Continuous `st` after `train!` / `collectstates` | **no carry** |
| Discrete `st` after `train!` | **has carry** |
| Discrete rewarm via `collectstates` | ≡ post-train `st` |

**Conclusion:** continuous AR always cold-starts today. There is nowhere in `st` to put a terminal ODE state. Public warm workaround for generic SciML reservoirs: `remake(prob; u0=terminal)`. `ContinuousESN` has no public `u0` hook without a custom AR loop.

---

## 2. Forecast quality — full matrix (the real analysis)

### Headline: short-horizon NRMSE (ContinuousESN, E5 / E1 horizons)

| Horizon | steps | t_λ | cold NRMSE | warm (train-terminal) NRMSE |
|---------|------:|----:|-----------:|----------------------------:|
| short | 28 | 0.5 | 0.75 | **0.21** |
| 1 t_λ | 55 | 1.0 | 0.69 | **0.15** |
| 2 t_λ | 110 | 2.0 | 1.34 | **0.085** |
| 3 t_λ | 166 | 3.0 | 1.51 | **0.093** |
| 4 t_λ | 221 | 4.0 | 1.66 | **0.14** |
| 6 t_λ | 331 | 6.0 | 1.51 | **0.67** |

Warm recovers the #456-class short-horizon gap: at 2–3 Lyapunov times, warm is ~**0.09** vs cold ~**1.3–1.5**.

Full-horizon NRMSE over all 1250 steps (≈22 t_λ) is high for both (cold **1.48**, warm **1.23**) — expected once chaos has diverged; the interesting regime is the first few t_λ.

### Valid prediction time (E3, threshold 0.5)

| Warmup K | VPT (t_λ) | full-horizon NRMSE |
|---------:|----------:|-------------------:|
| 0 (cold) | **0.22** | 1.48 |
| 10 | **4.87** | 1.14 |
| 50–2000 | ~4.13 | ~1.18–1.28 |

Even a short teacher-forced warmup (K=10) lifts VPT from ~0.2 → ~5 t_λ.

### Seed ranking (E4, full-horizon NRMSE)

| seed | NRMSE |
|------|------:|
| train_terminal | **1.23** |
| zeros (cold) | 1.48 |
| oracle test prefix (K=100) | 1.45 |
| randn / shuffled terminal | ~13.5 (catastrophic) |

Wrong non-zero seeds are much worse than cold. Warm is not “any u0” — it has to be a dynamically consistent terminal state.

### SciMLProblemReservoir eq.5 (E2)

| variant | full-horizon NRMSE |
|---------|-------------------:|
| cold | 1.47 |
| warm train-terminal | **1.31** |
| `remake(prob; u0=…)` | **1.31** (matches seeded) |

### Discrete control (E7)

| variant | full-horizon NRMSE |
|---------|-------------------:|
| cold (fresh `st`) | 1.55 |
| warm (post-train `st`) | **1.14** |
| rewarm via `collectstates` | **1.14** |

### Washout (E8, washout=200)

| variant | NRMSE |
|---------|------:|
| cold | 1.48 |
| warm full train | 1.18 |
| warm post-washout tail | **1.13** |

Post-washout tail is slightly better than full-train terminal.

---

## 3. Wall times (full, not a perf PR — order-of-magnitude only)

| stage | wall |
|-------|------|
| ContinuousESN train (`collectstates` + ridge), N=300, T=5000 | ~18 s |
| AR cold predict, 1250 steps | ~2 s |
| AR warm predict, 1250 steps | ~1 s |
| Warmup collect (full train pass for terminal u0) | ~train-scale |
| Full E1–E8 suite | ~275 s |

Warm vs cold AR cost is comparable; the extra cost of warm is mainly the one-time teacher-forced collect for `u0`. Not a throughput regression story — a correctness/usability one. Perf work stays on #467.

---

## 4. Smoke vs full (why smoke lied)

| | smoke (n=80, T=800) | full (n=300, T=5000) |
|--|---------------------|----------------------|
| ContinuousESN warm vs cold | warm **hurt** | warm **helps** (esp. short horizon) |
| SciML warm vs cold | helps a little | helps |
| Structural checks | same | same |

Smoke is fine for plumbing (`match_ok`, carry inspection). **Do not** use smoke for API justification.

---

## 5. API recommendation (updated after full run)

| Option | Verdict after full data |
|--------|-------------------------|
| `predict(...; initial_state=u0)` | **Yes — primitive.** Matches remake; E4 shows wrong seeds are dangerous so it should be explicit. |
| `predict(...; warmup_data=W)` | **Yes — sugar.** K≥10 already unlocks ~5 t_λ VPT; must use unit windows (`Δt=1`). |
| Persist terminal in `st` | Still attractive for discrete-like ergonomics; needs continuous `st` design — separate decision |
| Docs-only | Insufficient: cold looks like a broken model at 1–4 t_λ |

**Proposed first code PR:**
`initial_state` on continuous AR `predict` + unit tests that seeded zeros ≡ today’s cold path and that a known `u0` is actually used. Optional `warmup_data` in the same PR or immediately after. Lorenz short-horizon eye-test optional but strong.

---

## 6. Questions for @MartinuzziFrancesco

1. OK to treat short-horizon (1–4 t_λ) NRMSE / VPT as the acceptance metric for warmup, not full-horizon NRMSE?
2. Ship `initial_state` first, or `warmup_data` first, or both?
3. Should `train!` / `collectstates` start writing a continuous terminal into `st` (discrete-like), or keep state external via kwargs?
4. Keep this harness under `benchmarks/` long-term?

---

## File map

| Path | Role |
|------|------|
| `run.jl` | `--smoke` / full / `--only=` / `--n-res=` |
| `src/predict_variants.jl` | experimental seeded AR (no package API) |
| `src/experiments.jl` | E1–E8 |
| `results/summary_full.md` | latest full table |
| `results/FINDINGS.md` | this document |
Loading
Loading