investigate: continuous AR predict warmup (cold vs warm u0)

[Saswatsusmoy]

Note

CI is out of scope for this draft. Investigation-only harness under benchmarks/ (no package src/ / ext/ changes). Full test matrix / Runic / Downgrade are not meaningful here and runs on this branch have been cancelled. Please treat red checks as expected until an eventual code PR.

Summary

Investigation-only follow-up to the warmup discussion deferred from #456.

@MartinuzziFrancesco: “I think we can add the warmup as part of a separate PR, since I would like to investigate that a bit further.”

No package src/ / ext/ changes. Harness lives under benchmarks/continuous_warmup/.

Related: #397 · #456 · #467 (perf is separate)

Two run modes

mode	command	purpose
smoke	julia --project=. run.jl --smoke	plumbing (~20s, n_res=80)
full	julia --project=. run.jl	forecast analysis (~5 min, n_res=300, train=5000, predict=1250, #456 HPs)

Full config + numbers: results/FINDINGS.md · summary_full.md

Full-matrix headline (ContinuousESN, N=300)

Short-horizon NRMSE (warm = train-terminal u0):

t_λ	cold	warm
1	0.69	0.15
2	1.34	0.085
3	1.51	0.093
4	1.66	0.14

VPT (threshold 0.5): cold 0.22 t_λ → K=10 warmup 4.87 t_λ.

Structural: continuous st has no carry after train!; discrete does. Seeded u0=0 bit-matches package cold predict. remake(prob; u0=…) ≡ experimental warm on SciMLProblemReservoir.

Wrong seeds (randn / shuffled terminal) are catastrophic (~13 NRMSE) — worse than cold. Warm must be dynamically consistent.

Candidate API (after full data)

Option	Role
predict(...; initial_state=u0)	primitive
predict(...; warmup_data=W)	sugar (unit-window teacher-force)
Persist terminal in st	discrete-like; bigger design call
Docs-only	insufficient

Checklist

• Investigation harness only (no public API change)
• Smoke plumbing matrix
• Full feat(models): ContinuousESN #456-scale forecast matrix + FINDINGS
• Short-horizon / VPT / seed ranking / wall times recorded
• Agree API with @MartinuzziFrancesco before package PR

Questions for @MartinuzziFrancesco

1. Accept 1–4 t_λ NRMSE / VPT as the warmup success metric?
2. Ship initial_state, warmup_data, or both first?
3. External kwargs vs writing terminal into continuous st?
4. Keep harness under benchmarks/ long-term?

Draft until design is locked. Implementation is a follow-up PR.

[Saswatsusmoy]

Hey @MartinuzziFrancesco — following up on the warmup deferral from #456.

You asked to investigate warmup as a separate PR rather than land it inside ContinuousESN. This draft is that investigation: no package API change, harness under benchmarks/continuous_warmup/, and a full #456-scale Lorenz matrix (not smoke) so we can decide the API from real free-run behaviour.

cd benchmarks/continuous_warmup
julia --project=. -e 'using Pkg; Pkg.develop(path="../.."); Pkg.instantiate()'
julia --project=. run.jl          # full analysis (~5 min)
# optional plumbing check: julia --project=. run.jl --smoke

Writeup: results/FINDINGS.md · raw table: summary_full.md

---

Full matrix config (#456 scale)

n_res	300
train / predict	5000 / 1250
predict span	≈ 22.6 t_λ (dt=0.02, λ_max≈0.9056)
Wr radius / Win / bias	0.9 / 0.1 / 0.05
modifiers / ridge	NLAT2(), 1e-6
solver	Tsit5 (reltol=1e-6, abstol=1e-8)
seed	17
suite wall	~275 s on this laptop

Experimental AR loop (predict_ar_seeded) lives outside the package so we can inject initial_state without touching ext/. Sanity: seeding u0 = 0 reproduces package predict with max abs diff = 0.

---

The actual story: short-horizon free-run (ContinuousESN)

Full-horizon NRMSE over all 1250 steps is high for both once chaos has diverged (cold 1.48, warm 1.23). That is not the interesting metric. The free-run quality gap shows up in the first few Lyapunov times:

t_λ	steps	cold NRMSE	warm (train-terminal)
0.5	28	0.75	0.21
1.0	55	0.69	0.15
2.0	110	1.34	0.085
3.0	166	1.51	0.093
4.0	221	1.66	0.14
6.0	331	1.51	0.67

At 2–3 t_λ, warm is ~0.09 vs cold ~1.3–1.5 — same qualitative gap as the #456 Lorenz probe (cold ~1.5 → warm ~0.11). Cold AR looks like a broken model; it is mostly a missing warm-start.

Valid prediction time (error threshold 0.5)

warmup length K	VPT (t_λ)
0 (cold)	0.22
10	4.87
50–2000	~4.1

Even a short teacher-forced warmup (K=10) lifts usable free-run from ~0.2 → ~5 t_λ. Warmup time grid must use unit windows (Δt = 1, tspan = (0, K)). Stretching K samples across the train model’s long tspan silently changes ODE pacing (fixed in the harness; any warmup_data API should bake this in).

Seed ranking (why warm must be explicit)

seed	full-horizon NRMSE
train_terminal	1.23
zeros (today’s cold path)	1.48
oracle test-prefix (K=100)	1.45
randn / shuffled terminal	~13.5 (catastrophic)

Wrong non-zero seeds are much worse than cold. Warm is not “any u0” — it has to be a dynamically consistent terminal state. That argues for an explicit initial_state rather than silent magic.

Cross-checks

setup	cold	warm
SciMLProblemReservoir eq. (5) full-horizon NRMSE	1.47	1.31 (remake(prob; u0=…) identical)
Discrete ESN full-horizon NRMSE	1.55	1.14 (post-train st ≡ rewarm via collectstates)
ContinuousESN + washout=200	1.48	1.13 (post-washout tail slightly beats full-train terminal)

---

Structural (why the API has nowhere to put this today)

Check	Result
Continuous st after train! / collectstates	no carry
Discrete st after train!	has carry
Discrete rewarm via collectstates	≡ using post-train st
ContinuousESN AR seed today	always zeros(out_dims)
SciMLProblemReservoir AR seed today	res.prob.u0 (usually zeros unless user remakes)

So continuous AR always cold-starts. Public warm workaround for generic SciML reservoirs: remake(prob; u0 = terminal). ContinuousESN has no public u0 hook without a custom loop.

---

Wall times (order-of-magnitude — not a perf PR)

stage	wall
train ContinuousESN N=300, T=5000	~18 s
AR predict 1250 steps	~1–2 s
suite E1–E8	~275 s

Warm vs cold AR cost is comparable; the extra cost of warm is mainly one teacher-forced collect for u0. This is a correctness / usability story, not #467 throughput work.

---

API lean (from the full data)

Option	Role
predict(...; initial_state = u0)	primitive — matches remake; E4 shows seeds must be intentional
predict(...; warmup_data = W)	sugar — unit-window teacher-force → seed; K≥10 already wins on VPT
Persist terminal in continuous st	discrete-like ergonomics; bigger design call
Docs-only	not enough — cold looks broken at 1–4 t_λ

Working preference: ship initial_state first, optional warmup_data on top. Acceptance metric for a code PR should be 1–4 t_λ NRMSE / VPT, not full-horizon NRMSE over 20+ t_λ.

---

What I’d like your read on

1. Does this structural + short-horizon picture match what you wanted to dig into after feat(models): ContinuousESN #456?
2. Prefer initial_state, warmup_data, st persistence, or a combo?
3. First code PR: API + unit tests only, or also a warm Lorenz short-horizon eye-test in docs/tests?
4. Keep this harness under benchmarks/ long-term (like the PR4 profile tree), or treat the branch as disposable evidence?

I’ll keep this PR draft until you’ve had a chance to push back on the shape. Happy to add more seeds / HP sweeps if you want a specific plot before locking the API.

Thanks!

[Saswatsusmoy]

CI note: this draft is investigation-only (benchmarks/continuous_warmup/, no package code). The full Actions matrix is not in scope — I’ve cancelled in-flight runs on this branch and will avoid further pushes that re-trigger them. Red Runic/format on the harness is expected; please ignore CI status on #476 until a real API PR.

@MartinuzziFrancesco — findings stand on the full-matrix comment above; no need to wait on checks.

[Saswatsusmoy]

Temporarily closing to stop the Actions matrix on this investigation-only draft (contributor can't cancel upstream runs — 403). Reopening as draft immediately; CI remains out of scope.

[MartinuzziFrancesco]

sorry it took me so long to get to this. my main question in this issue was why is the warmup needed in the first place. after all, ESN doesn't use it. I'd suggest reordering §5's plan: persist the terminal u into st after train!/collectstates as the first PR (mirrors discrete ESN's automatic threading), with initial_state shipping as the override/escape hatch on top of that rather than initial_state first and st-persistence deferred as a separate decision.

also nice idea on the initial_state in general, the library does not provide that functionality, we only have the weird resetcarry! which does the job but creates a strange workflow. If you end up building it we can wire it up in all the cases in the library and just dispatch it on AbstractVecOrMat for normal states

[Saswatsusmoy]

Hey @MartinuzziFrancesco — thanks for the read, no rush.

On “why warmup if discrete doesn’t need it?”: continuous doesn’t need a special warmup concept — it’s missing the thing discrete already does. After train! / collectstates, discrete st carries the terminal state; continuous doesn’t (zeros / bare prob.u0). So AR always cold-starts. The harness “warm” path is just recovering that terminal u.

Fully with you on the reorder:

1. First PR: write terminal u into continuous st after train/collect, and seed AR from it (mirror discrete).
2. Then: initial_state as the override on top (happy to do this library-wide on AbstractVecOrMat — nicer than resetcarry!).
3. warmup_data sugar later if we still want it.

This draft stays evidence-only. I’ll open the product PR for st persistence off current master next.

Thanks!