|
| 1 | +# swiss-river × {DLinear, PatchTST} entity-identifier results (multi_channel, 2026-06-14) |
| 2 | + |
| 3 | +DLinear and PatchTST on the three Swiss datasets in **multi_channel** split |
| 4 | +(each station is a channel — identity is already implicit in the channel |
| 5 | +layout, unlike the per_entity LSTM where the identifier is the *only* |
| 6 | +identity signal). Single seed 2026, 50-trial HPO per cell, batch_size fixed |
| 7 | +32, 30 epochs. Numbers read from real `results.json` by |
| 8 | +`tools/plot_swiss_mc_results.py` (no hand-copied values); |
| 9 | +CSV: `figures/swiss-mc-2026-06-14/swiss-mc-rmse.csv`. |
| 10 | + |
| 11 | +> These results only became measurable after fixing four multi_channel bugs |
| 12 | +> on 2026-06-14 (commits `fdecec1`, `82f0903`): before that, the transparent |
| 13 | +> identifier modes were a silent no-op (== none) and the 2010/zurich cells |
| 14 | +> were all-NaN. The 2026-05 dlinear/patchtst transparent results are |
| 15 | +> **retracted** (they measured the no-op). |
| 16 | +
|
| 17 | +## Results — test RMSE (°C, denormalized) |
| 18 | + |
| 19 | + |
| 20 | + |
| 21 | +| model | dataset | none | embedding | onehot | random | sinusoidal | coordinates | |
| 22 | +|---|---|---|---|---|---|---|---| |
| 23 | +| DLinear | 1990 | 1.281 | 1.286 | 1.286 | 1.283 | 1.282 | 1.279 | |
| 24 | +| DLinear | 2010 | 1.355 | 1.356 | 1.356 | 1.361 | 1.358 | 1.357 | |
| 25 | +| DLinear | zurich | 1.391 | 1.401 | 1.396 | 1.393 | 1.393 | 1.385 | |
| 26 | +| PatchTST | 1990 | 1.374 | **1.304** | 2.189 | 2.059 | 2.108 | 1.815 | |
| 27 | +| PatchTST | 2010 | 1.488 | **1.387** | 2.505 | 2.345 | 2.340 | 2.036 | |
| 28 | +| PatchTST | zurich | 1.480 | **1.388** | 2.738 | 2.631 | 2.690 | 2.377 | |
| 29 | + |
| 30 | +## What we can and cannot claim (research-critic audit) |
| 31 | + |
| 32 | +**Defensible (single seed; val mirrors test, so not overfitting):** |
| 33 | + |
| 34 | +- **DLinear shows no measurable identifier effect** on swiss multi_channel: |
| 35 | + all six modes land within ~1% of `none` on every dataset (HPO val is |
| 36 | + equally flat, e.g. 1990 val 0.1103–0.1105 across modes). A linear |
| 37 | + decomposition model over channels has no head-room to exploit per-channel |
| 38 | + identity, and the wrapper's fusion projection adds none. |
| 39 | +- **On PatchTST, the injection POINT — not the identifier type — drives the |
| 40 | + result.** `embedding` via `add_after_patch` (injected *after* patch |
| 41 | + embedding) helps (~−5% vs none). The transparent modes, injected via |
| 42 | + `concat_to_x` + `ChannelTransparentWrapper`'s per-channel |
| 43 | + `Linear(1+D,1)` fusion *before* patching, hurt substantially (+30–60%), |
| 44 | + and they hurt in HPO val too (1990 onehot val 0.231 vs none 0.124). |
| 45 | + |
| 46 | +**NOT claimed (would over-reach):** |
| 47 | + |
| 48 | +- *"Transparent identifiers are harmful."* Too broad — `embedding` is also |
| 49 | + an identifier and it helps. What hurts on PatchTST is the **pre-patch |
| 50 | + `concat_to_x` fusion**, which perturbs the per-channel series that |
| 51 | + PatchTST then slices into patch tokens. `add_after_patch` avoids this. |
| 52 | +- *"DLinear is immune to identity."* We observe no effect; we cannot tell |
| 53 | + apart "the linear model can't use identity" from "the fusion projection |
| 54 | + discards it" without inspecting the projection weights. |
| 55 | +- Anything cross-domain — all three datasets are Swiss water temperature. |
| 56 | + |
| 57 | +**Key caveat (read before citing):** the multi_channel transparent results |
| 58 | +reflect the behaviour of **`ChannelTransparentWrapper`** (concat + learnable |
| 59 | +per-channel fusion projection), *not* the intrinsic value of the identifier |
| 60 | +information. This wrapper is structurally different from the per_entity |
| 61 | +`EntityTransparentWrapper` (zero-parameter concat). Cross-split comparisons |
| 62 | +must account for this. |
| 63 | + |
| 64 | +## Contrast with the per_entity LSTM (2026-06-13) |
| 65 | + |
| 66 | +| split | model | identifier effect | |
| 67 | +|---|---|---| |
| 68 | +| per_entity | LSTM | every identifier helps (−25 to −35% vs none) — identity is the ONLY identity signal | |
| 69 | +| multi_channel | DLinear | none (≈ flat) — identity already in the channel layout | |
| 70 | +| multi_channel | PatchTST | `add_after_patch` embedding helps; pre-patch transparent injection hurts | |
| 71 | + |
| 72 | +The dominant variable across these is the **split mode**, not the model: |
| 73 | +per_entity makes each sample identity-blind (so the identifier is decisive), |
| 74 | +while multi_channel already encodes identity in the channel index (so an |
| 75 | +extra identifier is at best redundant, at worst disruptive to the |
| 76 | +architecture). This matches the split-mode confound flagged in the 2026-05 |
| 77 | +report. |
| 78 | + |
| 79 | +## To harden / explain (follow-ups) |
| 80 | + |
| 81 | +- **Ablation isolating injection point from identifier type**: run PatchTST |
| 82 | + + transparent + `add_after_patch` — if it stops hurting, the cause is the |
| 83 | + pre-patch fusion, confirming the audit's reading. |
| 84 | +- **Inspect the DLinear fusion-projection weights** to tell "can't use" from |
| 85 | + "discards". |
| 86 | +- **Multi-seed** (≥3) for variance bands (shared with the lstm follow-up #32). |
| 87 | + |
| 88 | +## Reproduce |
| 89 | + |
| 90 | +```bash |
| 91 | +python tools/plot_swiss_mc_results.py |
| 92 | +``` |
0 commit comments