Skip to content

Commit 59f44bb

Browse files
jajupmochiclaude
andcommitted
docs(research): swiss mc dlinear/patchtst results + research-critic audit
36-cell multi_channel results (read from real results.json). Audit-corrected framing: DLinear shows no measurable identifier effect (flat, val mirrors test); on PatchTST the INJECTION POINT drives it — add_after_patch embedding helps (~-5%), pre-patch concat_to_x transparent fusion hurts (+30-60%). NOT claimed: "transparent harmful" (embedding is also an identifier and helps) or "DLinear immune" (can't separate model from projection). Key caveat: mc transparent results reflect ChannelTransparentWrapper's fusion projection, not intrinsic identifier value. Contrast table vs per_entity LSTM notes the split-mode confound. Follow-up ablations listed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 82f0903 commit 59f44bb

4 files changed

Lines changed: 179 additions & 0 deletions

File tree

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# swiss-river × {DLinear, PatchTST} entity-identifier results (multi_channel, 2026-06-14)
2+
3+
DLinear and PatchTST on the three Swiss datasets in **multi_channel** split
4+
(each station is a channel — identity is already implicit in the channel
5+
layout, unlike the per_entity LSTM where the identifier is the *only*
6+
identity signal). Single seed 2026, 50-trial HPO per cell, batch_size fixed
7+
32, 30 epochs. Numbers read from real `results.json` by
8+
`tools/plot_swiss_mc_results.py` (no hand-copied values);
9+
CSV: `figures/swiss-mc-2026-06-14/swiss-mc-rmse.csv`.
10+
11+
> These results only became measurable after fixing four multi_channel bugs
12+
> on 2026-06-14 (commits `fdecec1`, `82f0903`): before that, the transparent
13+
> identifier modes were a silent no-op (== none) and the 2010/zurich cells
14+
> were all-NaN. The 2026-05 dlinear/patchtst transparent results are
15+
> **retracted** (they measured the no-op).
16+
17+
## Results — test RMSE (°C, denormalized)
18+
19+
![RMSE by mode](figures/swiss-mc-2026-06-14/swiss-mc-rmse-by-mode.png)
20+
21+
| model | dataset | none | embedding | onehot | random | sinusoidal | coordinates |
22+
|---|---|---|---|---|---|---|---|
23+
| DLinear | 1990 | 1.281 | 1.286 | 1.286 | 1.283 | 1.282 | 1.279 |
24+
| DLinear | 2010 | 1.355 | 1.356 | 1.356 | 1.361 | 1.358 | 1.357 |
25+
| DLinear | zurich | 1.391 | 1.401 | 1.396 | 1.393 | 1.393 | 1.385 |
26+
| PatchTST | 1990 | 1.374 | **1.304** | 2.189 | 2.059 | 2.108 | 1.815 |
27+
| PatchTST | 2010 | 1.488 | **1.387** | 2.505 | 2.345 | 2.340 | 2.036 |
28+
| PatchTST | zurich | 1.480 | **1.388** | 2.738 | 2.631 | 2.690 | 2.377 |
29+
30+
## What we can and cannot claim (research-critic audit)
31+
32+
**Defensible (single seed; val mirrors test, so not overfitting):**
33+
34+
- **DLinear shows no measurable identifier effect** on swiss multi_channel:
35+
all six modes land within ~1% of `none` on every dataset (HPO val is
36+
equally flat, e.g. 1990 val 0.1103–0.1105 across modes). A linear
37+
decomposition model over channels has no head-room to exploit per-channel
38+
identity, and the wrapper's fusion projection adds none.
39+
- **On PatchTST, the injection POINT — not the identifier type — drives the
40+
result.** `embedding` via `add_after_patch` (injected *after* patch
41+
embedding) helps (~−5% vs none). The transparent modes, injected via
42+
`concat_to_x` + `ChannelTransparentWrapper`'s per-channel
43+
`Linear(1+D,1)` fusion *before* patching, hurt substantially (+30–60%),
44+
and they hurt in HPO val too (1990 onehot val 0.231 vs none 0.124).
45+
46+
**NOT claimed (would over-reach):**
47+
48+
- *"Transparent identifiers are harmful."* Too broad — `embedding` is also
49+
an identifier and it helps. What hurts on PatchTST is the **pre-patch
50+
`concat_to_x` fusion**, which perturbs the per-channel series that
51+
PatchTST then slices into patch tokens. `add_after_patch` avoids this.
52+
- *"DLinear is immune to identity."* We observe no effect; we cannot tell
53+
apart "the linear model can't use identity" from "the fusion projection
54+
discards it" without inspecting the projection weights.
55+
- Anything cross-domain — all three datasets are Swiss water temperature.
56+
57+
**Key caveat (read before citing):** the multi_channel transparent results
58+
reflect the behaviour of **`ChannelTransparentWrapper`** (concat + learnable
59+
per-channel fusion projection), *not* the intrinsic value of the identifier
60+
information. This wrapper is structurally different from the per_entity
61+
`EntityTransparentWrapper` (zero-parameter concat). Cross-split comparisons
62+
must account for this.
63+
64+
## Contrast with the per_entity LSTM (2026-06-13)
65+
66+
| split | model | identifier effect |
67+
|---|---|---|
68+
| per_entity | LSTM | every identifier helps (−25 to −35% vs none) — identity is the ONLY identity signal |
69+
| multi_channel | DLinear | none (≈ flat) — identity already in the channel layout |
70+
| multi_channel | PatchTST | `add_after_patch` embedding helps; pre-patch transparent injection hurts |
71+
72+
The dominant variable across these is the **split mode**, not the model:
73+
per_entity makes each sample identity-blind (so the identifier is decisive),
74+
while multi_channel already encodes identity in the channel index (so an
75+
extra identifier is at best redundant, at worst disruptive to the
76+
architecture). This matches the split-mode confound flagged in the 2026-05
77+
report.
78+
79+
## To harden / explain (follow-ups)
80+
81+
- **Ablation isolating injection point from identifier type**: run PatchTST
82+
+ transparent + `add_after_patch` — if it stops hurting, the cause is the
83+
pre-patch fusion, confirming the audit's reading.
84+
- **Inspect the DLinear fusion-projection weights** to tell "can't use" from
85+
"discards".
86+
- **Multi-seed** (≥3) for variance bands (shared with the lstm follow-up #32).
87+
88+
## Reproduce
89+
90+
```bash
91+
python tools/plot_swiss_mc_results.py
92+
```
69.5 KB
Loading
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
model,dataset,none,embedding,onehot,random,sinusoidal,coordinates
2+
dlinear,swiss-river-1990,1.2814,1.2860,1.2860,1.2827,1.2817,1.2790
3+
dlinear,swiss-river-2010,1.3545,1.3560,1.3558,1.3606,1.3576,1.3572
4+
dlinear,swiss-river-zurich,1.3905,1.4008,1.3964,1.3927,1.3933,1.3851
5+
patchtst,swiss-river-1990,1.3737,1.3035,2.1889,2.0590,2.1075,1.8148
6+
patchtst,swiss-river-2010,1.4884,1.3865,2.5051,2.3452,2.3402,2.0356
7+
patchtst,swiss-river-zurich,1.4803,1.3884,2.7380,2.6311,2.6900,2.3773

tools/plot_swiss_mc_results.py

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
#!/usr/bin/env python3
2+
"""Plot swiss multi_channel (dlinear/patchtst) entity-identifier results.
3+
4+
Reads denorm test RMSE from real results.json (no hardcoded numbers) and
5+
emits one grouped-bar figure per model (3 datasets × 6 modes) with the
6+
`none` baseline drawn as a reference line. Also writes a CSV.
7+
"""
8+
from __future__ import annotations
9+
10+
import csv
11+
import glob
12+
import json
13+
from pathlib import Path
14+
15+
import matplotlib
16+
17+
matplotlib.use('Agg')
18+
import matplotlib.pyplot as plt
19+
20+
ROOT = Path(__file__).resolve().parents[1]
21+
ART = ROOT / 'artifacts' / 'entity_identifier'
22+
OUT = ROOT / 'docs' / 'research' / 'figures' / 'swiss-mc-2026-06-14'
23+
OUT.mkdir(parents=True, exist_ok=True)
24+
25+
DATASETS = ['swiss-river-1990', 'swiss-river-2010', 'swiss-river-zurich']
26+
MODELS = ['dlinear', 'patchtst']
27+
MODES = ['none', 'embedding', 'onehot', 'random', 'sinusoidal', 'coordinates']
28+
TAG = {d: f'swiss-mc-{d.split("-")[-1]}-20260614' for d in DATASETS}
29+
SHORT = {'swiss-river-1990': '1990 (28)', 'swiss-river-2010': '2010 (63)',
30+
'swiss-river-zurich': 'zurich (15)'}
31+
COLORS = {'none': '#9aa0a6', 'embedding': '#1a73e8', 'onehot': '#34a853',
32+
'random': '#a142f4', 'sinusoidal': '#ea8600', 'coordinates': '#d93025'}
33+
34+
35+
def rmse(tag, ds, model, mode):
36+
hits = glob.glob(f'{ART}/{tag}/{ds}-{model}-{mode}-*/**/results.json', recursive=True)
37+
if not hits:
38+
return None
39+
m = json.load(open(hits[0]))['metrics']['test']
40+
return float(m.get('denorm_rmse', m.get('rmse')))
41+
42+
43+
data = {model: {ds: {mode: rmse(TAG[ds], ds, model, mode) for mode in MODES}
44+
for ds in DATASETS} for model in MODELS}
45+
46+
with open(OUT / 'swiss-mc-rmse.csv', 'w', newline='') as fh:
47+
w = csv.writer(fh)
48+
w.writerow(['model', 'dataset', *MODES])
49+
for model in MODELS:
50+
for ds in DATASETS:
51+
cells = data[model][ds]
52+
row = [f'{cells[m]:.4f}' if cells[m] is not None else '' for m in MODES]
53+
w.writerow([model, ds, *row])
54+
print('wrote', OUT / 'swiss-mc-rmse.csv')
55+
56+
fig, axes = plt.subplots(1, 2, figsize=(13, 4.8), sharey=False)
57+
for ax, model in zip(axes, MODELS):
58+
n = len(MODES)
59+
bw = 0.82 / n
60+
for j, mode in enumerate(MODES):
61+
xs = [i + (j - n / 2 + 0.5) * bw for i in range(len(DATASETS))]
62+
ys = [data[model][ds][mode] for ds in DATASETS]
63+
ax.bar(xs, ys, bw, label=mode, color=COLORS[mode], edgecolor='white', linewidth=0.5)
64+
# none reference line per dataset (thin) — visual anchor
65+
for i, ds in enumerate(DATASETS):
66+
nb = data[model][ds]['none']
67+
ax.plot([i - 0.45, i + 0.45], [nb, nb], color='#444', lw=0.8, ls='--', alpha=0.6)
68+
ax.set_xticks(range(len(DATASETS)))
69+
ax.set_xticklabels([SHORT[d] for d in DATASETS])
70+
ax.set_title(f'{model} (multi_channel)')
71+
ax.set_ylabel('Test RMSE (°C)')
72+
ax.grid(axis='y', alpha=0.3)
73+
for sp in ('top', 'right'):
74+
ax.spines[sp].set_visible(False)
75+
axes[0].legend(ncol=6, fontsize=8, loc='upper center', bbox_to_anchor=(1.05, -0.12), frameon=False)
76+
fig.suptitle('swiss-river × {dlinear, patchtst}: entity-identifier modes '
77+
'(single seed 2026, 50-trial HPO; dashed = none baseline)', fontsize=11)
78+
fig.tight_layout()
79+
fig.savefig(OUT / 'swiss-mc-rmse-by-mode.png', dpi=150, bbox_inches='tight')
80+
print('wrote', OUT / 'swiss-mc-rmse-by-mode.png')

0 commit comments

Comments
 (0)